View a Live XR Broadcast in Apple Vision Pro?

John LePore’s specialty is envisioning futuristic gadgets based on present-day technology. Film studios often call on him to imagine, create and deliver on such visions. He and his team at Perception conceived and brought to life the Wakandan Vibranium sand in the film “Black Panther” and Tony Stark’s touch-enabled holographic devices in “Iron Man 2,” among others. He has also contributed to projects at Google and Hummer, but on those, his lips are sealed, due to their confidential nature.

In January, he proposed what he believed to be a winning formula in a tweet. He wrote, “Apple Vision Pro + Formula 1 = Killer App?” The tweet’s view count has reached 989k as of this writing; it was reshared more than 1,400 times. “Across various social media platforms, it racked up about 55 million views,” he says.

What he was proposing was an extended reality (XR) application that brings real-world Formula 1 events into a virtual tabletop setup, delivered via Apple Vision Pro.

“I love that Formula 1 is a tech-centric competition. I believe it should be the benchmark for how we combine sports and technology,” he says. “What if I can take advantage of the physical view of the space and the geography of the event itself? There’s something so appealing about an architectural model, alive and in perfect sync with the reality unfolding thousands of miles away.”

For this article, we spoke to LePore and several technology leaders to explore the possibility of such an app, and the hurdles involved.

Live XR Broadcast for F1

In 2018, LePore submitted his app idea to Formula 1’s My F1 Challenge. His idea was named “One of the five most innovative entries of the year.” What he had created was “a proof of concept, a non-interactive prototype,” LePore says. In a sense, the prototype was his calling card, announcing what he and his consultancy could do.

LePore has some ideas about making the app a reality. “There’s an abundance of live data when an F1 event is happening. It’s very easy to create dots on a map, but our aspiration is to have precise information for where all four tires of the car are making contact with the racing surface. It’s a higher level of granularity. There are some cheats and tricks we can use to get there,” he says.

LePore recognized 3D representation of the racecourse and the vehicles might be a challenge, but he thought “it’s one of the easiest challenges to tackle.” For the purpose of the app, he won’t need manufacturing-standard CAD models; he only needs representative miniature models.

“We would take the hyperdetailed surgically precise data, and convert that into something much more lightweight, while retaining the critical pieces of accuracy,” he says.

LePore’s app idea is applicable not just to Formula 1 but to many other live sporting events, from basketball to boat racing. But depending on the event, the details that matter change. “With a football game, it’s important to have the human players in high-fidelity visuals—the way they bend their knees, the way their hair flows when they’re chasing the ball, and so on. But today, volumetric capture won’t be able to capture and relay that in real time, but we will get there,” he says.

In that sense, the F1 app’s visual hurdles are much easier to overcome. “In 3D modeling, hard, solid bodies [like race cars] are easier to model than human bodies,” he says. For his app, cockpit-level details are less important; its focus is “the helicopter point of view,” in LePore’s words.

How would the game handle car collisions, cars veering off the ramp, and other common race events?

“Things like damage to the cars, or cars going into the grass add to the drama of the event, but they’re not a great detractor for this app. In future iterations, we could add them, but for now, I’d say they’re nice-to-have items, but not critical,” says LePore.

For his work, LePore says he relies on the Adobe suite of products, Unreal and Epic game engines, and Maxon and Cinema 4D software, among others. His app, LePore says, could be delivered on a number of hardware platforms, not limited to XR headsets, but “we gravitated toward the Apple Vision Pro because we feel it raises the bar in fidelity and quality of experience,” he adds.

Valeo Racer

At this year’s South by Southwest 2024 conference in Austin, TX, mobile technology provider Valeo debuted Valeo Racer, a new XR app that gives in-car gaming experience for passengers. The game was developed in the Unity game engine.

“We used Unity Runtime to process sensor data in real time, within the vehicle, to create the ‘fun’ of the virtual racecourse,” says Geoffrey Bouquot, Valeo’s executive vice president of Strategy and chief technology officer, “In fact, we are still exploring the numerous features offered by Unity, and as our project progresses, we will be working with Unity to better design and optimize our template.”

Reflecting on his experience developing the Valeo Racer, Bouquot says, “One of the biggest challenges we’ve seen is in finding ways to standardize both the data formats and data transfer protocols across both present and future vehicles. Right now, each vehicle represents vehicle data differently, and game developers are not going to want to develop a different game for every individual vehicle brand, or every individual vehicle model within a brand.”

Lenovo ThinkReality VRX F1

Last Year, Lenovo and Formula 1 teamed up to deliver mini-games in VR revolving around the Japanese Grand Prix in Suzuka, Japan, and the U.S. Grand Prix in Austin, TX. With Lenovo ThinkReality headsets, players could “drive an F1 car around a replica of the Suzuka International Racing Course or the Circuit of the Americas, controlling its speed with the ThinkReality VRX controllers’ buttons … Drivers can drift around corners, but if they accelerate too much in a curve, the car can leave the track, and even fall off the table!,” explains Vishal Shah, general manager of XR and Metaverse, Lenovo.

With ThinkReality, Lenovo hopes to capture the AR/VR-powered training and visualization market. The headset is augmented with a set of cloud services and software for enterprise deployment.

“The racing game is an opportunity to showcase the power and value of VR experiences and the ThinkReality VRX, Lenovo’s new all-in-one virtual reality (VR) headset engineered for the enterprise,” Shah says.

Shah also reviewed LePore’s game concept, commenting, “Similar apps and more could be developed for use with ThinkReality A3 smart glass or the ThinkReality VRX VR headset with full-color passthrough. On premise 5G or 6G networks and edge servers supporting smart cameras and other sensors could capture and process data to feed the variety of applications.”

Beyond the game concept, Shah thought XR could also become a viable mechanism for remote tech support and training. “Race teams can use XR for training drivers and crews, vehicle development and maintenance with the use of 3D models and digital twins, as well as race team management with a steady stream of data and communications via a virtual digital workspace,” he says.

Shah’s view is that XR has the potential to become a regular feature in live event broadcasts, much like the yellow scrimmage line displaying down and distance info in broadcast NFL games. He argues, “That was one of the first widely experienced applications of AR technology in sports. Now, it’s hard to imagine watching a game without it.”

For San Diego Padres fans, telecom firm Motorola offers a multiplayer demo for two to four players, placing users in a virtual model of Petco Park. The app is powered by Motorola phones and ThinkReality VR gear.

“In gameplay, fans are offered the chance to take the field and test their skills by aiming baseballs at targets placed throughout the ballpark to score,” writes Motorola in its blog (“Motorola and the Padres Offer Fans a Major League Experience in Augmented Reality,” July 2022).

Verizon also released a proof of concept for the future of football training in VR, powered by ThinkReality hardware. Verizon writes, “The Verizon Future of Training POC showcased how AR can elevate traditional 2D instructions in a football playbook into a more engaging and immersive 3D format.”

More Lenovo Coverage

Making the Case for Engineering Workstation Upgrades

In this Making the Case whitepaper, Lenovo outlines how upgrading to the latest generation of professional workstations can provide a return on investment through increased engineering efficiency and greater flexibility.