5G Turbo-Charges Streaming VR, AR and VFX

By CHRIS McGOWAN

With its blazing speed, 5G – the next generation of wireless technology – is expected to spark plentiful innovations and burgeoning growth in entertainment and the arts, especially for streaming media, video games, virtual reality and augmented reality. These innovations will create new opportunities for content creators and visual effects artists.

“All of a sudden, people will have more access to high-quality data streams, very fast, with low latency,” according to David Bloom, a Santa Monica-based journalist and consultant who tracks the collision of technology and entertainment. For television, there will be options galore as 5G will bolster IP services that stream “all your different video content.”

That different content may include new forms of interactive TV, music and advertising. One can easily imagine the interactive potential of shows like Dancing with the Stars and American Idol, as well as live music concerts and sporting events. There will be narrative that follows in the footsteps of Charlie Brooker’s 2018 branching episode Bandersnatch in the Netflix series Black Mirror. And there will be a variety of interactive ads, some inspired by the Proctor & Gamble 2020 Super Bowl commercial “When We Come Together,” powered by interactive video company eko. In that commercial, visitors to the ad’s website made narrative choices that determined the ad’s final version. 5G-enabled location awareness may add another layer to the mix for advertising, TV events and other shows, according to Bloom. For most content, 5G will impact the production of the VFX underpinning it all.

THE CLOUD AND THE THIN CLIENT

Neishaw Ali, President and Executive Producer at SpinVFX, explains that since 5G streams high bandwidth with low latency (little or no lag time), “You are basically pulling data from your ‘studio in the cloud,’ thus allowing you to work from your laptop and a tablet anywhere in the world as long as you are connected to a 5G network. In a world where security is critical, 5G also allows you to secure your data in the cloud and only stream images to your mobile device. It also helps to reduce cost by allowing you to ‘spin up’ a different level of services only when you need it. This flexibility is exciting and challenging to VFX studios as it streamlines costs, provides efficiency and reduces barriers to entry which in turn creates global competition.”

Chris Healer, CEO of VFX studio The Molecule, comments, “I think the first thing that comes to mind is that the ‘thin client’ revolution will happen because of this technology [5G], where it’s cheaper and easier to send a screen share or transmit a UI than it is to process on a device locally. I would say that more software will join that revolution.” He describes one possible example, “It’s really not a far stretch for Adobe to release a 5G-enabled version of Photoshop that works on an iPad Pro, where Photoshop is running in the cloud and the interface is transmitted to the iPad.” 5G could change other things in the studio, says Healer. “One simple thing that would be improved tremendously would be the ability to view dailies, previs, VFX drafts, etc., while on set. As a VFX supervisor you don’t get much time on set with a director between setups, so being able to hit play on a video and see it instantly in full resolution without waiting for it to load would be huge. I envision VFX and post-production being untethered.”

Mathias Chelebourg, director of the VR film Doctor Who: The Runaway, comments, “The race for rendering power has always been at the center of our pipeline designs. The recent rise and evolution of streamed remote workstations in the studio with the addition of real-time engine tools are revolutionary breakthroughs that are deeply impacting the way we approach filmmaking from set to delivery.”

ENABLING VR AND AR

Virtual reality, which fully immerses users in another place/reality, and augmented reality, which overlays text or images on what is in the real world, are immersive media that have applications ranging from arts and entertainment to education and industry. “5G will be a huge enabler for AR/VR,” says Tuong H. Nguyen, Senior Principal Analyst for research and advisory company Gartner, Inc. “Having 5G opens up many possibilities. It provides high bandwidth, low latency, massively collaborative, on-demand, contextualized experiences. In AR, this could mean facilitating content delivery and consumption – providing contextualized experiences – giving the user relevant, interesting, actionable information based on their context.”

Lower latency and stable connections, thanks to 5G, will enhance the sense of presence in virtual reality – that you are really “there” inside the experience. In addition, 5G will cut the cord connecting many headsets to PCs, with the cloud taking over the heavy lifting. High-quality VR and AR will become possible where 5G is present. “The ability to rely on a strong 5G network to harvest the power of cloud computing will definitely be a key milestone in the evolution of VR broadly,” Chelebourg comments. “It needs to happen.”

David Bloom, journalist/consultant

Neishaw Ali, President and Executive Producer, SpinVFX

Vicki Dobbs Beck, Executive in Charge, ILMxLAB

Chris Healer, CEO, The Molecule

Mathias Chelebourg, VR Director

Tuong H. Nguyen, Senior Principal Analyst, Gartner, Inc.

Jason Ambler, Vice President of Digital Media, Falcon’s Creative Group

Bei Yang, Technology Studio Executive, Walt Disney Imagineering

“For VR and AR, the onus will be on a remote machine, without lugging around the hardware,” says Healer. “For instance, the Oculus Quest will basically not need much, if any, local storage. Taking it one step further is the interoperability of users who [become] untethered. With inside-out tracking you won’t need to reconfigure your living room to have a VR chat with your family. You could go to a park or the seaside and know you won’t be bumping into things.”

“Whether it’s AR or VR, 5G can enable massive collaboration. Think about something like The World of Minecraft, but everyone using it at once, or social VR, with thousands of virtual people interacting within a space,” says Nguyen. “Games that have multiple players in them are going to be superpowered,” adds Bloom. And, in terms of AR, 5G will allow for an overlaying of what’s happening in the game with the reality around you,” according to Bloom.

5G could tap into the interactive potential of shows like Dancing with the Stars and American Idol, as well as live concerts and sporting events. (Images courtesy of American Broadcasting Company)

Future interactive narratives could follow the example of Bandersnatch, the 2018 branching episode in the Netflix series Black Mirror. (Image courtesy of Netflix)

“Whether it’s AR or VR, 5G can enable massive collaboration. Think about something like The World of Minecraft, but everyone using it at once, or social VR, with thousands of virtual people interacting within a space.”

—Tuong H. Nguyen, Senior Principal Analyst, Gartner, Inc.

Next-generation interactive ads may be inspired by Proctor & Gamble’s 2020 Super Bowl commercial “When We Come Together,” powered by interactive video company eko. In that commercial, visitors to the ad’s website made narrative choices that determined the ad’s final version. (Image courtesy of Proctor & Gamble)

5G will allow multiple-player games like The World of Minecraft to be ‘superpowered,’ with potentially thousands of people using it at once. (Image courtesy of Mojang and Xbox Games Studios)

FIXED LOCATIONS

5G will also impact the world of fixed location entertainment. Jason Ambler is VP of Digital Media at Falcon’s Creative Group, which specializes in theme park and attraction design. “We certainly see 5G as part of our future content development strategy for location-based entertainment and immersive experiences,” relates Ambler, “not only for AR, VR and other mobile-based applications, but also for large-format and large-scale interactive content as well. One of the most exciting aspects of 5G for us is the potential to stream massive amounts of data from the cloud over extremely low latency networks, thus enabling the possibility of personalized foveated rendering [eye tracking] from the cloud. This could help to streamline the guest – or user – facing technologies.

“In addition, 5G could allow for what have been traditionally offline or pre-loaded VR and AR experiences to now become wide-area online experiences with the added benefits of real-time socialization, layers of customization and spontaneity that have not yet been seen in our industry. More and more of our projects are incorporating levels of real-time interactivity, mobile functionality, and other advanced networking systems that could all greatly benefit from the promise of 5G and allow you to engage with your environment and personalize your experience like never before.”

Bei Yang, Technology Studio Executive at Walt Disney Imagineering, comments, “On-device rendering fidelity has always been an issue with AR, and 5G can shift that thinking entirely as cloud-based rendering becomes a reality. Light fields streaming, rather than frames, made possible by larger bandwidth and lower latency will enable fairly lag-free AR experiences. More accurate locations from 5G will allow AR to be solidly based on the real world and allow crowd-sourced mapping of actual structures. We [Walt Disney Imagineers] create physical manifestations of these ‘virtual worlds’ for Disney theme parks, and we are constantly exploring new and emerging technologies, like 5G, that enhance the storied experiences within.”

“One of the most exciting things about a robust 5G network and Edge compute is how they’ll allow consumers to engage with high-fidelity immersive experiences both indoors and outdoors, where high-bandwidth data streams haven’t previously been available,” says Vicki Dobbs Beck, Executive in Charge at ILMxLAB, which together with The VOID created the fixed-location VR experience Star Wars: Secrets of the Empire. The immersive studio strives, as per its motto, to let audiences “step inside our stories.” “5G technology could allow us to do that on an even grander scale, across a number of platforms and locations, threading interactive narratives through multiple aspects of our daily lives,” says Beck. VR and AR’s digital effects will foster new types of artists who will overlay things on the real world in semi-real-time. Bloom comments, “So what does that mean for a VFX artist? You’re creating this stuff. You’re animating the real world. Our definition of what a VFX artist is will expand and shift. There will be new types of media possible with 5G.”

Ian Hambleton, CEO of the immersive studio Maze Theory, notes that there may be glitches with 5G in the beginning, “but as with many new technologies, and particularly connectivity, often things move forward far quicker than expected.”

Bloom observes that before 4G, “we couldn’t anticipate the rise of things like Uber and Lyft, and the ability to dispatch cars wherever we were and to track them and pay and do all of that. 4G enabled them. Just as that seemingly came out of nowhere with 4G, there’s a bunch of things we can’t anticipate quite yet that can be enabled with 5G.”