VP ROUNDTABLE: ON THE SURGING NEW TECHNOLOGIES THAT ARE CHANGING THE WORLD

By JIM McCULLAUGH

The acclaimed science fiction novelist Arthur C. Clarke once said, “Any sufficiently advanced technology is indistinguishable from magic.” It’s not a reach to say that virtual production has not only created magic for the movies, but VP will create magic for most of society in the years to come. VP and its cousin technologies – VFX, virtual reality, AR and others – are destined to transform the world.

Each of these technologies is in some progression toward altering, not just the world of entertainment, but education, medicine, science, military, architecture, real estate, travel, you name it. In some instances, VP, VFX, VR and other technologies are working together to accelerate this new world. What will the landscape look like in 10 years? We gathered a virtual roundtable of those on the front lines and asked them to gaze into their crystal balls.

Sam Nicholson, CEO and Founder, Stargate Studios Virtual production, virtual reality and artificial intelligence are rapidly changing the way we live, work and play. In the next 10 years, these technologies will exponentially accelerate and merge to disrupt a wide range of fields.

In arts and entertainment, AI will greatly reduce the cost of creating original content, enabling realistic virtual characters and photoreal synthetic environments. VR and VP will increasingly employ these assets to create immersive, interactive movies, games and new media. This will also greatly affect education with realistic simulations and personalized learning experiences.

In the healthcare industry, AI will analyze massive data sets in the cloud to instantly diagnose patients and recommend treatments. Immersive VR will be used to train surgeons and treat complex mental conditions. In architecture, real estate and travel, VR and VP will increasingly be used to allow users to virtually explore building designs and virtual locations, which will optimize industrial design and environmental planning. In the military, AI will interpret large amounts of historical and real-world data, then employ VR and VP to run multiple simulations for various tactical scenarios.

In the near future these technologies will become widely adopted, blurring the line between physical reality and virtual reality and offering new opportunities which will dramatically change our lives in myriad ways.

Alex Webster, Head of Studio, U.K., Pixomondo

Virtual production tools have changed visual content and entertainment forever, but the extrapolation of real-time rendering technology into medicine, education and beyond is also revolutionary. VR and AR offer a solution to visualize complex, stressful or dangerous situations in a real-world environment. Doctors and surgeons can prepare and review clinical skills to remotely build pre-operative experience, as well as remotely connect with colleagues to view and share how they execute a procedure and democratize skills and experience and the sharing of life-saving information in a tangible way. Virtual production also allows manufacturers to simulate entire production lines and visualize potential points of failure. Architects can fabricate engineering and design choices in real-time and contextualize those decisions in a virtual simulation of a real-world environment.

As for how film VP will look in 10 years’ time, it’s almost impossible to predict because of the current speed of innovation. But, we can assume that VP will behave as an intelligent ecosystem that links pre-photography development, previsualization to final pixel images in one engine, and leverages AI to enhance and optimize critical thinking in filmmaking practice. Machine learning will automatically apply tropes of film craft into pre-production, VFX and in-camera VFX, and latency in tracking will be reduced to nanoseconds, improving the performance of on-set, in-camera VFX profoundly. Previsualization of sequences and environments and the characters within them will be almost photoreal and will usher in a paradigm shift in how film and episodic content is conceived and created. We have seen the sudden advances in AI-informed concept art; expect this to translate to moving images in our near future.

On one hand, it’s unnerving, as preconceived notions of production are upended by game engine technology. On the other hand, technology and innovation have been inherent in the film industry since its inception, and we are at the early tipping point of the next fundamental change in how filmmakers can tell their stories. The ability to quickly create realistic, real-time renderings of complex structures or objects via game engine technology will empower adjacent industries to expedite the development, training and delivery of services.

Brian T. Nowac, Founder and CEO, Magicbox

Everyone is so excited about VR, AR, VFX and VP because they each possess attributes that have been present in just about every major advancement throughout history. I’m talking about increases in power capability, greater storage capacity, hardware miniaturization and mobilization. These are the reasons why we can expect this tech to change our work, our lives and our world. All these attributes are incredible on their own, but when they are combined, they give us the power of scale. This is the key I believe we will use to unlock the secrets of the universe.

When I say scale, I’m talking about the size of the world around us and the ability to change it instantly, zooming in and zooming out, like the scale tool in Adobe Illustrator, with infinite resolution. Imagine the perspective we will gain when we harvest real-world data and use it to build virtual environments we can then safely dive into and swim through as we get smaller and smaller, passed the atomic level. What will we discover when we use these technologies to see the quantum world up close? What will we learn about ourselves when we look back? This ability is not that far away, and it’s going to make our universe a much smaller place.

Kim Davidson, President and CEO, SideFX

Virtual production is in its infancy with production tools and techniques constantly evolving. Currently, VP is done with a patchwork of hardware and software from multiple vendors glued together by proprietary tools from a SWAT team of talented pioneers. In 10 years, the VP workflow will be standardized, common and available for rent to independent content creators. There will still be several vendors of hardware/software solutions, but they will be adhering to VP standards.

As virtual production becomes more commonplace, the fruits of the advancements, pioneered by high-end film, will have benefits beyond entertainment. Imagine walking around inside a 3D display dome and having the perspective of the projected environment adjust correctly to your position and respond to your gestures. Military, archeologists and tourists will familiarize themselves with sites before visiting. First responders will be given several scenarios to train in safe and highly realistic settings. Non-moving vehicles set up in the center of the dome will allow for effective training for bus drivers and operators of heavy machinery as they move through traffic and obstacles.

Outside the dome, actors or teachers can be captured in real-time and placed in the projected environment for viewers and students to engage with. Truly, the ability to create highly realistic spaces where anything is possible is only limited by our imagination.

Tom Debenham, Director of Photography, Visual Effects Supervisor, Co-Director, One of Us

The first thing to say is that virtual production already means several things which apply to different areas of the production process. The production process is already changing although often at a slower pace than the technologies around it. It’s tempting to think, as many people optimistically did around the pandemic, that virtual production might replace aspects of location shooting and make them cheaper and easier. Or, that the side of virtual production geared around prep might replace the need for scouting, recces and getting people together to share reactions to a location or a set.

I personally hope that it doesn’t replace either of these valued, real and human processes – but that instead, it can give us a wider range of tools to conceive putting the visual pieces of a puzzle together more flexibly and creatively.

One of the aspects of virtual pre-production, which has already developed hugely over the last 10 years, has been the digitization of the art department, and the convergence of the design worlds of art department and visual effects. This requires quite a lot of readjustment in terms of department structures and ways of communicating, but it can and will make us stronger together in terms of the coherence of the worlds we plan and create together. Virtual production in the sense of sharing models and visualizing collectively what the director’s vision can be is definitely an evolving and powerful principle. More a principle of communication than technology, but certainly enabled by modern tools.

On the other end of things, the convergence of photography and visual effects in the in-camera VFX side of virtual production is a powerful means of getting creative decisions made by the right people in collaboration on set. The ability to conceive of lighting and backgrounds as part of the same language, as we do in response to a location with available light, but with control over the time of day and weather, is definitely interesting!

It is only an answer to a relatively small set of shooting circumstances, but if it can be part of replacing what we have traditionally used blue and green screens for without completely upending traditional shooting schedules, it will be a step in the right direction.

I don’t foresee this technology replacing traditional filmmaking in any significant sense, but hope it will continue to enable us to re-align creative disciplines and visualize how they converge – in prep, on set and in post.

Irena Cronin, CEO, Infinite Retina

Virtual production is still in its infancy. Within 10 years, in combination with VFX, VR, generative AI and technologies not even thought up yet today, VP will have a wide range of uses in addition to entertainment. VP will be useful anywhere where a virtual environment is needed that is made relatively fast, adjusted in real-time and flexibly swapped out, in addition to the perks of being created and updated remotely. This includes realistic visualization for architecture planning, education, training exercises (military, first responders, law enforcement, etc.), marketing and advertising, and staging and backgrounds for real estate. It could be that the expense of VP will come down so much that it will become even more commonplace and be featured in restaurants and public spaces. The time for realistic visuals on demand has come and will increasingly dazzle.

Rene Amador, CEO and Co-Founder, ARWall

VP has completely changed the way genre blockbusters are shot. It’s currently the fastest-growing segment of the movie industry. ARwall’s technology has been designed to democratize VP tools for independent creators and home studios. We envisage creators having a complete toolset to create blockbuster-quality effects with their existing hardware, to the extent of being able to win major film awards from their living room within the next few years.

In 10 years’ time, I expect to see the lines blur between filmmaker, audience and performer. The ultimate goal of entertainment is as of yet unrealized: epic, world-spanning stories that can be enjoyed collectively, but are also meaningful on a personal level. Through XR, viewing experiences could be tailored to interweave details of an individual’s real life into the narrative. We’re not far from a society in which these virtual worlds may become more meaningful and ‘real’ than the physical world, and this is both an opportunity and danger I hope ARwall can help navigate.

Steve Calcote, Director and Partner, Butcher Bird Studios

Imagine turning a script into a blockbuster movie in a single day – with one filmmaker – within the next decade. That’s the inevitable level of creative power being unleashed by the current fusion of virtual production, AI-augmented creativity, real-time game engines and VFX, generative machine learning, reality scanning, digital doubles, automated cloud-based post-production and a global hunger to share visual storytelling like never before.

What place then for the artist? The same place as always: at the forefront of using revolutionary tools to invent visions the world has never seen and inspire humanity anew. We’ve been through this recently: smartphones allow every person on the planet to make perfect photos. Yet relatively few photos cut through the noise to truly move us. The current creativity singularity will empower individuals across every industry and every aspect of society to share their vision more clearly. But it will also enable artists and storytellers to create transformative works that take our breaths away… all over again.

Steve Griffith, Executive Producer, Virtual Production, DNEG

Virtual production is changing the world with digital humans and in how we produce content and tell stories in general. VR relies heavily on the capabilities of hardware. The drive to photorealism is completely dependent on it. As the technology progresses to where digital content is indistinguishable from reality, this is where the possibilities become endless. AI and ML aid in reducing artist time and help bridge the gap in photorealism by using libraries of content, but having the ability to create fully photorealistic, close-up humans, either based on real people or completely invented, means we can have posthumous interactions with people from the past, tell stories with celebrities from any point in their lifetime and tell pretty much any story we can think of.

The trick is then, how do we create the worlds and characters for storytelling? Our whole current production workflow is very much “building the car as we drive it,” at least in film and television. Storytellers shoot their content, edit and then fix it all in post. However, with worlds and characters becoming more digital, the work to create them is now needing to be done up front. Producers and creatives are asked to make decisions at the start and plan ahead before production begins. It’s something many people are very resistant to. We find those prepared to make decisions early on are few and far between. Most creatives are just not used to it, and it is hard to adapt to.

Libraries of photoreal content will eventually be accessible to anyone, and the ability to create all types of digital content will be democratized, in much the same way digital phones now have decent cameras and allow anyone to shoot films and tell stories. We are headed toward a Ready Player One future. It’s not that far off. I can’t even imagine what storytelling will be like when it’s happening in real-time and we’re both creating and interacting with worlds and characters that are photoreal, with the potential of simulating empirical data through neural networking. This is all the stuff you see in Black Mirror or read about in science fiction, but hey, looking around the world today, most of what I see were the things of science fiction when I was a kid.

Rob Bredow, Senior Vice President and Chief Creative Officer, ILM

Virtual production, or using digital tools in places we may have traditionally used analog techniques, continues to change the way movies and shows are made. In just a few short years, the LED Volume technique has gone from one stage in Manhattan Beach driven by a visionary filmmaker, Jon Favreau, and his team, to many stages all around the world supporting productions of all scopes and budgets. Ten years from now, we’ll look at an LED stage as one of the many powerful tools in the VFX practitioners’ growing arsenal of techniques, alongside bluescreen, on-set compositing and real-time AI tools that will further transform filmmaking in the future.

Casey Schatz, Head of Virtual Production, The Third Floor

Virtual production is changing the world by opening the door to computer graphics for creative individuals from the arts and sciences who may not have prior experience with CG software. The tech has become more intuitive and easier to engage with in the day-to-day work of many different personnel. The interface to technology has become more accessible. Within moviemaking, we may have an actor in a mocap suit, a cinematographer holding a virtual camera, a production designer scouting with VR and a key grip doing a tech scout with AR. In the sciences, we currently have car engineers using AR to collaborate on design and assembly, medical students using VR to practice heart surgery and motion capture being used in biomechanics. At home, we can use AR within our phones to preview furniture, put a funny face on someone and translate languages.

Virtual production is only just getting started. It’s difficult to imagine any industry that wouldn’t benefit from immediate feedback, infinite iterations and the democratization of these tools beyond the tech savvy.

Michael Ford, CTO, Sony Pictures Imageworks

As seen in the VFX and virtual production areas, visual complexity is increasing and the speed that we are needing to calculate and render pictures is trying to keep pace. We will need a massive amount of compute and network bandwidth to power the next generation of visualization, simulation and spatial computing that will dramatically impact the virtual and augmented worlds that will be created and ultimately explored. Our digital future, whether in virtual reality, augmented reality and whatever is next, is going to be produced for us not from a machine that is close to us, but from a network of machines that are distributed around us.

Mariana Acuña Acosta, Senior Vice President, Virtual Production, Technicolor Creative Studios

Virtual production unifies how content is made and brings more types of content into the mainstream media. Just like we have seen streaming services blur the lines between feature films and episodic, virtual production will blur the lines between film and less linear types of media such as games. It’s likely that in five years, A-list film directors will be directing games. In 10 years, they may be involved with content production and even further removed from film. This is not because content across different media types is becoming more similar. However, storytelling has always been a common core of all these media types, and virtual production is unifying how we can create them.

Joe Letteri, VES, Senior VFX Supervisor, Wētā FX

Rather than primarily entertainment and advanced design workflows, we’re likely to see improvement in the layering of digital “realities” on top of our everyday experiences. For Avatar: The Way of Water we created a new on-set depth compositing workflow that allowed James Cameron to see digital characters with accurate positioning and occlusion on a live-action set. These types of mixed-reality techniques are often pioneered in VFX and quickly adopted for more consumer applications, aiding everything from autonomous vehicle navigation, to advanced medical imaging and surgical capabilities and daily wayfinding.