23 January 2024
Industry Q&A Roundtable
By JIM McCULLAUGH
By JIM McCULLAUGH
VFX Voice asked a cross section of industry executives, artists, practitioners and visionaries this question:
What single creative process do you think will most define and shape VFX for the new decade, and why?
Here are their responses.
Lindy De Quattro, Production VFX Supervisor, MPC:
“Virtual Production has actually been around for years already, but after the success and publicity of The Lion King and the emergence of the AR/VR industry, it’s clearly the new hot catchphrase in our industry. I think the next 10 years will see it really take off and be embraced by many more filmmakers and at a variety of budgets. We will also see it used in more areas of the filmmaking process, not just previs and postvis – which were the earliest uses of VP – or to create entire films like The Lion King, but throughout the VFX pipeline.
“The big breakthrough for Virtual Production was the integration of real-time rendering engines from the game industry, like Unreal Engine and Unity. It took a while for these engines to mature enough to be able to handle film quality renders in real-time, but they’re basically there now. While it does require some extra planning and prep work to get a virtual scene set up, once that work is done then the filmmaker has amazing flexibility to not only visualize a scene, but also to modify the virtual elements as they work in real time. After the initial setup, it’s a very fast and cost-effective way to work.”
“We will see [virtual production] used in more areas of the filmmaking process, not just previs and postvis… or to create entire films like The Lion King, but throughout the VFX pipeline.”
—Lindy De Quattro, Production VFX Supervisor, MPC
Thilo Kuther, Founder and President, Pixomondo Global:
“Previsualization. VFX, which has long been considered a supporting industry in the film community, will become its own industry as disruptor of the status quo because it is using technology at every level of the process: AI, Game Engine, Virtual Reality. These tools will create a new type of filmmaker.
“With previs, the post-production phase is now becoming the pre-production phase as VFX houses offer technological tools to filmmakers who can make production decisions without risking financial consequences. When we build our previs set, as a filmmaker you can see it and make any adjustments before it’s built. It is similar to the process of industrial design: build a prototype, test the prototype, fine-tune the prototype, and then build the final product. Moving forward, this process will be capable of taking over production, from start to finish, not just pre-production. Within five years, VFX will literally overtake productions, and no longer be just a supporting player.”
Rob Bredow, Sr. Vice President, Executive Creative Director and Head of ILM:
“We are seeing a dramatic shift enabled by the maturing of virtual production technologies where – with close collaboration with filmmakers, production designers and DPs – VFX can create effects in-camera again. This benefits everyone. Actors aren’t standing in a sea of green expected to imagine the environment they are meant to be reacting to. Cinematographers can compose and light their scenes as they would on a traditional production with all of the elements right in front of them, each reacting in real-time to changes as they are made. Filmmakers are able to focus on the story that the shot needs to tell without the unknowns. To enable this, visual effects teams are involved from the very beginning of production. Savvy filmmakers have been engaging their visual effects supervisors early in the production cycle for years, to advise on shooting methodologies, determine best approaches, and help design shots for maximum effect – that’s not new. What is new, however, is the close collaboration between all the departments to achieve the vision during the production cycle, often in camera.”
“As the industry sprints and the audience reacts, the one single creative process that will define and shape the VFX industry is a well-thought-out and considered approach to the allotment of time: time managed, time dedicated, time saved, time respected, for an artist’s inclination to create from the heart – be that with code, or pencil and paper.”
—Karen Dufilho, Producer
Karen Dufilho, Producer:
“You’ve heard of the Butterfly Effect, right? Where a small change in input – the flap of a butterfly’s wings – can have a large effect on outcome: a tornado occurs in another part of the world.
“Things are chaotic. The entertainment industry is no exception. Tools change, studios’ power shifts, tax exemptions dictate work/life choices. Meanwhile, we’re inundated with more shows, content and choices now than ever before. The U.S. media and entertainment industry is expected to reach more than $825 billion by 2023. The U.S. gaming industry is expected to reach nearly $26 billion in 2019 in revenue. VR gaming has grown more than 14% since 2018.
“But in the chaos, there is a constant. Every line of code, every word written (and rewritten), every sketch sketched, traces back to that proverbial wing that resulted in the tornado. Never underestimate the power of pencil and paper on the world.
“As the industry sprints and the audience reacts, the one single creative process that will define and shape the VFX industry is a well-thought-out and considered approach to the allotment of time: time managed, time dedicated, time saved, time respected, for an artist’s inclination to create from the heart – be that with code, or pencil and paper. The results could blow you away.”
Kim Davidson, President & CEO, SideFX:
“I’m a big believer in immersive worlds and the impact they will have on the creative process – particularly for VFX. Imagine being on the set of a castle siege with armies charging, trebuchets launching fireballs and walls crumbing. However, in this case the set, armies and VFX are 100% CG, and the director and cinematographer are in free-roam VR outside the castle. They can slow, pause or rewind the action. They walk around and interact with the armies and VFX – adding more soldiers, increasing the size of the fireballs, or changing the landing spot of a falling stone block.
“Our industry is already well along the path to immersive filmmaking as many of the necessary components have been adopted or are being rapidly advanced. And leveraging proceduralism along with AI will allow art directors and set designers to readily and stylistically populate their themed immersive worlds – worlds where actors perform and stories unfold. Visual effects involving earth, wind, water and fire complete the illusion of this reality. Visual effects living inside the bigger visual effect of an immersive world.”
“Today, the cutting edge of technology development is in deep learning and AI. Most of us have seen demonstrations of the striking potential of these technologies to produce imagery without an artist driving the creative process. The missing piece in this technology is the integration of deep learning AI techniques into an artistically driven creative VFX process.”
—Xye, Founder, Yannix/ynx.vfx
Xye, Founder, Yannix/ynx.vfx:
“Creativity is always enabled by the tools and technologies used to express that creative idea or concept. In VFX, this grew out of the optical effects of the pre-digital matte paintings and projection technologies and was redefined by the computer and the digital manipulations it enabled. By giving artists access to tools that could precisely manipulate color and texture in three dimensions, this revolutionized the special effects in movies. Today, the cutting edge of technology development is in deep learning and AI. Most of us have seen demonstrations of the striking potential of these technologies to produce imagery without an artist driving the creative process. The missing piece in this technology is the integration of deep learning AI techniques into an artistically driven creative VFX process. In professional movies and video, it isn’t sufficient to create a compelling image. The essential need is to create the imagery that best captures the imagination of the filmmaker. While face replacement is a constant need in VFX, ‘deepfake’ technology is useless to a creative process if the filmmaker can’t call out the details they want adjusted and refine the performance accordingly.”
Norman Wang, CEO and Co-founder, Glassbox Technologies:
“The VFX industry has been the pioneer of many groundbreaking innovations, and, as with any industry, true transformation comes from advances in technology making great ideas from the last decade faster, better and more accessible, such is the case with the latest advances in real-time production. Real-time or Virtual Production is now the bridge between pre-production, production and postproduction. Moving forward, it could simply replace these separate parts thanks to recent and forthcoming advances in real-time and extended reality technology, as well as the power of real-time rendering engines. I anticipate we will see more post-production-quality visual effects created with the efficiency and rapid turnaround of pre-production, all while under the intensity of an on-set production environment.”
“Visualize, communicate and execute is, and will continue to be, the single most important creative process for anyone in VFX —and many other fields. … If no one can understand what you are going to create then they cannot understand how it helps the story and, more than likely, it will be rejected. But to effectively communicate these ideas is essential – and nine times out of 10 will lead to acceptance.”
—Jeffrey A. Okun, VES, Visual Effects Supervisor
Jeffrey A. Okun, VES, Visual Effects Supervisor:
“Visualize, communicate and execute is, and will continue to be, the single most important creative process for anyone in VFX (and many other fields).
“Visualize: To be able to close one’s eyes and see the words on the page bloom into images that continue to change and evolve until the shot, sequence or project takes on a life that is bigger than yourself.
“Communicate: If no one can understand what you are going to create then they cannot understand how it helps the story and, more than likely, it will be rejected. But to effectively communicate these ideas is essential – and nine times out of 10 will lead to acceptance.
“Execute: Obviously, you have to be able to make your idea real. To do so one needs to understand the ever-changing techniques and processes, as well as how to work with others effectively and with an eye toward budget and time.”
Fredrik Limsater, CEO, ftrack:
“The next 10 years promise a significant period of evolution. Technical developments like web 3.0 and cloud, and creative innovation in real-time and spatial computing, promise a change not just in visual effects, but in how we perceive the world.
“The one technological progression I’m most interested in at ftrack, however, is that of machine learning. AI has the potential to open up new pathways for creative work, but also to bring more adaptive, agile workflows to production pipelines. Machine learning could mine data to predict the length of specific tasks or forecast workloads, for example, and thus enable a more fluid approach to studio schedules. At ftrack, we’re looking at AI-powered analytics to perfect project bids and provide automated, historical insight on tasks as they’re assigned.
“Machine learning has the power to disrupt established VFX production models and empower faster, more efficient approaches to tackling work.”
“The next 10 years promise a significant period of evolution. Technical developments like web 3.0 and cloud, and creative innovation in real-time and spatial computing, promise a change not just in visual effects, but in how we perceive the world. … Machine learning has the power to disrupt established VFX production models and empower faster, more efficient approaches to tackling work.”
—Fredrik Limsater, CEO, ftrack
Neishaw Ali, President and Executive Producer, Spin VFX:
“For decades, visual effects has been considered a black hole for some directors and producers whereby they need to wait for the magical creation of their vision upon shoot wrap and often feel alienated from the post process. But what if the filmmakers can create and see their vision come alive during the shoot albeit at 70% quality level instead of having to wait until the wrap of visual effects? I feel the single most creative process that will redefine and shape VFX in the next decade will be the immersive way in which the director and key creative team can drive the creation of visual effects during prep and shoot, ahead of post. We are fortunate to be on the leading edge of technology developing emerging tools for realtime creation of VFX on set.
“Early examples of this were seen in Avatar movies and The Jungle Book and more recently on The Lion King, in which the production team created a complex virtual representation of the scene that is intelligent and intuitive, designed to work in the same way that is familiar to the filmmaker and allows for true real-time collaboration, whereby they can design and create the layout, shots and even select the most dramatic lighting during the shoot, applicable to their storytelling.
However, in order for this to be successful, producers have to spend more time during early prep engaging the director, DP, visual effects and key creative team to construct the best approach for their show.”
Saker Klippsten, CTO, Zoic Studios:
“If I were to take a guess I would say that the single most important process will be the use of game engine technology such as Unreal Engine as a means to facilitate a faster final look development process, thus giving the creatives better tools to envision their story. Game and TV/film industries have been on a collision course for at least 20 years. The current process involves having highly skilled artists use Maya, Nuke and other similar tools that are very complicated to achieve a look which then needs to be composited and rendered on CPUs. Game engines allow for faster real-time and near-real-time rendering capabilities, giving instant feedback and the ability to iterate rapidly. Many now support ray-tracing directly in the engine which has put the quality level on par with traditional methods.”
Eliot Sakharov, Business and Technology Strategist, Media & Entertainment, Microsoft:
“The cloud and intelligent edge will provide the compute power necessary for user experiences that defy today’s tech limitations. You’ll engage directly with your entertainment platform through AI and sensory interactions, with holographic and virtual environments nearly indistinguishable from the real thing. AI is being infused into every aspect of our experiences today, ranging from what to watch and listen to how we travel. Editorial and VFX are no different. Over the next decade we’ll see further and further advancements in capture technology, where volumetrics and photogrammetry will radically change how we immerse ourselves. With these lifelike captures we can begin to use AI to generate text-to-speech voices, gaze adjustments and interactive elements in xR that we never thought possible. Having a single interaction automatically translated into 80 languages will increase inclusion, immersion and engagement. No single advancement will define it, as it will be an ecosystem of technologies that are only just beginning to take shape. The end result? Immersive storytelling that expands the human/machine interface.”
Lauren McCallum, Global Managing Director, Mill Film:
“Virtual Production will be the bleeding edge of VFX innovation in this next decade, giving filmmakers the tools to bring their real work toolkits into the virtual, digital stages. Getting DPs and directors comfortable with virtual asset creation and virtual cameras potential is key as the industry really embraces virtual workflow. At Mill Film we believe when you combine technical innovation with the most diverse creative teams, you’re bringing every possible resource and tool to your client.”
“Virtual Production will be the bleeding edge of VFX innovation in this next decade, giving filmmakers the tools to bring their real work toolkits into the virtual, digital stages. Getting DPs and directors comfortable with virtual asset creation and virtual cameras potential is key as the industry really embraces virtual workflow.”
—Lauren McCallum, Global Managing Director, Mill Film
Aaron Weintraub, Sr. VFX Supervisor, MR. X:
“There are so many emerging technologies that it’s difficult to choose just one, but I believe that the explosion of applications that utilize the recent advancements in Deep Learning are probably the most influential and pervasive developments that we’ve seen in this industry in recent years. It’s been stated before, even in the pages of this magazine, but it bears repeating: Deep Learning is going to turn the way we do VFX on its head. The current cost of entry for the average artist is somewhat steep, requiring significant coding ability and computing power, but as the technology becomes adopted and packaged into off-the-shelf applications, and the processor speeds continue to accelerate, we will see these obstacles decrease dramatically. We are going to see tools that we could literally only dream about before. For every time you sat frustrated, trying to force software to do something that you – as a human – knew intuitively how the result should appear, wondering why it couldn’t just know how to do it, you will find a solution in Deep Learning.”
Aleksandar Pejic, VFX Supervisor, Cinesite:
“The quality and complexity of visual effects has quadrupled over the past several years by utilizing the latest technology in computer graphics. What has not changed much in my view is the process itself that is deeply connected to the filmmaking process. There is a story a director wants to tell and a shot which often requires some visual effects in order to visually support the narrative, and that’s where we help. Depending on the movie, shots are invisible if we work on a drama or character-driven project or sometimes very apparent when we help superheroes to save the world, but the creative process is very similar in both instances. It involves a number of steps in order to get the final result. Some of those steps, like rotoscoping, camera and object tracking, and compositing, are very labor intensive and will tremendously benefit from the latest software and hardware developments in AI and ML [machine learning]. We can also see in some areas of animation where character tools have combined physical simulation with AI/ML. This significantly speeds up the iterations, producing amazing results that once took days, if not weeks.”
Janelle Croshaw Ralla
“Many of the laborious and/or repetitive tasks in VFX have the potential to be solved and executed more efficiently by machines. This might replace today’s typical outsourcing workflows. Even simple examples like face swap show the potential of the impact machine learning has, particularly in regards to digital humans, de-aging and entirely virtual characters.”
—Janelle Croshaw Ralla, VFX Supervisor
Janelle Croshaw Ralla, VFX Supervisor:
“Based on the latest tech right now, the answer might be AI and machine learning, as well as AR and its impact on virtual production. Many of the laborious and/or repetitive tasks in VFX have the potential to be solved and executed more efficiently by machines. This might replace today’s typical outsourcing workflows. Even simple examples like face swap show the potential of the impact machine learning has, particularly in regard to digital humans, de-aging and entirely virtual characters.
“In the grand scheme of things, 10 years isn’t that far away. Ten years ago, after coming off of The Curious Case of Benjamin Button, we thought fully digital humans would quite possibly take over filmmaking and become widespread. The opposite ended up happening in many regards.”
Kim Libreri, CTO, Epic Games:
“Virtual production will define and shape VFX for the new decade. While it may be a stretch to describe virtual production as a single creative process, it is the production methodology that will single-handedly impact the role that VFX plays in modern filmmaking. Real-time technology is making it possible to integrate processes that were formerly relegated to post-production cycles and bring them into the live-action shoot. Rather than passing assets or recreating them for each step in the production process, digital shots can be shared and updated anytime – even on set, in real-time – to satisfy the creative vision of the director or DP. Virtual production brings a level of creative freedom and spontaneity to the set that we haven’t seen in decades.”
Adam Myhill, Creative Director, M&E Innovation Group, Unity Technologies:
“The future of filmmaking, especially in the VFX industry, is real-time. Up until now, in almost all cases, if a studio or director wanted to explore a ‘what if?’ scenario, they are probably facing a staggering amount of extra work. Re-rendering that section, compositing – you’d see the results days later, or more. With the shift to real-time tools, you’re able to try out those ‘what ifs’ and see the changes right before your eyes, whether you’re doubling the size of a digital dinosaur, changing the lighting in a scene, moving some set around for a more dynamic shot or trying out a totally different camera approach. Creators are now free to experiment, test, iterate, and see everything in context, without waiting for a traditional pipeline to render it all out. Adam: The Mirror director Neill Blomkamp famously changed the time of day for a scene on the last day of production. These tools are also breaking down silos across departments, allowing everyone to be involved in the process from the start. The types of creative input on a project are no longer limited by your stage in development. You can now easily shift between storyboarding, color grading, working on the edit, trying out different music, and experimenting with different lens language, all in one location.”
Steve May, Chief Creative Officer, Pixar Animation Studios:
“The technological ability to produce high quality CG content in real-time will have a dramatic impact on our creative process. It will enable artists, producers and directors to iterate creatively in the moment. This will result in a significant change to traditional production pipelines, artist tools and job descriptions. Production pipelines will become far less linear. Walls between artists of different disciplines will be blurred, which will increase the demand for generalists and tools that enable generalist workflows.”
“When I think of things that will revolutionize the industry, the first thing that comes to mind is game engines, specifically in relation to their use in virtual production workflows. … We’re quickly approaching a day where the visual quality of real-time rendering will match the quality levels that you would expect from a traditional VFX finals pipeline.”
—Chris Ferriter, CEO, Halon Entertainment
Chris Ferriter, CEO, Halon Entertainment:
“When I think of things that will revolutionize the industry, the first thing that comes to mind is game engines, specifically in relation to their use in virtual production workflows.
“At Halon we’ve been using game engines for all our visualization needs for several years now. The real-time nature allows for high-level look development, rapid iteration, and a very effective feedback loop with our clients. The real power lies in interactivity though, allowing you to scout virtual sets or engage in multi-user collaborative sessions using VR or virtual cameras. Taken one step further, we can push all the content created using those tools into a virtual production setup utilizing LED panels to provide that interactivity on set. This includes on-set visualization, but also the potential to up-res and shoot final pixels for in-camera VFX.
“We’re quickly approaching a day where the visual quality of real-time rendering will match the quality levels that you would expect from a traditional VFX finals pipeline.”
Ed Ulbrich, President and GM, Method Studios:
“Creative processes evolve alongside technological advancement, and we’re on the cusp of a massive paradigm shift with growing access to real-time tools, cloud-based compute power and low-latency 5G networks. These core technologies will transform how artists work in a way that may be unrecognizable in five years from now, let alone the next decade. The continued adoption of machine learning, AI and neural networks will also heavily influence the creative process moving forward, automating repetitive tasks that free artist bandwidth so that they can focus on creative ideation.
“We’re already seeing many of these core technologies in production, and the timeframe to navigate difficult tasks is going to collapse as tools and technology continue to improve. It’s incredible what artists can accomplish today with real-time rendering technology and a desktop, nimbly handling work that used to require a hardcore renderfarm. Rapid prototyping can be done in nearly full resolution, and temp versions look more like finals.”
“Creative processes evolve alongside technological advancement, and we’re on the cusp of a massive paradigm shift with growing access to real-time tools, cloud-based compute power and low-latency 5G networks. These core technologies will transform how artists work in a way that may be unrecognizable in five years from now, let alone the next decade.”
—Ed Ulbrich, President and GM, Method Studios
Leslie Chung, VFX Supervisor, Crafty Apes:
“I think the expansion of collaborative ability through real-time virtual production will change the way filmmaking and VFX marry together.
“As fantastical films require more epic demands from VFX, the ability for directors, DPs, visual effects supervisors and editors to choreograph a scene in real-time together will become the biggest game-changer of our industry. I foresee it enabling a more streamlined and inspiring workflow for the impossible to be directed by the creative team and then executed by a visual effects house. You’d have so many pieces of the concept puzzle able to be generated and molded much more quickly than you ever could before. With this reduction in iteration, so much of the guesswork can be taken out of how best to hit a director’s vision for his or her film. Ideally then, both VFX studio, director and supervisor could riff more quickly and creatively together.”
;
