VFX Voice

The award-winning definitive authority on all things visual effects in the world of film, TV, gaming, virtual reality, commercials, theme parks, and other new media.

Winner of three prestigious Folio Awards for excellence in publishing.

Subscribe to the VFX Voice Print Edition

Subscriptions & Single Issues


January 09
2024

ISSUE

Winter 2024

PUTTING THE AI IN THE ANIMATION INDUSTRY

By TREVOR HOGG

Wētā FX built a new facial system for Avatar: The Way of Water using a series of neural networks. (Image courtesy of Wētā FX and 20th Century Studios)

Wētā FX built a new facial system for Avatar: The Way of Water using a series of neural networks. (Image courtesy of Wētā FX and 20th Century Studios)

On the positive side, AI represents the automation of laborious tasks allowing additional time to be creative as well as producing new revenue streams and more immersive virtual experiences, while the negative part of the equation has it restricting creativity to an algorithm, plundering the efforts of others and eliminating jobs. Where does the truth lie for the animation industry? To find the answer, the expertise of Framestore, Technicolor Creative Studios, Wētā FX and Cinesite were consulted, and the general attitude about AI is not apocalyptic, but optimistic.

“AI is baked into many of the tools that we use at Framestore. In our own research and development, we are guided by the principle of using it to get to the creative faster,” states Tim Webber, Chief Creative Officer at Framestore. “Animation is a craft, and it’s the subtleties, mannerisms, nature and idiosyncrasies that an animator brings to a character and the performance that makes it great. Many animators consider themselves to be actors, bringing invisible performances to the characters they create. This is inspired by the animator’s own life experiences, watching people, animals or the study of reference material. They bring sympathy, empathy, emotional accuracy and deep creativity to their work, often facilitated and sped up by AI, but something it can’t replicate.”

AI and machine learning are fundamentally changing every type of technology. “Users are expecting everything to be smarter, responding to simpler inputs and helping users stay directed towards the users’ goals,” remarks Michael O’Brien, Senior Vice President of Research and Development at Technicolor Creative Studios. “Animation is no different. The tools we use need to become more intuitive and intelligent. AI will augment and assist artists so they can accomplish their goals in a shorter time by leveraging the learned intelligence of how artists worked in the past. Animators will still animate. They will still be creative. They will be helped along the way by algorithms providing assistance to common tasks, assurances that the approach is correct and communication of how the work is going.”

“AI does not replace artistry,” O’Brien observes. “It is another tool for us to make better pictures faster. New ideas will be connected and generated from algorithms, but artistry will always be the artist. The field of AI is growing at such a speed that we need to make sure we continue to assure the technology is used with full visibility to what the algorithms do and how they are trained, but the opportunities are amazingly vast. Animators starting today will benefit from decades of artistry, and standing on those shoulders will create new worlds, iterate into new directions and create an industry unlike the one we know today.”

A shift in focus is needed to better understand AI. “First off, it’s helpful to forget about the idea of ‘Intelligence,’ artificial or otherwise, and think about the function, which is a neural network,” notes Joe Letteri, Senior VFX Supervisor at Wētā FX. “What a network allows you to do is control multiple channels of information at the same time. For example, if you put a spline curve on an elbow joint rotation, you may want the shoulder to have some degree of sympathetic rotation. Then you go up the skeleton and add whatever joint rotations you need to make the movement look natural. A neural network allows you to encapsulate and manage those relationships, so that adjusting one joint will also adjust related joints in a plausible way. In this example, you could also use an IK [Inverse Kinematics] change to do something similar. It is not clear that a neural network would be better, but it is another tool that animators could use.”

Wētā FX built a new facial system for Avatar: The Way of Water using a series of neural networks to manage the relationship between facial muscles and expressions. “The tools help capture sympathetic secondary motion in the face, so animators are able to connect their creative decisions more intuitively,” Letteri states. “It was slow going at first, but has proven to be a powerful tool for animators.” Special attention is paid to the training dataset. “When we train a neural network, we are careful to only use data that we own or have the right to use for a particular show,” Letteri notes. “We’re fortunate that we have a partnership with Unity where they are looking into how our technology can reach a wider group of artists outside of Wētā FX. They have announced AI projects in beta and are already doing interesting things in this space.”

Cinesite has been talking a lot about various generative and machine learning applications and what they mean for the company. “Almost across the board it’s efficiency gains,” remarks Hank Driskill, Head of CG for Feature Animation at Cinesite. “There are certain tasks like rotoscoping, which is extreme manual labor and time-consuming. I admire the people who have the patience to do it, but that will go away in the coming years and be replaced with tools that can do it automatically. There is a whole layer of art design that is busy work, such as ‘Give me the artwork for what the trashcan in the corner looks like.’ You’re going to use generative AI tools to generate a thousand different trash cans and pick one.”

The term “AI” is a misnomer. “AI isn’t intelligent,” Driskill notes. “It is a useful tool that is broadly applicable. There are all kinds of aspects of how we work that modern machine learning and AI tools are going to have a big impact on, but right now they’re not going to replace the minds that are at work there.” A prevailing question is whether AI will cause everyone to get stuck in the algorithm of creativity. “It runs that risk, especially because generative AI is only as good as the dataset that you trained it on. What is absolutely coming is an inbetween program in the style of, let’s say, Glen Keane. But then you’re stuck in the style of Glen Keane. That isn’t the same as what you get today where artists can create any style of motion.”

The cost-effectiveness of AI will enable more animated movies like Nimona to be made that are not beholden to the traditional studiofour-quadrant marketing strategy. (Image courtesy of Netflix)

The cost-effectiveness of AI will enable more animated movies like Nimona to be made that are not beholden to the traditional studio four-quadrant marketing strategy. (Image courtesy of Netflix)

Nimona is an example of the diversification occurring in animated storytelling that Hank Driskill, Head of CG for Feature Animation at Cinesite, finds exciting. (Image courtesy of Netflix)

Nimona is an example of the diversification occurring in animated storytelling that Hank Driskill, Head of CG for Feature Animation at Cinesite, finds exciting. (Image courtesy of Netflix)

The major theme for Big Hero 6 is that technology is not inherently bad, which was demonstrated by the different ways the microbots got used by Hiro and villain Yokai aka Professor Robert Callaghan, who stole Hiro’s mind-controllable microbot technology.(Image courtesy of Disney)

The major theme for Big Hero 6 is that technology is not inherently bad, which was demonstrated by the different ways the microbots got used by Hiro and villain Yokai aka Professor Robert Callaghan, who stole Hiro’s mind-controllable microbot technology. (Image courtesy of Disney)

Machine learning was utilized by Framestore to help create a digital double of Yara Shahidi, who portrays Tinkerbell inPeter Pan & Wendy. (Image courtesy of Framestore and Disney)

Machine learning was utilized by Framestore to help create a digital double of Yara Shahidi, who portrays Tinkerbell in Peter Pan & Wendy. (Image courtesy of Framestore and Disney)

AI is confined to the dataset it has been trained on, which is not the same as artists who can create any style of motion for the characters in Teenage Mutant Ninja Turtles: Mutant Mayhem. (Image courtesy of Cinesite and Paramount Pictures)

AI is confined to the dataset it has been trained on, which is not the same as artists who can create any style of motion for the characters in Teenage Mutant Ninja Turtles: Mutant Mayhem. (Image courtesy of Cinesite and Paramount Pictures)

AI will assist in producing the in-between frames for characters that will speed up productions like Teenage Mutant Ninja Turtles: Mutant Mayhem. (Image courtesy of Cinesite and Paramount Pictures)

AI will assist in producing the in-between frames for characters that will speed up productions like Teenage Mutant Ninja Turtles: Mutant Mayhem. (Image courtesy of Cinesite and Paramount Pictures)

AI will assist productions such as the Disney+ series Iwájú to create generic background props that are not central to the storytelling.(Image courtesy of Cinesite and Disney)

AI will assist productions such as the Disney+ series Iwájú to create generic background props that are not central to the storytelling. (Image courtesy of Cinesite and Disney)

Generative AI tools have not been incorporated into the current workflow. “The concern with generative AI is that you don’t know where the reference came from or how it is being interpreted,” Driskill notes. “There are all of these legal challenges that are cropping up because like a lot of times with new technology, the law hasn’t caught up yet. The more that you’re building your own AI models and training it on material that you’ve created, then you’re golden.” Another legally sensitive issue is AI being utilized for vocal performances. “It is when they use AI to generate whole new lines, you get into the grey area fast. You’re now putting words in the actor’s mouth quite literally. That’s where it gets dangerous. Retooling a line is easy. I also see background characters being replaced by AI in the coming years, for sure.”

AI will provide suggestions. “Often when designing a character, you produce a series of model sheets about what this character looks like with different facial expressions and poses,” Driskill states. “The artist’s job is often to hit and dial-in those shapes, a lot of keeping it on model and point. AI is going to help clean it up.” An area of animation that could be majorly improved is cinematography. “I suspect that it will move towards the live-action paradigm where you set up the lights, cameras, have your environment in place, then you bring your actors to perform. You can have AI tools saying that you’re moving the camera too fast through this section, a real camera couldn’t do that. AI is going to give you more ways to check your work and make sure the thing you’re doing is within the bounds you’re setting, except when you don’t want them to,” Driskill says.

AI tools will help to make sure the facial expressions and poses stay on model, like with Blazing Sam in Paws of Fury: The Legend of Hank. (Image courtesy of Cinesite and Paramount Pictures)

AI tools will help to make sure the facial expressions and poses stay on model, like with Blazing Sam in Paws of Fury: The Legend of Hank. (Image courtesy of Cinesite and Paramount Pictures)

The quest for expanding revenue streams is constant. “People are always looking for the next revenue stream because theatrical has been struggling in recent years,” Driskill observes. “With the advent of streaming, a lot of folks aren’t going to movie theaters. However, people said the same thing when television arrived. They’re always looking for new revenue streams and new ways to produce content and put that content in front of people. When you generate a film, it’s partly about telling a story that means something to the world, but you also want to create experiences that people come out of wanting more. AI will help with producing more supplemental content. We have already talked about producing 60-second TikTok bits.”

Technological efficiencies, such as AI, are helping to lower production costs. “With the advent of streaming and their appetite for content, we’re able to create content that can be meaningful to a subset of the population,” Driskill remarks. “That excites the hell out of me, for somebody who has been doing this for a long time. We’ve gotten to the point now where we can deliver content to a broad audience without it having to be sanitized for being four quadrants across the planet. There are a lot of things being produced now that are absolutely not for everyone, and that’s okay because there are other people where a movie like Nimona will change their lives.”

AI can be utilized to provide design suggestions. Cinesite created the dragon for Season 3, Episode 4 of The Witcher. (Image courtesy of Cinesite and Netflix)

AI can be utilized to provide design suggestions. Cinesite created the dragon for Season 3, Episode 4 of The Witcher. (Image courtesy of Cinesite and Netflix)

AI and machine learning are fundamentally changing every type of technology, which will prove to be useful when constructing massive environments such as those found in Dungeons and Dragons: Honor Among Thieves. (Image courtesy of MPC and Paramount Pictures)

AI and machine learning are fundamentally changing every type of technology, which will prove to be useful when constructing massive environments such as those found in Dungeons and Dragons: Honor Among Thieves. (Image courtesy of MPC and Paramount Pictures)



Share this post with

Most Popular Stories

2024 STATE OF THE VFX/ANIMATION INDUSTRY: FULL SPEED AHEAD
09 January 2024
Animation
2024 STATE OF THE VFX/ANIMATION INDUSTRY: FULL SPEED AHEAD
Despite production lulls, innovations continue to forward the craft.
CHANNELING THE VISUAL EFFECTS ON SET FOR THE WHEEL OF TIME SEASON 2
23 January 2024
Animation
CHANNELING THE VISUAL EFFECTS ON SET FOR THE WHEEL OF TIME SEASON 2
In the case of the second season of The Wheel of Time, the on-set visual effects supervision was equally divided between Roni Rodrigues and Mike Stillwell.
GAIA BUSSOLATI: CONNECTING HOLLYWOOD AND ITALIAN CINEMA – AND STAYING CURIOUS
09 January 2024
Animation
GAIA BUSSOLATI: CONNECTING HOLLYWOOD AND ITALIAN CINEMA – AND STAYING CURIOUS
VFX Supervisor bridges Italian cinema, Hollywood blockbusters.
HOW TIME KEEPS SLIPPING AWAY IN LOKI SEASON 2
19 December 2023
Animation
HOW TIME KEEPS SLIPPING AWAY IN LOKI SEASON 2
Created by Michael Waldron for Disney +, the second season of Loki follows notorious Marvel villain Loki (portrayed by Tom Hiddleston), Thor’s adopted brother and God of Mischief.
DIGITAL DOMAIN GOES INTO MACHINE LEARNING MODE FOR <b>BLUE BEETLE</b>
05 December 2023
Animation
DIGITAL DOMAIN GOES INTO MACHINE LEARNING MODE FOR BLUE BEETLE
Pushing the boundaries of digital humans has become standard practice for Digital Domain, with the title character of Blue Beetle providing an opportunity to fully test a cloth system driven by machine learning and a custom neural system known as ML Cloth.