PUTTING THE AI IN THE ANIMATION INDUSTRY

By TREVOR HOGG

On the positive side, AI represents the automation of laborious tasks allowing additional time to be creative as well as producing new revenue streams and more immersive virtual experiences, while the negative part of the equation has it restricting creativity to an algorithm, plundering the efforts of others and eliminating jobs. Where does the truth lie for the animation industry? To find the answer, the expertise of Framestore, Technicolor Creative Studios, Wētā FX and Cinesite were consulted, and the general attitude about AI is not apocalyptic, but optimistic.

“AI is baked into many of the tools that we use at Framestore. In our own research and development, we are guided by the principle of using it to get to the creative faster,” states Tim Webber, Chief Creative Officer at Framestore. “Animation is a craft, and it’s the subtleties, mannerisms, nature and idiosyncrasies that an animator brings to a character and the performance that makes it great. Many animators consider themselves to be actors, bringing invisible performances to the characters they create. This is inspired by the animator’s own life experiences, watching people, animals or the study of reference material. They bring sympathy, empathy, emotional accuracy and deep creativity to their work, often facilitated and sped up by AI, but something it can’t replicate.”

AI and machine learning are fundamentally changing every type of technology. “Users are expecting everything to be smarter, responding to simpler inputs and helping users stay directed towards the users’ goals,” remarks Michael O’Brien, Senior Vice President of Research and Development at Technicolor Creative Studios. “Animation is no different. The tools we use need to become more intuitive and intelligent. AI will augment and assist artists so they can accomplish their goals in a shorter time by leveraging the learned intelligence of how artists worked in the past. Animators will still animate. They will still be creative. They will be helped along the way by algorithms providing assistance to common tasks, assurances that the approach is correct and communication of how the work is going.”

“AI does not replace artistry,” O’Brien observes. “It is another tool for us to make better pictures faster. New ideas will be connected and generated from algorithms, but artistry will always be the artist. The field of AI is growing at such a speed that we need to make sure we continue to assure the technology is used with full visibility to what the algorithms do and how they are trained, but the opportunities are amazingly vast. Animators starting today will benefit from decades of artistry, and standing on those shoulders will create new worlds, iterate into new directions and create an industry unlike the one we know today.”

A shift in focus is needed to better understand AI. “First off, it’s helpful to forget about the idea of ‘Intelligence,’ artificial or otherwise, and think about the function, which is a neural network,” notes Joe Letteri, Senior VFX Supervisor at Wētā FX. “What a network allows you to do is control multiple channels of information at the same time. For example, if you put a spline curve on an elbow joint rotation, you may want the shoulder to have some degree of sympathetic rotation. Then you go up the skeleton and add whatever joint rotations you need to make the movement look natural. A neural network allows you to encapsulate and manage those relationships, so that adjusting one joint will also adjust related joints in a plausible way. In this example, you could also use an IK [Inverse Kinematics] change to do something similar. It is not clear that a neural network would be better, but it is another tool that animators could use.”

Wētā FX built a new facial system for Avatar: The Way of Water using a series of neural networks to manage the relationship between facial muscles and expressions. “The tools help capture sympathetic secondary motion in the face, so animators are able to connect their creative decisions more intuitively,” Letteri states. “It was slow going at first, but has proven to be a powerful tool for animators.” Special attention is paid to the training dataset. “When we train a neural network, we are careful to only use data that we own or have the right to use for a particular show,” Letteri notes. “We’re fortunate that we have a partnership with Unity where they are looking into how our technology can reach a wider group of artists outside of Wētā FX. They have announced AI projects in beta and are already doing interesting things in this space.”

Cinesite has been talking a lot about various generative and machine learning applications and what they mean for the company. “Almost across the board it’s efficiency gains,” remarks Hank Driskill, Head of CG for Feature Animation at Cinesite. “There are certain tasks like rotoscoping, which is extreme manual labor and time-consuming. I admire the people who have the patience to do it, but that will go away in the coming years and be replaced with tools that can do it automatically. There is a whole layer of art design that is busy work, such as ‘Give me the artwork for what the trashcan in the corner looks like.’ You’re going to use generative AI tools to generate a thousand different trash cans and pick one.”

The term “AI” is a misnomer. “AI isn’t intelligent,” Driskill notes. “It is a useful tool that is broadly applicable. There are all kinds of aspects of how we work that modern machine learning and AI tools are going to have a big impact on, but right now they’re not going to replace the minds that are at work there.” A prevailing question is whether AI will cause everyone to get stuck in the algorithm of creativity. “It runs that risk, especially because generative AI is only as good as the dataset that you trained it on. What is absolutely coming is an inbetween program in the style of, let’s say, Glen Keane. But then you’re stuck in the style of Glen Keane. That isn’t the same as what you get today where artists can create any style of motion.”

Generative AI tools have not been incorporated into the current workflow. “The concern with generative AI is that you don’t know where the reference came from or how it is being interpreted,” Driskill notes. “There are all of these legal challenges that are cropping up because like a lot of times with new technology, the law hasn’t caught up yet. The more that you’re building your own AI models and training it on material that you’ve created, then you’re golden.” Another legally sensitive issue is AI being utilized for vocal performances. “It is when they use AI to generate whole new lines, you get into the grey area fast. You’re now putting words in the actor’s mouth quite literally. That’s where it gets dangerous. Retooling a line is easy. I also see background characters being replaced by AI in the coming years, for sure.”

AI will provide suggestions. “Often when designing a character, you produce a series of model sheets about what this character looks like with different facial expressions and poses,” Driskill states. “The artist’s job is often to hit and dial-in those shapes, a lot of keeping it on model and point. AI is going to help clean it up.” An area of animation that could be majorly improved is cinematography. “I suspect that it will move towards the live-action paradigm where you set up the lights, cameras, have your environment in place, then you bring your actors to perform. You can have AI tools saying that you’re moving the camera too fast through this section, a real camera couldn’t do that. AI is going to give you more ways to check your work and make sure the thing you’re doing is within the bounds you’re setting, except when you don’t want them to,” Driskill says.

The quest for expanding revenue streams is constant. “People are always looking for the next revenue stream because theatrical has been struggling in recent years,” Driskill observes. “With the advent of streaming, a lot of folks aren’t going to movie theaters. However, people said the same thing when television arrived. They’re always looking for new revenue streams and new ways to produce content and put that content in front of people. When you generate a film, it’s partly about telling a story that means something to the world, but you also want to create experiences that people come out of wanting more. AI will help with producing more supplemental content. We have already talked about producing 60-second TikTok bits.”

Technological efficiencies, such as AI, are helping to lower production costs. “With the advent of streaming and their appetite for content, we’re able to create content that can be meaningful to a subset of the population,” Driskill remarks. “That excites the hell out of me, for somebody who has been doing this for a long time. We’ve gotten to the point now where we can deliver content to a broad audience without it having to be sanitized for being four quadrants across the planet. There are a lot of things being produced now that are absolutely not for everyone, and that’s okay because there are other people where a movie like Nimona will change their lives.”