03 April 2024
Cover
By BARBARA ROBERSTON
By BARBARA ROBERSTON
“How did they do that?” may not be your typical reaction to scenes in an animated feature. But consider this shot with two characters from Disney/Pixar’s latest film Soul: A tall stick figure, a “counselor,” is trying to catch a “soul,” a small, soft three-dimensional character with a big round head and almost no body. The counselor is also 3D, but looks like a partially shaded, partially transparent line drawing with a Picasso-esque face. His nose and mouth point in one direction, his eyes in another. One eye is outside the lines entirely. As the counselor frantically scrambles after the little soul, he grows extra arms for a moment, then stretches one arm w-a-a-ay out, grabs the soul, lifts it up… and drops it. As it falls, the soul loses shape entirely and stretches, eyes growing big, until it lands on another soul, the character Joe, and melts over his head.
Joe Gardner (Jamie Foxx) is a middle-school music teacher in New York, who, on the very day his dream to be a jazz musician is about to come true, steps into an open manhole. Joe’s soul escapes “The Great Beyond” and lands in “The Great Before” where new souls train before birth. There, he meets 22 (Tina Fey), a soul not yet born who looks with skepticism at the Earth. It befalls Joe, a soul who has already lived and wants to live again, to show 22 the promise of life. And thereby hangs the tale – a soul who doesn’t want to live meets a soul who doesn’t want to die.
But what do souls look like and where are they before they are born? “The characters started as design problems and then quickly became technical challenges,” says Soul writer/director/executive producer (and Chief Creative Officer at Pixar) Pete Docter. “In literature and traditionally, people think of souls as ethereal. We wanted to see through them like fog but not be distracted by what was behind. How do we film that? And, that bled into location. Most religions talk about the afterlife. Few talk about before life. It’s one of those things that if you think about it too hard you’ll break your brain.”
The creative team decided the story would put blank-slate souls in a kind of training camp designed with grand abstract pavilions of knowledge. Mentoring and sorting those souls would be counselors like Jerry (Richard Ayoade), the tall line drawing who grabbed 22 and dropped her on Joe’s head.
BODY AND SOUL
“The souls are not physical characters,” says Michael Fong, Visual Effects Supervisor. “They can change shape, elongate and stretch. They have no physical form.”
Designers gave the souls faces, but the character’s ambiguity led them to lean on the technology group to help create foggy volumes that would sustain interest. “We needed to be inspired by technology,” Docter says. “We had weekly reviews. What if we push this way? What if we put a line around the fingers? It was cyclical art that was possible because we could be iterative and adaptable and change things quickly.”
Docter didn’t want the souls to look like the emotions in Inside Out or otherworldly characters depicted in other Pixar films, so the team created souls that catch light and reflect it in an atypical way. “The characters are volumetric, like fog,” Fong says. “But Pete didn’t want the volumes to look like volumes that have used correct math. He wanted pink here, blue there, green in the middle. We didn’t know how to do that.”
Over the past 10 years or so, physically-based lighting and rendering techniques have become standard for look dev, lighting, and rendering in most VFX and animation studios. They simplify lighting setups and bring synthetic worlds closer to reality. At Pixar, artists use RenderMan, a homegrown toolset based around Katana, for lighting and Nuke for compositing, a typical set of tools for many studios. But these characters weren’t typical.
“We wanted the characters to feel luminous, not emissive,” says Ian Megibben, Director of Photography. “Joy, a character in Inside Out, was a volume, but she was a light source. Soul’s Joe and 22 are also volumes, but they are affected by an external light source. We used volume rendering for a good portion of the soul world and characters. But we didn’t want them to take on local color. We didn’t want the shadows to look dark. And we didn’t want them to come across as a ghost. I remember asking the designers what their expectation was for light entering one end of a volume and coming out the other end. We narrowed down on the theme of prismatic light.”
BLUE IN GREEN
For reference, the artists looked at opalescent glass, beetle shells, rainbows in the mist – anything that showed light splitting. Then, rather than shadows, they used variations of color to create something that would feel like shadows.
“The rainbow was our surrogate for light and shadow,” Megibben says. “Warm colors represented light and cooler colors represented shadow. The characters shift from yellowish green to turquoise or teal to, on the unlit side, deep blue as shadows. On the edges, they have other rainbow colors – magenta, orange, yellow. We call that
their ethereal helmet. To preserve the luminance so they felt glowy, we relied on the hue of the color.”
Shading technical directors created a system in Houdini, Katana, Mari and Pixar’s own shading tool called Flow. Working with the shading team. the lighting team devised a method for casting shadows that was independent of light falling off a rounded object.
“I relied heavily on techniques from 20 years ago for this project,” Megibben says. “How do you ensure a mouth that is a translucent volume occludes light? Well, it doesn’t actually do that. At the end of the day, the rules of composition and creating an image still applied, but the techniques and tools were totally different, home grown and home conceived. We had to create our own lighting system. We used old techniques that were new to some of the younger artists. I hope that people in the industry will scratch their heads a little bit and think, ‘I don’t know what they did.’”
As might be imagined, rigs for these volumetric characters were tricky. Facial features were more important than the bodies – their head-to-body ratio was almost one-to-one and sometimes their legs would disappear. 22 has fingers only when she needs them; otherwise she has “mittens.” Sometimes there are lines around her fingers.
“It put a strain on our workflow,” Docter says. “In some cases, we could push the rig to the limit of what it could do and then go in and try to fix things. But we also had to build stunt versions of the characters.” Compositing techniques helped the final volumetric characters completely break from the physical world.
ALL OF YOU
Breaking from the physical world was only the half of it when it came to the counselors, which can completely break apart. “I usually don’t freak out when the art folks come up with ideas for characters, but I started to panic when they came up with the counselor characters,” Fong says. “They look different from different angles, maybe not even like a character. They’re doing their best to look humanoid, but don’t quite get it right. They can pop out arms and faces wherever they want. Their faces break, mouths break off. And we had to fit membranes between the lines. They’re not just planar.”
The design looks simple, but it’s deceptively difficult to create shape-changing, three-dimensional characters formed using lines and a membrane that appear to be two-dimensional.
“We had to break down the situations they would be in,” Fong says. “Some turn into buildings or are buildings. The arms could go up and down, but the shoulder joint could be anywhere in the wire frame. We had to build and rebuild the rig five times. We cheat by flattening [the 3D models] down in screen space, but that didn’t give us a viable membrane. So we build the membrane on the fly per frame. We had rules establishing where the membrane needed to be more solid and where it needed to be transparent. We’d never seen characters like this before. There was a lot of handwork.”
Concept artists, riggers and animators worked simultaneously for a year and a half to figure out what the counselors were and to develop systems that would work. Lighting artists created rules for what key light means on characters that appear to be flat, and shading artists created complex systems in Houdini that determined when to color inside the lines. Sometimes you can see through the characters; sometimes the lines are filled.
“This film is about me re-examining the worth of it all, about what are we doing with our time, the reason for doing it at all. Those are the things we’re exploring. What are we waking up for every day? Let’s examine what this is leading to.”
—Pete Docter, Writer/Director/Executive Producer
AIN’T MISBEHAVIN’
“Creating a system that kept everything consistent was fascinating,” says Jonathan Hoffman, Character Shading Technical Director. “I would dream about it at night. There were all these constraints: If a counselor swept a left arm in front of another arm, anything within would disappear. It was like a Venn diagram.”
“The souls are not physical characters. They can change shape, elongate and stretch. They have no physical form.”
—Michael Fong, Visual Effects Supervisor
Animators had the freedom to move the counselors as they wished. A counselor could become a staircase that it walked up. It could suddenly have multiple arms. An arm could cross under an armpit. The shading system needed to know what to render. “Imagine taking a pen and drawing it across the character,” Hoffman says. “You count the number of times it touches a line and remove the even lines. If it hits one, that’s positive, keep that. Hit another line, that’s two so cut that. That’s how we’d march a ray across a two-dimensional shape. But the character has lines going down the middle that break the simple logic. The right side and left weren’t always right and left because sometimes the character would twist. Then the membrane had to exist within the scene as a 3D shape. We try to render everything in a single frame. So after we flatten the characters, we return them to 3D space so the membrane can exist there.”
SPACE IS THE PLACE
The souls and counselors exist in The Great Before. Creating the non-physical environment they wander around in became another design adventure. “We wanted a place that felt soft, that had a sense there were no boundaries,” Docter says. “It extends into an undefined horizon. A lot of the shapes and choices we made were trying to reinforce the idea that this place inspires knowledge. The souls are blank slates, but when they come to Earth they have a personality. So we have pavilions like in ancient Greece and in
world’s fairs.”
Motif, an in-house tool for set dressing, provided real-time feedback to artists trying to visualize the ideas in Docter’s and the art director’s minds. “We could build fields of grass and pavilions and get feedback in reviews, and then make changes in real-time,” Fong says. “The grass might not be wispy enough or too opaque, or need to be taller and longer. We could change that on the screen.”
NEW YORK
Contrasting with The Great Before is Earth – specifically, New York City, the place Joe wants to return to and 22 wants to avoid. To Joe – and to the film – The Great Before is ethereal. New York is jazz.
“I grew up in a musical family,” Docter says. “My parents were musicians and teachers and my two sisters are professional musicians. But instead of listening to classical music, I listened to ’30s and ’40s jazz. As we were looking for a main character who would be born with a passion, I thought about a lot of occupations, even animator. But it seems like a musician, especially a jazz musician – there’s a nobility without selfishness. It’s not like someone is going to get rich being a jazz musician. You do it because you have a passion. That jazz is being written as it’s played spoke directly to the theme of the film. You’re given a tune. What do you do with it?”
To create the New York that a Black jazz piano player like Joe Gardner would know, Docter relied on a “culture team” drawn from, among others, African-American artists at Pixar, jazz musicians in New York and co-writer Kemp Powers who became a co-director. Powers is the same age as Joe, is a jazz musician, grew up in New York, and he’s Black.
“I had a lot to learn,” Docter says. “The peek into the America that’s right next to us was eye-opening, not only the culture but the things African-Americans have brought to America and their experience of being marginalized. There was a lot I didn’t know that I didn’t know. We found musicians and teachers in New York and asked them, ‘What’s your lifestyle? What posters are hanging on your wall?’ Initially, I thought 90% of the film would be in The Great Before. As it turned out, most of the film focuses on New
York.”
Just as jazz influenced the theme of the film, it also infused the style of New York, the detailed sets and environments such as the West Village, Queens, a barber shop, a tailor shop, the Half-Note jazz club, the middle school classroom. “When you look at a fence, you can see syncopation in the rhythm of the slats,” Hoffman says. “We wanted no straight lines. No repeating patterns. Long, elegant, exaggerated shapes, even in Joe’s design.”
The set designers looked at 101 Dalmatians as they considered giving Joe’s New York an improvisational style. “Our sets team loved working on New York and put a lot of time into making sure everything had enough detail,” Fong says. “We wanted to see if we could combine the real world with something almost impressionistic, rather than having cobbles or bricks everywhere we can suggest it.”
TAKE FIVE
But when it came to the musicians, the crew wanted to make sure that whether in Joe’s classroom or the jazz club, the musicians were really playing music. “We set up about eight million cameras all around the musicians,” Docter says. “We have close-ups of their fingers, close-ups of details all around.”
“We needed to be inspired by technology. We had weekly reviews. What if we push this way? What if we put a line around the fingers? It was cyclical art that was possible because we could be iterative and adaptable and change things quickly.”
—Pete Docter, Writer/Director/Executive Producer
Then, with help from Disney, they converted the music to analyze it electronically. “We ran that music into the digital instruments,” Fong says. “You can see the keys light up on the virtual piano. Animators can know which key is pressed at what time for how long. We would sync that with the video so they could see the way the musicians use their fingers. If you watch Joe’s hands you will see how he lifts his fingers off and presses with the sides of his fingers.”
“Creating a system that kept everything consistent was fascinating. I would dream about it at night. There were all these constraints: If a counselor swept a left arm in front of another arm, anything within would disappear. It was like a Venn diagram.”
—Jonathan Hoffman, Character Shading Technical Director
“At the end of the day, the rules of composition and creating an image still applied, but the techniques and tools were totally different, home grown and home conceived. We had to create our own lighting system. We used old techniques that were new to some of the younger artists. I hope that people in the industry will scratch their heads a little bit and think, ‘I don’t know what they did.’”
—Ian Megibben, Director of Photography
They also hooked a midi device into a keyboard so they could port the music into Pixar’s animation system. “The animators probably cheated some things, but they created
fantastic authenticity,” Docter says.
MILESTONES
Soul is the third feature film Pete Docter has directed, and he has received major honors for his previous: Oscar, Annie and BAFTA awards for Inside Out, plus nominations for Best Original Screenplay from all three organizations; Oscar, Annie and BAFTA awards for Up along with nominations for Best Original Screenplay from the three organizations and a VES award for outstanding animation; an Oscar nomination and a BAFTA Children’s Award for Monsters, Inc., the first film he directed. He also has an Annie Award for writing Toy Story 2, an Oscar nomination for writing Toy Story, and an Annie for Toy Story animation.
So for Docter to consider whether what he has been doing is worthwhile comes as a surprise. But that was indeed the motivation for this film.
“I bet a lot of people in visual effects are like that, too, in that you go into a line of work that appeals to you, you become passionate about what you’re doing and it becomes
your world,” he says. “This film is about me re-examining the worth of it all, about what are we doing with our time, the reason for doing it at all. Those are the things we’re exploring. What are we waking up for every day? Let’s examine what this is leading to.”
Those questions could have been explored – and have been explored – in live-action films. But an animated film makes it possible to do so in unique, creative and visually stunning ways. A Great Before filled with blank-slate souls. Who else could have imagined such a place and have the technology and artistic experience to put it on “film?”
“One of the joys of working here at Pixar is to have this amazing bunch of artists bring this stuff to life,” Docter says. “They allow us to open deep doors.”
;
