By TREVOR HOGG

Kate Winslet as Ronal, left, and Sigourney Weaver as Dr. Grace Augustine taking part in underwater and surface performance capture, which were stitched together into a singular performance. (Photo: Mark Fellman)

Images courtesy of 20th Century Studios

After recreating a famous nautical disaster that became the highest-grossing film of all time, filmmaker James Cameron turned his focus towards the skies and imagined what galactic colonialism would look like if humans discovered other inhabitable planets and moons. Avatar went on to unseat Titanic at the box office and is now being expanded into a franchise consisting of four planned sequels, the first of which, Avatar: The Way of Water, takes places 14 years after the original movie, where wheelchair-bound mercenary Jake Sully (Sam Worthington) leaves behind his crippled body for a genetically engineered human/Na’vi hybrid body to live with the indigenous Neytiri (Zoe Saldana) on the lunar setting of Pandora.

“Visual effects allow us to put up on the screen compelling and emotive characters that could not be created with makeup and prosthetics, and cannot be as engaging with robotics, and it allows us to present a world that doesn’t exist in a photoreal way as if that world really exists,” Producer Jon Landau remarks. “Those two things, combined with the story that we have, creates a compelling cinematic experience.”

The biggest technical advancement has been in facial capture. “On the first movie, we recorded facial performance with a single standard-definition head rig,” Landau details. “This time around we’re using two high-definition head rigs. We’re capturing quadruple, or more, the type of data to drive the performance. Wētā FX has a smart learning algorithm that trains on what the actors do after we put them into FACS session.”

A template is constructed before principal photography commences. “We probably have 160 people here in Los Angeles that build these files, that are a slightly cruder representation of the movie but describe exactly what it is we want to achieve,” Production Visual Effects Supervisor Richard Baneham explains. “We acquire our performances, go through an editorial process selecting the preferred performances, and put them into what we call camera loads, which are moments in time as if they are happening. A scene might be made up of 10 ‘camera loads,’ or just one depending on the consistency of the performances that we need to deal with and the intended cutting pattern. It is a true representation of our intended lighting, environments and effects to the point when we’re done with it, Jim often asks, ‘Does it match the template?’”

Returning as the main vendor to the franchise is Wētā FX, which handled the vast majority of the 3,200 visual effects shots, with ILM contributing 36. The performance capture took into account that a number of scenes take place above and below water. “When actors are performing underwater, almost always you can use the body capture, which is helpful for getting that sense of how characters would be behaving in water,” Joe Letteri, Senior Visual Effects Supervisor for Wētā FX states. “The facial gets more complicated because the face rig that we can use above ground won’t hold up below ground. We used the light system underwater, like a single GoPro, rather than a pair of high-resolution stereo cameras that we would normally use. If there was a particular emotional beat where we were on the characters and Jim needed their performance, as soon as he got the takes done in the water, he would have them come up, put on the normal head rig and repeat the performance for their facial capture above the water. Then we would stitch the two together later.”

Adopted into the Sully family is the human character of Spider, portrayed by Jack Champion. “We had Jack onstage for performance capture and captured everything with him in scale, which meant building multiple scale sets and getting the interaction from character to character in a nonuniform scale, which is a hard thing to do,” Baneham remarks. “Hopefully that culminates into a singular performance that has an integrated representation of Spider at the right scale. Then, Jim is able to take the cameras that he or I did and repeat it on set with the live-action crew. Jack as Spider live-action is often heavily informed by what he did in the capture session, but there is lots of room to maneuver or make choices on set that are different from what we did. Exterior forces impounding on a character is actually how you trick an audience into believing that something is really integrated when in fact it’s a full CG representation of Quaritch and a live-action representation of Spider. Pantomime is your enemy when it comes to integration.”

Each sequel will introduce new cultures, with the coastal-dwelling Metkayina making an appearance in Avatar: The Way of Water. “The Metkayina skin is slightly greener and has stripes that have more of an aquatic feel, while the forest Na’vi are bluer and have a stripe pattern based on tiger stripes,” Letteri explains. “The Metkayina have a stronger dorsal ventral coloring, where they are lighter in the front and darker in the back, more similar with what you would see with aquatic creatures. And they are evolved for swimming, so they have tails that have a wide end to them that can be used to help propel them through the water. The Metkayina have what we call stripes on their arms and legs that are like thicker skin, almost like flanges that can be used for propulsion. There were a number of design changes to make them distinctly adaptive to their environment, and that is part of the story. When Jake and his clan go to visit, they have to adapt, despite not being physically built for the environment.”

Marine life has been incorporated into the world-building of Pandora. “You figured that something like a fern would have probably evolved on another planet,” Letteri notes. “There are a lot of plants that we spread out to give you that familiarity, and interspersed are what we call the exotic plants, which are the Pandora-only plants. That got set up during the daylight scenes, and then for the nighttime scenes we turned on the bioluminescence, which was on both plants [ferns and exotic], which gave it that distinction. That was definitely inspired by bioluminescence underwater, and bringing it above ground and having it illuminate this whole forest. Underwater we took a similar approach by having different types of coral-like fan or tree corals mixed with new ones that give it more of that Pandora feel, and adding bioluminescence.”

Returning as an antagonist is the character of Miles Quaritch, played by Stephen Lang.

From left: Ronal, Tonowari and the Metkayina clan have a skin tone that is greener than the bluer forest-dwelling Na’vi.

“Visual effects allow us to put up on the screen compelling and emotive characters that could not be created with makeup and prosthetics, and cannot be as engaging with robotics, and it allows us to present a world that doesn’t exist in a photoreal way as if that world really exists. Those two things combined with the story that we have creates a compelling cinematic experience.”
—Jon Landau, Producer

A sentient creature called Payakan has a pivotal role. “Don’t call him a whale!” Baneham laughs. “He’s a Tulkun. What we do is try to never allow design to be for design’s sake. The kinematic structures, environmental aspects of how and where they live, what they would eat and how they would hunt, all inform the design, which is the outward expression. What is bringing them to life is the motion side of things. The motion design is understanding how something locomotes, emotes and expresses itself on a physical level. We may need to make changes to where the fins are and how big they are. Then you ask, ‘What would make sense for this creature and character?’ You can start to manipulate the design into its final end stage. I always try to ground everything in terrestrial reference, and Jim is the same. Even though we know that it’s not real, our job is to make the audience walk away feeling that place and those characters can exist [on Pandora].”

On set was Wētā FX Senior Visual Effects Supervisor Eric Saindon, who had a close working relationship with Cinematographer Russell Carpenter. “In The Hobbit, there were a few shots in the troll sequence where we put a background plate behind a bluescreen and did a quick Ncam situation so you could see what was going on,” Saindon remarks. “In this film, every live-action shot was captured with Simulcam using depth-based compositing within the live-action camera. When you are looking through the lens, our live-action and CG elements are composited together at the correct depth and placed in the proper space. We know exactly where Spider is walking in and amongst five or six Na’vi. You know where the background falls and where the eyelines of the live-action characters could be. It’s not even using the bluescreen.”

Carpenter found his third collaboration with Cameron different from lensing True Lies and Titanic. “On a regular film, the director of photography is working with the production designer from the word ‘go,’” Carpenter observes. “Here, I come in and had to make sure that the language is already in the virtual world, because I was bringing live lighting to that. Towards the beginning of the film, there were definitely scenes populated by humans that had hardly anything virtual about them, but as the story branches out into the Na’vi world, then you’re into this world of compositing. That’s the point where it gets painstaking. Especially painstaking are the jungles, because what kind of light is coming down through the canopy? We don’t have a canopy onstage, but we do have branches that we can put that are fairly close. And if you’re running through a jungle, your actor is hit by this kind of light, and he goes 15 feet and it’s a different kind of light, and then there is something else. Not only does the light have to happen at the right time, but it also has to be the proper color temperature and right quality.”

Action takes place out at sea. “That big Picador we actually built with two HamiltonJet engines in it,” Saindon reveals. “The boats were put on the same gimbal that we use for flying, but at a much bigger scale. We did simulations of those boats in the water so we got the proper motion of them going over the waves, how the bow goes up over the waves, and the way they jump the waves and move through.” Water interaction was achieved by utilizing spray cannons. “We could have faked it, because we had to add water anyway for the bow of the boat,” Saindon notes, “but what we wanted was Mick Scoresby [played by Brendan Cowell] up on the bow being hit by a wave, clearing his mask, being absolutely drenched, and knowing how to act if he got hit by a wave. We hit him with four spray cannons that just about knocked him over, because it was a lot more water than he was expecting! It puts you into the shot.”

Coordinating everything was a massive enterprise. “What was difficult for me, though it was satisfactory watching what it eventually became, was not having that payoff at the end of the day when you’ve done a live-action film and go, ‘That was such a great scene. I loved watching the actor do that,’” Carpenter states. “The payoff comes much later when you’ve seen the whole thing done by Wētā FX.” The important aspect is making sure that the technology does not overshadow the storytelling. “That’s the thing,” Carpenter emphasizes. “This goes back to Jim. The miracle is that there was five and a half years of blood, sweat and tears that went into this, and there is ton of technology, and when I look at the scenes, I don’t see any of the technology. It’s just an immersive experience. I see Jim as a North Pole explorer. He tries some challenges that he doesn’t know how to do, but is betting that he can do it by the time the film is finished.”

James Cameron Conducts a Deeper Exploration of Pandora

The human character role of Spider, portrayed by Jack Champion, was expanded by James Cameron, resulting in extensive integration between the live-action and animated performances. (Photo: Mark Fellman)

Unlike the marker-based facial capture used by Robert Zemeckis for Beowulf and The Polar Express, James Cameron decided to go the image-based route for Avatar. “It was an intense learning curve,” Cameron admits. “When we finished Avatar, what I requested was for the studio to continue to pay everybody for a couple of months so that we would have time to do a full and complete download and debrief. We did a three-day retreat during which I asked everybody to bring a white paper and notes of their three or four years of work and R&D so we could figure out how to do it better this time. Better in two ways: the end result on the screen looking better and a process that was more straightforward, efficient, intuitive and user friendly for artists.”

Along with merging their pipelines, Lightstorm Entertainment (a production company founded by Cameron and producer Lawrence Kasanoff) focused on the performance capture in water while Wētā FX looked after the CG water. “Wētā FX was responsible for developing all of the tools necessary for the computational fluid dynamic simulations that were necessary to simulate water and figuring out exactly the layers of simulation technology that would be required for that,” Cameron remarks. “In the early stages, my in-house team had to develop the methodology for capturing in the water. Our air volume used infrared. However, infrared doesn’t propagate underwater. We wanted something from a nonvisible spectrum so that our reference camera lighting didn’t interfere with the performance-capture camera system. We tested ultraviolet and that turned out to work quite well. Then we had to create the code to knit the two volumes together in real-time so that we would solve for the entire figure.”

“When people see this film, they’ll be captivated by the world-building, the visuals, and all of the things that we couldn’t do with the camera. But what they’ll be astonished by is the sense of connection to the characters.”
—James Cameron

James Cameron describes visual effects as being the fabric of his craft when it comes to making Avatar movies. (Photo: Mark Fellman)

Edie Falco joins the Avatar franchise as General Ardmore. (Photo: Mark Fellman)

For James Cameron, visual effects are part of the image-making process.

Ocean conditions had to be recreated to get the proper performance capture interactions. “We created an underwater wind tunnel so we could have actors riding on their creatures and acting and interacting with each other where we could shoot the reference camera so that we could see their facial and body performances clearly,” Cameron states. “Capture data is useless if you don’t have great reference, because that reference footage is what we cut with as editors and is how we ground truth the dataset. It’s like a triangulation process. In the end, we used that quite extensively in the animation phase.” The technology enhances rather than hinders performances. “In live-action, an actor has to maintain a performance across multiple times, whereas with performance capture they only have to do it once. When we do additional takes, those takes are exploratory, they’re not about matching. When people see this film, they’ll be captivated by the world-building, the visuals, and all of the things that we couldn’t do with the camera. But what they’ll be astonished by is the sense of connection to the characters.”

Helping to drive the storytelling is a group of in-house concept artists at Lightstorm Entertainment. “On the sequels, I started by working for six months generating 800 pages of notes on the characters and the general ideas of the story,” Cameron explains. “I gave those notes to my co-writers and was already in the design process. It was a parallel processing between the story-building and the world-building. I knew that they were going to need creatures to ride, so I came up with two creatures – an Ilu and a Skimwing. I said, ‘Start playing with these ideas.’ Meanwhile, I’ve started to write characters jumping on an Ilu and doing this and that. It emerged that the Skimwing would be more of a warrior’s mount and the Ilu is more of a local horse-like creature, not in appearance but in behavior. We have this idea from Hitchcock on down that auteurs already have the movie running in their head and it’s just of process of communicating it to everybody else. It’s not like that at all. You have this fuzzy picture. That’s why I work so well with all of these artists, because they know that there’s something there and that their input will be the final thing that people see.”

“As I was coming up as a director, there were the regular sequences and then there were the visual effects sequences,” Cameron observes. “An Avatar movie works differently. The Way of Water is three hours long, and there is not one second of that which is not a visual effect. It more plays by the rules of an animated film, except the end result looks like photography. We used to call it special effects; however, when they’re not special anymore, what do you call them? To me, they’re not visual effects anymore, but the image-making process. My live-action cast of Jack Champion, Brendan Cowell and Edie Falco worked in the performance-capture volume first, roughing in the scene so that later they wouldn’t be disoriented when shooting in front of a bluescreen and a partial set. When you ask me, ‘What is my attitude towards visual effects now?’ It’s my craft. Do I fantasize about just going out and grabbing a camera because I love to handhold it and shoot a live-action film down and dirty? Yeah, I love that. But the truth is that I get to do that within the greater story of an Avatar production.”