VFX Voice

The award-winning definitive authority on all things visual effects in the world of film, TV, gaming, virtual reality, commercials, theme parks, and other new media.

Winner of three prestigious Folio Awards for excellence in publishing.

Subscribe to the VFX Voice Print Edition

Subscriptions & Single Issues


January 18
2018

ISSUE

Web Exclusive

Credit Where Credit is Due – Motion Capture

By MICHAEL GOLDMAN

Andy Serkis with a facial-capture rig on steals a precious serum from a laboratory as the Ape Caesar in Rise of the Planet of the Apes. (Photo copyright © 2011 Twentieth Century Fox Film Corporation. All rights reserved.)

About three years ago, the visual effects’ community debated actor Andy Serkis’ use of the term “digital makeup” while attempting to explain the nature of his collaboration with the Weta Digital visual effects’ team when doing performance-capture acting work to create the iconic ape character Caesar in the Planet of the Apes series and in previous projects, such as Lord of the Rings. That was part of a larger industry conversation that has continued and expanded in recent years as performance-capture has become increasingly sophisticated on the one hand, while simultaneously, studios are pursuing acting award campaigns for actors like Serkis, whose work forms the foundation for completely CG characters. In fact, Serkis was named Best Actor by the San Francisco Film Critics Circle, heralding a watershed moment for motion capture performance.

Indeed, the process has evolved at such a high level that, increasingly, A-list actors are donning lycra motion-capture suits and facial markers at a rapidly increasing rate. Besides veterans like Serkis, Bill Nighy, Terry Notary, and Tom Hanks, actors like Benedict Cumberbatch, Christian Bale, and Cate Blanchett, among others, are now participating in the upcoming film directed by Serkis—Warner Bros. version of The Jungle Book, which unlike the 2016 Disney version of the story, is relying heavily on performance capture methodologies.

Thus, the question of how best to apportion credit for authoring such performances—how to define, properly acknowledge, and honor the visual effects’ contribution to the final on-screen character—has become a hot-button issue across the industry.

On the one hand, as John Knoll, chief creative officer and a veteran visual effects’ supervisor at ILM, states, “If you look at the credits of any of these movies, and see how many animators are listed, it’s clear those animators must be doing something pretty important.”

John Knoll (Photo Credit: Brent Bowers)

“But the performance is only going to be as good as who drives it. A Bill Nighy [Davey Jones from the Pirates of the Caribbean franchise] or an Andy Serkis are still making the same emotional, dramatic choices as any other actor would make. The timing, intent, tone and all those things are things that are being captured, and those are choices made by the actor on set. But I dislike the term ‘digital makeup’ even though I understand the desire to communicate the idea of what is being done, because there is the danger of trivializing the contribution of all the [visual effects’] people who work really hard, and put a lot of art into it.”

That issue of trivializing their contribution is something visual effects’ artists passionately care about. Many of them emphasize that with performance capture characters, no matter the high level of an acting performance, “there is never a direct translation” from acting to the digital realm, as veteran visual effects supervisor Jeffrey A. Okun, VES puts it.

“The whole point is to get something [on screen] that looks good,” Okun says. “So, unless you are going to photograph it directly, if it is in the CG world, the artist therefore has to do a ton of work one way or another. There never is a direct motion-capture translation, so you want to honor the performance, but also the artists who follow the performance and do stuff to [characters] that actors frequently never hear about. It’s a collaborative art.”

Today, Serkis calls that collaboration “a marriage of skills, no question about it.” But he adds that what he and others have been trying to suggest is that during the course of creating a fully realized performance in a digital character using performance capture methodologies, “the animator’s role is, in this particular type of filmmaking, very specific. That is obviously different from [what animators do] in creating a more traditional-style CG movie, or another animation-pipeline style movie, where the animator decides all the character’s expressions, and so forth. In this work, they don’t own the character in the same way that an actor does.”

When contextualized that specifically, many visual effects’ professionals concede the animator’s role has changed for performance-capture projects, but they emphasize it remains just as important.

Hal Hickel

Actor Bill Nighy as tentacle-faced Davey Jones in facial-capture rig for Pirates of the Caribbean: Dead Men’s Chest (Photo credit: Industrial Light & Magic. Copyright © 2006 Disney Enterprises, Inc. and Jerry Bruckheimer, Inc. All rights reserved.)

“There is a big overlap between what an animator does and what an actor does—it’s a Venn diagram,” says ILM animation supervisor Hal Hickel, who was part of the team crucial to creating the Davey Jones character, among others. “There is a big overlap in what they are trying to do in terms of creating a character you can believe in. The actor, however, is doing it in real time, improvising, and so on, while animation is deliberate and planned, a whole different process. But there is that overlap and a lot of common language between the disciplines. Yes, we do use a great deal of motion-capture, but at least in visual-effects films, we use it for things it makes sense for, so why wouldn’t we use it?

“But for animators, what has been great about that is the fun of partnering with an actor, partnering with the performance they created on set, honoring that performance and doing our best to make sure it is faithfully represented in a digital character. It’s never plug-and-play, after all, so you still see almost all the gifts of the animator coming into play. I say ‘almost’ because there is one fun aspect of animating that does sort of get removed when dealing with [performance capture], and that involves conceptualizing the performance. In [such movies], the animator doesn’t get to do that anymore.”

However, the visual effects’ team has not only the responsibility for “honoring” the original performance, as Hickel stated, but also for maintaining its effectiveness in the midst of all manner of changes, enhancements, and what works best for the evolving story. Knoll points to Davey Jones, a character that debuted in 2006’s Pirates of the Caribbean: Dead Men’s Chest built on Bill Nighy’s acting performance. That character features creature-like tentacles on his face—elements that had to be woven into Nighy’s performance, Knoll emphasizes.

“We wanted to honor what Bill Nighy was doing on set,” Knoll explains. “But that said, there were many things Bill could not do, involving the tentacles especially. So they had to be fully animated and yet part of Bill’s performance. When the character gets angry, the tentacles writhe around in an agitated fashion. Also, the design of the character meant he didn’t have a nose. So if he does a sniffing-like action, it is up to Hal’s [animation] team to figure out what Bill Nighy was trying to communicate with that. So there are a lot of things that are animator choices, but as much as possible, we try to base them on what the actor was doing as we apply them to the CG character.”

Okun insists that any attempt to totally separate the technology, technique, and animator’s artistry from the acting performance is likely to miss the mark—they are inexorably tied together, he believes. Indeed, he argues another, typically forgotten aspect of the visual effects’ contribution is the tireless work done by ever-present development teams at major facilities around the industry and by leading manufacturers to invent and evolve the tools that make it possible to capture an actor’s on-set performance to begin with, and insert it into a CG character seamlessly. “You can have the greatest motion-capture performance of all time, but if the motion-capture technology is so crude that it does not pick up the details, then the visual effects’ artist has a heck of a lot more work to do,” he reminds. “Without [sophisticated technology development], there would be no Gollum or Caesar or Avatar.”

Many point to the development and expert usage of facial capture techniques, in particular—motion-captured acting performances simply didn’t translate properly without extensive keyframe animation until those techniques were perfected by the visual effects’ industry. That brought an end to the Uncanny Valley’s zombie-like faces and moved the state-of-the-art closer to stunning realism than ever before, but animation still remains crucial to making such characters complete, industry professionals emphasize.

Jeffrey A. Okun, VES

Joe Letteri, VES

“These are really detailed, intricate models of the face that we build, and the way the face changes through all the different expressions and phonemes is complicated, and still not well understood, even by those of us sitting there looking at it dozens of times each day,” Weta Digital senior visual effects supervisor Joe Letteri, VES says. “So there is a lot of artistry that goes into interpreting it, and getting it ready for animation, and then the actual animation itself. There are a lot of choices you have to make that if you get right, you get a great performance, and if you get slightly wrong, it looks dead. The difference between the two is not really always obvious, why one thing is working and one thing is not.

“There is still quite a lot that has to get done by hand to make [digital faces] work. The facial capture process involves recording data and using some sort of an algorithm that you have created to translate that data into the performance. There is no perfect algorithm to do that. So the final judgment comes down to us, as artists, looking at it and saying, does it work or does it not work? And if it doesn’t work, we throw away data or we modify it and we do whatever it takes to make it work. To be clear, throwing away data doesn’t change the fact that we are trying to get exactly what the actor performed. It’s just that the algorithm can’t always give you that, so you have to go in there and do it by hand. This is where it can become confusing for people. There is no one button that says ‘performance in, character out.’ It still requires a large team of artists to do that every step of the way.”

Actors on horseback in motion-capture suits evolved into fully realized digital apes for a scene from Dawn of the Planet of the Apes. (Photo copyright TM and © 2017 Twentieth Century Fox Film Corporation. All rights reserved.)

Those unique tools, in combination with the tireless labor of artists at facilities like Weta Digital, ILM, Digital Domain, Moving Picture Company (MPC), and others, therefore are inseparable from a fully-realized performance-capture success. Indeed, some visual effects’ professionals insist that the entire invention, conceptualization, execution, enhancement, and finalization of a performance so that it fits perfectly into a character according to the filmmaker’s requirements is so heavily dependent on visual effects that it would be out of context to honor an actor for such performances without the VFX team alongside. Others think honoring such actors is fine, but they still want the world to know the full story of how their performances were constructed.

“The awards’ subject is tricky—are you talking about the character or the acting?” Letteri asks. “You can break it down any number of ways. If you look at pure acting, you have to say what Andy did [as Caesar] on [the Planet of the Apes movies] was phenomenal. If you saw it on stage, without the digital Caesar, you would still think so. But you look at the total package of taking the performance and creating a character, then a translation has to happen—not just the performance, but the skin, the hair, the eyes, everything that makes the character look like a real ape you photographed on set. So that is not an easy question to answer.”

Actor Alan Tudyk in motion-capture suit play plays K-2SO, an Imperial droid who was reprogrammed and finds himself with the rebellion in Rogue One: A Star Wars Story. (Photo copyright © 2016 Industrial Light & Magic, a division of Lucasfilm Entertainment Company Ltd., All Rights Reserved.)

Hickel adds that “I’ve done a lot of thinking about this topic, and I feel like I still have more to do about it. But my basic feeling at this point, and I’m an Academy member, is that the awards’ categories we have now are currently adequate. I think if people feel a performance as a character is worthy of an acting nomination, OK, then they should nominate [the actor], but then the visual effects’ team should be nominated in the visual effects’ category. It’s like the [1980] Elephant Man movie. John Hurt was nominated as an actor, but then it seems to me, they should have nominated the makeup artist in the makeup category. Costumes and makeup all contribute in that regard. But like I said, I have more thinking to do on this.”

Knoll says he has no problem with an actor receiving an award “for delivering a really good performance, even if you never directly see the actor if that performance is applied to a CG character. Artists are working on top of that and people should know that, but I don’t want to disqualify an actor if he happens to work to drive a CG character.”

And that is part of Serkis’ argument. For better or worse, his collaboration with the visual effects’ world has put him front and center in this discussion, with studios pushing for some method of honoring performances by actors like him. When asked if a hybrid award or “some kind of joint category” should eventually be created, he points out that “if an actor puts on prosthetics and plays a role, there is no prejudice there, even though his work does revolve around a team of artists disguising the actor. If an actor is playing a role from page one to page 120 of the script, and he’s on set every day, should they be disqualified from consideration [for an acting award] because they are participating in [performance-capture work]?”

And so, a dichotomy exists. Is it “the actor’s” performance alone? Is the animator the “guardian of the performance,” as Serkis suggests, or an equal participant in its creation to begin with, as many in the VFX industry believe? As with many areas where technology and artistry intersect in unexpected ways, there may be no quick or easy way to answer such questions, or adding or adjusting how honors for such work are distributed any time soon. But, in either case, VFX artists and actors alike insist that such characters work best when there is a seamless collaborative process in the best filmmaking tradition between artists from both sides.

Karin Konival

Indeed, Karin Konival, the actress who played the orangutan Maurice in the three modern Planet of the Apes movies, suggests that, in her experience across three movies working closely with the WETA Digital team, “it’s an ensemble really. We are all part of the same crew. Just as with the cameraman and the grips and lighting and sound, everybody is working together, with the focus remaining on our performance under guidance from a director. It doesn’t feel separate at all.”

”The artists and technicians that work with us all the way through the filming process to make sure every muscle in the face, every movement is incredible are so sophisticated, and it is incredible what they have done. I don’t think [honoring] one should take away from the other.”

“The idea of apportioning credit for motion capture performances comes down to a few central themes,” states Mike Chambers, VES Chair. “It’s about garnering respect and recognition for the visual effects professionals who lend their vision and expertise to bring fully realized CG characters to life.  And it’s about honoring the inseparable collaboration that takes place among the actors and VFX artists, without diminishing the vital contributions either makes to a final on-screen performance.”

“VFX technology is evolving at a rapid pace, and we’re able to produce great imagery in profound new ways,” Chambers adds. “While the subject of establishing new or hybrid award categories to keep pace with sophisticated digital performances is a worthwhile discussion – one likely to intensify as the industry advances – the crux right now is to ensure some kind of parity in crediting both partners in the effort.  We all take pride in the outstanding work we are able to create together and it stands that we all benefit when that authenticity carries through the lifespan of the work – from concept to accolades.”


Share this post with

Most Popular Stories

PROGRAMMING THE ILLUSION OF LIFE INTO THE WILD ROBOT
01 October 2024
Exclusives, Special Report
PROGRAMMING THE ILLUSION OF LIFE INTO THE WILD ROBOT
For The Wild Robot Director/writer Chris Sanders, achieving the desired visual sophistication meant avoiding the coldness associated with CG animation.
REALISTIC FACIAL ANIMATION: THE LATEST TOOLS, TECHNIQUES AND CHALLENGES
01 October 2024
Exclusives, Special Report
REALISTIC FACIAL ANIMATION: THE LATEST TOOLS, TECHNIQUES AND CHALLENGES
Facial animation has come a long way since the days of animatronics, as filmmakers now have access to a choice of advanced facial motion capture systems.
META QUEST 3 AND APPLE VISION PRO SPARK SURGE IN VR/AR HEADSETS
01 October 2024
Exclusives, Special Report
META QUEST 3 AND APPLE VISION PRO SPARK SURGE IN VR/AR HEADSETS
While sales of VR and AR headsets were gloomy in 2023, 2024 has seen a solid recovery.
VFX IN ASIA: BOOM TIME
01 October 2024
Exclusives, Special Report
VFX IN ASIA: BOOM TIME
Asia is playing an evolving role in shaping the global VFX ecosystem.
ARTISTIC ALCHEMY: THE PERSONAL CREATIONS OF VFX TALENT
01 October 2024
Exclusives, Special Report
ARTISTIC ALCHEMY: THE PERSONAL CREATIONS OF VFX TALENT
In the world of visual effects, digital toolsets reign supreme, but that does not mean that traditional mediums have been tossed to the wayside.