VFX Voice

The award-winning definitive authority on all things visual effects in the world of film, TV, gaming, virtual reality, commercials, theme parks, and other new media.

Winner of three prestigious Folio Awards for excellence in publishing.

Subscribe to the VFX Voice Print Edition

Subscriptions & Single Issues


December 10
2019

ISSUE

Web Exclusive

How the AVENGERS Movies Impacted, and United, the World of Visual Effects

By CHRIS MCGOWAN

“It’s interesting to look at the major leaps in visual effects over the years,” observes Dan DeLeeuw, Marvel Studios Visual Effects Supervisor. “In many cases, it’s a single film that does something revolutionary, and those techniques are incorporated into other films – The Curious Case of Benjamin Button, Willow with one of the first morphs, Terminator 2: Judgment Day and The Abyss all come to mind. But there are also the big franchises that push a whole collection of techniques forward. The Star Wars trilogy, all of the Harry Potter films, and now the MCU [Marvel Cinematic Universe] have all been instrumental in changing the dynamic of the industry over the course of the series.” DeLeeuw served as Visual Effects Supervisor for Avengers: Infinity War (2018) and Avengers: Endgame (2019). Those two films, directed by Joe and Anthony Russo, represent a culmination of the visual effects developed and refined for the MCU, beginning over a decade earlier with Iron Man. With Infinity War and Endgame, the MCU – more than ever – brought together the global visual effects community in what was almost like a “war-time effort” of cooperation, commitment and innovation.

Dan DeLeeuw, Visual Effects Supervisor, Marvel Studios

ILM, Weta Digital and Digital Domain led the charge on both films, along with Method Studios on Infinity War and Framestore on Endgame, according to DeLeeuw. Cantina Creative, capital T, Cinesite, DNEG, Exceptional Minds, Framestore, Lola Visual Effects, Perception, Rise FX and Technicolor also worked on Infinity War, while those firms (with Stereo D substituting for Method Studios) also contributed to Endgame.

DeLeeuw worked on Infinity War and Endgame for Marvel with Jen Underdahl (Visual Effects Supervisor), Kelly Port (Visual Effects Supervisor), Dan Sudick (Special Effects Supervisor) and Victoria Alonso (Executive Vice President, Physical Production). DeLeeuw, Port, Sudick and Russell Earl (ILM Visual Effects Supervisor) were nominated for the 2019 Academy Award for Best Visual Effects for Infinity War, which also garnered four 2019 Visual Effects Society awards, including Outstanding Visual Effects in a Photoreal Feature (honoring DeLeeuw, Underdahl, Port, Sudick and Weta Digital’s Matt Aitken).

Russell Earl, Visual Effects Supervisor, ILM

“It’s a fantastic franchise to get to work with. Marvel has a real knack for bringing together incredibly creative, collaborative people. It’s amazing to be part of a team that is continually striving to bring great stories to fans, while at the same time creating new fans. Every film builds off the prior films, and we are challenged to always improve upon the work done on earlier films,” comments Russell Earl, ILM Visual Effects Supervisor on Infinity War and Endgame. “The MCU has consistently required leading-edge VFX and pushed the capabilities of the companies that have worked on the movies. Each movie has provided its own set of interesting challenges,” notes Stuart Penn, Framestore Visual Effects Supervisor on Endgame.

Infinity War. Left to right: Star-Lord/Peter Quill (Chris Pratt), Mantis (Pom Klementieff) and Groot (voiced by Vin Diesel). Photo: Film Frame

BY THE NUMBERS

Overall, visual effects accounted for a significant portion of Infinity War and Endgame’s combined budget of approximately $675 million. Infinity War hired 14 VFX houses to produce 2,623 shots while Endgame also utilized 14 houses to create 2,496 shots, according to DeLeeuw. Comments Drew Jones, Chief VFX Business Development Officer at Cinesite, “One of the biggest impacts that MCU films have made on the VFX landscape has been the number of VFX shots per show. This is illustrated best when you look at the number of VFX shots in the original Iron Man, approximately 1,000, and compare it to Avengers: Endgame, which had nearly 2,500 VFX shots. As you can see, VFX has come to dominate movies in this genre, and the impact of their use of VFX is felt throughout the industry.”

Matt Aitken, Weta Digital Visual Effects Supervisor for Infinity War and Endgame, remarks, “The MCU films are visual-effects driven. They encompass all aspects of high-end visual effects, from the digital performance work on Thanos, Hulk, Rocket and Groot, to very large-scale simulation and destruction work, hero digital doubles, fully CG environments, epic battle sequences and invisible de-aging work. The settings are usually contemporary, so the work is typically grounded in a physical reality, which can make it even more complex to achieve.”

Matt Aitken, Visual Effects Supervisor, Weta Digital

“The impact on the global visual effects business of having all this high-end work available has been hugely significant in supporting the current buoyant state of the industry.”

—Matt Aiken, Visual Effects Supervisor, Weta Digital

Infinity War. Thanos (Josh Brolin). (Photo: Film Frame)

 THE VFX AVENGERS: SHARING THE CHALLENGES

In the MCU to date, a significant percentage of the total visual effects workload in any given year has been shots in Marvel films. “The impact on the global visual effects business of having all this high-end work available has been hugely significant in supporting the current buoyant state of the industry,” observes Aitken. “The challenges presented in this work for Marvel have necessitated developments of new tools and techniques in a wide range of visual effects including, for example: integrating motion capture-based digital performance with live-action performance, and animation-driven physically-based simulation work – to cite two examples of the many available.”

Earl comments, “The MCU films have gotten bigger and bigger in terms of scope, scale and expectations. The films have covered the gamut of VFX – characters, environments, destruction, simulation and vehicles. We are challenged with it all. The MCU has really shown how the VFX community can pull together and share assets, shots and even technology in service to these great stories. The films are packed with so much story and so many characters, and an equally massive amount of effects. All the major VFX houses work on the films, so we are all continually pushed to raise our game.”

How did Marvel coordinate all the VFX? “Having a great producer and a great crew” were key, according to DeLeeuw. “I’ve worked with Jen Underdahl since Captain America: The Winter Soldier, and together we have worked on approximately 10,000 shots! We were lucky enough to work with a crew that spanned Winter Soldier, Captain America: Civil War, Infinity War and Endgame. The crew included our production-side team and all the awesome VFX houses that worked on the film. In some ways it came down to practice and familiarity. We developed a shorthand with all the VFX vendors. They would instinctively know how to integrate notes. That teamwork was the only way we could get through so many shots on such short schedules.”

Infinity War. Thanos. (Photo: Film Frame)

“One of the biggest impacts that MCU films have made on the VFX landscape has been the number of VFX shots per show. This is illustrated best when you look at the number of VFX shots in the original Iron Man, approximately 1,000, and compare it to Avengers: Endgame, which had nearly 2,500 VFX shots. As you can see, VFX has come to dominate movies in this genre, and the impact of their use of VFX is felt throughout the industry.”

—Drew Jones, Chief VFX Business Development Officer, Cinesite

Infinity War. Falcon (Anthony Mackie) flying over Wakanda battlefield. (Photo: Film Frame)

Endgame. Tony Stark/Iron Man (Robert Downey Jr.) (Photo: Film Frame)

Endgame. Thor (Chris Hemsworth). (Photo: Film Frame)

Alessandro Cioffi, VFX Supervisor, Trixter

In order to cope with the large scale of the projects and their complexity, the Avengers saga helped introduce a strong network of collaborations among vendors, which Marvel has promoted and brought to a new level of efficiency, according to Alessandro Cioffi, Trixter Visual Effects Supervisor. Cioffi has worked on numerous MCU movies. “The impact of this new strategy has been enormous and highly beneficial for all parties involved, from small VFX boutiques up to large studios in the field: the sharing of experience, technology and methodologies has allowed an overall growth of quality in the [making of] VFX, without mentioning the opportunity for most studios, especially the medium-small ones, to better manage their budgets.”

The MCU “forced VFX in general to up its game,” according to Exceptional Minds Visual Effects Artist Mason Taylor. As Kenneth Au, Visual Effects Supervisor for that firm, adds: “[The Marvel films] have pushed the VFX industry into a new level of possibilities, realism and quality.”

Infinity War. Left to right: Spider-Man/Peter Parker (Tom Holland), Iron Man/Tony Stark (Robert Downey Jr.), Drax (Dave Bautista), Star-Lord/Peter Quill (Chris Pratt) and Mantis (Pom Klementieff). Photo: Film Frame

A Few (of Many) MCU VFX Highlights

Infinity War and Endgame featured many notable VFX achievements that built upon the visual effect heavy-lifting of the previous Avengers films – The Avengers and Avengers: Age of Ultron – and the many MCU films that focused on individual Avengers. Highlights included the Iron Man suits and Heads Up Display (the three Iron Man movies), Skinny Steve (Captain America: The First Avenger), the teaming up of the Avengers to battle the Chitauri (The Avengers), the aging effect (Old Peggy), plus the imposing Helicarrier of Captain America: The Winter Soldier, the Ego planet (Guardians of the Galaxy Vol. 2) and the final battle scene of Avengers: Age of Ultron.

Iron Man had an impressive evolution from the start of the MCU to the (current) end. DeLeeuw explains, “The Iron Man suit and the practical suits’  legacy would influence the entire [Marvel Cinematic] universe. I think the MVP of all the effects had to be Iron Man’s HUDs [Heads Up Display]. It was an effect that captured the feel of Iron Man and solved the problem of letting an actor hidden by a helmet act.”

Endgame. The women. Left to right: Pepper Potts in Rescue Suit (Gwyneth Paltrow), Valkyrie (Tessa Thompson), Scarlet Witch (Elizabeth Olsen), Captain Marvel (Brie Larson), Mantis (Pom Klementieff) and Shuri (Letitia Wright). (Photo: Film Frame)

Infinity War and Endgame

VFX highlights for Infinity War and Endgame are numerous and include aging/de-aging craft; Hulk, Smart Hulk and Thanos (important landmarks in photoreal humanoid characters, according to Framestore’s Penn); the battle on Titan in Infinity War; and the final battle scene in Endgame.

Infinity War. (Photo: Film Frame)

Titan VFX

The battle on Planet Titan in Infinity War has become a landmark in visual effects and received many awards, including Outstanding Effects Simulations in a Photoreal Feature and Outstanding Compositing in a Photoreal Feature at the 17th Annual Visual Effects Society Awards. Aitken comments, “The combination of cutting-edge digital performance work, complex simulations, and the creation of a large, varied digital environment that was art directed to be both alien and believably naturalistic set a new benchmark for compelling 3rd-act battle sequences.”

 

Aging/De-Aging (Peggy and Steve)

Lola VFX was called upon to age Steve Rogers’ love interest, Peggy Carter, in Captain America: The Winter Soldier and would be asked to do the same for Rogers in Endgame. “For Old Steve, we went back to Lola,” notes DeLeeuw. “We again used an old-age double as a source for age spots and wrinkles and merged those onto Old Steve. What took us months on Winter Soldier took us weeks on Endgame.”

Endgame. Smart Hulk (Mark Ruffalo) (Photo: Film Frame)

Smart Hulk

DeLeeuw explains, “I am extremely proud of what we were able to accomplish with Smart Hulk and Thanos. I feel that the VFX teams were able to raise the bar in bringing the soul of a performance to our digital characters.” ILM’s Earl adds, “From the first Avengers film to the most recent, the Hulk has been a compelling character to work on. We’ve partnered with Mark Ruffalo to create Hulk’s performance, and each time we’ve been able to employ cutting-edge technology to ensure the performance Mark gives shines through. It’s really been an incredible team effort.”

Smart Hulk provided an additional challenge. Whereas Thanos’ facial movements were very restrained and intense, Smart Hulk’s movements were elastic and broad. “We started with how much of Mark Ruffalo we could incorporate into this new version of Hulk. The final design ended up looking rather handsome,” recalls DeLeeuw. “ILM used data from the Disney Research Facial Capture System, Medusa, to derive the high-resolution tracking information used to drive their facial rig. In addition, they had new software called Blank, which allowed them to rewrite their retargeting and deformation tools.”

Infinity War. Thanos. (Photo: Film Frame)

Thanos

“We knew Infinity War would live or die based on our ability to bring Thanos to the screen in a convincing way,” DeLeeuw adds. “We had the opportunity to work on a villain that could reach the heights of a character like Darth Vader. It just had to work, and because of the dedication of everyone involved it did. We started with blend shapes and ended with machine learning algorithms recognizing incredibly detailed movements on Josh Brolin’s face and transferring those moments to the digital model.”

Says DeLeeuw, “When we started Infinity War, Ryan Meinerding, Marvel’s Head of Visual Development, asked what we could do to make Thanos successful. I immediately said, ‘Let’s incorporate more of Josh Brolin into the design.’ If we could see Brolin’s eyes in Thanos, I knew it would be a great starting point for retargeting the facial capture. Ryan came back with one of the most amazingly detailed ZBrush sculpts I have ever seen. We sent the Zbrush file to Digital Domain and Weta to start developing the character. Each company brought something special to the character and we were able to merge the two. We captured Brolin during a test session, and Digital Domain came back with a test using their machine learning tools that allowed Thanos to reproduce Josh Brolin’s incredibly subtle performance. Thanos’ character was able to grow into one of our most memorable villains. In the end, it all came down to the size of the work. Without having gone through the process of making Infinity War, we could never have finished Endgame.

Endgame. The charge. Tony Stark/Iron Man (Robert Downey Jr.) (Photo: Film Frame)

The Portals

“In Avengers: Endgame, the moment when the portals open, bringing all our heroes to the battlefield, had to be faultless to allow the emotion of that sequence to play out for the audience,” says Aitken. “This sequence combines complex plate-based work with epic full-CG shots. The timing of the reveal of Black Panther, Shuri and Okoye, when the first portal opens and they step out of the bright hazy glow of the Wakanda light and into the cool dark light of the compound bomb crater, went through multiple revisions. Teasing out their reveal, which confirms that the heroes’ plan for reversing Thanos’ blip has actually worked, was instrumental in allowing the full emotional intensity of this landmark scene to play out.”

 

Final Battle

“If Infinity War was our master’s degree, Endgame was our doctorate. We used everything we learned from the previous movies and more for the final battle,” comments DeLeeuw. “One of the biggest challenges was dealing with the sheer size and scope of the battle. We split the battle between Cinesite, ILM and Weta, dealing with “thousands of digital doubles, over 50 hero characters, complex fluid sims, digital explosions, an entire digital set, each of the heroes’ special powers, dust from the blip, wet raccoon hair, Doctor Strange magic – the number of effects are almost impossible to list. Without the heavy lifting of Weta, ILM and Cinesite, we never would have finished.” DeLeeuw was more than pleased with the finished result. “Watching the final battle with an audience and listening to their reaction is one of the most awesome experiences of my career.”

Matt Aitken on Infinity War and Endgame

Matt Aitken, Weta Digital Visual Effects Supervisor for Infinity War and Endgame, worked on four MCU films and was “an avid audience member” for the others. He talks with VFX Voice about Weta’s work on the final two Avengers movies, which took the franchise’s advancements in visual effects to a new level.

VFX Voice: What was Weta’s main work on Infinity War?

Matt Aitken: Weta Digital was responsible for the 3rd-act battle sequences on Planet Titan — the Q-Ship crash, Iron Man, Doctor Strange, Spidey, Star-Lord and company planning the attack on Thanos as well as the extended fight with Thanos on Titan and the deaths of most of the heroes on Titan once Thanos gets the last Infinity Stone.

Endgame. Thanos. (Photo: Film Frame)

VFX Voice: How did you build Thanos to deliver such an emotionally and physically compelling performance in Infinity War?

Aitken: We made significant refinements to our digital character pipeline for Thanos. A lot of this work focused on making sure we could accurately re-create Josh Brolin’s performance on Thanos. We built two facial puppets, one of Josh Brolin and one of Thanos, each with a complete facial animation system. Our process involved solving the captured performance motion onto the Josh Brolin actor puppet first, which allowed us to check that we were matching the live-action performance accurately. Once we were happy with the facial animation on the actor puppet, we migrated it across to the Thanos puppet. We carefully calibrated the two puppets to allow for seamless migration of the actor’s performance onto the character. Getting this calibration right was challenging technically and involved the use of forensic data to accurately capture the density of facial tissue between the skin and the skull at various points across the head. Once that calibration was complete, the process of migrating the performance onto the character puppet was straightforward.

VFX Voice:  You were also called on to do large-scale destruction events for Infinity War.

Aitken: [This included] the Q-Ship crash and Thanos pulling a moon down onto Iron Man, a sequence that became iconic. The challenge in creating this sequence was the huge scale of the event. The Russos referred to this as a ‘global destruction’ event, and the atmosphere on Titan is filled with dust and debris for the rest of the movie. Our destruction pipeline is set up to facilitate animation-driven timing and overall shape of events like this, so our animators can define things like the speed and scale of the meteors and the speed of the shockwave that propagates when the largest meteor hits the ground. Then the FX team takes over to add simulations for the heat shield effects, fire, smoke and destruction, using the distributed sparse volumetric solver within our in-house simulation framework called Synapse. Synapse is ideal because it allows us to distribute our sims over multiple machines, which means we can preserve a very high level of local detail across this large-scale event while still adhering to a tight production schedule.

Endgame. Thanos. (Photo: Film Frame)

VFX Voice: What else did Weta work on in Infinity War?

Aitken: We developed the look of Iron Man’s new Mark 50 Bleeding Edge suit with its nano-tech particles, which we generated with procedural geometry. We also designed the look of the blip effect when Thanos uses the Infinity Stones to turn half of all life in the Universe to dust.

VFX Voice: What were some of the challenges of Endgame?

Aitken: Our first shots in the film are of Thanos’ H-Ship attacking and destroying the compound, and our work goes through to Tony’s snap and his death from the effects of the gauntlet. The end battle sequence posed several unique challenges, not least of which was the large cast of digital characters.

VFX Voice: For the final battle in Endgame, you had to deal with many heroes and villains.

Aitken: On our heroes’ side, we have pretty much everybody who has ever appeared in an MCU movie join the battle. In total, there were 47 hero digital characters in the end battle. All of them had to be available as hero digital assets, including high-level digital doubles of the live-action characters. Thanos’ army is entirely digital, including the Black Order, whom we had to prep for dialogue. Thanos needed to have a compelling level of presence, and we had to be able to identify with his decisions at an emotional level for the narrative to work.

VFX Voice: What was the single most complex shot in Endgame?

Aitken: It appears in the sequence where the two armies charge one another and clash for the first time. It’s 45 seconds long, primarily CG, and takes place in a fully CG environment, but also incorporates three separate plate elements at various moments throughout the shot. The shot includes many complex VFX simulations.

Endgame. Thanos. (Photo: Film Frame)

VFX Voice: What were the challenges of creating the bombed-out environment?

Aitken: At the studio’s request, we replaced the set with a CG environment that included building rubble, wrecked cars and chunks of concrete. The extra work also had the benefit of allowing us to take over lighting the environment, so we could replace the set key lights, which are necessarily somewhat overhead, with a low, raking, distant sun bleeding through the smoke left by the attack. It also freed us up to modulate the environment lighting in support of the emotional dynamic of the sequence – sunny before the attack, and then after the compound is destroyed, it’s overcast with a pall of smoke and dust hanging over the area. When the portals open up the battlefield becomes brighter again.

VFX Voice: How was Thanos different for Endgame?

Aitken: Thanos on Titan, in Infinity War, was somewhat of a philosopher, meditating on past actions. This Thanos is dressed in armor and ready to do battle, and his motion had to reflect that. Motion capture from Josh Brolin and the stunt team gave us a great starting point for his body motion. We heavily augmented these performances using keyframe animation to make sure he always felt believably heavy and, at the same time, agile.

VFX Voice: What about the work on his face?

Aitken: We added refinements to his facial rig to get around some shortcomings that we had experienced on Infinity War but didn’t get the chance to fix during production on that show. In particular, we added more control to the corners of his mouth. That was an area that we had to develop a work-around for in order to get the detail correct on Infinity War, so we took the opportunity to apply that fix between the two shows.

Endgame. Left to right: War Machine (Don Cheadle), Rocket (voiced by Bradley Cooper) and Pepper Potts in Rescue Suit (Gwyneth Paltrow). (Photo: Film Frame)

VFX Voice: How did Deep Shapes help you?

Aitken: We used some new facial tech that has been developed recently at Weta Digital – a tool called Deep Shapes to add further complexity to Thanos’s facial performance. Deep Shapes adds another octave of detail to the facial performance without requiring any extra effort from the facial models team or the animator. These additional shapes are derived analytically and applied procedurally. They are not a simulation. Deep Shapes adds complexity to the way one expression migrates into the next, but the start and end of the transition aren’t affected, so the animator still retains control over the shape of the face and the facial performance can’t go off-model.

VFX Voice: What improvements were made to the blip effect from the end of Infinity War?

Aitken: We redesigned the techniques we had designed to generate the blip effect, both to add support for the vast number of blip events that occur when Thanos’ whole armor disappears, and to add much greater visual complexity to the look of the effect in the long, lingering shot of Thanos blipping away.

We dealt with the large number of blip events in the shots where Thanos’ whole army disappears – a part of the movie we called the mega blip – by coming up with a couple of different level-of-detail solutions. For background blip events, we created a version of the blip that ran in Eddy, a volumetric simulation and rendering tool that runs in Nuke. The Eddy blip solution allowed us to turn around many blip events very quickly.

VFX Voice: Were there visual effects you were able to achieve in Endgame that weren’t possible in Infinity War, even with one film production following right after the other?

Aitken: There were improvements to our tools and techniques, which resulted in a lift in the quality of the visual effects we delivered on Endgame compared to Infinity War.


Share this post with

Most Popular Stories

The Miniature Models of <strong>BLADE RUNNER</strong>
02 October 2017
Exclusives, Film
The Miniature Models of BLADE RUNNER
In 1982, Ridley Scott’s Blade Runner set a distinctive tone for the look and feel of many sci-fi future film noirs to come, taking advantage of stylized production design, art direction and visual effects work.
The New <strong>Artificial Intelligence</strong> Frontier of VFX
20 March 2019
Exclusives, Film
The New Artificial Intelligence Frontier of VFX
The new wave of smart VFX software solutions utilizing A.I.
Converting a Classic: How Stereo D Gave <strong>TERMINATOR 2: JUDGMENT DAY</strong> a 3D Makeover
24 August 2017
Exclusives, Film
Converting a Classic: How Stereo D Gave TERMINATOR 2: JUDGMENT DAY a 3D Makeover
James Cameron loves stereo. He took full advantage of shooting in native 3D on Avatar, and has made his thoughts clear in recent times about the importance of shooting natively in stereo when possible...
THE PEARL: THE SUPER ALIEN MODELS OF<strong> VALERIAN</strong>
02 August 2017
Exclusives, Film
THE PEARL: THE SUPER ALIEN MODELS OF VALERIAN
Among the many creatures and aliens showcased in Luc Besson’s Valerian and the City of a Thousand Planets are members of the Pearl, a beautiful...
2018 – Year of the Rapidly Expanding <b>VFX Moviemaking Vocabulary</b>
03 April 2018
Exclusives, Film
2018 – Year of the Rapidly Expanding VFX Moviemaking Vocabulary
Industry leaders discuss trends and issues.