03 April 2024
Film
By TREVOR HOGG
By TREVOR HOGG
Whenever Robert De Niro and Martin Scorsese collaborate with each other gangsters seem to be part of the storyline, whether it be Mean Streets, Goodfellas, Casino and now The Irishman. The biographical crime drama is based on the book I Heard You Paint Houses: Frank “The Irishman” Sheeran and Closing the Case on Jimmy Hoffa by Charles Brandt, which recounts the life of mob hitman and World War II veteran Frank Sheeran, who claimed to have killed labor leader Jimmy Hoffa.
Netflix has invested $200 million into the production that features De Niro starring alongside Al Pacino, Joe Pesci, Harvey Keitel, Bobby Cannavale, Anna Paquin and Ray Romano. Complicating matters for Production Visual Effects Supervisor Pablo Helman (War of the Worlds) is the fact that Frank Sheeran (Robert De Niro), Jimmy Hoffa (Al Pacino) and mafioso Russell Bufalino (Joe Pesci) appear throughout the 300 scenes at different ages, and Scorsese did not want to utilize traditional motion-capture techniques as he felt they would hinder his filmmaking process.
Helman previously collaborated with Scorsese on Silence. “It’s incredible to work with and learn from such a master,” acknowledges Helman. “Marty blocks everything on paper to give everybody who is part of the production a sense of how he’s thinking about scenes. Marty has an incredible eye and memory. Then, oddly enough, it’s all about the gut and feeling. Sometimes when Marty looks at the work that we’re doing he starts to remember how he felt about this or that performance or why he selected a particular performance. It’s up to us to interpret what he has in mind and make him happy.”
The Irishman is a performance-heavy movie. “The cinematic narrative that Marty uses is that of a masterwork,” remarks Helman. “There is a lot of dialogue and close performances that have to do with the eyes, the chin, how the nostrils move, and the timing choices made by the actors. There were some previs pieces that we had to make, and besides the facework that we’re doing in the movie there are numerous complicated situations that had to do with starting the shot on location and marrying that into a set or vice versa. Those types of things are difficult to improvise. The previs informed everybody about the methodology and resources needed to make the shot work.”
The story spans a period of 40 years with a theatrical runtime of 210 minutes. Of the 2,300 shots, about 1,700 had visual effects with 1,000 featuring de-aged faces. ILM served as the main vendor with the work divided between facilities in San Francisco and Vancouver, while 2D makeup fixes were provided by SSVFX and Vitality VFX. “We see Robert De Niro when he is 24, 30, 36, 41, 50, 65, and after that the makeup takes over to get to 83 when he dies. Joe Pesci is more limited in that he starts at 50 and ends at 83. Al Pacino starts at 44, ends at 62, and was a completely CG performance.”
Because of the frequent cross-cutting between the various ages of Sheeran, Hoffa and Bufalino, it was impossible to cast different actors for each of them and still have the audience making a connection with the characters. “Marty is all about performances and character. It has been a difficult effort extracting performances from the actors and translating them into the younger versions.”
“We see Robert De Niro when he is 24, 30, 36, 41, 50, 65, and after that the makeup takes over to get to 83 when he dies. Joe Pesci is more limited in that he starts at 50 and ends at 83. Al Pacino starts at 44, ends at 62, and was a completely CG performance.”
—Pablo Helman, Visual Effects Supervisor
De Niro, Pacino and Pesci made different acting decisions for their roles. “Robert De Niro is the quietest of the three,” observes Helman. “His performance is subtle, with his eyes, the movement of the chin, and the way his face wrinkles into showing us an expression. He internalizes everything going on around him. Joe Pesci is subtle but sarcastic. You always know that there’s something behind what he said. Al Pacino is boisterous because Jimmy Hoffa was a hyper person.” No keyframe animation was utilized for the de-aging. “Everything was captured in 3D and re-targeted into a younger version of the actor,” states Helman. “The process enabled us to understand how a performance gets put together in the faces of these actors. It has been an incredible experience just to learn what De Niro means when he raises an eyebrow.
“I come from having worked with motion capture for five years and having markers on the actors’ faces as well as being in a controlled environment,” explains Helman. “After our first meeting with De Niro and Marty we were told that they weren’t going to wear any markers or helmets on set. The system that we researched and developed is called Flux, which is performance capture based on lighting and textures. We also created a three-camera rig so wherever the center RED DRAGON camera with a Helium sensor was moving, there were two ALEXA Mini witness cameras situated to the left and right of it that were capturing infrared footage. The two witness cameras were outfitted with infrared lights in order to throw infrared light onto the actors that was not seen by the center camera. What that did was neutralize the light direction so Flux could take those three cameras, triangulate the actor in front of the camera, and generate geometry that captured every facial nuance.”
Because Flux is dependent on lighting and texture, De Niro, Pacino and Pesci did not wear any makeup. “In order to create a younger version of the actors we spent two years building a library of performances of the targeted age for all of these actors,” reveals Helman. “Every one of those performances were catalogued by content, such as moods. That was also part of the software. Flux captures and interprets the performance at the contemporary age of the actor. The actors are 76 and 78 years old. We also had versions of the actors at all of the ages that needed to be represented. There was an effort to put all of those renders through our library, and that gave us an output of several faces from different projects they had been in, taking into consideration their performance, lighting and lens size.”
At the beginning of the project, a Medusa session was held where De Niro, Pacino and Pesci were asked to do specific performances. An interesting discovery was that their approach towards acting had changed from when they were younger. “There were some expressions that we couldn’t find any examples of,” states Helman. “That is also part of how Scorsese subtly designs a character. Four years ago, when starting The Irishman, I thought, ‘We’ll make De Niro look like he did in Goodfellas or Casino.’ But it wasn’t the case. We’re making someone who appears as the character of Frank Sheeran. When De Niro played Al Capone in The Untouchables, he was a completely different person.” Adjustments were made to the body posture to create a sense of history. “We also had to do the hands as well as mix digital hair with what was on set in order to get rid of wig lines.”
Sets and locations had to be seamlessly blended together. “There was a difference in exposure and light,” notes Helman. “The interiors used LEDs because the infrared light does not contaminate the spectrum. But when you go outside there is a lot of infrared light that comes from the sun, so we had a specific set of filters. It has been incredible collaboration with Rodrigo Prieto ASC, AMC (Silence) as what we were trying to do had to be exact and match his lighting and the performance of the actors.” The Phantom camera was utilized for slow-motion shots. “Some shots were shot 300 or 400 frames per second. It’s great to see that, because the motion blur is minimized and you can really see the performances on the faces.”
A close partnership was had with production designer Bob Shaw (The Wolf of Wall Street) in determining what needed to be practically and digitally built to convey the proper time period. “Scorsese is a method director, meaning that if there are 700 extras on set and the camera is looking in one direction, he still wants them to be performing even if they’re not in the frame. He is trying to build a world as much as possible that is real. But obviously there is the economics part of it and the logistics of shooting in New York.” As a production designer recreates a period with set dressing and props, the same approach was taken in making age-appropriate characters. “Before Marty saw any work, he asked me, ‘What is this going to do to my movie?’ I answered, ‘It’s going to create the context for the period that you’re trying to depict.’”
“The main achievement here is that we captured the actors’ performances on set under theatrical light without markers on their faces,” believes Helman. “Also, there was no other performance capture in a controlled environment. What you got was what you got. It’s taking technology away from the facial performance that takes place and translating those performances into younger versions of themselves. You’re going to see an hour and a half worth of work where Robert De Niro, Al Pacino and Joe Pesci age in front of your eyes. I’m looking forward to going to the back of the theater and listening to people react to what we’re trying to do. It was an incredible project for us to work on because the visual effects are completely at the service of the live-action shoot.”
Watch this Netflix VFX reel on “How The Irishman’s Groundbreaking VFX Took Anti-Aging to the Next Level”:
Primary VFX vendors contributing to the de-aging of the main characters and providing support for The Irishman were Dublin-based SSVFX and Vitality Visual Effects in Vancouver.
SSVFX looked after 750 shots, of which over 300 were full de-aging and 300 vanity/prosthetic support/eyes, eye bags and neck clean-up across all three main characters, as well as 50 environment work shots and 100 general composites.
“Our VFX totaled about 75 minutes of the film’s running time and was completed by a crew of over 90 within SSVFX,” recounts Ed Bruce, Head of VFX/Visual Effects Supervisor for SSVFX. “We came on board as a partner vendor of ILM’s to look after some of the de-aging. We were able to demonstrate that our 2D de-aging process was developed around a very technical approach to aging versus an artistic route, and this was key to our involvement on the project.
“SSVFX has a series of processes and procedures utilizing our bespoke toolsets specifically around the anatomical and technical approach of reducing a person’s visible age. It’s less about the art and more about the science of aging.”
SSVFX also looked after CG set extensions in most of the locations the film is set in. “We did this using either fully CG sets or 2.5D matte paintings,” Bruce reports. “We also used vintage CG cars to populate streets as well as added glass to driving shots where cast were being de-aged.”
Prior to working on The Irishman, SSVFX had invested a lot of time in re-developing their de-aging workflow. “We built our system around the principles of no blurs or tracked 3D patches, which ensures a natural level of skin-pore detail and skin movement,” explains Bruce. “Our system is all 2D and doesn’t require any Lidar scans, photogrammetry or 3D object tracking. It also doesn’t require any facial tracking markers or shooting age- appropriate doubles. It’s a system built for flexibility in post and shooting.”
In the past, establishing continuity in de-aging was often challenging. It was important that SSVFX’s 2D de-aging results matched with ILM’s 3D results. “Our technical approach matching ILM’s younger look for a character – having been provided their CG asset render references and a library of photographic back catalog reference of each actor – was straight forward, clinical and avoided the challenges of the past. It also enabled [Production VFX Supervisor] Pablo [Helman] and Marty to have the flexibility for nuanced and specific notes without difficulty.
“Being able to replicate from shot to shot and scene to scene becomes less about what looks right and more about following the set defined characters de-aging workflow,” Bruce continues. “This generally gets an artist 90% there before any creative discussion is needed around scene specific attributes like performance in expressions or lighting. Each character ages throughout the film, and therefore from scene to scene our de-aging decreases until we began adding age to [Jimmy] Hoffa and Frank [Sheeran].”
One “fun” challenge for Bruce was working on very long shots at 5K. “As we’re working on high detail on faces, proxies weren’t really an option. Speed and playback were solved by ensuring each artist station had a fast SSD disk to localize and cache on. Now anytime our crew works at 2K they’re marveling at the speed.”
Bruce found working with Helman and the ILM team on The Irishman “a wonderful experience like no other. After reading the script, we knew this was going to be a very special film, and the visual effects component was so integral to the storytelling and key to the film’s success.”
Vitality totaled 444 shots excluding the omitted shots – 115 shots de-aging the main character’s faces and almost 100 shared shots with ILM where they used a 3D approach to achieve the de-aging and Vitality de-aged the hands to match the age. They also did around 100 shots of wig fixes for different characters, greenscreen shots for a driving scene, and a handful of miscellaneous VFX work such as painting out objects that should not have existed during that time period and adding things to a scene to ensure it reflected that period. For example, they added a cupola to the Howard Johnson Hotel to match the look from the 1960s.
“Although we are a veteran team for high-end age reduction and digital makeup work, we were anxious to live up to expectations,” acknowledges Vitality Visual Effects Supervisor Jiwoong Kim. “It was very important for us to retain every nuance of an actor’s performance in all lighting conditions. Everything ended up going really smoothly and we were given the time to get the job done properly.
“We felt incredibly honored to have been chosen as a vendor for this film and we are thrilled to be a part of cinematic history. De-aging work using VFX has become very sophisticated these past few years, and we are committed to continuing to push the boundaries.”
;
