How Framestore Fleshed Out ‘Smart Hulk’ for AVENGERS: ENDGAME

By TREVOR HOGG

Machine-learning software was trained to capture micro-moments that made for a more believable facial performance. Front is Chris Evans, who plays Steve Rogers/Captain America. (Images courtesy of Framestore and Walt Disney Pictures/Marvel Studios.)

Along with ILM, Framestore was responsible for creating Smart Hulk, a peaceful coexistence between scientist Bruce Banner and his volatile, green alter ego that makes his first appearance in Avengers: Endgame.

“ILM started developing Smart Hulk probably during Infinity War,” states Framestore Visual Effects Supervisor Stuart Penn. “Before the beginning of Endgame they turned over the base model, textures and some of the face shapes. From that, we developed our own muscle system, added to the face shapes, built our own face rig, and were provided reference by the client of turntables and facial workouts that ILM had done to make sure our facial performances were matching.”

Stuart Penn, VFX Supervisor, Framestore. 

Smart Hulk was the most complex facial rig created by Framestore. The two key areas to translate were the movement of Mark Ruffalo’s mouth and the way his eyes moved. 

“It was down to the level where [Mark Ruffalo] has specific muscle structures in his face, and it was translating those movements into Hulk. The two key areas were the movement of his mouth and the way his eyes move. It was the essence of Mark’s performance coming through on quite a different face.”

—Stuart Penn, Visual Effects Supervisor, Framestore

Machine-learning software was trained to achieve certain keyframes in specific shots.  “That allowed us to generate a quick first-pass animation for the entire sequence,” explains Penn. “We would quickly render it out and send it back to the client so they had something quickly for their edit. We then moved to keyframe animation on the facial stuff to get what we weren’t getting from the solves. The solves also gave us a micro-movement pass that we could extract from the machine-learning data.”

Because Smart Hulk is a hybrid of Hulk and Bruce Banner, there was a heavier reliance on the performance capture of Mark Ruffalo. “It was down to the level where he has specific muscle structures in his face, and it was translating those movements into Hulk. The two key areas were the movement of his mouth and the way his eyes move. It was the essence of Mark’s performance coming through on quite a different face.”

Smart Hulk is slightly less green in Endgame, as the desire was to have a more human subsurface skin coming through underneath. At right is Scarlett Johansson, who plays Natasha Romanoff/Black Widow, and left is Chris Evans.

“It was the most complex facial rig that we’ve ever done. It had many more shapes and additional controllers for the animators to get that fine level of performance, and then all of the dynamic stuff and machine-learning micro-movement that we then plugged in on top.”

—Stuart Penn, Visual Effects Supervisor, Framestore

“It was the most complex facial rig that we’ve ever done,” continues Penn. “It had many more shapes and additional controllers for the animators to get that fine level of performance, and then all of the dynamic stuff and machine-learning micro-movement that we then plugged in on top. The body rig was similar to other rigs that we’ve done before, but we didn’t have a full, working muscle system in there, which we needed.  Marvel was keen that he’s muscly and quite bulky, so when he moves and flexes his arms you actually see those muscles moving and sliding underneath the costume. We had to overdo the muscles to read them through the cloth of the costumes.”

Smart Hulk needed to be able to convey a wide range of emotions. “The hardest thing was that he was doing comedy performances quite a bit. Comedy is all about timing and subtle changes in expressions. We would always go back to watch what Mark did to see what it was about his performance told you what his emotional state was.”

A more finesse approach was adopted for the animation of Smart Hulk since he is capable of graceful and delicate movements. At right is Chris Evans.

Smart Hulk (Mark Ruffalo) was rigged to believably handle a wide range of emotions, including comedy, like in this moment with Chris Evans.

“Marvel was keen that he’s muscly and quite bulky, so when he moves and flexes his arms you actually see those muscles moving and sliding underneath the costume. We had to overdo the muscles to read them through the cloth of the costumes.”

—Stuart Penn, Visual Effects Supervisor, Framestore

Unlike his unruly predecessor, Smart Hulk is capable of graceful and delicate movements. “You wouldn’t have caught him picking up a pencil and poking the keyboards, but now he is,” states Penn. “These are precise actions, rather than Hulk smashing, grabbing a car and throwing it. It was a different level of animation. It was much more finessed.”

Having Smart Hulk interacting with the console was straightforward. “We found and filmed people with chunky hands to see the way all of the muscles and tendons moved and simulated that on top as well. We had him eating ice cream as well. We were trying to work out what sort of instrument does he use? Does he use a big or little spoon? His hands are disproportionately big to his mouth. It was tricky finding the right size spoon for him to hold in his fingers without it looking uncomfortable.

The entire hangar environment was built in CG along with the exterior view of the Avengers’ compound. 

“You wouldn’t have caught him picking up a pencil and poking the keyboards, but now he is. These are precise actions, rather than Hulk smashing, grabbing a car and throwing it. It was a different level of animation. It was much more finessed.”

—Stuart Penn, Visual Effects Supervisor, Framestore

The time-travel suits were a digital creation. 

“Our animators love to perform and filmed each other acting out things so they could then use that as reference beyond what we had,” adds Penn. “We got the designs of the costumes and built real copies of them. We dressed up the biggest and chunkiest in the office and filmed them to give us reference as to how the costume might move. Mark isn’t anywhere as heavy as Hulk and doesn’t move in the same way. His footsteps are nowhere near as big as Hulk, so it’s a definite interpretation of Mark’s body performance into something much bigger. Trying to keep it alive and weighty was definitely a challenge.”

Smart Hulk is slightly less green in Endgame, according to Penn. “There was a desire that we feel the human subsurface skin coming through underneath. We put a pinkish, subsurface glow underneath. We pushed that quite a lot, especially in the cheek area and around the ears, to give him more of a human, fleshy surface underneath the green that’s on top.”