In addition to the work mentioned on Pokémon: Detective Pikachu, and some additional machine learning techniques used for their Endgame Smart Hulk, Framestore has also been looking at how the technology that sits behind deep fakes can be extended for other purposes. “We’re continuing to do a lot of research and development into A.I. and machine learning,” McGee adds, “looking to adapt and integrate new functionality and ways of working into existing tools like facial capture, performance capture, creature animation and fast rendering tool sets.”
The impact on talent-for-hire
As CG humans appearing in filmed entertainment became more popular – and more lifelike – many commentators suggested that ‘digital synthespians’ might some day replace actors altogether. Of course, this has not happened, and actors have regularly informed, especially via body and facial capture, their CG selves or other synthetic characters.
But could it be deep fakes, rather than completely CG humans, that spell the death knell for actors, or significantly change the game in the talent-for-hire industry?
Li’s view is that well-made deep fakes can become extremely beneficial when there is a need for a digital actor. “The main impact of deep fakes,” he says, “will be the ability for filmmakers to not be limited by the storyline due to the extreme cost for using digital actors. They now have a tool where they can find a double, and use A.I. face replacement to turn their actor into any person they want at a very low cost, or make the person younger.”
“We’ll always need performances from actors, even if it’s for a completely digital character,” offers Bartlett on this subject. “Motion capture and references are invaluable for any VFX team, especially animators. Deep fakes are adding to the topic of VFX and digital humans putting actors out of work, something we’re always asked about. But what we’re applying is digital makeup, not replacing anyone’s performance. So, as much as actors and performers are needed now, they’ll be needed tomorrow even as this technology continues to evolve.”
McGee agrees. “A deep fake A.I. can be a useful tool to do a bit of the heavy-lifting in a shot or sequence that VFX teams can build upon. This is what we did in Detective Pikachu and it worked in that instance because of the ultimate output. But if we were to go into a project where a stunt performer does an entire performance and their face had to be replaced, deep fake isn’t the way to go. At least not yet.”
Watch Framestore showcase and explain its deep fake process in shooting an actor and face swapping Boris Johnson and Donald Trump onto him.
Pinscreen also made a Donald Trump deep fake. For this video, it used its PaGAN neural face renderer for face replacement.
Watch ctrl shift face’s most popular deep fake video featuring face swapping between Bill Hader and Arnold Schwarzenegger.