23 January 2024
Exclusives
By IAN FAILES
By IAN FAILES
We’re reaching a time when traditional animation, CG techniques, virtual production, real-time rendering and motion capture are all coming together in useful tools for artists in visual effects and animation. Indeed, we may already be there, as visual effects studio The Mill’s new Mill Mascot system shows.
Mill Mascot is a toolset that allows artists to puppeteer CG characters through hand and facial gestures. Making that possible is a connection of real-time game engine tech, animation tools and motion sensors. The idea is that directors, clients and artists can ‘jump in’ and get creatively involved in the development of a character themselves – live.
VFX Voice asked The Mill about how the Mascot system was developed, how it works and what it has been used for so far.
Mill Mascot began life at the studio as The Mill’s ‘real-time animation system,’ or RTAS, bringing together software and hardware tools aimed at enabling live animated production. The results were a number of in-house Mill demos and some ‘Vonster’ spots for client monster.com, which saw a hairy creature puppeteered to perform human-like antics.
“The rudimentary beginnings of Mill Mascot allowed characters to be controlled in the very traditional sense of puppeteering by only moving their mouths open and shut,” outlines Jeff Dates, Creative Director at The Mill New York. “We also had to be aware of character design in the early stages, as the system allowed for minimal movement of the body and extremities.”
Since that time, says Dates, Mill Mascot has evolved in its abilities in significant ways. “It now allows for a great deal more flexibility and creativity. Characters are far more dynamic in their look, and the system generates final imagery at a much higher fidelity. We also have much finer control over the movement, from the entire body and extremities down to the eyelids and nuanced facial expressions. This has also greatly been improved by the presence of facial tracking in addition to hand controls.”
Behind those abilities is a combination of Epic’s Unreal Engine, Leap Motion for hand tracking, and an iPhone X for facial tracking. However, The Mill continues to develop and iterate the system so that it will ultimately be software and hardware agnostic.
“The [Mill Mascot] system behaves much like a live-action puppet. We’re able to introduce performance controls for facial emoting and hand dexterity for the character. The ability to generate useful animation is limited only by what the character is able to do. So in reality, we found that once the character is ‘active’ and being performed, it’s as alive as any puppet. So we had literally hours of animation from our recording session. Not to mention real candid ‘outtakes.’”
—Jeff Dates, Creative Director, The Mill New York
It’s all part of a push at The Mill to approach visual effects and animation work differently. The studio has the benefit of working on a range of different creative projects, from commercials to shorts to motion graphics and animated pieces. In terms of real-time, The Mill has been at the forefront of this technology, having partnered with Epic on a virtual production demo called ‘The Human Race.’ Here, The Mill’s tracker-covered Blackbird car was utilized to deliver car commercial-like shots for a Chevrolet vehicle demo. A virtual car was composited live over the Blackbird, and could be re-skinned in real-time.
This and other real-time projects are designed to provide flexibility and choice for directors and clients. A much more traditional shooting, CG and animation pipeline requires time for iterations. Real time is intended to allow for quick changes in choices, and a chance to explore more options.
Importantly, The Mill isn’t aiming, just yet, for a full-body motion-capture solution with Mill Mascot. Clearly there are systems capable of doing that incredibly well. Instead, Dates makes a distinction between real-time performance animation and real-time motion capture, with Mill Mascot intended to be a tool that can drive a fun character performance, not necessarily one completely grounded in reality (even though that ability does exist).
“At The Mill we use theatre-trained puppeteers and real-time animation artists to inject human essence and personality into our characters, but anyone can take the reins.”
—Jeff Dates, Creative Director, The Mill New York
The ‘Vonster’ spots were some of the first glimpses into what The Mill’s real-time animation system was capable of. Other instances with live-animation performances were seen in ‘Seanna’s Ocean Buddies,’ a Sesame Studios animated short. Then, at a recent conference for HPE (Hewlett Packard Enterprise), The Mill devised a real-time interactive installation with Mascot.
“After creating the CG mascot for HPE’s ‘Tame The IT Monster’ TV campaign, they wanted to re-purpose the asset and use Mill Mascot in order to interact with their audience at the HPE Discover Conference in Las Vegas,” recounts Dates. “We created three activations using Mill Mascot in Vegas. First, the IT Monster performed live on stage, interacting with the HPE CEO during his Keynote presentation and ‘interrupting’ his speech. Because the character was being performed live by puppeteers using Mill Mascot backstage, the IT Monster could also interact with the audience, proving the content wasn’t pre-recorded.
“Additionally, we built two custom booths on the conference floor, one that allowed attendees to play with and interact with the IT Monster, and one that allowed them to puppeteer and animate him themselves. It was hugely rewarding to see people interacting with Mill Mascot on such a large scale and the client was extremely happy.”
“Using a system like this really allows [clients] to be a part of the process. Not only can they see the action, but they can also direct and give feedback live, and in some cases perform the characters themselves. This adds a whole new level of accessibility to the art.”
—Jeff Dates, Creative Director, The Mill New York
Dates has been the driving force behind Mill Mascot, a project he says came out of the challenges of many projects he’d worked on thus far. “I’ve always been a storyteller. One of the frustrating things for writers is getting work produced, so I came up with the idea as a way to produce my own stories and sketches to try out new material. If you are familiar with the saying, ‘Fail fast, fail often,’ well, Mill Mascot allowed me to get ideas out of my system and onto the screen quickly. Animation became less precious, and experimentation became the currency.”
To get a character working in Mill Mascot, artists start by generating a high-quality CG character asset using traditional modeling, rigging and texturing tools they are already familiar with. Then they can import the character into Mill Mascot, connecting it to the gestural controls (the Leap Motion controller and iPhone X) and allowing it to be animated and rendered in real-time via Unreal Engine.
“The system behaves much like a live-action puppet,” explains Dates. “We’re able to introduce performance controls for facial emoting and hand dexterity for the character. The ability to generate useful animation is limited only by what the character is able to do. So in reality, we found that once the character is ‘active’ and being performed, it’s as alive as any puppet. So we had literally hours of animation from our recording session. Not to mention real candid ‘outtakes.’”
Consideration of the gestural controls is important to then allow a performer to use hand and facial movement to drive the character. “Here at The Mill,” says Dates, “we use theatre-trained puppeteers and real-time animation artists to inject human essence and personality into our characters, but anyone can take the reins.”
So far, the kinds of characters created have appeared as final animation in advertisements and also in live audience situations. But Mill Mascot can also be used as a previs tool or for exploring performances, either by animators within the studio or directors and clients demonstrating what they would like to see.
“This doesn’t make traditional animation redundant, it just provides a new way in which we can animate characters in a world that demands fast and interactive content all the time.”
—Jeff Dates, Creative Director, The Mill New York
So what does Mill Mascot offer for directors, clients and artists that is different from the usual way they bring CG characters to life? Dates suggests that often clients do not know or understand what goes on behind closed doors in the world of visual effects.
“However,” the creative director notes, “using a system like this really allows them to be a part of the process. Not only can they see the action, but they can also direct and give feedback live, and in some cases perform the characters themselves. This adds a whole new level of accessibility to the art.
“The key benefit for all parties is creative flexibility,” continues Dates. “With the ability to treat character animation more like a shoot, capturing content and performance live, directors and animators can make creative decisions on the fly, develop new content and ideas as they go, and even capture ‘bloopers.’”
Will it take away the jobs of animators? Not at all, cautions Dates. “This doesn’t make traditional animation redundant, it just provides a new way in which we can animate characters in a world that demands fast and interactive content all the time.”
“The IT Monster performed live on stage, interacting with the HPE CEO during his Keynote presentation and ‘interrupting’ his speech. Because the character was being performed live by puppeteers using Mill Mascot backstage, the IT Monster could also interact with the audience, proving the content wasn’t pre-recorded.”
—Jeff Dates, Creative Director, The Mill New York
Moving forward, Dates says he hopes Mill Mascot can be used to help generate performances for some famous brand mascots. “This system has huge potential in the episodic entertainment world, and I’d really love to see Mill Mascot used in this space. We managed to ‘shoot’ the two-minute animated film for Sesame Studios last year in just one day, so this could be of huge benefits to brands such as Nickelodeon and Cartoon Network.”
;
