THE RISE OF VR/AR/VFX AND LIVE ENTERTAINMENT

By CHRIS McGOWAN

In one breakthrough after another, AR, VR and VFX are augmenting live entertainment, from ABBA’s avatars to XR concerts to Madonna dancing live on stage with her digital selves.

ABBA: THEIR ‘70S SELVES

When the Swedish group ABBA returned to the stage last May after a 40-year hiatus, they did so digitally with the help of ILM. The foursome – Björn Ulvaeus, Benny Andersson, Anni-Frid Lyngstad and Agnetha Fältskog – appeared in ABBA Voyage via their de-aged digital avatars, virtual versions of themselves on huge screens in the purpose-built, 3,000-capacity ABBA Arena, which was constructed in Queen Elizabeth Olympic Park in London. (ABBA Voyage won the 21st Annual VES Award for Outstanding Visual Effects in a Special Venue Project.)

ABBA’s 20-song “virtual live show,” over five years in the making, is a hybrid creation: pre-recorded avatars appear on stage with a physically present 10-piece band to make the experience more lifelike and convincing. The avatars meld the band’s current-day movements with their appearances in the 1970s.

ILM supplied the VFX magic, with more than 1,000 total visual effects artists in four studios working on the project, according to the show’s spokespersons. ILM Creative Director and Senior Visual Effects Supervisor Ben Morris oversaw the VFX of the show, which was directed by music-video veteran Baillie Walsh.

First, Morris and his team scanned thousands of original 35mm negatives and hours of old 16mm and 35mm concert footage and TV appearances of the band. The supergroup quartet spent five weeks singing and dancing in motion-capture suits as ILM scanned their bodies and faces with 160 cameras at a movie studio in Stockholm. The same process was undertaken with younger body doubles, who followed their moves, guided by choreographer Wayne McGregor, and whose movements were blended with those of ABBA to give the band more youthful movements.

The digital versions of ABBA appear on the stage and to the sides of the arena on towering ROE Black Pearl BP2V2 LED walls, powered by Brompton Tessera SX40 4K LED processors. Each screen is 19 panels high, and there are an additional 4,200 ROE LED strips in and around the area. Solotech supplied the LED walls. Five hundred moving lights and 291 speakers connect what is on the screens to the arena. The result is spectacular and suggests that many large-scale digital shows may be on the way for music stars who are getting old or simply don’t like touring.

VIRTUAL TUPAC, VIRTUAL VINCE

Digital Domain created digital representations of the rap star Tupac Shakur and legendary Green Bay Packers football coach Vince Lombardi in 2012 and 2021, respectively, which raised the visual-quality bar for virtual appearances projected live.

On April 15, 2012, at the Coachella Valley Music & Arts Festival in Indio, California, Tupac Shakur appeared in a CGI incarnation on stage at the Empire Polo Field in Indio, California. The virtual Tupac sang his posthumous hit “Hail Mary” plus a “duet” of “2 of Amerikaz Most Wanted” with Snoop Dogg, who was on stage, in the flesh.

The computer-generated realistic image of Shakur was shown to some 90,000 fans on each of two nights; YouTube videos of the event reached 15 million views, according to Digital Domain. Some called it the “Tupac Hologram” – it wasn’t a hologram, but it was 3D-like. Unlike ABBA Voyage (2022), which featured the participation of the band in creating the group’s avatars, the Shakur on stage was created long after the singer’s death in 1996. The project took about two months to complete, with 20 artists of different disciplines, according to Digital Domain’s Aruna Inversin, Creative Director and VFX Supervisor. The virtual Tupac was the vision of Andre “Dr. Dre” Young, and Digital Domain created the visual effects content. AV Concepts, and an audio-visual services and immersive technology solutions provider, handled the projection technology.

The virtual Shakur was a two-part effect, according to Kevin Lau, Digital Domain Executive Creative Director. “In the first, we recreated Tupac using visual effects, and the other was a practical holographic effect created using Hologauze. For the performance, we started by filming a body double performing the set. The actor had an incredible likeness in both stature and movement to Tupac. Once that performance was in the can, we began digitally recreating a bust of Tupac’s likeness. This digital head was then combined with the body performance and animated to match the song. The two were blended together in composting to create a seamless recreation of a seemingly live performance.”

Lau continues, “The performance asset was then synched to the live band and projected on Hologauze. This material has the ability to reflect bright light, while remaining fairly transparent in areas that are dark. The result is a figure that appears to exist physically in a three-dimensional space.”

“The Vince Lombardi project was similar to the Tupac hologram, but we enlisted a slew of new techniques – many of which we created ourselves – that were not available at the time,” comments Lau. The virtual Lombardi – evoking the NFL football coach as he was in the 1960s – appeared on the jumbotron at Raymond James Stadium in Tampa on February 7, 2021.

Lau explains, “We began by filming a body double to do the bulk of the performance. But instead of having to create a full digital recreation of Coach Lombardi’s head, we enlisted Charlatan, our machine learning neural rendering software. This allowed us to train a computer off a data set of available Vince Lombardi images. The software can then synthesize those images and recreate a likeness of the subject – in this case, Coach Vince Lombardi – based off our actor’s [the puppeteer] performance.”

For the virtual Lombardi, a slightly different projection technique than before was used, according to Lau. “Because of the nature of the Super Bowl, we didn’t have access to a sequestered location or a secured space to prep screens and projectors. The stage had to be wheeled out into the end zone and set up, then broken down within a matter of minutes. For this, we enlisted more of a ‘Pepper’s Ghost’ effect, in which a translucent mylar screen is put at a 45-degree angle from a large LED screen. The resulting image is reflected back to the viewing audience, allowing you to see through parts of the screen giving the holographic illusion.”

The digital Shakur and Lombardi are two “tentpole projects” for Digital Domain in that particular area, and they have also done “a handful of other executions for promotional and concert events,” says Lau. “We are always looking for new opportunities to surprise and delight viewers. AR/VR and holographic immersive entertainment have a huge role to play in the future. Being able to re-contextualize our environment will truly bring the audience in and allow for higher engagement with fans.”

MADONNA AND MARIAH

At the 2019 Billboard Music Awards, five Madonnas and Colombian singer Maluma performed the reggaeton-pop song “Medellín.” Or, more precisely, one Madonna and four avatars sang with Maluma, in an AR-enhanced performance that utilized volumetric capture and Unreal Engine.

“To the surprise and shock of fans around the world, four copies of Madonna – each wearing one of her signature outfits from over the years – appeared and danced alongside the original, creating a show unlike any other and leaving fans to wonder how she pulled it off,” says Piotr Uzarowicz, Head of Partnerships & Marketing for Arcturus, a leader in volumetric video technology.

Arcturus’s HoloSuite software “was instrumental in the post-production of the four pre-recorded Madonna performances that were integrated into the live TV broadcast,” says Uzarowicz. “This was the first time that volumetric video was used in a television broadcast, and it signaled an emerging new era of entertainment.”

The augmented reality content could be seen by TV viewers of the telecast and was presented alongside live components, including Madonna herself, Maluma, and over a dozen dancers.

Sequin AR was tasked with bringing the Madonna clones to life in AR. “We worked with Jamie King [Madonna’s Creative Director], Madonna and several fantastic companies to create the performance, including Dimension Studios [where the volumetric capture of Madonna took place], StYpe [camera tracking], Reflector Entertainment [co-founded by Cirque du Soleil creator Guy Laliberté] and Brainstorm [InfinitySet],” says Sequin AR CEO Robert DeFranco.

“The ‘Madonnas’ were created with volumetric capture, and additional elements were created, including rain, stage extensions and clouds,” explains DeFranco. “The elements were rendered in real-time using Brainstorm and Unreal Engine.”

Sequin AR was founded as a virtual production and augmented-reality solutions company, and it has extended its capabilities to include immersive web3, mobile and VR solutions for its clients. “We consider ourselves a 3D immersive company,” says DeFranco. “AR and real-time render technology allow content producers the ability to tell new stories that were previously only possible in post-production. This helps them make a greater emotional impact and connect with audiences in more engaging ways.”

DeFranco explains, “AR viewing can be accomplished with a number of screen types, whether it be AR glasses, mobile, monitors or broadcast. Broadcast AR leverages the TV, [and] we integrate the real-time technology into the standard production pipeline, creating the virtual production pipeline for AR.”

Sequin AR provided virtual production and AR broadcast services for Mariah Carey’s Magical Christmas Special on Apple TV+ on December 4, 2020. The production utilized “virtual sets being rendered in real-time with Zero Density and Unreal Engine for seven cameras. The shoot was done in five days, which would have not been possible if it were not for virtual production,” according to DeFranco.

Ego, which Sequin worked on, “was all AR, motion capture and facial capture. We provided virtual production technical engineering on the show supporting the technical pipeline and playout in real-time,” says DeFranco.

Regarding tech utilized, he notes, “In addition to our technical pipeline and processes, we leverage Unreal Engine and a variety of solutions depending on the production needs, including Mo-sys [virtual production and camera tracking], StYpe [camera tracking technology for AR or virtual studio], Zero Density [real-time visual effects, real-time broadcasting and a markerless talent tracking system] and Silver Draft [supercomputing high-end rendering], to name a few.”

DeFranco adds, “Augmented reality and virtual production continue to push the boundaries with innovative companies creating new content and experiences. We are thrilled to be helping advance the adoption of these technologies and creating new ways to use them.”

XR CONCERTS

Last year, Snap signed a pact with Live Nation Entertainment for audiences to access AR experiences via Snapchat at Live Nation music festivals such as Lollapalooza, Bonnaroo, Governors Ball and Electric Daisy Carnival. Snapchat users will be able to find their friends in the audience, locate landmarks on the festival grounds, try on merchandise and experience other AR content. Snapchat has a large potential audience for its AR experiences – it had 363 million active daily users worldwide as of the third quarter of 2022, according to the firm.

“There is growing potential for attending concerts and live VR theater and performance from the comfort of your home with VR. Just as we are finding a shift in the film industry due to the growth of streaming, I believe we will find audiences who seek XR entertainment. The possibility of being able to connect with your favorite performers and bands while also attending with your friends and doing so from your home environment is compelling.”

—Kiira Benzing, Founder and Creative Director, Double Eye

U2, Eminem and Maroon 5 are among the artists who have added various types of AR experiences to their shows via smartphones and apps. In 2022, for Flume’s show, Coachella partnered with Unreal Engine to add AR elements to the Coachella YouTube live stream. In addition, the Coachella Valley Music and Arts Festival’s Coachellaverse app offered various immersive AR effects, including geo-specific experiences created in partnership with Niantic. Last December, the virtual band Gorillaz performed AR concerts in both Times Square in New York City and Piccadilly Circus in London.

In October 2020, in a peak part of the pandemic, Billie Eilish’s “Where Do We Go?” XR concert from XR Studios in Hollywood served up “a meticulous visual affair, replete with lofty LED screens and extended-reality [XR] effects, that felt determined to recapture weary viewers’ attention,” according to Amy X. Wang of Rolling Stone. She continued, “With the feel of a highly produced music video, the show, which charged $30 a ticket, hit on all the strengths of livestreaming. Enormous animated creatures and chimeric landscapes whirled by around Eilish, her brother Finneas and her drummer Andrew Marshall as they played into multiple roving cameras from a 60-by-24-foot stage; the trio’s sparse physical presence made for a striking silhouette to the rapidly shifting scenery.”

BTS, The Weeknd and Metallica are among a few of the big names who have offered VR songs or concerts. And MelodyVR, NextVR, VeeR, Meta Platforms and Wave are some of the platforms that offer XR concerts and have large libraries of such fare.

Moreover, visual effects are responsible now for an increasing number of virtual performers. South Korea’s Eternity is a virtual K-pop band that uses AI technology, while Aespa has both physical and virtual members.

VR VARIETY SHOW

Kiira Benzing is Creative Director and Founder of Double Eye, which is presenting interactive theater and live entertainment in virtual reality. Her latest effort is the VR variety show Skits & Giggles, which is published by Meta Platforms and available on Horizon Worlds. “We wanted to make a variety show and test how comedy in a scripted format might play out in VR for a live audience,” says Benzing.

The entire show is interactive, according to Benzing. “The show is mostly scripted between the skits, monologues and musical numbers; but there are moments of spontaneity and improv that also arise in every show.” Skits & Giggles was a 2022 nomination for Best Immersive Performance at the Raindance Film Festival.

Benzing comments, “VR is an incredible medium for live performances. I see a wonderful niche deepening in its growth as more and more live performances are being created across different Social VR platforms. Now there are so many new solutions with the hardware becoming more affordable and the Social VR platforms growing in numbers. All of these elements together make an ecosystem more possible for live VR performance to flourish.”

Benzing continues, “VR, AR and XR as a whole are amazing evolutions of experiences we can share with our audiences. The player’s journey is quite different if we use AR to overlay onto the physical world or VR to take them into a brand-new world, but both forms of the tech can be transformative.” She observes, “We are seeing festivals, concerts, bands and theater companies venture into these mediums. Since XR has such a power to transform an experience, I believe the more audiences are introduced to AR and VR. the more they will begin to expect live performances to include an XR extension.”

And, Benzing notes, “There is growing potential for attending concerts and live VR theater and performance from the comfort of your home with VR. Just as we are finding a shift in the film industry due to the growth of streaming, I believe we will find audiences who seek XR entertainment. The possibility of being able to connect with your favorite performers and bands while also attending with your friends and doing so from your home environment is compelling.”

She concludes, “As audiences have a taste for the combination of XR and live performance, they will crave more and more.”