23 January 2024
Exclusives, VR Trends
By CHRIS McGOWAN
Abraham Lincoln strides through the cemetery in Lincoln in the Bardo. (Image copyright © 2017 Graham Sack and Sensorium)
Virtual reality’s creative vocabulary has things in common with that of film and theatre, as well as aspects that are all its own.
“Imagine a Venn diagram that incorporates cinematic and theatrical elements and elements that are native to virtual reality,” says Graham Sack, the writer and director of the 360 VR video Lincoln in the Bardo (released in 2017 through the New York Times VR app) and of the four-part series The Interpretation of Dreams (released through Samsung VR Video in 2018).
The task of Sack and other VR directors is to understand what virtual reality has in common with other media, and to develop its particular tools.
Lincoln in the Bardo is a good example of this process. The much-lauded 10-minute piece is based on the George Saunders novel of the same name, and was produced by Graham Sack, Plympton (a literary studio), the new-media studio Sensorium, and The New York Times VR. The Molecule provided VFX work. The cinematic 360 video (with spatialized sound) takes place on a February night in 1862, during the Civil War. A grief-stricken President Abraham Lincoln pays a visit to a marble crypt in a Georgetown cemetery, where his 11-year-old son Willie Lincoln has been laid to rest. The president mourns as his beloved son’s spirit futilely tries to communicate with him. Dead souls – visible to us but not to Lincoln – offer up both piteous and comedic monologues. Everyone is in limbo – hence the title’s use of bardo, Tibetan Buddhism’s name for a transitional state between death and rebirth.
Abraham Lincoln at night in the cemetery in Lincoln in the Bardo. (Image copyright © 2017 Graham Sack and Sensorium)
“The absence of frame is a big anxiety for traditional film directors that are starting to work in virtual reality. They worry that the audience isn’t going to be able to follow a story if you can’t directly manage where they’re looking.”
—Graham Sack, VR Writer/Director
The performances in Lincoln are rather theatrical, being somewhat “elevated,” and delivered by seasoned New York theatre actors like Pete Simpson (Lincoln). And while plays use certain methods to guide the audience’s attention, VR has its own ways to cope with a 360-degree “stage.”
Sack observes, “The absence of frame is a big anxiety for traditional film directors that are starting to work in virtual reality. They worry that the audience isn’t going to be able to follow a story if you can’t directly manage where they’re looking.” This is arguably more of a concern in single-narrative VR, as opposed to interactive VR, with multiple storylines or experiences. Sack adds, “There are a lot of tools for guiding viewers around the space and gently nudging them towards what is narratively significant while still giving them the freedom to look elsewhere if they want to.”
The assembled ghosts look on as Abraham Lincoln carries his dead son in Lincoln in the Bardo. (Image copyright © 2017 Graham Sack and Sensorium)
“[Lincoln in the Bardo] feels like one shot. It unfolds, in a sense, almost like a 10-minute play, with very heavy visual effects. The way we did that, we shot these giant, continuous shots on greenscreen with all of the various characters. We were incredibly careful about the spatial positioning. It was the first time we had the whole crew dealing with greenscreen in a VR setting like that.”
—Graham Sack, VR Writer/Director
Those tools include sound elements and visual cues. “A good example is where Lincoln enters, maybe 140 degrees over the viewer’s left shoulder. With no cuing you probably would not think to turn around and watch the entrance, but we give a series of subtle signals that make turning and looking behind you automatic. One of them is simply sound. You hear his footsteps and some ambient environmental sounds that cue you, like the hooting of an owl. There are a series of things happening in that direction.
“And second, you have the attention of the characters on screen, where they’re looking. Viewers will naturally follow the sightlines of the primary characters on screen. Third, there’s the use of movement and shadow. So there was some dramatic expressionistic lighting. The way Lincoln is lit from behind by this light throws a very long shadow all the way into the foreground directly in front of the viewer. You catch the shadow and the movement of the shadow, and it cues you that something must be happening behind you.
“So there are at least three or four different elements. The result is people feeling like they’re naturally looking behind them but, of course, it’s orchestrated directorially.”
Abraham Lincoln at night in the cemetery in Lincoln in the Bardo. (Image copyright © 2017 Graham Sack and Sensorium)
Abraham Lincoln strides through the cemetery in Lincoln in the Bardo. (Image copyright © 2017 Graham Sack and Sensorium)
Sack had already done some experiments in VR when he approached George Sanders at a 2015 book event in Austin, Texas. He demonstrated the medium for the first time to Sanders, who was immediately intrigued. The two kept in touch, discussing ways to collaborate. Sack read the manuscript of the author’s new book Lincoln in the Bardo (winner of the Man Booker Prize for 2017) and found it “stunningly beautiful.” He adds, “It was set in a single location, a cemetery at night. It was the sort of immersive, transportive location I thought would both work in virtual reality and be evocative. The content was supernatural but still poignant, being about the living and the dead, particularly the living president and the ghost of his dead son. So I thought there was a lot of visual effects potential in terms of the treatment of the spirits, which would be really visually engaging in VR. So that’s sort of what attracted me to doing it.”
“It was very effects heavy,” continues Sack. The dead souls in the cemetery were given a “spectral quality and transparency” and were painstakingly gathered together with extensive greenscreen shots, stitching and layering. “Lincoln and the body of his son were the only people shot on location [at the Green-Wood Cemetery in Brooklyn, N.Y.]. Every other character in the piece was shot separately on greenscreen.
The actors had to work extensively with the green screen in “Lincoln in the Bardo”. © 2017 Graham Sack and Sensorium.
“There was some dramatic expressionistic lighting. The way Lincoln is lit from behind by this light throws a very long shadow all the way into the foreground directly in front of the viewer. You catch the shadow and the movement of the shadow, and it cues you that something must be happening behind you.”
—Graham Sack, VR Writer/Director
“If you watch the piece, it feels like one shot. It unfolds, in a sense, almost like a 10-minute play, with very heavy visual effects. We shot these giant, continuous shots on greenscreen with all of the various characters. We were incredibly careful about the spatial positioning. It was the first time we had the whole crew dealing with greenscreen in a VR setting like that. The principals, including the soul of Willie and a speaking chorus, were shot basically independently. The giant crowd scenes were shot in layers and then composited together. We shot it by recycling the same dozen supernumeraries in different costumes, different positions, etc.”
The Molecule’s CEO, Chris Healer, adds, “A lot of times we were shooting things and we’d map things out, and we’d have people looking in the strangest directions. But we didn’t really know until post-production was complete and everything was composited together whether people would be looking in the right direction. One of the hardest things is getting the sightlines right with greenscreen in this context.”
Filming at the cemetery in Lincoln in the Bardo. (Image copyright © 2017 Graham Sack and Sensorium)
“I thought there was a lot of visual effects potential in terms of the treatment of the spirits, which would be really visually engaging in VR. And so that’s sort of what attracted me to doing it.”
—Graham Sack, VR Writer/Director
Sack has also teamed with Geoff Sobelle, John Fitzgerald and Matthew Niederhauser (of Sensorium) to create objects in mirror AR closer than they appear, a fusion of augmented-reality technology and an immersive theatre installation, presented in May 2018 at Tribeca Storyscapes, the interactive media division of the Tribeca Film Festival. The Molecule and Sensorium provided technical direction.
Tribeca also presented another VR production by Sack last May, one that was inspired by Sigmund Freud rather than Abraham Lincoln. Sack comments, “Samsung was interested in whether virtual reality could support serialized content, almost like television-style episodes. After Lincoln came out they approached me.” The result was the four-episode The Interpretation of Dreams, written and directed by Sack, who teamed again with Sensorium on the production. “I went back to Sigmund Freud’s The Interpretation of Dreams,” says Sack. “Each episode is based on one of Freud’s original case studies in dreams and the unconscious.”
Graham Sack’s The Interpretation of Dreams. (Image copyright © 2018 Graham Sack and Sensorium)
About the challenge of utilizing traditional narrative techniques vs. developing immersive storytelling tools, Sack says, “That’s been the most fun in terms of working in the medium. What parts of the directorial vocabulary naturally carry over from other media, like film, and which ones need to be invented from scratch?”
;
