23 January 2024
VFX Profile
By NAOMI GOLDMAN
By NAOMI GOLDMAN
What if you could get inside the headlines and be one with the story? What if you could go beyond observing things on the street and experience emotion as if a situation is happening to you? Bearing witness to a violent attack, feeling the power of gunfire, being in the room with prisoners in jail cells and patients in clinics – these are real-life stories being brought to life through the dynamic field of virtual reality journalism.
Nonny de la Peña, named “The Godmother of Virtual Reality” by The Guardian and Engadget and one of the 20 most influential Latinas in tech by CNET, is a pioneer of virtual reality and immersive journalism – a phrase she coined to describe the groundbreaking application of VR for social impact. As the founder and CEO of Emblematic Group, a leading VR/AR company that focuses on creating empathetic engagement, she has collaborated with PBS Frontline, Wall Street Journal, The New York Times, Planned Parenthood, the True Colors Fund and other organizations to create impactful virtual reality experiences depicting real-life events.
Her VR projects include Across the Line, which helps viewers understand what some women go through to access reproductive health services, After Solitary, which takes viewers inside the Maine State Prison to experience a harrowing story of solitary confinement, and Out of Exile, which uses VR to draw attention to the plight of homeless LGBTQ youth. Other projects have explored Guantanamo Bay Prison, experiences of refugees and, most recently, the impact of climate change on the landscape of Greenland.
VFX Voice experienced several of Emblematic’s projects in their Santa Monica, California studio and sat down with de la Peña to talk about the value of interactivity and her mission as a visionary storyteller seeking to foster empathy.
VFX Voice: You have a reputation as a boundary breaker who has changed the face of multimedia journalism. Tell us about your trajectory from mainstream news journalist to using VR for social impact.
de la Peña: As a correspondent for Newsweek and then a traditional documentary filmmaker, I was always interested in conveying stories about the underdogs or people you might not have access to in your day-to-day life. Much of journalism is about reporting the facts, the scene on the ground, describing the reactions and aftermath of often vexing situations. As journalists we work to generate critical engagement, but I wanted to go beyond the reporting and focus on the experience wherein people could virtually transport themselves.
In 2004, I wrote and directed a documentary entitled Unconstitutional: The War on Our Civil Liberties, which examined the subjugation of American civil liberties following the events of 9/11. It had a big segment on Guantanamo Bay Prison, as I felt that the media was not adequately covering Gitmo. I was awarded a grant from The MacArthur Foundation and Bay Area Video Coalition to translate an existing documentary project into digital media. How do you report on a destination where you don’t have access? That was the catalyst. We built a virtual but accessible version in Second Life in 2007 (and later rebuilt it for our premiere at Unity Moscow Museum) that threw people into detainee positions as they heard tormenting encounters with guards, and had them face the concept of habeas corpus. That technology is something I embraced and advanced to allow for that frame-breaking user experience.
“What if I could present you with a story that you would remember with your entire body and not just your mind? With VR, I can put you in a scene in the middle of a story you normally see on the news and lessen the gap between actual events and personal experience.”
—Nonny de la Peña
I think that anything that exposes you to differing viewpoints or culture or lifestyle is important and enriches our understanding of the human condition. So I moved further into negotiating the boundaries between conveying the news and creating opportunities for people to fully immerse themselves – sometimes as a victim, sometimes as a witness, or sometimes as the protagonist, but in all cases getting a real-life view from the ground.
“I moved further into negotiating the boundaries between conveying the news and creating opportunities for people to fully immerse themselves – sometimes as a victim, sometimes as a witness, or sometimes as the protagonist, but in all cases, getting a real-life view from the ground.”
—Nonny de la Peña
VFX Voice: You’ve addressed many of the world’s most pressing social issues. Give us some insight into your most recent projects and what you were hoping they accomplish.
de la Peña: Across the Line was produced in partnership with the Planned Parenthood Foundation of America and premiered at the 2016 Sundance Film Festival as a walk-around experience. It helps viewers understand what some women go through to access abortion services. We wanted to place viewers in the shoes of a patient entering a health center to experience what they encounter and depict the toxic environment that many healthcare providers, health center staff and patients must endure to provide or access care on a daily basis. A true multimedia project, we used real audio gathered at protests, scripted scenes and documentary footage. We created a 360-degree version so the piece could be distributed as widely as possible.
Out of Exile: Daniel’s Story is a powerful reminder of the kind of hostility faced by so many in the LGBTQ community. In this piece we wanted to shine a light on the fact that 40% of homeless youth in America identify as LGBTQ, with the majority coming from communities of color. We take you into the story of Daniel Ashley Pierce as he is confronted about his sexual orientation by his family in a “religious intervention,” an event that turns dramatic and violent. It was important to us to create a powerful reminder of the kind of hostility faced by so many in the LGBTQ community, but also end with a sense of optimism as Daniel and others share their personal accounts of triumphing over despair.
We used videogrammetry to create holograms of Daniel and his peers and recreate the event using video captured by Daniel at the time. It was a challenging and emotionally raw experience because we were asking Daniel to recount the brutal attack, blow by blow, sharing with us who hit him and how, so that we could reenact the event with motion capture and stay as true to the actual experience as possible. Having his words as a guidepost is a really important part of our process to instill the experience with integrity and authenticity.
And on the heels of the United States’ withdrawal from the Paris Climate agreement, we created Greenland Melting in collaboration with PBS Frontline and NOVA. The piece provides a rare, up-close view of the icy arctic scenery that’s disappearing faster than predicted. We used a 360 Ozo Camera clamped onto the edge of a helicopter allowing us to create that vantage point so that users are viewing the landscape as if they are in the helicopter themselves.
VFX Voice: You’ve said that ‘as the world becomes increasingly global and our online and offline lives become increasingly integrated, it is critical to convey stories that create empathy and preserve our humanity.’ What can virtual reality teach us about empathy?
de la Peña: What if I could present you with a story that you would remember with your entire body and not just your mind? With VR, I can put you in a scene in the middle of a story you normally see on the news and lessen the gap between actual events and personal experience.
In 2012, we premiered the first virtual reality experience at the Sundance Film Festival, Hunger in Los Angeles. In the piece, animated characters stand in line outside a soup kitchen that has run out of food as real audio plays – and you watch as a man collapses in a diabetic seizure. We had no idea what to expect. But the emotion was overwhelming. People were crying their eyes out in a way I haven’t seen since then on any of our projects. I think a lot of the power comes from what happens after you take off the goggles and have to confront your feelings with others around you. People are pushed out of their comfort zones and cannot look away from difficult situations. I think that sense of empathy drew others to the field. It was during this project that I introduced Alejandro (Academy Award-winning director Alejandro González Iñárritu) to the medium, and I’m really inspired by the powerful immersive works he has created.
“I think a lot of the power comes from what happens after you take off the goggles and have to confront your feelings with others around you. People are pushed out of their comfort zones and cannot look away from difficult situations. I think that sense of empathy drew others to the field.”
—Nonny de la Peña
Here’s another good example. Planned Parenthood took the Across the Line piece into communities with a strong anti-abortion stance and after seeing it, many came out feeling that women should not have to experience that kind of hateful vitriol. It didn’t change their views on abortion per se, but in some cases it opened their eyes to the treatment of these woman and healthcare workers – and that is significant in terms of creating a meaningful sense of understanding by literally walking in someone else’s shoes.
VFX Voice: What is your approach to starting a VR project?
de la Peña: At the beginning of any project, I think about how I would experience a situation with my whole body, using all of my senses. I literally close my eyes and place myself right into the scene. I walk around the scene and think about what I would feel and hear and see and what emotions would be running through my body as the story unfolds around me. I am always thinking about the spatial nature of the narrative and try to capture and absorb the situation.
Think about how your experience changes if you see something through a car window vs. being right there on the street. Or how your point of view changes if you’re scared or hungry or tired. If you’re a bystander or a victim or a perpetrator of a crime. Telling people about something you have heard or watched is just less visceral than describing it first-person – and that sense of presence is what we strive for.
In creating After Solitary, we used photogrammetry. We took a huge number of photos of a solitary cell from every conceivable angle and then stitched them together. Then we used volumetric capture photography using multiple cameras and stitched the frames together. We captured video of an actual inmate and then dropped him into the solitary cell with you so that he and you are moving together around the cell as he tells his story in that confined space. Viewers come out with a very raw understanding of that restricted environment and its psychological impact.
VFX Voice: Are there best practices for VR journalism? What’s an example of the balancing act you managed between VR technology and authenticity?
de la Peña: Maintaining authenticity and integrity are of the utmost importance as we create these virtual experiences. Our projects are guided by real-life accounts, and having captured audio and video footage from real events is a critical marker for our simulated environments and animated characters. It’s something I’m always thinking about as we negotiate boundaries.
For example, in Greenland Melting we needed to drop in holograms of NASA scientists. The tricky question – what should they wear? We didn’t want them to dress in parkas since they were shot against a greenscreen, and we thought that wearing a parka would suggest the experts were actually shot at the glacier. It’s details like that can raise questions about journalistic integrity that we take painstaking measures to address properly.
We are currently working on a paper on best practices, and I have recently joined The Aspen Institute and Knight Foundation, Knight Commission on Trust, Media and Democracy to confront important questions around responsible immersive journalism and journalistic integrity due to the subjectivity of experience and editorial control. One of our priority goals is to identify the perennial and emerging values and social obligations that should guide those who produce distribute and consume news and information to ensure a functioning democracy.
“At the beginning of any project, I think about how I would experience a situation with my whole body, using all of my senses. I literally close my eyes and place myself right into the scene. I walk around the scene and think about what I would feel and hear and see and what emotions would be running through my body as the story unfolds around me. I am always thinking about the spatial nature of the narrative and try to capture and absorb the situation.”
—Nonny de la Peña
VFX Voice: What’s next for you and Emblematic Group?
de la Peña: We have another project in the works with PBS Frontline on cross-contamination of DNA that we’re excited to share. And we just got a $250,000 grant to start building a platform where people can film their own stories. Called REACH, it’s a platform for hosting and distributing 3D models of locations that news organizations can use to create innovative and cost-effective “walk-around” VR content. You film the story and we will provide the volumetric locations that can be dropped in.
And something fun and totally different – we’re working on SLASH Presents Trashed! A VR Game of Epic Hotel Destruction. So we’re building a game MC’d by legendary rocker Slash that includes a lot of crazy antics, from cars being driven into swimming pools to throwing TVs over balconies. And of course it will also be multiplayer, because, come on, you can’t nail a bed to the ceiling without your friends, right?
VFX Voice: What excites you about the future of immersive journalism?
de la Peña: I’m excited by the potential of VR to change the way that mainstream media tell stories. I think that as this area evolves and technology becomes ever more accessible, journalists will more organically pivot towards VR as the best way to tell a particular kind of emotional, place-based story. And I’m excited that one day you’ll be interviewing Virtual Nonny – a fantastic avatar that I’ll puppeteer from home!
;
