Immersive Encounters

Partners

Show & Tell! research group at AP University College Antwerp
Anastasia Pash
Lee Erdman
Ronan Lynch

Challenge

Verena, our Head of Research, was offered the opportunity to participate in a 10-day intensive Storytelling in Virtual Reality course hosted by the University of Antwerp. The competitive programme attracted diverse talent from across Europe and North America, including artists, producers, developers, researchers and students.

Focused around hands-on activities and collaboration, participants worked in small teams to develop a working VR prototype, responding to a number of challenges: limited time, limited resources, working with people you just met, identifying skills within the team, and, of course, working in a medium so new that storytelling rules and conventions are still being written.

Process

To introduce the cohort to the basics of immersive storytelling, the teaching team had prepared a series of lectures along a fantastic lineup of industry guest speakers from journalism, marketing, and the cultural industries. Feeling inspired, we were ready to tackle our own project:

Identifying a story

Equipped with pens, paper and post-it notes, we started brainstorming story ideas. Among the wealth of possible characters, settings and events we prototyped, one idea, inspired by the ferris wheel outside Antwerp central station, began to stick. We explored frameworks for linking individual “story pods” to create bigger, interactive narratives, and after several iterations settled on a concept involving a hotel elevator.

Writing the script

We developed three mini stories exploring themes like choices, status, social mobility and gender inequality. Read-throughs with the actors helped us refine the tone, addressing these sometimes difficult topics with a pinch of humour. At its core, our experience allowed participants to put themselves in someone else’s shoes and become a character within the story. Other characters would react to their presence, speak to them or hand them objects. The final script contained instructions on how actors should approach and interact with the camera/viewer. Our process more closely resembled writing for the stage than writing for film.

360 vs. 180

To prepare for the live action shoot, team mate Anastasia, who regularly films 360 and 180 videos for her travel website, did some test recordings around campus. When we reviewed the footage, we were surprised how effective 180 stereoscopic VR can be. It felt just as immersive as 360 and our test viewers wouldn’t constantly look around to see if they were missing anything (FOMO in VR is real). 180 also had some obvious production benefits: Crew could remain on set during takes, and less stitching was needed in post.

Team roles

Playing to each other’s strengths, team roles came quite naturally. Ronan who lectures in character design wrote most of the script, Anastasia who produces virtual tours for her VR travel startup became camera woman, Lee, a former dancer and performer, was in charge of rehearsals and directing actors on set, and Verena took on Unity development. Having a multi-skilled team is key to immersive productions. However, unlike commercial projects, all four of us were heavily involved in all aspects of the production and had the chance to gain experience outside our usual areas of expertise.

Production

We scheduled our shoot right at the beginning of week two. Our scenes were quite short (1-2 minutes each) and single take. We managed to film them all in one day, which left us with enough time to refine interactivity, media playback and 3D graphics. Shooting day was intense but also a great deal of fun. With no budget and limited time we had to be pragmatic about logistics, exploiting Anastasia’s Airbnb, a school meeting room and a basement corridor as shooting locations. Anke, a Conservatoire student who was on the course with us, helped recruit some of her actor friends to help out on the project. None of them had acted for virtual reality before, making it a great learning experience for both sides.

The next set of production workflows included stitching and editing the footage, creating 2D interface graphics, sourcing music and sound effects, creating the 3D elevator environment, and programming for interactivity. Software development had to start prior to filming, and Verena created a clever system to swap footage in and out without breaking the experience.

Outcome

The final prototype is an interactive, non-linear VR experience that combines 180 stereoscopic video with real-time 3D graphics rendered in game engine. The experience starts inside a hotel elevator. By pushing buttons on the panel (basement, executive, penthouse), viewers select different stories in which they become the main character (with limited agency). Participants can experience these stories in any order of their choosing. After each story, they return to the elevator to discover new messages written on the mirror, and a final identity reveal after they experienced all three stories.

The prototype is quite advanced considering the short development time. We now use it regularly to discuss diversity experiences with clients. It is also a great tool to demonstrate modular/episodic content formats for virtual reality.