In-Depth Feature: VR (R)Evolution

By Jason Anderson, Published March 4, 2016 

Creators and developers grapple with new ways to tell stories in the rush to sate appetites for VR and immersive video.

Any explorer of virtual reality’s swiftly expanding frontiers may be reminded of screenwriter William Goldman’s withering line about the business of making movies: “Nobody knows anything.” But whereas the scribe behind Butch Cassidy and the Sundance Kid meant that as a putdown of Hollywood execs who mistakenly believed they understood what it took to make hits, here the line is more suggestive of the bewilderment felt by newly minted VR content-creators at just how swiftly the ground is shifting beneath their feet.

“It really is all so fresh, it makes your brain hurt,” says Patrick Milling Smith in his Los Angeles office. As a co-founder of the studio, he’s one of the key figures helping to usher VR into the media big leagues, a process that may accelerate very rapidly as the Oculus Rift, PlayStation VR, HTC’s Vive and other slick new rigs transform VR from tech-expo darling to mass consumer device. According to one industry forecast, shipments of headsets could top two million units in 2016 and reach ten times that by 2018. In another piece of speculation, the whole VR industry may be worth US$150 billion by 2020.

The signs of its swift mainstreaming are many. A few weeks before Samsung’s Gear VR headset sold out its initial release in November, the New York Times distributed cardboard viewers to allow subscribers to see The Displaced, a portrait of three refugee children that is the first of a new series of documentary shorts the newspaper is creating with Milling Smith has witnessed a maelstrom of activity since showcased a rapturously received round of VR projects at the Sundance film festival in 2015. (Several new experiences also premiered at the 2016 festival, which included 30 VR demos in all plus several more immersive installations.)

“Everything good that is done in the medium redefines what people thought you could do,” he says. “We’re in that experimental stage, where creativity and the desire to tell story and push story will define the language of the medium. And it’s all happening at great speed.”

Even those who’ve been in the digital media sphere for decades have been impressed by VR’s velocity. “It’s an embryonic moment,” says Michael Naimark, the media artist and researcher who’s been in the interactive-media field since helping develop the earliest incarnations. (He’s just coming off a four-month artist residency in Google’s VR group, too.) Says Naimark, “I’ve never seen anything move so fast in my career – it’s faster than mobile, faster than Web 2.0, faster than interactive video.”

It’s not all happening in a vacuum, mind you – indeed, the current craze for all things interactive and immersive may really be the next stage of a longer process. Ana Serrano, Chief Digital Officer at the Canadian Film Centre and Founder of CFC Media Lab believes that the new wrinkle here is that filmmakers have belatedly discovered a field that has previously been furrowed by computer scientists, graphic designers and other media artists. “Now it’s the Game of Thrones folks and their studios who are moving into it in droves,” she says, “as well as the documentary filmmakers.”

That makes for a crowded space bustling with people who are thrilled and not a little bewildered by the possibilities at hand. Indeed, it can feel like “the wild west,” in the words of Andrew MacDonald. A cameraman and filmmaker who recently crafted a cheeky 360-video Hitchcock homage called Agnus Die!, he’s hustling to develop VR content for Toronto’s Cream Productions. As he says, “People are just making up shit as they go.”

And that’s as true for the major American VR producers like and Jaunt as it is for Cream, Occupied VR, Secret Location and other scrappier outfits eager to stay on the leading edge. With everyone from Disney to VICE getting in on the action and the hunger for new content intensifying, the new players must contend with thorny questions about the kinds of stories and experiences that best utilize VR. After all, the medium may prove to be as different to films and TV as those were to radio, as Milling Smith notes. No one even knows the adoption rate, or how soon those adopters will graduate from the two- to five-minute-long VR experiences that are available now to works lasting 30 minutes and beyond.

VR content creators must devise new how-to manuals that may demand a radical rethink of the fundamentals of telling stories on screen. That means we may be entering an era of vast experimentation, some of which is bound to not work out so well.

The digital media theorist whose 1997 book Hamlet on the Holodeck remains the seminal primer on narrative forms and strategies in the digital age, Janet Murray is understandably wary of VR over-hype among the film folk. “All this opens up a large palette but people can draw on the wrong parts of it,” she says. “We don’t have a viewer now – we have an interactor. And since the interactor is bringing all that movie sophistication to the table because of the highly elaborated visual vocabulary and storytelling we’ve experienced in film, that raises our level of expectation.”

Thankfully, many creators sound more excited than daunted. “When I was growing up I loved the Choose Your Own Adventure books,” says Nicolas Billon, a Toronto playwright and screenwriter at work on several VR projects. “I feel like VR could be the perfect tool for something like that.”

With its desks full of giant HD monitors and speakers, piles of cables and gear, and beat-up sofas, the Toronto workspace for Occupied VR is the very epitome of the innumerable offices that informally serve as the VR world’s R&D division. Occupied VR is involved with a variety of projects for the medium (and holograms, too) that range from games to short films to “branded experiences” to whatever other uses people can dream up.

In fact, the field is so new that no one’s even settled on the terminology. Some reserve the term “virtual reality” for more interactive and viewer-directed experiences, which makes it more apt for the emerging breed of games like Oculus’ long-awaited multi-player shooter EVE: Valkyrie. (A forthcoming Neuromancer-inspired game named Technolust is one of Occupied VR’s flagship projects.) The phrase “360 video” may become the preferred one for the immersive, stereoscopic short films (whether doc and narrative) and music videos that users of the Vrse, Oculus and Jaunt apps are already seeing with their phones and Google Cardboard viewers. (To save any further confusion in this article, “VR” covers both.)

Though the cardboard contraptions are fine for the time being, they’re no substitute for the headsets. That much is clear once I get my inaugural immersion in a much-coveted Oculus Rift (priced at a daunting US$599, the consumer edition is released at the end of March). First, Occupied VR president J. Lee Williams has to conquer the usual set-up hassles. “VR conferences are hilarious,” he says, “because it’ll be a three-day conference and the first two-and-a-half days are just people trying to get the machines to work and saying, ‘I swear it’s awesome!’ Then for the last two hours of the show, everyone’s got the stuff running and they’re scrambling to show it to you.”

Produced by Ana Serrano at the CFC Media Lab and directed by J.Lee Williams and Blair Renaud of Occupied VR, David Cronenberg: Body/Mind/Change Redux is the VR extension of TIFF’s recent exhibition on the Canadian director. Drawing heavily from imagery in The Fly and Videodrome -- whose “Image Accumulator” helmet-like device could’ve been an ‘80s prototype for the Oculus Rift -- the short is an eerie yet ingeniously effective example of VR at its most cinematic. That quality is suggested by the same oddly languorous establishing shots Cronenberg has long used as well as the intricate sound design.

Indeed, spatial audio may prove to be the most valuable tool for storytellers hoping to control the perspective of viewers who can suddenly look in all directions and whose attentions may be impossible to corral. In the case of the Cronenberg piece, noises call attention to particular corners of the virtual environment. More insidious is how the buzzing of a fly near your ears (and then seemingly between them) toys with spectators on a more subconscious level. Says Williams, “The best directors for VR aren’t going to be the biggest directors in Hollywood – they’ll be the ones who understand the human mind and psyche.”

Given the more visceral nature of the viewing experience fostered by scary movies, it’s not so surprising that horror is the predominant genre for the narrative short films either already available in VR or just around the corner. Canadian studio The Secret Location won an Emmy for their VR teaser for Sleepy Hollow and created a similarly eerie promotional piece for Insidious: Chapter 3 – it now has a whole VR horror series in the works.

“It feels like an incredible medium to really immerse someone in a world that is full of dread,” says Patrick Milling Smith. “It can make people feel very uneasy.” A favourite of expos and festivals since its debut at SXSW in March, Catatonic is’ first contribution to the genre. Here, the viewer is placed in the role of a newcomer to a creepy old mental hospital that must be two doors down from the one in American Horror Story: Asylum. Flickering lights and freaky figures abound but as in Jaunt’s two mini-thrillers thus far – Zombie Purge and The Black Mass – the narrative elements are too thinly conceived for the result to feel like more than a teaser. Such exercises may succeed as first-time experiences but once VR’s novelty value fades, content has to become more compelling.

Andrew MacDonald’s already wary of the glut of POV experiences in the first wave of VR fare. “It’s all about ‘Hey, we’ll put the rig on a bobsled or go paragliding with it or put it in a racecar,’” he says. “There’s only so much of that you can take.”

What he hopes to see is more efforts to integrate some old-school cinematic vocabulary into the VR world. That’s what he tried to do with Agnus Die!, a VR homage to Psycho’s shower scene whose racy, campy trappings belie its crafty consideration of the challenges of film editing in the new medium.

Says MacDonald, “One of the things I was conscious of is that in the 360 world, you have no control over where anyone is looking when they’re watching the piece. I find that with pieces that are either cut too quickly or aren’t thought out, viewers are often looking somewhere when the cut happens and then it’s confusing. They’re like, ‘Oh, I was looking at a lamp and now I’m looking at a car -- what’s going on?’”

With Agnus Die!, MacDonald wanted the action to drive the viewing. Thus when the Janet Leigh surrogate is stabbed and falls to the ground, the viewer’s eyes follow her downward. “That gave me the opportunity to cut in that iconic Hitchcock image of the blood going down the drain,” says MacDonald. “Ideally you’re using story to really drive the viewer to be looking in the right spot so you can throw an edit in that makes sense.”

Alas, one shot that’s missing is an equivalent to the close-up of Leigh’s screaming face. Really getting into actors’ faces will remain a challenge with 360-degree cameras, which is one reason why J. Lee Williams has started experimenting with 180. “I think the whole gotta-be-360 thing is going to fall away,” he says. People may demand it to “make them feel for the characters they’re watching.”

Indeed, with their cumbersomeness, 360-degree camera rigs present a whole series of complications that may drive content creators to adopt nimbler creative set-ups. One of Michael Naimark’s current projects is the “IMU VR camera,” which is essentially a pair of glasses equipped with two wide-angle tiny HD cameras, binaural mikes and an inertial measurement unit in the middle. Says Naimark, “I track VR cameras very carefully and people are building cameras that weigh several hundred pounds made out of as many as 14 RED Dragon cameras.” He laughs. “My thing costs nothing, weighs nothing and is tiny.” He is just as happy to challenge the emerging gotta-be-360 orthodoxy.

On the other hand, close-ups may be the last thing viewers want to experience in VR. Janet Murray had that thought when she experienced Henry, one of several new works by the Oculus Story Studio that combine the emerging aesthetics of VR with the more tried-and-tested tactics of Pixar-style animation. She notes that in developing Henry, the Story Studio team realized that giving viewers a closer view of the titular hedgehog hero when he’s feeling sad could produce a very different result than if it happened in a film. Says Murray, “It’s actually disturbing when you are in the space because you want to reach out to him but you don’t know where the fourth wall is. What you want is for him to be further away!”

She considers that a very smart insight, as well as a potentially key one. “It’s great that people are making stuff and articulating these insights,” she says, “but it’s going to take thousands of such insights in order to come up with the vocabulary for storytelling that we take for granted in a novel or television show or movie.”

Though Occupied VR’s Cronenberg piece and MacDonald’s Psycho riff show how VR can function like juiced-up movies with a hugely expanded (and now essentially endless) frame, other examples suggest that the new medium may demand a more radical overhaul of narrative conventions. Created by Canadian filmmaker Samir Mallal (Nollywood Babylon) with visual effects artist Aron Hjartarson for, New Wave hardly seems earth-shattering, being a modest vignette about a beachside spat between twenty-something lovers. Yet by allowing the viewer to decide which character’s voice to hear -- like so much in VR, it all depends on where you swivel your head -- the film toys with an innovation that may have wider ramifications for storytellers.

Perhaps the VR environment should be regarded not so much as a new place to tell stories like the kinds we already know – rather, it could be a container for an set of overlapping narratives, with the viewer now in charge of what paths to take and which characters to observe. According to Milling Smith, has more such films in the works, where “you’ll have a split narrative and you can choose which journey to have.”

Ironically, with VR’s potential favouring of more open storytelling modes, its emphasis on immersive experiences that erase the proverbial fourth walls and its various efforts of corralling viewers’ roaming attentions, it may have less of a connection to films and TV than it does to theatre. As an art form, it’s been contending with many of these same issues for centuries. No wonder that’ ever-expanding of collaborators now includes Felix Barratt – he’s the mastermind behind Sleep No More, an immersive theatrical event that takes place over many floors of several New York warehouses and allows attendees to decide how they proceed through the stories. As Milling Smith says, “This immersive world is what Felix has been working in for north of a decade so it’s a very natural step for him to find out how to capture it.”

Nicolas Billon also believes that his experiences with his hit plays The Elephant Song and Butcher give him tools that may be especially useful in this brave new world. Says Billon, “I don’t really think in cuts -- I think in scenes. That’s something that will come in handy for VR.”

Besides the chance to realize his fantasy of devising his own Choose Your Own Adventure scenarios, he’s intrigued by the impact of how the audience will approach when they’re presented with a dramatic incident, like a couple in a heated argument. “How far away do you stay from it?” he wonders. “How close up do you get? And who are you are you looking at during that argument, who do you choose to look at? It’s all of those things because I think the story changes depending on who you look at.You’re not necessarily looking at the two people arguing there, for instance -- if there’s a child there, do you watch the child watching the argument? In VR, I feel like you have that choice.”

These shifts between macro and micro narratives and between character perspectives could entail changes of genre as well. “You can watch the tragedy,” says Billon, “or you can move over to the comedy.”He’s right to wonder whether VR stories will “use mashups in a way that we’ve never seen it before.”

More VR could also follow a path that’s already been interactive storytelling circles. One project in development in the CFC Media Lab explores the potential of “layered video.” As Ana Serrano explains, “That allows people to seamlessly move between scenes or various points of views of scenes based on the movement of their heads. As an audience member, you’re watching a film that you think is the linear film and can only be the same film over and over again. But then when your friend does it, they see a different take of that film because they move their head differently.”

The idea of turning the user into a film’s de facto editor was central to earlier CFC Media Lab projects for web and mobile platforms. “I think VR is another medium where you can play around with that,” says Serrano. “An interaction model based on where your head, your gaze is placed might even be more natural than point-and-click.”

Janet Murray is especially intrigued by the ways in which VR content creators will create “dramatic compression” in the environments they’re building, ensuring that viewers have reasons to stay and explore these new spaces even when they can’t touch or manipulate objects within them. But she also emphasizes that it could take VR just as long as cinema did to develop a truly mature and sophisticated visual vocabulary. As she says, “You have to figure out what’s equivalent to changing the lens and cutting the sound and acting for the camera – there’s a whole host of engaging design questions that will just take a lot longer to sort through. I see that there’ll be people in this for a long period of time and there’ll be productive development but that some people who are engaged in doing it now will be leaving because they won’t be able to make money from it in the short run.

“It’s on an artistic timeline,” she adds. “It’s not like making a new toaster. It’s inventing a new medium and you don’t do that in four months or six months or two years.”

Serrano believes that those adoption and innovation rates could be greatly affected if ordinary consumers can begin making their own 3D/360-degree movies and sharing them with the same ease they have with iPhone cameras and YouTube. “If we have the part of the ecosystem that allows the regular Joe who already likes to use Instagram and make little movies to make immersive cinema easily, then I think you’ll have more of an audience that wants to see what the pros do with the form.”

For his part, Michael Naimark cautions against the “herd mentality” that goes along with any zeitgeist moment like the kind VR is having. “The outlier work is something I would rather encourage than jumping on the bandwagon and saying, ‘Start making VR stories! Shoot 360 all the time!’”

For the moment, so many tantalizing strategies remain in the theoretical or R&D stages at best. But while Goldman’s line might right true about nobody knowing anything for sure, dribs and drabs of the big picture are emerging fast for VR content creators. What’s more, these tools and tactics may not even really be for the storytellers reared on movies and TV. As J. Lee Williams says, “We’re not building VR for our generation, we’re building for the next. They’re the ones who will adapt to it faster.” He laughs. “We don’t have the young brains anymore!”’ Patrick Milling Smith on VR’s great expectations

What signs do you see that VR is having its tipping point?

I think the New York Times partnership and the giving away of the Google Cardboards was one. So was the selling out of the Samsung Gear VR -- the Sony PlayStation VR will be another big moment. There’s definitely a hunger for the headsets and the adoption is happening quicker than anyone could’ve imagined. Now the medium depends on compelling content and that comes down to storytelling. I think people have to craft and execute good stories and good experiences and that will super-charge the adoption. Again, gaming will be a big part of that, and genre films, too. This will be an enormous year for VR and really cement it as a mainstream medium.

Do you foresee it becoming the dominant entertainment medium?

It’s just an addition. I’m intrigued to see how people use the headsets. If you look at Netflix with their app where they look it as a large TV set with fantastic spherical 3D capabilities. So you can go to the Netflix app and watch their traditional 16x9 content through your headgear. I’m curious to see how some people find it unnatural to strap this thing onto their eyes and others do it with great ease and it feels very natural.

Is it even fair to expect people to consume VR like they do with movies and TV?

It’s a different pace and a different language to traditional film. And there’s already a demographic today that finds it hard to watch a 90-minute film -- 44 minutes seems to be the max for the new generation. If you look at viewing habits online, people are embracing shorter and shorter forms of content. So I don’t know if the mission is to replicate traditional film. It will land on a number that is natural to the format and to the story you’re trying to tell. I don’t think there are too many barriers right now. But the more people immerse themselves in VR, the more they get used to it, the more comfortable they’ll be for longer periods of time.

Why is documentary film such a hot area for VR?

It’s a natural medium for it. Rather than showing somebody through an editorialized, edited point of view with choices of shots, you’re really dropping someone else’s environment, letting them walk in someone else’s shoes. The barrier of entry into someone else’s world is removed and you’re really making someone feel like they’re there and they have that sense of presence. They’re bearing witness to events that have so much more truth to them because you feel like you have the freedom to stand amongst them.

But do you at all fear that VR users will lose that collective experience they may enjoy as moviegoers because of the nature of the headsets?

I don’t think Facebook would’ve spent the money they spent if they weren’t planning to make it a more social experience so I think that’s coming. We’ve done demos where we’ve triggered 250 headsets to all play the same content at the same time and sure, it’s a very immersive experience, but if you go to a movie theatre and watch a great film, you’re sucked into that film and it’s only really when the credits come up that you then turn to whoever you’re with and you talk about it and you share your feelings about that shared experience. I don’t think that’s such a step away in VR. It’s just that much more immersive. I don’t think it stops you having the same experience and having a connection with others about what you just went through. I think that kind of things happens with every new medium -- when books first came out, they thought it was the end of conversation.

Further takes on the VR (r)evolution

Now that VR is having its zeitgeist moment, there’s an inevitably diverse set of opinions about what exactly is supposed to be coming around the corner. Will it become the ne-plus-ultra in entertainment media or fall free to the over-hype and disappointment that snuffed out VR’s first flickering in its ‘90s? One problem is that there’s just not enough material to know. As Michael Naimark puts it, “We need more samples.”

Even so, some film-biz veterans believe that VR’s already had a chance to get a cultural foothold and been found to be wanting, at least when it comes to competing with other means of storytelling. In an interview with The Guardian, Pixar’s Ed Catmull was frank about its limitations: “It’s good but it’s not storytelling.”

Some filmmakers aren’t sure what to make of it yet. In an interview with The New Yorker, Werner Herzog points out that part of the problem with predicting anything about VR is that unlike so many pivot points in the history of culture, the technology is driving the content rather than the other way around. By his typically Herzogian assessment, VR is a new container we don’t have “any clear idea how to fill.”

Nevertheless, content creators are inching closer to a clearer picture. In a video made for the Future of Storytelling summit in New York last October, Oculus Story Studio’s Sascha Unseld shared some of the fascinating lessons learned during the development of the animated short Henry. In many respects, what’s being challenged is our basic understanding of our own roles as spectators. As Unseld says, “A book works through inner monologue – you read what the character thinks. In a film you understand a character through their actions. And in VR, I think you understand the story more through what you feel in a situation.”

That’s an idea with massive implications. Janet Murray wonders if VR content creators will get the time they need to discover enough them to know what really belongs in the container.

“What I started predicting in the fall -- and I already see it happening -- is that people are going to announce they’re disappointed, that it failed, that it was overhyped,” she says. “This is a familiar pattern to me from my work with interactive TV development all through the 2000s and with conversations about digital technology for the past 30 years, really. The early adopters see the potential but it takes a long time for a community of practice to arrive that creates the conventions that make for coherence.”

Murray notes that the biggest challenge is having the patience to see a new technology through all these necessary iterations. In other words, creating a whole new medium can take some time. “The thing I always say it’s always easier to tell the direction of change than to predict the pace of change!”

Jason Anderson is a journalist, lecturer and film programmer in Toronto.