From 1 to 100: A Conversation with Oscar-Winning Film Pioneer John Gaeta

By CFC Staff March 25, 2016 14:25

The brightly-lit conference rooms, even at the Hilton, are not conducive to discussions about immersive media. Panelists and participants are subjected to a fixed AV set up, which prevents speakers from creating a feeling of immersion. So when Ana Serrano (Canadian Film Centre’s Chief Digital Officer) asked us to close our eyes and listen at a VR panel at SXSW, we were game.

Serrano played that familiar pivotal scene from the movie The Matrix, the one where Morpheus (Laurence Fishburne) meets Neo (Keanu Reeves) for the first time and asks him to choose between the red pill or the blue one. That’s a scene that John Gaeta is very familiar with. After all, the Academy Award-winning VFX supervisor and Creative Director of ILMxLAB (Lucasfilm's new immersive entertainment division) created The Matrix’s bullet time

Gaeta is currently extending onto a variety of platforms, including VR, the entire Star Wars franchise. Serrano kicked off the conversation with him by contextualizing VR as being at an inflection point, similar to when the Netscape Browser launched in 1994. Creators, she said, have the power to determine exactly what kind of immersive media ecosystem to build: one that can lead us to a more Matrix-like future, or one which intentionally moves away from it. She then turned to Gaeta and asked, “What pill would you take?”

“The purple pill,” he answered. 

Gaeta went on to talk about a future where users/citizens can have control over how much virtual content can seep into their real worlds. (“Imagine going from 1 to 100.") He described a scenario where participants can control how they move from augmented reality experiences, where information is overlaid on top of what we see, to a more mixed reality environment (akin to the Microsoft HoloLens demo videos), and eventually to full-blown VR, where you can completely shut out the real world.

This reframing of VR as part of a continuum is an important one. Gaeta and Serrano referred to “control and privacy” throughout the interview and this vision puts the user very much in control of his or her own immersion. The future, according to them, should be as much about how the user manages her own immersion into media as what she experiences.

This nuanced vision clearly informs much of the work being done at ILMxLab. There’s a thoughtfulness to the way Gaeta and his team design immersive media experiences. Referring to them as responsible designers, Serrano shined a light on their graduated approach to immersion, a framework which Gaeta referred to as “story portals.”

Eighteen months ago, way before all the VR hype hit, Gaeta developed a prototype for this framework to work on both tablets and Oculus Rift. The demo he showed the SXSW audience was impressive, all the more so because what seemed to be an interactive cinematic film was actually being rendered in real time.

The “story portals” prototype works with seminal scenes in the following way:

  • The scene itself provides the context for the user, giving meaning to the experience. (In the prototype we were shown, the scene was new yet all too familiar, as it took place in the Star Wars universe and revealed R2-D2 and C-3PO on Tatooine trying to find a means to escape.) 
  • The user can explore the actual scene by pausing the film, moving around in 360°, choosing a direction, un-pausing and then resuming through a totally different path in the film. In this manner, the user can either simply explore, or intentionally follow secondary characters and plot lines. 
  • Should the user wish to participate more actively in the environment, maximum agency gives him or her the opportunity to control certain elements of the scene (i.e. flying an X-wing through the scenic environment).

Each of these interaction models are facilitated by an intuitive, almost invisible UI that Gaeta is further developing. Audiences were wowed by the technical prowess of the ILMxLAB team during the demo, given the flawless finish of the experience, which virtually captured the feeling of walking around a film. 

From “Story Portals” to “Holo-Cinema”

Walking around a film is ultimately what we long for. Although there seems to be a disconnect between Passive VR (and all those 360° videos the New York Times is pushing), and the Active VR games (which will undoubtedly be the next killer apps in the foreseeable future for Oculus, Sony and HTC), there’s a reason why experimentation in both is important: fusion of cinema and games is where it’s at. And ILMxLAB is definitely close to consummating this marriage.

With “Holo-Cinema” and the prototypes they’ve built to showcase this marriage (recently launched at Sundance), Gaeta and his team hacked together existing technologies – sensors, stereoscopic glasses, Mo-cap tech, Cave-type projection – to create a petting zoo like no other. Imagine being able to “touch” a life-size Jurassic Park baby dino. The demo video they showed at SXSW revealed just how close Gaeta and his team are to creating an experience like the Holodeck.

As Serrano pointed out, having story worlds like Star Wars and Jurassic Park to build on puts Gaeta and ILMxLAB at a distinct advantage, way above the stratosphere compared to other VR creators. Not only do they have access to all that amazing material, but setting the context for the experience (arguably one of the most difficult things to do with new properties on new platforms) is a non-issue since audiences are intimately familiar with the worlds. 

What's Next: Tips from Gaeta for Those Creating VR Properties from Scratch 

  • Think series -- Since VR experiences tend to be between one to five minutes in length, context and engagement can be built over time and across episodes.
  • Think social -- The social dimension of VR will make or break it. Be one of those creators who take a crack at what a VR social experience could be.
  • Think storymaker -- Stanley Kubrick had a vision for what kind of story worlds he wanted to create and explore and he built new rigs and machines to realize them. VR demands that we literally become storymakers, hacking away at conventions, technologies, systems and processes.
  • Think team -- If you’re a 360° video maker, find a game engine developer (Unity or Unreal) and work with them. Filmmakers need to understand VR films as spaces to explore. Game designers need to understand how stories are told.
  • Think design with intention -- There will be VR made that may unintentionally invade people’s privacy, easily facilitate addiction, and/or guide people irresponsibly across experiences they may not be ready for physically or psychologically. Don’t be one of those people.