We have to talk about Virtual Reality.

It’s VR talk time, you’re old enough.

Let’s get this out of the way – 3D television died, Google glasses died, everything is bad, why try anything let’s just sit around bitching. Great, thanks for the opinion. But that argument is based around success I’m not looking for. There is zero chance or interest that I will be “a media tycoon by getting in at ground level”. People that think that way are calculating profit and loss, not making fun things. VR will probably die, so did a heap of things we’ve enjoyed.

But you were against Google Glasses and AR! Yes that’s right. I don’t want to build something that tells me what I am seeing. I want to build entertainment. VR and AR are only related in the most superficial way.

I’m learning how to handle VR by first remaking the 2010 video for Greater Reward.

all

Until you actually make something it’s all theory. Then the problems come crashing in fast. Let me add my voice to the very sensible advice already there.

  • Until you see the shot on a VR helmet you have very little idea what you are getting. The distortion moves everything away, your instinct to make things fit the image is wrong. Something that looks OK there is about 1cm from the viewer’s face and that sucks.
  • The inter pupil distance really matters. If your camera has pupils 6.5cm apart then everything must be scaled to that. If it’s wrong you end up with things being kilometres in size and that hurts. There is no zoom. Prime lens only.
  • There are no edges to the frame and so you have to design in all directions. But there is a centre of attention, only 70° in the middle. Your viewer is seeing a little bit at a time, you don’t know which bit. So most scenes should be simpler than you first thought. You only place complex things where you want the viewer to look. So like the way you place furniture in a room.
  • I tried tilting the camera. Nope. The viewer feels gravity pulling them down, the scene doesn’t, so it tilts, not the camera. It just doesn’t work. That and the previous point mean most of your camera skills are dead. So if you want the viewer to look up, you have to move something up there to get their attention. Same for most directions. You use the same tricks as stage plays: light sound action.
  • Editing seems OK so far but jump cuts are worse than usual. You need to move toward the new position, or pre-empt it. I’m testing fades now.
  • Technically you are making 4K but the bit the audience is seeing is about 640×480. All textures should be severely anti-aliased. Like Gaussian. No grids that will moire with the pixels. Think 1990’s computer graphics. Everything smooth.
  • Drop the light levels down, keep the lighting comfortable. You can’t use normal filters on a VR image – e.g. blur. So you have to get this right in camera.
  • I’m not seeing too many problems with motion but I am keeping it slow and steady. Also I am used to the helmet. One day maybe people will all be used to VR and some of these rules will be broken.

So is all this pain worth it? Depends on the material. I generally would steer the vast majority of film making away from VR (I wish my students would listen). You cannot perform cinema with VR. If you ever write ‘we see’ or ‘moves to’ in your scriptment do not use VR. As for Greater Reward, all the scenes are positioned in spheres of some sort and so the translation becomes possible. But what else will work – I just don’t know yet.

(Nothing here about sound design – a different post for that).

Back on the VR rollercoaster

Here you see a CINERAMA screen, from 1952.This.Is.Cinerama.1952a

CINERAMA was a big hit in that year. A standard film of the time would fit in the middle screen only. So you can see what an impression it must have made. I’m interested by the kind of films that bloom with any new technology. There must be a roller coaster film. It’s likely to be the very first thing that gets shot in the new (cumbersome) format.

rollercoaster3603d

Standard VR Roller Coaster film

And close behind are what we now know as a “Go Pro movies”. Sporting feats.

This.Is.Cinerama.1952b

And aerobatics.

This.Is.Cinerama.1952c

Once these technical demonstrations are done it’s time for a few experimental works by corporate funded artists, some stadium sports and a slowly wilting realisation that a good story works just as well on an old analogue TV, so what are you going to do? George Pal did good business with The Wonderful World of the Brothers Grimm in 1962.

cinerama-grimm-3films

Eventually the name CINERAMA became more important than the process. By the late 60’s it was a 70mm print stretched out. But the stories were better for it.

20015s

I would have liked to have seen a CINERAMA print, but standard Super Panavision was pretty cool. (My folks took me to see it soon after it came out.) This film isn’t right on analogue TV. Here’s a site devoted to wide screen film.

So instead of just going through this cycle, maybe we can think about it. What benefits can be found in the VR format? Obviously there’s interaction, but I’ll just leave that to one side for now please.

Consider this – no one can stand behind a spherical camera. There is no behind.

People working in VR are keen to point out that the frame is no longer there, and so the idea of composing an image in a frame is lost. So you cannot center the image, show something to one side – any of that. Edits are OK but you cannot know which direction the audience is facing. “Cut to:” is rendered useless when they could be looking at the sky for all you know.

The standard of ‘reversals’ – over the shoulder conversations – is dead. Next time you watch a film count how much of it is reversals. Then realise – that’s gone, all gone.

Nausea – it’s about the hairs in your ears that detect acceleration. Not movement – a scene in a moving train is fine – but the ease-in and ease-out motion of a standard motivated camera move. When the ears detect that you’re doing something impossible it’s time to prepare for emergencies and up comes lunch.

VR is about sound.

If this sounds as if VR is more problem that potential, and if you are a standard film maker please go elsewhere and let the experts take over. By experts I’m speaking about sound designers who have dealt with 360º for a very long time. We build realistic spatially coherent environments out of the materials of sound. We turn your head to the events you then watch. We signal that something is occupying a point, an arc, all of space. We signal that something went by, causing a Doppler. Our Foley gives substance to things in all directions.

VR demands that the old equation of vision first / sound second be turned on its head. Because you are not going to be able to navigate the worlds that you’re filming without first realising the air, the tone of the room, the placement of all the sound sources. If your set has three walls and relies on a particular orientation to work – you’re in deep shit.

But if you are a sound designer for film – don’t be smug because the game is going to get considerably harder. Get a VR helmet, learn how it works. Then start to build soundscapes that work in 360º. 5.1 won’t work any more there is no front/back. Listen to your mixes as somebody would in a room. What size room? Round? Carpeted? You are suddenly required to not only capture the air, but to construct it. I can’t tell you how we are going to do that. I can tell you that if we want to get past the demonstration films it’s going to be up to you.