What contests are we winning?

Talking with people in the AR/VR world there’s a constant, silly question buzzing in the air like a gnat? “Who’s winning – VR or AR?”. It is an interesting question, not for what the question is asking but what asking it implies in the first place. Is this all a contest? With a winner and a loser? Have we become so obsessed with the “gamified marketplace of ideas” that we can’t actually be motivated without some implicit or explicit conflict or large plush prize? But what even is the conflict? What is there to lose in this contest of AR v VR?

The contest implies they are the same project, suggesting that their finish lines are the same finish line. They are drawing upon a lot of the same technology for sure, but so are mobile phones, connected thermostats, smart TVs and watches. The basic antagonism seems to revolve around posturing –  for the best head mounted display, or the most pure vision of what is meant by “immersive” or “reality”, or who is the reigning champ of the “ultimate experience”. And this would all be as ludicrous a sideshow as it sounds, except for the number and stature of people involved on both “sides” who act like it’s a serious debate. In fact it was an actual debate at this year’s Augmented World Expo, and only a marginally tongue in cheek one.

And I get how important it is for one hardware maker to be able to capture market share, get funding or get acquired. Or for a game publisher to drum up marketing collateral to prep for a release. What’s a little bothersome is how easily this marketing spun copy is eaten up by people who should know better and then regurgitated as a real, pressing issue, when the real pressing issue is that people need to move past the towel snapping and make more complete things that are actually worth doing.

Manufacturers of HMDs want to demonstrate that each has the better display resolution, the better optics, better hardware integration – this makes perfect sense. It’s like competing computer chip makers claiming theirs is best because of clock speed, number of cores or instruction sets – it’s reasonable.   The differences in comfort, tradeoffs between configurability and convenience, and comparable aesthetics are like the PC v Mac debate – okay, I get that. Arguing whether VR or AR  will “win”, or is “better”, is like someone arguing that a realtime, embedded OS is inherently better than an interactive one like Windows, or that a freight train is better than a cargo ship.

If we take the crassly entrepreneurial measure of money – then AR has already “won”. It has market share, it’s profitable in products now, it generates revenue. But really, it’s a silly debate – we’ve been augmenting and virtualizing reality for years : the transistor radio, books, air freshener, hell even the rearview mirror. Timothy Leary is laughing at us all right now because he “won” the contest 50 years ago – and without a computer. So what should you do when someone asks you “who will win AR or VR?”– I think I know what Dr Leary would do.

Is there a VR equivalent of the whip pan?

There are many ways that a virtual reality cinematic experience differs from a traditional one. Much of the discussion is around whether or not certain techniques translate from one world to the other, and what can be done psychologically, physiologically, or mechanically without inducing pain or nausea in the viewer. But there’s less discussion about the narrative structuring elements of traditional filming and editing techniques and what their immersive counterparts might be. There are numerous camera and editorial decisions that we have “learned” to understand in watching movies – the so called cinematic language. Things like the over-the-shoulder point of view shot, the two angle dialog shot, or the sequence that moves from a long establishing shot to a medium shot to a closeup shot to introduce a place and/or character. But I’m going to choose the whip pan as the best example to illustrate some basic differences between traditional and VR cinema.

But why the whip pan? Well because it’s a very useful and ubiquitous film transition, that functions on many levels, yet it is almost entirely physically useless in an VR experience because that much camera motion is disorienting and causes motion sickness.

So is there a VR analog of the whip-pan? And what is a whip pan? Simply put – it’s where the camera spins very quickly from one point of view to reveal another one. This winds up putting the ending view in juxtaposition to the starting view. A common example might be a tense POV shot where the camera whips around to reveal the threat – a monster, the enemy, or antagonist of some kind. Also, because one byproduct of moving a camera very fast is an exaggerated motion blur that obscures what’s being filmed, a whip pan can be used to hide a transition to a radically different camera angle or even a different setting or time. The whip pan wrenches our perception and attention from one understanding of the story to another.

To illustrate the difficulty in finding a VR analog, let’s look at a whip pan example applied to both traditional projected cinema and immersive cinema.

Traditional: We are watching a horror movie, the setting is a forest at night, the shot is a POV shot and we are walking forward slowly, scared, seeing only what’s lit directly by our flash light. We hear a noise from behind – a breaking twig – the camera whips around to exactly where the monster is, and then drops down to just miss an enormous claw aimed right at where our heads would have been.

Immersive: We are experiencing the horror movie in a first person role, moving through the forrest at a rate that’s not too disorienting to be uncomfortable or distracting, maybe our attention is following the flashlight beam. There’s a sudden noise behind us, a breaking twig, we turn in our seats to, maybe see the monster in time to maybe duck underneath its giant claw. Maybe our reaction times are too slow and we miss the action of the monster entirely but are thrown into the next segment of the narrative anyway. Maybe we’re left confused and lose interest.

So what is it that makes the traditional whip pan so effective, what did it facilitate, and what does it lend to the overall emotional effect of the shot? The whip pan is the reaction of the audience; we have been conditioned as an audience to interpret this motion as our own action, something which on some level we feel we have created and are in control of. Our experience is that we are reacting to the sound by actively turning toward it and facing it, when in fact that’s very far from the truth. In reality the “camera” presciently reacts to a sound created at precisely the right moment, at exactly the right speed to get to the perfect point at the next perfect moment to both reveal the horrifying threat and simultaneously and miraculously escape it. This is not an exercise of free will, this a controlling cinematic device with a very specific narrative purpose and outcome: disorient, respond, reveal, escape.

There are also practical components of a whip pan that are often just as important. People in VFX know that you can hide a multitude of sins in the motion blur. Even in non-effects films, editors can hide a number of unlikely or incongruous transitions by dissolving through a fast camera move. Very often the camera angle or placement from the A part of the pan is not desirable for the B part. Or maybe you want to pan from being tall and looking down to being short and looking up. This can be very effective at making a revealed threat seem more ominous, or it might simply be an accommodation for practical size differences. The point is that the confusion and the disorientation of the pan itself is used to hide incongruities and inconsistencies that occur in getting from your origin to your destination. Because we know to accept this motion blur as transition from one thing to another it can also be used to transition from one place or time to another – we begin the pan and everything is green, we end it and everything is covered in snow – we know that we went from summer to winter and probably understand that we had a whirlwind autumn.

The whip pan goes beyond a simple camera move and becomes a complex mechanism of narrative cinema. How do we give a VR audience experience the same emotional experience? Should we? It may be that the whip pan is too idiomatic a part of traditional film to have an easy, direct counterpart – like one of those German words that has to be translated into an entire sentence in English.  Maybe it’s a matter of breaking entirely from a first person POV experience to an angle where we see the entire resolution of the action?  Maybe trying to do that at all is missing the point entirely. Maybe VR is not about controlling the view, but rather controlling the environment. Maybe narrative control is an outdated authoritarian construct. Maybe it all just requires our experience of VR to mature, and – like those people who stopped running out of early movies from oncoming trains – we’ll stop throwing up and learn how to read to VR’s abrupt new camera moves.

Transparency in AR advertising

It can be difficult to sift through the hype around Augmented Reality technology, and that’s not made any easier by misleading promotional videos that don’t fairly represent the final user experience. Granted, trying to represent what an AR experience will be to someone who has never had one can be difficult, but the over promising of marketing can actually hurt an industry still trying to establish its legitimacy.

So I was happy to see that Microsoft, in it’s latest promotional material for Hololens showing medical uses of the glasses, actually had images that depicted the transparency of the final effect. In the Windows Central article “New HoloLens video demos usage in medicine, is more honest about field of view” ( http://www.windowscentral.com/new-hololens-video-more-honest-about-fov ), we see that, in addition to a more honest field of view, that the image overlays are shown as being transparent. I’m glad, because up until now a lot of AR marketing material has the overlay images composited over the backgrounds as though they had opacity. Users expecting to see solid objects hovering in space would be justified in feeling bait-and-switched when all of the objects look ghostlike and ephemeral.

I hope that as AR hardware gets closer to consumer release that the accuracy of the marketing materials improves. The additive light technology is amazing and will be used for incredible things, but it won’t make the objects opaque – they won’t have that level of visceral tangibility. If users expectations are too high, their disappointment might match, and aside from high ROI research, technical, and industrial uses, the technology risks being seen as another fad.

NonSequitur 000 – Sports Social Media Aggregation

Here’s an idea – Sports Social Media Aggregation

I don’t have cable TV, and since broadcast television was “improved” into digital and away from being receivable, I’ve wondered how can I watch sports without going to a bar. Then it hit me.

There is, of course, no way to break the exclusive handshake between organized sports and sports broadcasting – it’s the money printing machine after all, but we might be able to look at it sideways. Is there enough coverage of sports on social media and internet radio to build reasonable aggregated coverage? This seems like something that four brogrammers sharing a studio apartment South of Market could whip up on a weekend bender. One site, many channels, windowed media streams – so have at it, and let me know the URL when it’s done.

Augmented Reality – it’s about to vanish

We’re about to witness a vanishing act. I was at AWE (Augmented World Expo) this year, and I had an even more distinct impression, than in previous years, that AR is about to vanish. And by vanish I mean it’s about to become invisibly present – just simply another component of user interface design.  I think this because what makes AR a “thing” to be noticed is just what a clunky, inconvenient pain in the ass it is to actually use.

But that’s improving every year. It’s still a clunky pain in the ass, don’t get me wrong, and it’s likely to be a clunky pain in the ass for a little while to come. But it is improving every year. The hardware is getting faster, lighter, thinner, with better batteries — less annoying, more fun — something to use. As the technology gets more ubiquitous, it becomes more integrated into everyday experience. And as the technology gets less uncomfortable it becomes less of its own thing.

We only need a separate name for it because it’s something “other” to general experience, and as it looses its distinct separation, it loses its need for a separate name.  AR will simply vanish into being a component in the way things are done.

Who am I? : Existential Crisis in Narrative VR

So exactly who am I here? There’s a story unfolding around me and I’m caught in the middle. The POV is my POV, is the protagonists POV, but I’m finding that I’m making decisions in this POV that I wouldn’t make. Walking to places I would never walk. And there isn’t the comfortable detachment given to me by a distancing frame — an over the shoulder shot where the POV is the protagonists, but my POV is that of observing the POV from an outside vantage.

So exactly who am I here? Suddenly I’m running, when I would not choose to run, or standing still when I would definitely be running. I’m not in control of my actions, because my actions are someone else’s actions supplanting and replacing my own. They become, in essence, my actions done to me. It’s as though everything I experience is taking place in a meta narrative where everything I see and hear are the experiences of someone else.

So exactly who am I here? Certainly not someone with agency. And not someone with self determination. I am not exercising my free will, I’m exercising the will of someone else – someone unseen. I’m trapped in someone else’s story. And while I’m experiencing it as though it were mine and real, my experience of it is dissociative and decidedly unreal. And my experience of it as a phenomenon, is anything but my experience of the proffered narrative.

AR It’s All an Add

I’ve seen a lot of very hyped “visualizations” of what new next generation AR experiences will be like over the next few years, and one thing none of them are covering is how they’re dealing with the compositing of the augmented content with the reality content.  In a mixed reality mode, it’s pretty straight forward because the entire image is synthetic and the new content is simply comped “over” a live feed from a device camera. But with augmented content, light field, OLED, etc, the augmented content is  reflected, refracted or projected over the background, essentially compositing the new content as an add – and so effectively it exists like a reflection on a window.

Which is great – it’s a very useful and impressive thing, but it’s not generally what’s shown in the product marketing collateral. To their credit Epson shows the simulated images for their Moverio glasses as being somewhat transparent, and maybe because they are used to dealing more directly with imaging professionals. So what does this disconnect between the promise and the delivery mean? Well it might be that no one really notices – they may be so blown away by seeing aliens attack their living room that they don’t care that their floor lamps are visible through the enemy hordes. However, they might just as easily be left feeling that they’ve just watched an early B monster movie. Will the take away from the AR revolution, be that it’s nothing more than just tech hype, over promised and under delivered?

What troubles me here is that it would be very easy for the marketing videos to be pretty darned accurate to the actual display characteristics of the devices – just composite the elements on the video feed as an Add rather than an Over and you’d be very close (+- some error in trying to match luminosity, etc). But they don’t. They display it as though it’s an opaque element – and well … that does look better, it’s more realistic and ultimately presents a far more compelling experience. And the decision to present it inaccurately probably means they know how much better it looks. So they must be a bit worried about showing the reality of what’s really, currently available.  And if they’re worried I’m worried.

Composting in real time AR/MR experiences actually offers some really cool development opportunities – let’s hope people start taking those on soon.