AR It’s All an Add

I’ve seen a lot of very hyped “visualizations” of what new next generation AR experiences will be like over the next few years, and one thing none of them are covering is how they’re dealing with the compositing of the augmented content with the reality content.  In a mixed reality mode, it’s pretty straight forward because the entire image is synthetic and the new content is simply comped “over” a live feed from a device camera. But with augmented content, light field, OLED, etc, the augmented content is  reflected, refracted or projected over the background, essentially compositing the new content as an add – and so effectively it exists like a reflection on a window.

Which is great – it’s a very useful and impressive thing, but it’s not generally what’s shown in the product marketing collateral. To their credit Epson shows the simulated images for their Moverio glasses as being somewhat transparent, and maybe because they are used to dealing more directly with imaging professionals. So what does this disconnect between the promise and the delivery mean? Well it might be that no one really notices – they may be so blown away by seeing aliens attack their living room that they don’t care that their floor lamps are visible through the enemy hordes. However, they might just as easily be left feeling that they’ve just watched an early B monster movie. Will the take away from the AR revolution, be that it’s nothing more than just tech hype, over promised and under delivered?

What troubles me here is that it would be very easy for the marketing videos to be pretty darned accurate to the actual display characteristics of the devices – just composite the elements on the video feed as an Add rather than an Over and you’d be very close (+- some error in trying to match luminosity, etc). But they don’t. They display it as though it’s an opaque element – and well … that does look better, it’s more realistic and ultimately presents a far more compelling experience. And the decision to present it inaccurately probably means they know how much better it looks. So they must be a bit worried about showing the reality of what’s really, currently available.  And if they’re worried I’m worried.

Composting in real time AR/MR experiences actually offers some really cool development opportunities – let’s hope people start taking those on soon.


Freeing Immersive Content Creators from App Trap

One of the biggest hurdles facing anyone wanting to deliver AR/VR content right now is that every different implementation requires a different packaging of content data. Some of this is a result of the “game” and “app” ecosystems that these experiences come from, but there’s also no other alternative.

Content cannot be delivered as a broadcast stream because there is no definition of what that stream is. Without that there is no standard viewing “environment” to leverage. There are some attempts to work on this – YouTube’s 360 video is an interesting way of delivering one component of immersive content, but it’s not an extensible or leverageable technology. It’s essentially only a movie player. A content creator cannot, for instance, embed a 360 video as one of many elements in a deliverable program.

And so content creators also have to be technologists capable of building worlds of mixed elements inside of an app or game metaphor. Each experience is a one-off, individually crafted delivery of heterogenous content. But most of this content is really just reconfigured instances of only a handful of different kinds of data – 2d, 3d, static, animated, geometry, images, navigable, etc. And this repetition could be exploited into not only a consistent data exchange “format”, but also a consistent experience environment. A content provider would construct, not an app or game, but a container of elements and descriptors, deliverable as a “unit” to any compliant experience environment. Like a broadcast network delivered TV shows, bounced off satellites, thrown across the airwaves or down cables to a TV set that decoded and displayed the experience.

But what would that package look like? How can we all agree? What are the NTSC, mpeg, jpeg, obj, wav of VR? Is it a file? Is it a file aggregation container? There are a lot of questions to answer, but the freedom afforded to content creators when they no longer have to worry about he technology of the viewing experience, could bring the freedom that other creators have had for years. Film makers don’t have to worry about the inner mechanical workings of projectors, writers don’t have to worry about how printing presses work, and AMVR content creators should not have to worry about writing apps.