Who will make the merchandise display cases for VR shopping?

Many are convinced that simultaneous, shared, social experiences in VR and other 3D immersive modalities are a foregone conclusion. Regardless of how deluded we might be in this, one thing becomes clear – in order for this to scale, we will need to have a consistent way of describing all of the stuff – much like how molecules are a consistent way of describing the real world. Luckily the virtual world is many orders of magnitude simpler than the actual physical world, and instead of the uncountable trillions of sub particle level interactions of matter, the virtual world needs only a truly astounding level of trackable events through a potentially manageable number of protocols and standards.

The problem is that even at many orders of magnitude simpler, the task of how to consistently describe “anything” so that we can share it, sell it, buy it, travel to it, hold it, toss it back and forth, etc. is still really, amazingly complicated. Much more complicated than say – the choice of game engine d’jour, OBJ or FBX or Collada, or whether or not you have a cool physics engine. But what really are the basics of virtual matter that need description so that they can be manipulated in the ways we expect? I was thinking about this and came up with a functional, if prosaic example to get me into a more pragmatic frame of mind than say – blasting space zombie outlaws.

Let’s assume we have simultaneous social immersive 3D experiences delivered over a common framework. And let’s say that within that space there are millions of stores. And in many of these stores is the virtual equivalent of a merchandise display case. And let’s say your company makes display cases for virtual environments. There are a lot of assumptions here for sure, and the “display case” here is really just a conceptual placeholder for whatever the virtual world might offer up as a kind of “durable good”. But let’s put all that aside for the moment and assume that your business is making virtual display cabinets.

In the real world, display cabinets have certain features that make them more suited for some purposes than others, and yours are very good and specialized. In your case they are jewelry cases that have buttons on top that let a shopper rotate the shelves around forward and back. You make high end cabinets that are very durable and come in standard sizes that fit in with other leading retail fixture manufacturers’ products. The doors operate smoothly allowing ample access for the sales associate to quickly retrieve even the most tiny items the customer might want. When a business orders cabinets from you, they pay for them, you ship them out, they are installed and exist physically in place. No one can really duplicate them beyond manufacturing a knock off product.

In a virtual world retail businesses will want display cabinets, and just like in the real world they won’t normally want to design and manufacture them themselves. They will expect to buy them and for them to just simply work. Customers will be able to easily peruse their options and make their choices. They may want to try things on, see how they match the color of their eyes before they buy them. Your display cases will have to use the same “trying on” mechanisms that the rest of the display cases in the store do, because the store will want to support the latest most accurate shopping reality capture avatar system available. Your display case needs to be installable within the store’s inventory control scheme, but also installable within the stores local cartesian coordinate frame. It needs to be addressable within their asset management system so that stock changes and merchandising decisions can be pushed to the cabinets from central databases. Your cabinets will need to be backward compatible with this stores stock and inventory system which is several versions out of date because “they like their system fine the way it is”, and they are a big customer so you need to keep their business.

And so let’s say now you’ve managed to make a a future proof, universally accessible and addressable, fully inter-functional display cabinet, backward compatible with old virtual mercantile standards, with compliant e-commerce security features, but you still have another issue. How do you make sure that the store isn’t making copies of your display cabinets and using them across all their wholly owned subsidiaries? Or selling them overseas to offset a flat Christmas sales season? Or being stolen by a nefarious shopper and resold on the lucrative display cabinet black market?

This is where it’s all about standards. All about the protocols that set out the expected behaviors and configurations that define and prescribe how all of these magical virtual interactions happen. It’s the subatomic glue that connects all the disparate experiences into coherent, navigable places, and continues to do so after the cowboys and star fighters have all gone home.

Inside out projection

I recently took place in a Dance/Hack at Kinetech Arts, here in San Francisco. Kinetech is a group of technologists and dancers around whom a whole host of people (me included) orbit and take part when we can. The Dance Hack is a yearly event where teams of dancers/technologists/musicians/visual artists, get together to create some expression of motion technology. This year it was also linked to similar events in London and Amsterdam. In addition to special programs and performances, Kinetech has an open studio every Tuesday.

One Tuesday I brought my Kodak PixPro 360 camera, which takes a dome image 360 degrees around and a little over 210 degrees horizon to horizon. It records the anamorphic projection onto a square. Dancer/Choreographer, Megan Meyer, was very interested in what it did. She was interested in how the spherical nature of the image related to depictions found on ancient pottery. So we decided to try and put something together for the Dance/Hack.


Greek pottery

The concept started out grand, of course – epic narrative, historical reference, comments on contemporary society, etc. The time constraints and the resource limitations meant scaling back the vision a bit.

I’ve been thinking about alternate projections since I was a kid and learned that Greenland wasn’t nearly as big as I’d been led to believe, that straws don’t actually bend in glasses of water, and that perspective is sort of skewing a cone into a cylinder. I am intrigued by the visual and narrative possibilities of full-dome projection, but the barrier to entry is very high – too high for the DIY spirit of the Dance/Hack. We needed a simpler, cheaper way of reconstructing the images onto some viewing surface that many people could view.

We started to think small. The idea started from Megan’s concepts of ancient pottery which, though now are priceless museum antiquities, were created to be utilitarian, domestic items. Could we make the experience happen on a domestic scale?

Kodak sp360 web site

The Kodak 360 images remind me of chrome-ball photos taken on set for visual effects filming. The angles of the projection are different, but visually the results are very similar. What if I invert that process? If I could project the image off of a reflective sphere, and onto a larger, translucent ball, the resulting image would rectify itself into a relatively undistorted final result. It worked in my head while we were drinking coffee and drawing on scratch paper – so what could possibly go wrong? (within tolerances)

Here’s the theory:

The Kodak sp360 takes images through a very wide fisheye lens – with an over 200 degree field of view. The idea was to invert the distortion of flattening the dome onto the picture plane by projecting the picture plane back off of a dome21022835408_688a34069d_z_d


Of course this worked great in my head and on paper. And in a perfect world with enough time and resources, it probably would have worked well in practice. But part of the fun of a hackathon is fielding the unexpected and working within the real world constraints of the day.



Here’s the reality:

To begin with we didn’t have a budget to buy a very round and reflective chrome ball like the kind used in VFX production, but Megan did find some very shiny silver christmas ornaments. We didn’t have a source for spheres made of a highly transmissive translucent material, but Ikea has a good deal on round paper lanterns.  Well – okay – we’re building a lamp, and the aesthetic direction is based around a wood and paper lantern. That’s great because the rig to hold the projector, the ornament, and the lantern was going to be built from wooden slats and dowels.

The key behind getting this all to work was establishing the correct alignment of projector beam to christmas ball, and then to the paper lantern. This seemed easy because it should just be a matter of stacking one directly on top of the other and then changing the distances between them to adjust for which section of the image hit where on the sphere. But I hadn’t counted on the built in convenience functionality of the little projector. Unlike the projectors I remember from the previous millennium, modern projectors have all kinds of corrections so that you don’t have to angle the projector or worry about “keystoning” the image.


This meant that all of that proper alignment was in practice shifted wildly off of the “axis” of projection. And, in fact, the image hitting the sphere wouldn’t even be a square. Most of the day was spent with a swiss army knife saw, a cordless drill, and masking tape, trying to shim and bend everything to hold the ornament in the right place. Luckily dowel rods are flexible.

Here’s the result:



It was a little clunky but had its own charm – like a cross between some wabi sabi rusticness and the tree from A Charlie Brown Christmas. But it worked, and definitely points to some interesting possibilities with bigger and better hardware.

What I find most intriguing about this projection is that inverts the notion of looking out and looking in. The camera “sees” out, but the projection allows us to look into that view. I really like this look into an impossible slice of perspective – a view into the infinite.



Freeing Immersive Content Creators from App Trap

One of the biggest hurdles facing anyone wanting to deliver AR/VR content right now is that every different implementation requires a different packaging of content data. Some of this is a result of the “game” and “app” ecosystems that these experiences come from, but there’s also no other alternative.

Content cannot be delivered as a broadcast stream because there is no definition of what that stream is. Without that there is no standard viewing “environment” to leverage. There are some attempts to work on this – YouTube’s 360 video is an interesting way of delivering one component of immersive content, but it’s not an extensible or leverageable technology. It’s essentially only a movie player. A content creator cannot, for instance, embed a 360 video as one of many elements in a deliverable program.

And so content creators also have to be technologists capable of building worlds of mixed elements inside of an app or game metaphor. Each experience is a one-off, individually crafted delivery of heterogenous content. But most of this content is really just reconfigured instances of only a handful of different kinds of data – 2d, 3d, static, animated, geometry, images, navigable, etc. And this repetition could be exploited into not only a consistent data exchange “format”, but also a consistent experience environment. A content provider would construct, not an app or game, but a container of elements and descriptors, deliverable as a “unit” to any compliant experience environment. Like a broadcast network delivered TV shows, bounced off satellites, thrown across the airwaves or down cables to a TV set that decoded and displayed the experience.

But what would that package look like? How can we all agree? What are the NTSC, mpeg, jpeg, obj, wav of VR? Is it a file? Is it a file aggregation container? There are a lot of questions to answer, but the freedom afforded to content creators when they no longer have to worry about he technology of the viewing experience, could bring the freedom that other creators have had for years. Film makers don’t have to worry about the inner mechanical workings of projectors, writers don’t have to worry about how printing presses work, and AMVR content creators should not have to worry about writing apps.

The Late 1940s Black and White TV of Virtual Reality Experiences

Everyone seems to be chasing some pretty lofty production goals in VR right now – fully immersive 360 cinematic visual experiences, with full body tracking and gestural input – and that’s great. It’s like the ultimate mind bending experience. But it’s missing a bigger, more achievable, and more deliverable alternative which is a lot more like black and white TV of the late 40s.

It’s not a sexy as the hard wired, high octane, dedicated immersive pipeline experience of an 8K surround, best seat in the house concert experience, or the subtly expressive and captivating world of an elegantly rendered narrative, but it’s deliverable, right now, and on cardboard or a simple smartphone.

If we let go of designing for the future hardware utopia – no not all of us, and certainly not all of the time – we can make experiences that we can deliver right now. How captivating they are will be based on how well the inherent limitations are embraced and become part of the experiences themselves. It’s like the $9.95 sculpture in design class – what’s the best sculpture you can make for $9.95? Not what’s the best approximation of $9,999 dollar sculpture you could have made if the assignment weren’t so damn frustrating, and not the $0.99 sculpture – you get no points for false economies. But the best that you can do while fully embracing the limitation of $9.99.

What can we do with limited resolution, limited bandwidth, limited tracking, limited capture? Can we make a simple experience that can be immersive, but not stereo? Can a viewer go to a web page, hold up their smart phone and be inside an engaging experience? What are the experiences that lend themselves most to these design constraints? News? Documentary? Sports? Conversations? Simple telepresence? Standup comedy? Variety shows? We are not at the readily available 8k video experience of VR yet; we aren’t even at the readily available Color TV NTSC 1950s experience of VR yet. How do we design compelling experiences for what we do have. There were compelling things on TV when it was black and white, on a tiny round screen, and the image was mostly ghosted, solarized, and smeared. Maybe people were just smarter in the 40s.