Freeing Immersive Content Creators from App Trap

One of the biggest hurdles facing anyone wanting to deliver AR/VR content right now is that every different implementation requires a different packaging of content data. Some of this is a result of the “game” and “app” ecosystems that these experiences come from, but there’s also no other alternative.

Content cannot be delivered as a broadcast stream because there is no definition of what that stream is. Without that there is no standard viewing “environment” to leverage. There are some attempts to work on this – YouTube’s 360 video is an interesting way of delivering one component of immersive content, but it’s not an extensible or leverageable technology. It’s essentially only a movie player. A content creator cannot, for instance, embed a 360 video as one of many elements in a deliverable program.

And so content creators also have to be technologists capable of building worlds of mixed elements inside of an app or game metaphor. Each experience is a one-off, individually crafted delivery of heterogenous content. But most of this content is really just reconfigured instances of only a handful of different kinds of data – 2d, 3d, static, animated, geometry, images, navigable, etc. And this repetition could be exploited into not only a consistent data exchange “format”, but also a consistent experience environment. A content provider would construct, not an app or game, but a container of elements and descriptors, deliverable as a “unit” to any compliant experience environment. Like a broadcast network delivered TV shows, bounced off satellites, thrown across the airwaves or down cables to a TV set that decoded and displayed the experience.

But what would that package look like? How can we all agree? What are the NTSC, mpeg, jpeg, obj, wav of VR? Is it a file? Is it a file aggregation container? There are a lot of questions to answer, but the freedom afforded to content creators when they no longer have to worry about he technology of the viewing experience, could bring the freedom that other creators have had for years. Film makers don’t have to worry about the inner mechanical workings of projectors, writers don’t have to worry about how printing presses work, and AMVR content creators should not have to worry about writing apps.

The Late 1940s Black and White TV of Virtual Reality Experiences

Everyone seems to be chasing some pretty lofty production goals in VR right now – fully immersive 360 cinematic visual experiences, with full body tracking and gestural input – and that’s great. It’s like the ultimate mind bending experience. But it’s missing a bigger, more achievable, and more deliverable alternative which is a lot more like black and white TV of the late 40s.

It’s not a sexy as the hard wired, high octane, dedicated immersive pipeline experience of an 8K surround, best seat in the house concert experience, or the subtly expressive and captivating world of an elegantly rendered narrative, but it’s deliverable, right now, and on cardboard or a simple smartphone.

If we let go of designing for the future hardware utopia – no not all of us, and certainly not all of the time – we can make experiences that we can deliver right now. How captivating they are will be based on how well the inherent limitations are embraced and become part of the experiences themselves. It’s like the $9.95 sculpture in design class – what’s the best sculpture you can make for $9.95? Not what’s the best approximation of $9,999 dollar sculpture you could have made if the assignment weren’t so damn frustrating, and not the $0.99 sculpture – you get no points for false economies. But the best that you can do while fully embracing the limitation of $9.99.

What can we do with limited resolution, limited bandwidth, limited tracking, limited capture? Can we make a simple experience that can be immersive, but not stereo? Can a viewer go to a web page, hold up their smart phone and be inside an engaging experience? What are the experiences that lend themselves most to these design constraints? News? Documentary? Sports? Conversations? Simple telepresence? Standup comedy? Variety shows? We are not at the readily available 8k video experience of VR yet; we aren’t even at the readily available Color TV NTSC 1950s experience of VR yet. How do we design compelling experiences for what we do have. There were compelling things on TV when it was black and white, on a tiny round screen, and the image was mostly ghosted, solarized, and smeared. Maybe people were just smarter in the 40s.

Tensegrity and Clothing Simulation

In the 1960s Buckminster Fuller coined the word tensegrity as a combination of tension and integrity, to describe a structure which holds its form through the balance of tension between its parts. It’s a great metaphor when thinking about how energy is distributed in a visual effects simulation at rest and I’ll misuse it as a shorthand for just that.

Let’s take clothing dynamics for example. Look at the clothes you’re wearing. Every fold and wrinkle is an expression or the outcome, of a complex set of interconnected forces – friction, tensile strength, elasticity, etc. The shape exists as it does, solely because of the physical forces of the pieces of fabric, how they’re attached to each other, and the mutual exertion between the cloth and its environment. This is its tensegrity, and the folds of a shirt are a system of balanced tensions, momentarily stabilized.

So what use is this to simulating clothing in animation? Well, because it explains why it’s such a pain in the ass. The cloth’s tensegrity is essentially a lot of forces to keep in check with one another. Let’s look at it backwards.

When a fold is modeled into a shirt for instance, to get an approvable geometric model, and that model is then used as the basis of a simulation – what’s modeled is not actually a fold, but a complex interconnected web of physical tensions and exertions, held together in a sate by, and according to, its tensegrity. When the forces to that matrix of physical interplays change, the system must rebalance, and since the original balance was not based on anything resembling the physics of cloth, it’s efforts to rebalance are not very cloth-like.

Traditional clothes start out as really weird flat shapes. What materials these shapes are made of and how these shapes are attached to one another establish their tensegrity. That results in specific shapes, folds, draping and motion in response to environmental forces – shape is motion and motion is shape. Most CG modeled clothing could never be “unstitched” to lie flat – it would have odd warping, buckling and distortions. The simulated forces act on those structural malformations as input, and the simulation math tries to make sense of it all, as if the warpings were intentional distributions of mass in space.

The visual results are weird, “bubbly”, oozing, and overreactive motions that fold unexpectedly, and keep crawling after the character stops. Those are the simulation engine’s efforts to reestablish balance in the energy of the clothing mesh. It’s just the simulation version of a computer never does anything you don’t tell it to do.

for an actual, more scholarly explanation of tensegrity start here (please)

an interesting example of a 3D printed dress that uses modeled forms rather that flat patterns (and moves more like a sim)

take a peek at pattern making

#simulation #clothsim #cfx #tensegrity #moviephysics #patterndrafting #3dprinting

Deep Learning – What’s Old is New Again

Went to the Hive Data presentation on deep learning at Nvidia the other night. A very interesting take on “what was old is new again”. Old theories and methods that used to be too slow to run, or at least at any useable scale, are running on faster machines, more RAM, and GPUs and are proving to be very useful. It’s interesting that raw compute power is enabling old ideas. So many of the realistically practical approaches we’ve taken as “best practice” can be seen as elaborate workarounds while waiting for hardware to catch up with simple, brute force approaches. In AI it’s deep learning neural networks and in computer graphics it’s monte carlo ray tracing.

Richard Socher from Metamind presented a live demo of their AI For Everyone, drag and drop neural networking web interface. What’s amazing is that it worked – a live demo that worked – this gives me faith that their technology is sound. And I now have some ideas for making some AI Art using cross referencing. I want to make a web site that takes random twitter comments, analyzes them for mood and content, and then matches them up to a CC flickr photo which has a similar classification profile – displaying the twitter comment as a caption to the photo. I’ll probably never have the time though :/

@metamind @hivedata #ai #deeplearning

Standard Delivery of AR/VR Packages

It’s easy to imagine the content and potential of future alternative and trans-media experiences – what they might look like, might sound like, might feel like, smell like? What’s more difficult is to imagine delivery channels of sufficient scale to get those experiences to an audience. At GDC this year there were a lot of companies with amazing demos of amazing technology that played amazingly. At least when all the proprietary hardware could connect to the proprietary “store” to play proprietary file formats. In an age of Open Source and Open Standards, the achilles heel of VR might just be an attachment to closed platforms and formats.

“It has to be as easy to use as a toaster.” , or a TV, but probably easier than a microwave oven. It reminds me a lot of the mid 90s “multi-media” revolution, which was a lot of promise plagued by driver problems, hardware incompatibilities, and an explosion of file formats.

As easy as TV – but why is TV easy? It’s because TV is a standard – NTSC, PAL, SECAM and now digitally with MPG. More than once I saw demos false-start with an error saying that the hardware couldn’t connect to the *brand-name* store. Not only is this the monetization tail wagging the physical playback dog, it also points out that the VR world is not only siloed, but apparently even garrisoned. Currently I can author content in Unity, or Unreal, or something else that turns into something that can only play back in a corresponding engine through a dedicated app, and while this is great for a game approach, it’s a pain in the ass if what I want is a TV/toaster level barrier to entrance.

How do we get there from here? We need the NTSC of VR. And not a megalithic beast that tries to be everything to everybody, but something that standardizes basic delivery of content. It’s the old 85% 10% 5% approach – figure out how to automate 85%, facilitate 10% and make 5% possible – if painful. Standardized delivery is that 85%. Just as broadcast TV never replaced movies, a VR standard won’t obviate the need for cutting edge experiences to be delivered on bleeding edge hardware. But what VR/AR and other immersive media experiences need is wide spread adoption, and the only way to get that is with a smart way of encapsulating most of the functionality needed in a standardized wrapper that can leverage existing online delivery channels. Youtube and Vimeo can both playback video files, or those files can be served up from another place. The files can be hosted from one place and served up through an embedded link. It’s a delivery ecosystem that the web knows, and knows how to monetize already; it’s just waiting for standard packages to deliver.

Borrowing a Historical Lens

The comment in this image that photo is “as well composed as the best renaissance art”, is interesting. Firstly – it’s as well composed as the best romantic neoclassical art, some 400 years after the Italian Renaissance (there’s more time between Da Vinci and Gericault, than Gericault and Picasso). But our collective apathy toward our own cultural history is not the most interesting part. The power in this image is not so much in its composition directly, but how much that composition makes the content of the image play in the same intellectual/emotional space as the old paintings it borrows them from. We read the brawl with all the heroic grandeur of neoclassical history painting because we are seeing it through the lens of neoclassical history painting. And so the struggle in the Ukrainian Parliament, becomes “The Struggle in the Ukrainian Parliament”, and the picture becomes a grand satire within a cognitive dissonance that exists between a schoolyard fight with old men in suits, and the “really important stuff” we see in “real” Art museums.

(need to find image attribution and update)

‪#‎artsnob‬ ‪#‎usingmydegree‬ ‪#‎agingpoststructuralism‬

#Siggraph2013 #effectsOmelet

The “Effects Omelet” presentation at SIGGRAPH is always a great source for inspired creativity on the ground by VFX artists and TDs.  David Lipton, Head of Effects at Dreamworks Animation, gave a particularly interesting talk about he achieved the Jack Frost frost effect in DWA’s “Rise of the Guardians”.

Interesting use of old school approaches to get more controllable artistic results in the expressive effect of Jack Frost’s frost in DWA’s Rise of the Guardians.  The frost needed to be a highly stylized, very art directable and expressive effect, where Jack’s staff would freeze objects by propagating elegant, icy arabesques that skated across surfaces, covering them in stylized frost patterns.

Lipton said that they were helped immensely by the copious notes, reference images and concept art prepared by the Art Department.  This gave him and his team a very distinct target to aim for, and helped to narrow the problem at hand.

The first approaches were simulation based, but proved to be hard to control, espescially because the effect itself needed to be an expressive actor in the film, with its performance often leading directing the eye through key story moments.  The winning approach was to look far back into the history of computer graphics to an old standby of cellular automata.  These are systems in which cells of a grid, like pieces on checker board, follow simple rules that determine how each cell becomes filled by its neighbors.  In this case the rules would determine how ice would grow from square to square as time progresses. The speed at which the squares were filled defined paths, like roadways, along which the delicate and stylized crystal patterns would be constructed.  Because the automata exist in a grid, the rules could be “painted” in like pixels in a digital photo providing a high degree of control.  The end result was a controllable, yet very organic looking crystal propagation that added a sense of magic and expressiveness to the scenes.