A Tale of Two Frame Rates

I had the good fortune of attending a local SIGGRAPH chapter talk by Bruna Berford of Penrose Studio, regarding production methodology and how they approached animation in their beautiful and emotionally compelling VR experience “Alumette”. She presented a good view into the difficulties, challenges,and rewards of adapting to working in this new medium. And let me say that to my thinking, they are actually embracing the new technology fully as a narrative medium.

But that’s not what this post is about. This post is about a misconception that people in V/M/AR have around the concept of frame rate. Specifically the holy grail of a 90+ fps redraw rate. This is held up as a metric which must be achieved for a non-nauseating viewing experience and is usually stated as an across the board dictum. Alumette, however, threw a very nice wrench into that, one which points to something that I’ve tried to articulate in the past. There are two different frame rates going on here. And that difference is apparent in Penrose’s use of stop motion frame rates for its animation.

The first “frame rate” is the one is the one that’s usually meant and I think of that as being the perceptual, or maybe even proprioceptural frame rate. This is the frame rate that corresponds to how well the environment is tracking your body’s movements. For instance when you turn your head, or when you walk around in a room scale experience. This is the one that tells your lizard brain whether or not you are being lied to. But a lot of people, including seasoned veterans, stop here, assuming that the matter is settled, but I think there’s a second frame rate at work.

The second is what I would call the exterior frame rate. This is the frame rate of the displayed content. And in Alumette this was definitely not at 90+ fps. In fact it was at a consciously much “slower” and less constant frame rate because it was being animated on hard, held keys with no interpolation. This was to emphasize the poses of the animation. The result was an elegant reference to traditional stop motion animation, with all of the artistic start/stop and a wonderfully surreal sense of time. And the overall experience in VR was not so much watching a stop motion animation, but rather existing in space with one. It was pretty cool.

The “content” was running at what I night guess averaged to ~12 fps, but the display of it, and therefore more importantly my perception of the experience was at the magic 90+ fps. This is an important distinction – especially when it comes to content creation. Would 360 video at a lower playback rate, say 18 fps, give us that old Super8 home movie feel as long as the the video sphere it’s projected onto was moving seamlessly? Could a game engine environment be optimized to hold frames of animation at 30fps allowing temporally redundant data to limit draw calls or GPU memory writes?

Inside out projection

I recently took place in a Dance/Hack at Kinetech Arts, here in San Francisco. Kinetech is a group of technologists and dancers around whom a whole host of people (me included) orbit and take part when we can. The Dance Hack is a yearly event where teams of dancers/technologists/musicians/visual artists, get together to create some expression of motion technology. This year it was also linked to similar events in London and Amsterdam. In addition to special programs and performances, Kinetech has an open studio every Tuesday.

One Tuesday I brought my Kodak PixPro 360 camera, which takes a dome image 360 degrees around and a little over 210 degrees horizon to horizon. It records the anamorphic projection onto a square. Dancer/Choreographer, Megan Meyer, was very interested in what it did. She was interested in how the spherical nature of the image related to depictions found on ancient pottery. So we decided to try and put something together for the Dance/Hack.

h2_31-11-10

Greek pottery

The concept started out grand, of course – epic narrative, historical reference, comments on contemporary society, etc. The time constraints and the resource limitations meant scaling back the vision a bit.

I’ve been thinking about alternate projections since I was a kid and learned that Greenland wasn’t nearly as big as I’d been led to believe, that straws don’t actually bend in glasses of water, and that perspective is sort of skewing a cone into a cylinder. I am intrigued by the visual and narrative possibilities of full-dome projection, but the barrier to entry is very high – too high for the DIY spirit of the Dance/Hack. We needed a simpler, cheaper way of reconstructing the images onto some viewing surface that many people could view.

We started to think small. The idea started from Megan’s concepts of ancient pottery which, though now are priceless museum antiquities, were created to be utilitarian, domestic items. Could we make the experience happen on a domestic scale?

Kodak sp360 web site

The Kodak 360 images remind me of chrome-ball photos taken on set for visual effects filming. The angles of the projection are different, but visually the results are very similar. What if I invert that process? If I could project the image off of a reflective sphere, and onto a larger, translucent ball, the resulting image would rectify itself into a relatively undistorted final result. It worked in my head while we were drinking coffee and drawing on scratch paper – so what could possibly go wrong? (within tolerances)

Here’s the theory:

The Kodak sp360 takes images through a very wide fisheye lens – with an over 200 degree field of view. The idea was to invert the distortion of flattening the dome onto the picture plane by projecting the picture plane back off of a dome21022835408_688a34069d_z_d

grec_urn_figE

Of course this worked great in my head and on paper. And in a perfect world with enough time and resources, it probably would have worked well in practice. But part of the fun of a hackathon is fielding the unexpected and working within the real world constraints of the day.

grec_urn_figA

 

Here’s the reality:

To begin with we didn’t have a budget to buy a very round and reflective chrome ball like the kind used in VFX production, but Megan did find some very shiny silver christmas ornaments. We didn’t have a source for spheres made of a highly transmissive translucent material, but Ikea has a good deal on round paper lanterns.  Well – okay – we’re building a lamp, and the aesthetic direction is based around a wood and paper lantern. That’s great because the rig to hold the projector, the ornament, and the lantern was going to be built from wooden slats and dowels.

The key behind getting this all to work was establishing the correct alignment of projector beam to christmas ball, and then to the paper lantern. This seemed easy because it should just be a matter of stacking one directly on top of the other and then changing the distances between them to adjust for which section of the image hit where on the sphere. But I hadn’t counted on the built in convenience functionality of the little projector. Unlike the projectors I remember from the previous millennium, modern projectors have all kinds of corrections so that you don’t have to angle the projector or worry about “keystoning” the image.

grec_urn_figD

This meant that all of that proper alignment was in practice shifted wildly off of the “axis” of projection. And, in fact, the image hitting the sphere wouldn’t even be a square. Most of the day was spent with a swiss army knife saw, a cordless drill, and masking tape, trying to shim and bend everything to hold the ornament in the right place. Luckily dowel rods are flexible.

Here’s the result:

 

grec_urn_figC

It was a little clunky but had its own charm – like a cross between some wabi sabi rusticness and the tree from A Charlie Brown Christmas. But it worked, and definitely points to some interesting possibilities with bigger and better hardware.

What I find most intriguing about this projection is that inverts the notion of looking out and looking in. The camera “sees” out, but the projection allows us to look into that view. I really like this look into an impossible slice of perspective – a view into the infinite.

OLYMPUS DIGITAL CAMERA