Film grain, persistence of vision, sensor tubes, and experienced resolution.

At various times in my career I’ve had to think about image resolution, film grain, how to pull a clean image from a noisy video, and orbiting gamma-ray observatory sensor tubes. The question is: how much digital resolution is needed to represent the analog resolution of motion picture film? Or maybe it’s more of a statement: How to problematize the question of – “how much digital resolution is needed to represent the analog resolution of motion picture film?”, or at least have fun thinking about it.

When I worked at the visualization lab at UCR, in the dark ages, I remember a physicist talking about a project he was working on for an orbiting gamma-ray observatory that was going to map background radiation from outer space. They wanted to do this at a very high resolution. The problem was that the gamma particle sensor tubes they used could only resolve to a couple degrees at a time – basically 1/180th of the sky in a circle. A 180 sample map is not very high resolution. But he had a trick. The orbiting platform would be incredibly stable and predictable, so they could take a bunch of overlapping coarse passes with such accuracy that his software would be able to synthesize a much higher resolution of the sky – to fractions of a degree. This is how things like synthetic aperture radar work.

A little while later a mathematician came into the lab and wanted a still from the X Files credit sequence “The Truth is Out There”, for a presentation slide, but we couldn’t get a clean screen grab from the VHS tape. It was weird – the text and the background were stable, and at playback speed the image seemed clean enough – it’s only when we’d freeze a frame that the text melted into a fuzzy blob of static. So then I thought about it – basically every frame is like a gamma particle collection tube, the TV screen isn’t moving and neither are the text or the background. What if I were to “blend” a number of these frames together? Would a get a much clearer result? The answer was yes. And not just because of an NTSC fields vs frames thing – the more frames I combined the clearer the text became. What that blending method was I’ll leave up to the reader – and if you figure out let me know because I’ve forgotten. The point was that accumulating noisy images over time emphasized their similarities – the text, and deemphasized the differences – the smearing static. I’ve also used a similar technique to accumulate/synthesize a high resolution still from multiple low resolution renders each with slight camera shifts.

Okay so back to film grain and the resolution needed to represent a movie film frame with a digital frame. Here’s the fun part to mull over. A digital image is a regular grid of pixels in rows and columns. A film image is an irregular matrix of physical grains of stuff holding onto dyes of different colors – they aren’t uniform in size and certainly not uniform in location – and this all changes, randomly for every frame. Your brain does a great job of blending all of this together over time, just like blending together the video frames. How much digital resolution is needed to represent movie film? I remember when a 2K image was going to be more than enough. Now we’re pretty sure we need 4K. I wonder what kinds of phantom resolutions happen in our minds from the accumulation of unpredictable tiny grains of color? What kinds of resolution apertures we can synthesise out of all that noise? How do we experience different presentations of visual resolution?

here’s a link with some neat diagrams illustrating synthetic aperture radar:  http://www.radartutorial.eu/20.airborne/ab07.en.html

 

Advertisements

An Old Map and Declarative 3D

This is an early  “work in progress” visualization of an 18th century map drawn by Giambattista Nolli, using @A-Frame declarative HTML mark-up extensions for VR/3D in WebGL – with procedurally generated geometry and baked lighting in Houdini. Lots more to do and learn. Eventually it will be part of an AR promo piece but I couldn’t resist.

aaa_nolli_splash

(better navigation in cardboard, varied building heights, and better global illumination)

 

A Tale of Two Frame Rates

I had the good fortune of attending a local SIGGRAPH chapter talk by Bruna Berford of Penrose Studio, regarding production methodology and how they approached animation in their beautiful and emotionally compelling VR experience “Alumette”. She presented a good view into the difficulties, challenges,and rewards of adapting to working in this new medium. And let me say that to my thinking, they are actually embracing the new technology fully as a narrative medium.

But that’s not what this post is about. This post is about a misconception that people in V/M/AR have around the concept of frame rate. Specifically the holy grail of a 90+ fps redraw rate. This is held up as a metric which must be achieved for a non-nauseating viewing experience and is usually stated as an across the board dictum. Alumette, however, threw a very nice wrench into that, one which points to something that I’ve tried to articulate in the past. There are two different frame rates going on here. And that difference is apparent in Penrose’s use of stop motion frame rates for its animation.

The first “frame rate” is the one is the one that’s usually meant and I think of that as being the perceptual, or maybe even proprioceptural frame rate. This is the frame rate that corresponds to how well the environment is tracking your body’s movements. For instance when you turn your head, or when you walk around in a room scale experience. This is the one that tells your lizard brain whether or not you are being lied to. But a lot of people, including seasoned veterans, stop here, assuming that the matter is settled, but I think there’s a second frame rate at work.

The second is what I would call the exterior frame rate. This is the frame rate of the displayed content. And in Alumette this was definitely not at 90+ fps. In fact it was at a consciously much “slower” and less constant frame rate because it was being animated on hard, held keys with no interpolation. This was to emphasize the poses of the animation. The result was an elegant reference to traditional stop motion animation, with all of the artistic start/stop and a wonderfully surreal sense of time. And the overall experience in VR was not so much watching a stop motion animation, but rather existing in space with one. It was pretty cool.

The “content” was running at what I night guess averaged to ~12 fps, but the display of it, and therefore more importantly my perception of the experience was at the magic 90+ fps. This is an important distinction – especially when it comes to content creation. Would 360 video at a lower playback rate, say 18 fps, give us that old Super8 home movie feel as long as the the video sphere it’s projected onto was moving seamlessly? Could a game engine environment be optimized to hold frames of animation at 30fps allowing temporally redundant data to limit draw calls or GPU memory writes?