Jon Baker, Graphics Programming

Cheap Lenticular/Holographic Rendering Using a 4D LUT

  A friend shared a very cool idea about lenticular rendering, based on a discussion about holograms around this video, explaining how they are constructed from coherent light and film-recorded wave interference patterns. By modelling the lenticular surface with SDF geometry, and doing the refraction logic for that material, they were able to simulate the optical behavior of these microlenses - this is super cool, because it actually creates the directionally-varying behavior of a lenticular surface. Lenticular printing is a technology for view-dependent printing, where an array of small lenses is molded into a plastic sheet on top that provides a selective view of the printed surface underneath. I have been doing some looking and depending on the curvature of these lenses and the thickness and IoR of the material, typical films support viewing angles from 30° to 50°. The plastic material does not support hemispherical viewing angles the way I'm doing here. I'm looking into options for what's called "fly-eye" lenticular surfaces or microlens arrays, which model spherical bumps rather than half-cylinder geometry - this enables variation over a 2D surface rather than the 1D options provided by more typical lenticular printing. This is referred to as an "integral image", and it is a form of hologram.

  This piece by David Roberts and some other followup pieces by the same author cover some history related to technological approaches to integral and lenticular imaging. I emailed Forward Optics and heard back from him, and I'm in the process of trying to find a supplier for this fly-eye/microlens material with a large enough pitch that I can print these on a regular printer. For context, I was asking about their MicroLux, which is 141 lenses per inch - my print capability tops out at 600 dots per inch, so I will not be able to sufficiently resolve the background image at that pitch. He referred me to Fresnel Technologies, the supplier he used for the lens array to create the image below on flim. This document is full of extremely interesting material, showing patent drawings for a number of approaches that used gratings and other optical techniques. One very interesting one is this camera that used 4 separate fixed-focus lenses to create stereo images directly onto a strip of film.

  I'm really interested in producing one of these fly-eye prints. Sufficiently sampling the pixel footprint and the angular divisions, you'll be able to resolve a nice image over the desired range of angles. You can see what the image under the lenticular sheet looks like. To implement this in a realtime renderer, I have taken a simplified approach to the rendering, and am not actually modeling the refraction through the specific lenticular geometry. Instead, I'm keeping a 4D LUT, with the first two dimensions in terms of pixel XY, and the remaining two corresponding to a hemispherical mapping over two angles Θ and Φ. It's basically just using Euler angles right now. With the underlying image sufficiently resolved, there should be minimal artifacting when separately linearly filtering over the nearest subpixels representing the angular mapping, and between these filtered angular results from the four nearest pixels. Nearest sampling on the pixels creates visible blockiness, and nearest sampling on angle creates discrete jumps in the image when varying across ranges of viewing angle. Interestingly enough, I've already implemented a compound eye camera for Daedalus, and with some tuning it will be appropriate for this application as well.

  What this represents is an array of discrete apertures at the pixel locations, which each represent the incident radiance at a point on the image. This is sampled over the entire hemisphere, which means that this is a form of hologram. They represent a small virtual aperture which admits the radiance arriving on the other side of the plane. This is actually moving towards the representation used by Alexander Sannikov's radiance cascades method - however, his mapping is not so uniform as this, he exchanges spatial and angular resolution dynamically over a range of depths - close layers have a lot of spatial resolution and a very limited angular resolution, while distant elements trend towards cubemap behavior, approximating the distant radiance with very limited spatial resolution and a lot more angular resolution. I'd like to understand the methodology there better, eventually, because I think it's a better solution.

 In this implementation, I started with a 4096x4096 atlas texture of 512x512 pixels with 8x8 subdivisions. This was not sufficient angular resolution when trying to cover the entire hemisphere, and there were large discrete jumps in the represented image. It may be practical, if you only needed to cover a smaller FoV through each "pixel aperture". To cover the hemisphere, I ended up with good results doing 64x64 pixels with 64x64 angular subdivisions, using linear filtering at the pixel level, and multisampling when initially creating the LUT. I will need to experiment with this angular coverage, because lenticular prints typically have significantly narrower viewing angles and will need to have the resolution tuned for the particular optics in front of the printed image.

 The data footprint is significant, with a 4096x4096 8-bit RGBA texture already taking up ~67mb. You typically hit API limits on texture sizes, about 4x larger than this (typical value of VkPhysicalDeviceProperties.limits.maxImageDimension2D returned from vkGetPhysicalDeviceProperties is 16384). Practically speaking, you only access a very small subset of that from any given view angle, so sampling it is still extremely fast. You can see the raw table of what each virtual aperture sees, in the screenshot above. In order to use this method in a pathtracing context, I think I need to keep higher-precision linear HDR data, and that will apply at least another 2X on top of that, potentially 4X if I want to use 32-bit. I'm excited to get another pathtracer going now that I've got a working Vulkan codebase, and I think there will be a lot of interesting applications for this tek there. For example:

• High Sample Count - you can arbitrarily sample over pixel footprint and angle, and run that process for as many samples as you want. The most I've tried here is 16, and each just jitters origin and direction and runs the ray-sphere test. In a pathtracer, you have the same goal, to resolve the incident radiance as well as possible. Extremely high sample counts are possible offline, and can be sampled for a low, constant cost in realtime, just as well as the sphere we see here. Constraining your range of angles and optimizing your use of the subpixel resolution is going to be important, especially when preparing an image to place behind a microlens array with a given FoV. I want to look at the octahedral mappings that are commonly used for normal vector compression, in order to uniformly cover the hemisphere (euler angles in the current mapping do not do this so uniformly).
• Usage as a Hologram - at this point I have a representation that can contain the entire hemisphere of incident radiance, with a certain precision. This is in fact a hologram, as it is encoding all the directionally and spatially varying information across this square aperture. This is quite beautiful, and for the result, I'm surprised the data footprint is so small. In the same way that we can sample this plane with the UV and view direction in the video above... we can actually use this same angular and spatial relationship to sample a planar emissive surface in a pathtracing context. When a ray hits, we can sample this radiance value for x,y,Θ,Φ (need to store HDR values, for this application). Effectively, what we have is a "portal", it's like we can record all the light that comes through a window. The viewer can move around, and by sampling this image, achieve the same perception that they would if they actually looked through the article.
• DoF - you could potentially bake DoF into this image, if you render the initial LUT with a DoF effect... I'm quite curious to see what this looks like when sampled... I'm not sure if this would map to something physically meaningful or not.
• Alpha Mask - what if you stored an alpha value, when creating the initial LUT? You could compute this from some kind of depth threshold, or material ID, depending on what the renderer producing it looks like.

  And something else that was mentioned around this is potentially trying to compress the data representation. This seems like a good opportunity, since you have a lot of similarity between the irradiance through these apertures... the low frequency spatial correlations correspond to distant objects, and high spatial frequencies, those which would vary pixel-to-pixel when viewed from the same direction, generally correlate to nearby objects.