Integral Photography Research Articles

3D light-field display with an increased viewing angle and optimized viewpoint distribution based on a ladder compound lenticular lens unit.

Three-dimensional (3D) light-field displays (LFDs) suffer from a narrow viewing angle, limited depth range, and low spatial information capacity, which limit their diversified application. Because the number of pixels used to construct 3D spatial information is limited, increasing the viewing angle reduces the viewpoint density, which degrades the 3D performance. A solution based on a holographic functional screen (HFS) and a ladder-compound lenticular lens unit (LC-LLU) is proposed to increase the viewing angle while optimizing the viewpoint utilization. The LC-LLU and HFS are used to create 160 non-uniformly distributed viewpoints with low crosstalk, which increases the viewpoint density in the middle viewing zone and provides clear monocular depth cues. The corresponding coding method is presented as well. The optimized compound lenticular lens array can balance between suppressing aberration and improving displayed quality. The simulations and experiments show that the proposed 3D LFD can present natural 3D images with the right perception and occlusion relationship within a 65° viewing angle.

How to Ensure a Good Depth Perception on a Light Field Display

<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">In this article, we define the technical requirements for a light field display to provide effective depth cues such as binocular disparity, motion parallax, and accommodation at a pixel resolution expected by the eye. These requirements are evaluated with respect to existing technologies (e.g.,integral imaging displays) or expected future ones (e.g., micro-light-emitting diode (LED) displays). Simulation of light field displays are proposed for different display sizes (smartphone, desktop, or TV), resolution per view, and pixel pitch. These simulations convey the main impact of the pixel pitch to reach the light field display goal in terms of depth cue requirements. The last part of the paper focusses on data generation issues: how to generate the necessary amount of views at the display side and what the adapted data formats are for such a purpose. The paper is associated with supplementary videos that will illustrate the view generation topic using a dedicated data format that is optimized for current graphics processing unit (GPU) processors. Video capture of early-stage light field displays will also be provided</i> .

Examining the utility of pinhole-type screens for lightfield display.

The use of microlens arrays for lightfield display has the drawback of providing images with strong chromatic aliasing. To overcome this problem, pinhole-type lightfield monitors are proposed. This paper is devoted to evaluating the capability for such lightfield monitors to offer the user a convincing 3D experience with images with enough brightness and continuous aspect. Thus, we have designed a psychophysical test specifically adapted for lightfield monitors, which allowed us to confirm the usability of pinhole-type monitors.

Foveated light-field display and real-time rendering for virtual reality.

Glasses-free light field displays have significantly progressed due to advances in high-resolution microdisplays and high-end graphics processing units (GPUs). However, for near-eye light-field displays requiring portability, the fundamental trade-off regarding achieved spatial resolution remains: retinal blur quality must be degraded; otherwise, computational consumption increases. This has prevented synthesizing the high-quality light field from being fast. By integrating off-the-shelf gaze tracking modules into near-eye light-field displays, we present wearable virtual reality prototypes supporting human visual system-oriented focus cues. An optimized, foveated light field is delivered to each eye subject to the gaze point, providing more natural visual experiences than state-of-the-art solutions. Importantly, the factorization runtime can be immensely reduced, since the image resolution is only high within the gaze cone. In addition, we demonstrate significant improvements in computation and retinal blur quality over counterpart near-eye displays.

Development of a scalable tabletop display using projection-based light field technology

We present a light-field display optical system design, state-of-the-art hardware technology, and computational software that are readily scalable (due to their modular structure) and provide naturally immersive and volumetric display systems. We explain the integration of 72 microprojectors into the tabletop display system. In addition, we used 6 workstations for all images and 18 high-performance graphics processing units. Sophisticated image generation through author tooling and the pixel re-arrangement algorithm in the Unity engine enabled us to produce complete three-dimensional images from the radiant rays of two-dimensional pixels using diffuser screens and microlens arrays.

Augmented Reality Vector Light Field Display with Large Viewing Distance Based on Pixelated Multilevel Blazed Gratings

Glasses-free augmented reality (AR) 3D display has attracted great interest in its ability to merge virtual 3D objects with real scenes naturally, without the aid of any wearable devices. Here we propose an AR vector light field display based on a view combiner and an off-the-shelf purchased projector. The view combiner is sparsely covered with pixelated multilevel blazed gratings (MBG) for the projection of perspective virtual images. Multi-order diffraction of the MBG is designed to increase the viewing distance and vertical viewing angle. In a 20-inch prototype, multiple sets of 16 horizontal views form a smooth parallax. The viewing distance of the 3D scene is larger than 5 m. The vertical viewing angle is 15.6°. The light efficiencies of all views are larger than 53%. We demonstrate that the displayed virtual 3D scene retains natural motion parallax and high brightness while having a consistent occlusion effect with natural objects. This research can be extended to applications in areas such as human–computer interaction, entertainment, education, and medical care.

35.1: Tabletop three‐dimensional floating autostereoscopic display system with 360‐degree continuous visualization

Background Tabletop 3D display is one of the most challenging and exciting 3D displays. Among different 3D display technologies, the integral photography (IP) based tabletop 3D display is a promising 3D display with full‐color and real‐time rendering. Still, few can offer the in‐situ interaction in multi‐user scenes.Methods In this study, we proposed a novel tabletop 3D floating autostereoscopic display system with 360‐degree continuous visualization according to the optical path and rotation design. As the brightness of the floating image will be changed due to the different angles between light source and AIP, we analyzed the AIP‐based light reflectivity and derived the relation formulas of the reflectivity. We also proposed a depth‐based multiple viewpoint rendering algorithm for real‐time face and gesture tracking to achieve the vertical parallax supplement and model control.Results We verified the continuous visualization and light reflectivity analysis results via simulation experiments. As a result, the brightest setting angle (the angle corresponding to the highest light reflectivity) of AIP was influenced by the AIP reflection unit parameters. The proposed tabletop display realized the aerial display of the 3D model and could provide a continuous 360‐degree observation point of view between different polyhedron faces.Conclusions We proposed a tabletop autostereoscopic 3D display system and analyzed the AIP‐based light reflectivity. The light reflectivity formulas were worthy of reference for designing the AIP 3D or 2D displays. The proposed system could meet the observation of multiple users and offer in‐air interactions.

39.1: Invited Paper: Super‐resolution light field display based on learned elemental image and designed compound lens

39.1: <i>Invited Paper:</i> Super‐resolution light field display based on learned elemental image and designed compound lens

49.4: Long Viewing Distance and Large Depth of Field Augmented Reality (AR) 3D Display Based on MEMS Laser Projection Array

The light collimation performance of each imaging unit in integral photography determines the viewing distance and depth of the field of the stereoscopic scene. MEMS laser projector has a good light collimation effect. Our work is to implement MEMS array extension through the subtle design of light paths. The single‐line MEMS scanning array and the multi‐mirror array are finely coordinated. Projection array realizes stereo display by transforming spatial pixels into angle information. The 3D light field is rendered using the space‐time multiplexing technology of MEMS laser scan projection. On this basis, we try to replace the multi‐mirror array with a semi‐transmission and semi‐reflection mirror array to achieve augmented reality. The mirror array can balance the number of views and pixels. The depth information of the real object is obtained using the structured light scanning technology of the depth camera. The calibration, registration, accurate fusion, and in‐situ superposition of the 3D reconstruction scene and the natural scene are realized.

Real-time pre-rectification of aberrations for 3D light-field display based on a constructed virtual lens and ray path tracing

Aberrations caused by optical lens refraction in a light-field display (LFD) are minimized based on real-time pre-rectification. A virtual optical lens with the high precision surface is developed the constructive solid geometry under minimum primitives to rectify the sampling rays, and the virtual lens exhibits the same response to different incident ray wavelengths as the real lens in the LFD optical domain. In addition, assembly misalignment is taken into consideration during the partitioning of the elemental image array so that all the elemental images and the covering lenses can be precisely aligned. After rectifying the sampling rays with the constructed virtual lens, the rays are launched in parallel in the GPU, and path tracing is conducted to shade the element image array. The proposed approach ensures the consistency of the integral photography and LFD processes, which minimize the distortion and increasing the depth of field for three-dimensional (3D) images. Experimental results verify the feasibility of our proposed method. The structural similarity index is improved by 0.1, and 3D images at different depths are correctly expressed. Only three primitives are introduced to the system as a whole, meaning that the extra computation has little impact on the rendering efficiency. Consequently, the real-time generation of 3D light-field images is possible.

Reduction of pixel deviation effect in three-dimensional light-field display based on the fitting algorithm with dense-viewpoints

In three-dimensional (3D) light-field displays, the quality of a reconstructed image degrades with increasing depth of field due to the pixel deviation effect. A fitting algorithm of dense-viewpoints is proposed to reduce the pixel deviation with high image quality. To analyze the pixel deviation effect, the human vision-based perspectives perceived from 3D light-field display system with the holographic functional screen are mathematically modeled. The optimized synthetic image is coded according to the mapping relationship from the display model and the optimal Gaussian distribution parameters achieved with the maximum likelihood estimation method. The effectivenesses of the proposed model and the optimization method are verified through simulations as well as optical experiments. A human vision-based high resolution 3D light-field display with 45-degree viewing angle and the large depth of field is achieved.

Performance improvement for compressive light field display based on the depth distribution feature.

Compressive light field (CLF) display using multi-layer spatial light modulators (SLMs) is a promising technique for three-dimensional (3D) display. However, conventional CLF display usually uses the reference plane with fixed depth, which does not consider the relationship between the depth distribution of the object and the image quality. To improve the quality of the reconstructed image, we further analyze the relationship between them in the paper. The theoretical analysis reveals that the object with a closer distance to the physical layer has a better reconstruction quality when the SLM layers have the same pixel density. To minimize the deviation between the reconstructed light field and the original light field, we propose a method based on the depth distribution feature to automatically guide the light field optimization without increasing the layered number or the refresh rate. When applied to a different scene, it could detect the dense region of depth information and map them as close to the physical layers as possible by offsetting the depth of the reference plane. Simulation and optical experiments with the CLF display are demonstrated to verify the proposed method. We implement a CLF display that consists of four-layer stacked display panels and the distance between two adjacent layers is 5cm. When the proposed method is applied, the peak signal-to-noise ratio (PSNR) is improved by 2.4dB in simulations and 1.8dB in experiments.

Comprehensive performance analysis of objective quality metrics for digital holography

Objective quality assessment of digital holograms has proven to be a challenging task. While prediction of perceptual quality of the recorded 3D content from the holographic wavefield is an open problem; perceptual quality assessment from content after rendering, requires a time-consuming rendering step and a multitude of possible viewports. In this research, we use 96 Fourier holograms of the recently released HoloDB database to evaluate the performance of well-known and state-of-the-art image quality metrics on digital holograms. We compare the reference holograms with their distorted versions: (i) before rendering on the real and imaginary parts of the quantized complex-wavefield, (ii) after converting Fourier to Fresnel holograms, (iii) after rendering, on the quantized amplitude of the reconstructed data, and (iv) after subsequently removing speckle noise using a Wiener filter. For every experimental track, the quality metric predictions are compared to the Mean Opinion Scores (MOS) gathered on a 2D screen, light field display and a holographic display. Additionally, a statistical analysis of the results and a discussion on the performance of the metrics are presented. The tests demonstrate that while for each test track a few quality metrics present a highly correlated performance compared to the multiple sets of available MOS, none of them demonstrates a consistently high-performance across all four test-tracks.

Render‐based factorization for additive light field display

AbstractAugmented Reality (AR) and Virtual Reality (VR) applications enable viewers to experience 3D graphics immersively. However, current hardware either fails to provide multiple viewpoints like projectors or cause conflicts between vergence and accommodation like consumable headsets. Prior researchers have designed multi‐view displays to solve the problem by enabling view‐dependent images and focus cues. Nevertheless, it requires minutes to calculate one frame, which is critical for real‐time AR/VR applications. In this paper, we propose a new render‐based factorization for additive light field display and further improve the performance by optimizing the initialization of layers. We next compare our work with the state of the art and the results show that our solution has competitive results while the calculation takes less than 20 ms for each frame.

Depth reconstruction for 3D light-field display based on axially distributed light field

Continuous depth maps are reconstructed based on depth estimation from light-field data of axially distributed image sensing (ADS). In the proposed method, the light field of ADS is introduced, and the light-field trajectory function is presented, which formulates the relationship between image point positions in different axially captured images and the depth of object point. The depth value of an object is achieved by searching the light-field trajectory through statistical regression, which leads to an accurate depth estimation by making use of the structure information among the densely sampled views in light-field data. Moreover, as regions with smooth texture and occlusions do not contain reliable depth information, an energy minimization-based depth map optimization is carried out from initial estimations to obtain continuous and robust depth maps. Experimental results demonstrate that the reconstructed depth maps are accurate, continuous, and able to preserve more details. Moreover, the synthesized light-field images calculated by reconstructed depth map show good effect in 3D light-field display, and the validity of the proposed method is verified.

Light field illumination: Problem-specific lighting adjustment

Abstract Choosing a proper lighting approach is a crucial task in designing visual inspection systems. It becomes especially challenging for complex-shaped, transparent objects, which change the directional distribution of incoming light in various ways. We overcome this challenge by constructing a light field display and deploy it as a highly tunable lighting device. Thereby, an object-specific light field can be generated, which highlights the features of the object under test with maximum contrast. We explain the calibration procedure, the rendering pipeline and present examples of customized illuminations.

48‐3: Light Field Display Using Virtual Imaging Mode

This study proposes a light field display (LFD) using a lens array and an aperture array. The viewing zone angle is larger for the proposed LFD than the LFDs using a lens array. The image brightness is higher for the proposed LFD than the LFDs using an aperture array.

48‐5: Eye‐sensing Light Field Display for Spatial Reality Reproduction

We have developed the eye‐sensing light field display (ELFD), which lets the user experience a virtual space as if it were real. This is made possible by real‐time light field rendering with its optical design based on eye‐sensing and by taking into account human perceptual characteristics.

Depth of field analysis for a three-dimensional light-field display based on a lens array and a holographic function screen

Compared with traditional integral imaging systems, light-field displays (LFDs) based on a holographic functional screen (HFS) shows significant advantages in terms of the number of viewpoints, large viewing angle, and high viewing resolution. LFDs usually suffer from a limited depth range, and the reason majorly originates from the pixel deviation effect in the reconstruction process. Here, an analysis method for the depth of field (DOF) based on the pixel deviation effect observed on the HFS plane is presented. Simulation experiments are carried out to show that the percentage of pixel deviation is a more accurate measure to calculate the DOF. The feasibility of this method is verified in optical experiments.

Wide viewing angle with a downsized system in projection-type integral photography by using curved mirrors.

To obtain a large viewing angle in conventional projection-type integral photography frameworks, multiple projectors need to be arranged at a particular angle to a lens array. Hence, the systems require a large space. This paper proposes a system that achieves a large viewing angle in a space-saving manner by using curved mirrors that face each other. To this end, a projector is placed directly behind a lens array, and curved mirrors are installed to surround the rays from the projector. The incident angle toward the lens array increases after there are multiple reflections between the mirrors, which increases the viewing angle. In addition, it is not necessary to install the projector at an angle to the lens array, which results in a space-saving system. With the proposed method, a viewing angle of ±60 deg can be achieved.