Conventional Integral Imaging Research Articles

Viewing-Angle-Enhanced and Dual-View Compatible Integral Imaging 3D Display Based on a Dual Pinhole Array.

Conventional integral imaging (InIm) three-dimensional (3D) display has the defect of a small viewing angle and usually presents a single 3D image. In this paper, we propose a viewing-angle-enhanced and dual-view compatible InIm 3D display system. The crosstalk pixel areas within the conventional elemental images (EIs) that result in image crosstalk were effectively utilized either for viewing angle enhancement or for dual-view 3D display. In the viewing-angle-enhanced 3D display mode, a composite elemental image (CEI) that consisted of a normal EI and two view-enhanced EIs was imaged by a dual pinhole array and formed an extended 3D viewing area. A precisely designed mask array was introduced to block the overlapped rays between adjacent viewing areas to eliminate image crosstalk. While in the dual-view 3D display mode, a CEI was composed of image information of two different 3D scenes. With the help of the dual pinhole array and mask array, two different 3D images were reconstructed for the left and right perspectives. Experiments demonstrated that both the left and right sides were increased by 6 degrees from the conventional 3D viewing angle, and also, a dual-view 3D display effect that retains the same viewing angle as the conventional system was achieved. The proposed system has a compact structure and can be freely switched between two display modes.

Orthoscopic elemental image synthesis for 3D light field display using lens design software and real-world captured neural radiance field.

The elemental images (EIs) generation of complex real-world scenes can be challenging for conventional integral imaging (InIm) capture techniques since the pseudoscopic effect, characterized by a depth inversion of the reconstructed 3D scene, occurs in this process. To address this problem, we present in this paper a new approach using a custom neural radiance field (NeRF) model to form real and/or virtual 3D image reconstruction from a complex real-world scene while avoiding distortion and depth inversion. One of the advantages of using a NeRF is that the 3D information of a complex scene (including transparency and reflection) is not stored by meshes or voxel grid but by a neural network that can be queried to extract desired data. The Nerfstudio API was used to generate a custom NeRF-related model while avoiding the need for a bulky acquisition system. A general workflow that includes the use of ray-tracing-based lens design software is proposed to facilitate the different processing steps involved in managing NeRF data. Through this workflow, we introduced a new mapping method for extracting desired data from the custom-trained NeRF-related model, enabling the generation of undistorted orthoscopic EIs. An experimental 3D reconstruction was conducted using an InIm-based 3D light field display (LFD) prototype to validate the effectiveness of the proposed method. A qualitative comparison with the actual real-world scene showed that the 3D reconstructed scene is accurately rendered. The proposed work can be used to manage and render undistorted orthoscopic 3D images from custom-trained NeRF-related models for various InIm applications.

Three-Dimensional (3D) Visualization under Extremely Low Light Conditions Using Kalman Filter.

In recent years, research on three-dimensional (3D) reconstruction under low illumination environment has been reported. Photon-counting integral imaging is one of the techniques for visualizing 3D images under low light conditions. However, conventional photon-counting integral imaging has the problem that results are random because Poisson random numbers are temporally and spatially independent. Therefore, in this paper, we apply a technique called Kalman filter to photon-counting integral imaging, which corrects data groups with errors, to improve the visual quality of results. The purpose of this paper is to reduce randomness and improve the accuracy of visualization for results by incorporating the Kalman filter into 3D reconstruction images under extremely low light conditions. Since the proposed method has better structure similarity (SSIM), peak signal-to-noise ratio (PSNR) and cross-correlation values than the conventional method, it can be said that the visualization of low illuminated images can be accurate. In addition, the proposed method is expected to accelerate the development of autonomous driving technology and security camera technology.

44‐2: Viewing Window Extension in Integral Imaging Based Head‐mounted Light Field Displays using Time‐Multiplexed Method

Conventional integral imaging based head‐mounted light field display systems suffer from a small viewing window to minimize crosstalk artifacts and achieve a balanced optical performance. In this paper, we present a method for expanding the eye‐box size through a time‐multiplexed scheme by using a liquid crystal aperture array.

A Deep Learning Framework to Remove the Off-Focused Voxels from the 3D Photons Starved Depth Images

Photons Counted Integral Imaging (PCII) reconstructs 3D scenes with both focused and off-focused voxels. The off-focused portions do not contain or convey any visually valuable information and are therefore redundant. In this work, for the first time, we developed a six-ensembled Deep Neural Network (DNN) to identify and remove the off-focused voxels from both the conventional computational integral imaging and PCII techniques. As a preprocessing step, we used the standard Otsu thresholding technique to remove the obvious and unwanted background. We then used the preprocessed data to train the proposed six ensembled DNNs. The results demonstrate that the proposed methodology can efficiently discard the off-focused points and reconstruct a focused-only 3D scene with an accuracy of 98.57%.

Wide-Viewing-Angle Integral Imaging System with Full-Effective-Pixels Elemental Image Array.

There exists a defect of the narrow viewing angle in the conventional integral imaging system. One reason for this is that only partial pixels of each elemental image contribute to the viewing angle and the others cause image flips. In this paper, a wide-viewing-angle integral imaging system with a full-effective-pixels elemental image array (FEP-EIA) was proposed. The correspondence between viewpoints and pixel coordinates within the elemental image array was built up, and effective pixel blocks and pixels leading to flipping images were deduced. Then, a pixel replacement method was proposed to generate the FEP-EIAs, which adapt to different viewing distances. As a result, the viewing angle of the proposed integral imaging system was effectively extended through the replacement of the pixels, which caused the image flips. Experiment results demonstrated that wide viewing angles are available for the proposed integral imaging system regardless of the viewing distances.

Integral imaging 3D display with enhanced viewing angle using two tilted folding mirrors

We propose a viewing-angle-enhanced integral imaging (InI) 3D display using two tilted folding mirrors. The tilted folding mirror generates a corresponding additional virtual tilted micro lens array (MLA) adjacent to the real MLA, which is similar to a curved MLA structure. The two virtual tilted MLAs emit tilted light beams with larger tilt angles than conventional InI, which results in an enhanced viewing angle. A modified elemental image synthesis algorithm is proposed to match the proposed scheme. Visualization simulation and optical experiment are performed to verify the proposed method.

High-Aperture-Ratio Dual-View Integral Imaging Display.

Low aperture ratio is a problem in the conventional dual-view integral imaging (DVII) display using a point light source array. A high-aperture-ratio DVII display using a gradient width point light source array is reported in this work. The elemental Images 1 and 2, which are alternatively aligned on a liquid crystal panel, are illuminated by the light rays emitted from an assigned point light source. The optical path is optimized by optimizing the widths of the point light sources. The aperture ratio of the proposed DVII display was demonstrated as 1.88 times the conventional DVII display. Experiments showed that the vertical viewing range is related to the vertical width of the first row point light source, whereas the aperture ratio is related to the vertical widths of all point light sources. By optimizing the widths of the point light sources, the aperture ratio is enhanced without loss of viewing range.

3D photon counting integral imaging by using multi-level decomposition.

In this paper, we propose three-dimensional (3D) photon counting integral imaging by using multi-level decomposition such as discrete wavelet transform to improve the visual quality and measurement accuracy under photon-starved conditions. Conventional 3D integral imaging can visualize 3D objects and acquire their depth information. However, the amount of irradiated light on the object causes the degradation of visual quality for 3D images under photon-starved conditions. To visualize 3D objects, photon counting integral imaging has been utilized. It can detect photons from 3D scenes by using a computational photon counting model, which is modelled by the Poisson random process. However, photons occur not only from objects but also in areas where objects do not exist. Moreover, photon fluctuation may occur in the scene through shot noise. Since these noise photons are measurement errors, it may decrease the image quality and accuracy. In contrast, our proposed method uses 2D discrete wavelet transform, which can emphasize the object photons effectively. Finally, our proposed method can enhance the visual quality of 3D images and provide more accurate depth information under photon-starved conditions. To prove the feasibility of our proposed method, we implement the optical experiment and calculate various image quality metrics.

Field-of-view enhanced integral imaging with dual prism arrays based on perspective-dependent pixel mapping.

A field-of-view (FOV)-enhanced integral imaging system is proposed by the combined use of micro-lens array (MLA) and dual-prism array (DPA). The MLA coupled with a DPA virtually function as a new type of the MLA whose FOV is much more increased than that of the original MLA, which enables the capturing of perspective-expanded elemental image arrays (EIAs) of input 3-D scenes and the FOV-enhanced reconstruction of them. For its practical operation, a two-step digital process called perspective-dependent pixel-mapping (PDPM) is also presented. With this PDPM method, picked-up EIAs with a couple of MLAs and DPAs are remapped into the new forms of EIAs to be properly reconstructed in the conventional integral imaging system. Operational performances of the proposed system are ray-optically analyzed. In addition, the feasibility of the proposed system is also confirmed from the computational and optical experiments with test 3-D objects on the implemented prototype. Experimental results finally show a two-times increase of the FOV range of the proposed system when it is compared with that of the conventional system.

Three-dimensional polarimetric image restoration in low light with deep residual learning and integral imaging.

Polarimetric imaging can become challenging in degraded environments such as low light illumination conditions or in partial occlusions. In this paper, we propose the denoising convolutional neural network (DnCNN) model with three-dimensional (3D) integral imaging to enhance the reconstructed image quality of polarimetric imaging in degraded environments such as low light and partial occlusions. The DnCNN is trained based on the physical model of the image capture in degraded environments to enhance the visualization of polarimetric imaging where simulated low light polarimetric images are used in the training process. The DnCNN model is experimentally tested on real polarimetric images captured in real low light environments and in partial occlusion. The performance of DnCNN model is compared with that of total variation denoising. Experimental results demonstrate that DnCNN performs better than total variation denoising for polarimetric integral imaging in terms of signal-to-noise ratio and structural similarity index measure in low light environments as well as low light environments under partial occlusions. To the best of our knowledge, this is the first report of polarimetric 3D object visualization and restoration in low light environments and occlusions using DnCNN with integral imaging. The proposed approach is also useful for 3D image restoration in conventional (non-polarimetric) integral imaging in a degraded environment.

3D image hiding using deep demosaicking and computational integral imaging

This paper proposes a 3D image hiding algorithm with deep learning based demosaicking and computational integral imaging methods. Compared with the previous color 3D image coding method, the proposed algorithm can greatly reduce the amount of ciphertext data by two-thirds through the color filter array (CFA). And the generated CFA is coded by our proposed block cellular automata coding method, which has high security. It is worth noting that The deep learning based demosaicking algorithm can effectively resolve the problem of low resolution that exists in the conventional integral imaging reconstruction algorithms.

Integral imaging display with enhanced depth of field based on bifocal lens array

AbstractIn this paper, we propose a depth‐enhanced integral imaging display based on bifocal lens array. The bifocal lens array consists of a small lens array with plano‐convex circular lens units and a large lens array with square lens units. The proposed display device generates two central depth planes and stitches two reconstructed 3‐D images in depth, which largely improves the depth of field. The experimental results indicate that the proposed structure reconstructed 3‐D image with the depth of field as large as 150.8 mm, which is double of conventional integral imaging display.

Dual-View Integral Imaging System With Wide Viewing Angle And High Spatial Resolution

In this paper, a dual-view integral imaging system is proposed to present two different 3D images for viewers in different viewing zones. For most of the conventional dual-view integral imaging systems, either the viewing angle or the spatial resolution of 3D image is decreased. In the proposed dual-view system, the viewing angle of each viewing zone is enlarged by introducing a switchable mask. Meanwhile, the spatial resolution of the 3D image is maintained by using the time-division multiplexing technology. A prototype of the proposed dual-view system is developed and it has wide viewing angle and high spatial resolution. The experiments coincide with the theoretical analysis.

Photon-counting 3D integral imaging with less than a single photon per pixel on average using a statistical model of the EM-CCD camera

We investigate photon-counting 3D integral imaging (PCII) with an electron multiplying charged-coupled device (EM-CCD) camera using dedicated statistical models. Using conventional integral imaging reconstruction methods with this camera in photon-counting conditions may result in degraded reconstructed image quality if multiple photons are detected simultaneously in a given pixel. We propose an estimation method derived from the photon detection statistical model of the EM-CCD to address the problems caused by multiple photons detected at the same pixel and provide improved 3D reconstructions. We also present a simplified version of this statistical method that can be used under the correct conditions. The imaging performance of these methods is evaluated on experimental data by the peak signal-to-noise ratio and the structural similarity index measure. The experiments demonstrate that 3D integral imaging substantially outperforms 2D imaging in degraded conditions. Furthermore, we achieve imaging in photon-counting conditions where, on average, less than a single photon per pixel is detected by the camera. To the best of our knowledge, this is the first report of PCII with the EM-CCD camera employing its statistical model in 3D reconstruction of PCII.

Dual-view holographic stereogram 3D display based on integral imaging

Conventional dual-view integral imaging (II) three-dimensional (3D) display has the disadvantage of information halving in each viewing zone. We propose a holographic stereogram (HS) type dual-view 3D display without information halving in each viewing zone. The HS can be generated fast by Fourier transforming the elemental images of II into hologram elements (hogels). With shifted Fourier transform, each hogel is generated with a rotated viewing zone. By adjusting the viewing zone of each hogel to make them converge at a specific distance, a specific viewing zone is generated. Then, by adding the two holograms together, each of which corresponds to a separate viewing zone, a dual-view HS is obtained without halving the information presented in each viewing zone. The proposed method is verified through simulation and optical experiment.

High-resolution augmented reality 3D display with use of a lenticular lens array holographic optical element.

An augmented reality (AR) three-dimensional (3D) display based on one-dimensional integral imaging (1DII), by using a lenticular lens array holographic optical element (LLA-HOE), is proposed. The 3D image of the 1DII has higher vertical resolution compared with the image of conventional integral imaging whose resolution is sharply reduced for providing quasi-continuous viewpoints in both the horizontal and vertical directions. The proposed 3D display consists of a projector and an LLA-HOE and is compact. As an image combiner, the LLA-HOE can diffract Bragg-matched light rays that have the same wavelength and incident angle as the original reference wave; it can also function as a lenticular lens array to reconstruct a 3D image but transmit other light rays emitted from the surroundings. In the experiment, an LLA-HOE of 80 mm×80 mm size is recorded, and a combination of a high-resolution 3D virtual image and a real 3D object is presented by the proposed AR 3D display.

Demonstration of an improved integral imaging system with large viewing angle using the micro-lens array mask

An improved integral imaging system with large viewing angle using the micro-lens array mask (MLAM) is demonstrated. Currently, the limitation of viewing angle is one of primary defects of the conventional integral imaging (InIm). Here, a novel MLAM is designed and fabricated to eliminate this limitation, and a mapping relationship of coding is presented to guarantee the perceived 3D image with correct occlusion relation. The viewing angle of the proposed InIm-improved system are significantly improved without decreasing the spatial display resolution of the displayed 3D images. In the experiment, a demonstrated prototype system using the MLAM and a 23.6-in. liquid crystal display (LCD) panel with the resolution of 3840 × 2160 is realized with 45° viewing angle, which can provide a natural 3D visual effect with high quality.

Effective orthoscopic integral imaging reconstruction via adjustable depth position

In this paper, we propose an effective orthoscopic three-dimensional reconstruction via adjustable depth position in integral imaging system. Compared to conventional integral imaging reconstruction method, the proposed method can directly reconstruct orthoscopic 3D images through one-step ray tracing with adjustable depth positions. Thus, a real, undistorted, orthoscopic three-dimensional images can be reconstructed various distances without the magnification and overlap process of conventional integral imaging reconstruction. To show the feasibility of the proposed method, the preliminary experiments are carried out and the some experimental results are presented.

Demonstration of an improved integral imaging with large viewing angle based on two crossed lenticular lens combined arrays

An improved integral imaging with large viewing angle based on two crossed lenticular lens combined arrays (LLCAs) is demonstrated. The limitation of the viewing angle of the perceived three-dimensional (3D) scene constructed by the conventional integral imaging (InIm) is one of the most primary drawbacks of the conventional InIm for its commercial application. Here, the LLCA is designed and fabricated to enlarge the viewing angle, and the perceived 3D scene with large viewing angle produced by the proposed improved-InIm with two crossed LLCAs is presented. In the experiment, an improved InIm with 44° viewing angle based on two crossed LLCAs and the 23.6-inch liquid crystal display (LCD) panel with the resolution of 3840 × 2160 is realized, which can provide natural 3D visual effect with high quality.