Elemental Image Array Research Articles

P‐8.3: Integral Imaging Display System with Enhanced Viewing Angle

A simple but effective method for increasing the viewing angle of the integral imaging system was proposed. Unlike the existing methods, the viewing angle was extended through the fully effective utilization of the pixels within the elemental image array rather than the modifications to the system structures or the importing of extra devices. The relationships between the pixel positions and the viewing zone distributions were analyzed, the algorithm model for replacing the invalid pixels was built and then a method for generating full‐effective‐pixel elemental image array (FEP‐EIA) was proposed. Experiment results demonstrated that the viewing angle of the integral imaging system was significantly increased with the proposed method.

P‐4.6: Efficient Synthetic Encoding Algorithm based on Depth Offset Mapping for Tabletop Integral Imaging Display

An efficient algorithm of three‐dimensional image rendering based on depth offset mapping for tabletop integral imaging display system is proposed. The two‐dimensional color image is used as the reference image, and the corresponding gray depth image is used to obtain the depth information. The offset of pixels in each elemental image array is calculated along the horizontal and vertical directions. According to the geometric relationship between the integral imaging display panel and the viewing position, the synthesized three‐dimensional image can be rendered directly. Compared with the traditional method based on virtual viewpoints generation, the proposed algorithm has fast computational speed and high rendering efficiency. The effectiveness of this method is verified by experiment, which can provide high quality three‐dimensional display effect.

Generalized Hough transform for 3D object recognition and visualization in integral imaging.

Object recognition is an automated image processing application of great interest in areas ranging from defect inspection to robot vision. In this regard, the generalized Hough transform is a well-established technique for the recognition of geometrical features even when they are partially occluded or corrupted by noise. To extend the original algorithm-aimed at detecting 2D geometrical features out of single images-we propose the robust integral generalized Hough transform, which corresponds to transformation under the generalized Hough transform of an elemental image array obtained from a 3D scene under integral imaging capture. The proposed algorithm constitutes a robust approach to pattern recognition in 3D scenes that takes into account information obtained not only from the individual processing of each image of the array but also from the spatial restrictions arising from perspective shifts between images. The problem of global detection of a 3D object of given size, position, and orientation is then exchanged under the robust integral generalized Hough transform for a more easily solved maximum detection in an accumulation (Hough) space dual to the elemental image array of the scene. Detected objects can then be visualized following refocusing schemes of integral imaging. Validation experiments for the detection and visualization of partially occluded 3D objects are presented. To the best of our knowledge, this is the first implementation of the generalized Hough transform for 3D object detection in integral imaging.

High-Quality 3D Visualization System for Light-Field Microscopy with Fine-Scale Shape Measurement through Accurate 3D Surface Data.

We propose a light-field microscopy display system that provides improved image quality and realistic three-dimensional (3D) measurement information. Our approach acquires both high-resolution two-dimensional (2D) and light-field images of the specimen sequentially. We put forward a matting Laplacian-based depth estimation algorithm to obtain nearly realistic 3D surface data, allowing the calculation of depth data, which is relatively close to the actual surface, and measurement information from the light-field images of specimens. High-reliability area data of the focus measure map and spatial affinity information of the matting Laplacian are used to estimate nearly realistic depths. This process represents a reference value for the light-field microscopy depth range that was not previously available. A 3D model is regenerated by combining the depth data and the high-resolution 2D image. The element image array is rendered through a simplified direction-reversal calculation method, which depends on user interaction from the 3D model and is displayed on the 3D display device. We confirm that the proposed system increases the accuracy of depth estimation and measurement and improves the quality of visualization and 3D display images.

Light field displays with computational vision correction for astigmatism and high-order aberrations with real-time implementation.

Vision-correcting near-eye displays are necessary concerning the large population with refractive errors. However, varifocal optics cannot effectively address astigmatism (AST) and high-order aberration (HOAs); freeform optics has little prescription flexibility. Thus, a computational solution is desired to correct AST and HOA with high prescription flexibility and no increase in volume and hardware complexity. In addition, the computational complexity should support real-time rendering. We propose that the light field display can achieve such computational vision correction by manipulating sampling rays so that rays forming a voxel are re-focused on the retina. The ray manipulation merely requires updating the elemental image array (EIA), being a fully computational solution. The correction is first calculated based on an eye's wavefront map and then refined by a simulator performing iterative optimization with a schematic eye model. Using examples of HOA and AST, we demonstrate that corrected EIAs make sampling rays distributed within ±1 arcmin on the retina. Correspondingly, the synthesized image is recovered to nearly as clear as normal vision. We also propose a new voxel-based EIA generation method considering the computational complexity. All voxel positions and the mapping between voxels and their homogeneous pixels are acquired in advance and stored as a lookup table, bringing about an ultra-fast rendering speed of 10 ms per frame with no cost in computing hardware and rendering accuracy. Finally, experimental verification is carried out by introducing the HOA and AST with customized lenses in front of a camera. As a result, significantly recovered images are reported.

Curved Computer-Generated Integral Imaging Based on Backward Ray-Tracing Technique

In traditional computer-generated integral imaging (CGII) methods, the generated elemental image array (EIA) is flat, and it doesn't fit for curved integral imaging (II) systems. Here, a curved CGII method was proposed for curved II system. In the proposed method, multiple virtual camera arrays (VCAs) were built up to determine the origins and directions of the traced rays. Combined with the backward ray-tracing (BRT) technique, a curved EIA was generated, and the rendering time was the same as the BRT-CGII of flat II system. A curved II system consisting of a curved organic light emitting diode display screen, a curved lens array and a curved optical diffuser screen was built up in the experiment. The proposed curved CGII method generated correct curved EIA for the curved II system, and the system presented vivid 3D images with the viewing angle as wide as 80° × 50°.

Wide-Viewing-Angle Integral Imaging System with Full-Effective-Pixels Elemental Image Array.

There exists a defect of the narrow viewing angle in the conventional integral imaging system. One reason for this is that only partial pixels of each elemental image contribute to the viewing angle and the others cause image flips. In this paper, a wide-viewing-angle integral imaging system with a full-effective-pixels elemental image array (FEP-EIA) was proposed. The correspondence between viewpoints and pixel coordinates within the elemental image array was built up, and effective pixel blocks and pixels leading to flipping images were deduced. Then, a pixel replacement method was proposed to generate the FEP-EIAs, which adapt to different viewing distances. As a result, the viewing angle of the proposed integral imaging system was effectively extended through the replacement of the pixels, which caused the image flips. Experiment results demonstrated that wide viewing angles are available for the proposed integral imaging system regardless of the viewing distances.

Generation of elemental image arrays for different reconstruction depths

Generation of elemental image arrays for different reconstruction depths

End-to-End Optimization of an Achromatic Diffractive Optical Element Array for Integral Imaging Three-Dimensional Display

Diffractive optical element (DOE) array promises compact shape for full-parallax integral imaging three-dimensional (3D) display. However, DOEs suffer from large chromatic aberration due to the strong wavelength-dependent nature of diffraction phenomena that degrade the quality of reconstructed 3D images. An end-to-end DOE optimization approach is proposed to reduce chromatic aberration for integral imaging. The end-to-end optimization framework includes RGB pre-processing convolutional neural networks and achromatic optics optimization design of rotationally symmetric DOE. An optical display model based on diffractive optics is proposed to analyze the integral imaging 3D display process for achromatic optical optimization design. The pre-processed elemental image arrays are modulated by an optimized DOE array to reconstruct the achromatic 3D images. A 3D artifacts scene without chromatic aberration is reconstructed in different views with the proposed method, and both peak signal to noise ratio (PSNR) and structural similarity (SSIM) are improved compared to the conventional Fresnel lens DOE.

Re-Calibration and Lens Array Area Detection for Accurate Extraction of Elemental Image Array in Three-Dimensional Integral Imaging

This paper presents a new method for extracting an elemental image array in three-dimensional (3D) integral imaging. To reconstruct 3D images in integral imaging, as the first step, a method is required to accurately extract an elemental image array from a raw captured image. Thus, several methods have been discussed to extract an elemental image array. However, the accuracy is sometimes degraded due to inaccurate edge detection, image distortions, optical misalignment, and so on. Especially, small pixel errors can deteriorate the performance of an integral imaging system with a lens array. To overcome the problem, we propose a postprocessing method for the accurate extraction of an elemental image array. Our method is a unified version of an existing method and proposed postprocessing techniques. The proposed postprocessing consists of re-calibration and lens array area detection. Our method reuses the results from an existing method, and it then improves the results via the proposed postprocessing techniques. To evaluate the proposed method, we perform optical experiments for 3D objects and provide the resulting images. The experimental results indicate that the proposed postprocessing techniques improve an existing method for extracting an elemental image array in integral imaging. Therefore, we expect the proposed techniques to be applied to various applications of integral imaging systems

Light-field 3D encryption based on the monocular depth rendering.

The real-time performance of light-field 3D encryption technology based on the integral imaging principle is restricted by the acquisition speed and the data of the elemental image array (EIA). Herein, we propose a light-field 3D encryption scheme based on monocular depth rendering. With the help of a convolution residuals network (CRN), the proposed scheme can generate the corresponding depth map from a single RGB image and simplify the pickup process of the EIA according to the image mapping. For encryption, using reversible state loop cellular automata (RSL-CA) to encrypt a single RGB image updates traditional 3D encryption, greatly improving the security and efficiency of the encryption algorithm. It is experimentally demonstrated that optical 3D reconstruction is clear and brightly colorful and also has a good parallax effect. The proposed method can open a brand-new research perspective for light-field 3D encryption.

3D medical images security via light-field imaging.

This Letter proposes a selective encryption scheme for three-dimensional (3D) medical images using light-field imaging and two-dimensional (2D) Moore cellular automata (MCA). We first utilize convolutional neural networks (CNNs) to obtain the saliency of each elemental image (EI) originating from a 3D medical image with different viewpoints, and successfully extract the region of interest (ROI) in each EI. In addition, we use 2D MCA with balanced rule to encrypt the ROI of each EI. Finally, the decrypted elemental image array (EIA) can be reconstructed into a full-color and full-parallax 3D image using the display device, which can be visually displayed to doctors so that they can observe from different angles to design accurate treatment plans and improve the level of medical treatment. Our work also requires no preprocessing of 3D images, which is more efficient than the method of using slices for encryption.

High-resolution integral imaging display with precise light control unit and error compensation

3D resolution is an important display performance of integral imaging system. In this paper, an integral imaging system with improved resolution was proposed by using precise light control and error compensation. In the proposed system, a precise light control unit was used instead of the conventional diffused backlight unit, so as to reduce the voxel size. Additionally, an error compensation method was proposed to avoid the mismatch between the pitches of the lens array and the elemental image array, so as to eliminate the crosstalk of reconstructed 3D images. In the optical experiment, a prototype was developed using the proposed structure and error compensation method. It reconstructed vivid 3D images with higher resolution and less crosstalk compared to the conventional system.

Extrinsic crosstalk suppression based on a calibration convolutional neural network for glasses-free 3D display

Extrinsic crosstalk degrades the 3D imaging quality and it occurs due to fabrication errors of the light control optical component and assembling errors of the display system. Here, an extrinsic crosstalk suppression method based on a calibration convolutional neural network (C-CNN) is demonstrated for glasses-free 3D display. The C-CNN is established to calibrate the 3D imaging with extrinsic crosstalk, which can obtain the high-order nonlinear decoding function of the light control optical component with fabrication errors and assembling errors. Furthermore, model for 3D imaging with both extrinsic and intrinsic crosstalk can be mathematically established using the C-CNN integrated with the frequency response calculation. Additionally, according to the calibration for real-life 3D imaging with extrinsic crosstalk, pixels of elemental image array (EIA) contributing to viewpoint crosstalk are extinguished to suppress extrinsic crosstalk. The experiment verifies the feasibility and superior of the proposed method. A 5-degree viewing angle and 5 viewpoints glasses-free 3D display with improved 3D imaging quality is demonstrated.

Deep-learning-based 3D object salient detection via light-field integral imaging.

This Letter proposes an effective light-field 3D saliency object detection (SOD) method, which is inspired by the idea that the spatial and angular information inherent in a light-field implicitly contains the geometry and reflection characteristics of the observed scene. These characteristics can provide effective background clues and depth information for 3D saliency reconstruction, which can greatly improve the accuracy of object detection and recognition. We use convolutional neural networks (CNNs) to detect the saliency of each elemental image (EI) with different viewpoints in an elemental image array (EIA) and the salient EIA is reconstructed by using a micro-lens array, forming a 3D salient map in the reconstructed space. Experimental results show that our method can generate high-quality 3D saliency maps and can be observed simultaneously from different angles and positions.

Field-of-view enhanced integral imaging with dual prism arrays based on perspective-dependent pixel mapping.

A field-of-view (FOV)-enhanced integral imaging system is proposed by the combined use of micro-lens array (MLA) and dual-prism array (DPA). The MLA coupled with a DPA virtually function as a new type of the MLA whose FOV is much more increased than that of the original MLA, which enables the capturing of perspective-expanded elemental image arrays (EIAs) of input 3-D scenes and the FOV-enhanced reconstruction of them. For its practical operation, a two-step digital process called perspective-dependent pixel-mapping (PDPM) is also presented. With this PDPM method, picked-up EIAs with a couple of MLAs and DPAs are remapped into the new forms of EIAs to be properly reconstructed in the conventional integral imaging system. Operational performances of the proposed system are ray-optically analyzed. In addition, the feasibility of the proposed system is also confirmed from the computational and optical experiments with test 3-D objects on the implemented prototype. Experimental results finally show a two-times increase of the FOV range of the proposed system when it is compared with that of the conventional system.

Overlapping-free dual-view integral imaging display

In this paper, an overlapping-free dual-view integral imaging (II) display system was proposed. The light rays emitted from the left and right parts of the compound element images were deflected into opposite directions by using a prism array, thus the left and right viewing zones were completely separated. In addition, a compound elemental image array generation method was also proposed, in which a non-uniform camera array was built up according to the viewpoints distribution of the proposed system. Compared with previous dual-view II displays, our proposed system avoided the overlapping images between two viewing zones, and the crosstalk was removed. In the experiment, we designed and developed a prototype whose left and right viewing zones were spatially separated. The left and right viewing angles were both 28°, and the angular spacing between them was 54°. Both viewing zones presented vivid three-dimensional display performances. Our proposed dual-view II display system is eminently suitable for the vehicle display.

Demonstration of a novel multi-cameras light field rendering system and its application

Large scale light field rendering of real scene is very important for the area of image-based modeling and parallax-based 3D display. With this in mind, we tried to use simple cameras, lenses, motion platform and a host PC to capture the light field and render it for 3D display. It extends our ability to capture the 3D information of the real scene in an effective and compact way, elemental image array for 3D display can be generated without the need of any pixel interpolation or virtual view synthesis. Optical experiment verified the validity of the designed system.

Dual-mode light field display with enhanced viewing range: From far distance viewable to near distance touchable

Based on full analysis of the viewing characteristic of integral imaging and multiview display, a dual-mode light field display with viewing-distance detecting is proposed by changing the pixel clustering of the elemental image array and re-locating the center of the element image accordingly, the width of the viewing zone at target viewing distance is maximized and the viewing distance range of the light field display is enhanced. The optical experiment verified the effectiveness and validity of the proposed method. Our reported work will make a step further in light field display from far distance viewable to near distance touchable.

Fast responsive 2D/3D switchable display using a liquid crystal microlens array.

A fast responsive two-dimensional/three-dimensional (2D/3D) switchable display is demonstrated based on an active liquid crystal (LC) microlens array and a twisted nematic (TN) cell. Compared with the traditional LC microlens array, the fabricated LC microlens array has more ideal phase profile and better focusing effect. The TN cell can switch the polarization direction of the incident light with a very short switching time (4.3 ms) and a small driving voltage (5Vrms). By introducing the tilted elemental image arrays and tilted LC lens array, the moiré patterns are eliminated. The fast switching of the 2D/3D display can be realized by applying or removing voltage to the TN cell. The fast responsive 2D/3D switchable display with the LC microlens array and the TN cell is thin and compact without moiré pattern compared with other 2D/3D switchable display devices.