Elemental Image Array Research Articles

Effect of Viewpoints on the Accommodation Response in Integral Imaging 3D Display

Integral imaging three-dimensional (3D) display provides quasi-continuous viewpoints within a certain viewing angle. A single human eye can obtain several parallax images corresponding to the viewpoints. In this paper, we studied the effect of viewpoints received by a single human eye on the accommodation response when viewing the 3D image reconstructed by integral imaging 3D display. We analyzed the viewpoints distribution of an integral imaging 3D display and the correspondence relationship between the viewpoints and pixels information in elemental image array. In the experiment, the accommodation responses of human eyes when viewing the 3D image with different viewpoint quantities and dimensions are measured. The statistical results reveal that the more viewpoints received by a single human eye, the closer the accommodation response of the 3D image is to that of the real target. This tendency is obvious when the viewpoint quantity reduces from 10 × 10 to 6 × 6. For situations of 2 × 2 and 1 × 1 viewpoint, the accommodation response is unstable and different from the real target. When considering the viewpoint dimensions, two-dimensional viewpoints can provide a more natural accommodation response than one-dimensional viewpoints do. For the one-dimensional viewpoint situation, the horizontal arranged viewpoints and vertical arranged viewpoints have no statistical discrepancy.

Light field resampling method for elemental image array generation of integral imaging

AbstractIn this paper, we proposed a light field resampling method for generating elemental image array (EIA) of integral imaging. The proposed method can break through the constraints between parameters of traditional integral imaging record device and display device and allow to generate a new set of EIA suited to be displayed in an integral imaging display device with arbitrary parameters from any given recorded EIA. Three‐dimensional display results show the correctness and superiority of the proposed method.

Real Scene Pickup Method of Elemental Image Array Based on Convergent Camera Array

Limited available methods for the generation of an elemental image array (EIA) may be the most severe bottleneck for the promotion and application of integral imaging. To obtain rich information of three dimensional (3D) real scene at a relatively low cost, and solve the pseudoscopic problem in real display mode, we propose an EIA generation method based on convergent camera array through establishing the optical mapping model between the proposed method and the classical computer virtual generation method. We expound the basic principle of solving the pseudoscopic problem by changing the relative position of object and center of the convergent camera array in this paper. The experimental results indicate that the generated EIA can be displayed without the pseudoscopic problem in real display mode, relative position relationships of the objects in 3D real scene can also be accurately restored. Both preliminary optical experiments and the optical analysis are conducted to prove the feasibility and validity of the proposed method.

集成成像3D信息获取技术

As a novel auto-stereoscopic three-dimensional (3D) display technology, integral imaging is able to acquire complete 3D information of the 3D scene and reconstruct it. Thus, integral imaging is one of the most promising technologies of the glasses-free 3D displays. In the integral imaging 3D information acquisition process, different perspectives of the 3D scene are recorded to generate the elemental image array to be the 3D content for the integral imaging display. On the other hand, the integral imaging 3D acquisition can be a multi-dimensional information pickup approach applied to the detection and imaging of occluded scenes, target recognition, two-dimensional (2D) / 3D information encryption and microscopic 3D imaging fields. The basic principles, different types of the integral imaging 3D information acquisition approaches, and real-time 3D information acquisition technologies were reviewed in this paper. Some challenging problems and future development trend of the technology were also discussed.

Adaptive periodic δ-function array-based three-dimensional optical refocusing from elemental image array

In order to optically refocus three-dimensional (3-D) objects on their real depths, we design an adaptive periodic δ-function array (A-PDFA)-based 3-D optical refocusing method. An A-PDFA kernel with adaptively determined impulse number is first generated, and next by convolving this A-PDFA with the elemental image array (EIA) captured using an integral imaging system, the depth-filtered EIA is obtained to finally refocus the objects on their real depths. Based on simulations and experiments, the performance of the A-PDFA-based 3-D optical refocusing method is confirmed with the widest effective process zone and the highest image quality compared to other classical 3-D optical refocusing methods. Thus, it is believed the proposed method potentially can be used in various 3-D imaging and display applications.

Elemental Image Generation Algorithm Using Reverse Iteration Along Optical Path

We propose a fast elemental image generation algorithm for integral imaging display based on depth information and a two-dimensional scene image. Existing elemental image generation algorithms establish many-to-one mappings from the scene to the elemental image, thereby including redundant calculations. The improved algorithm entails establishing a single mapping from elemental image to the scene. Each pixel in the elemental image matches a unique scene pixel by inverse iteration along the optical path. This establishes a high precision matching and eliminates hole caused by depth discontinuities. In this paper, we obtain a unique mapping solution by inverse iteration along the optical path using a one-pixel step. The time complexity of this algorithm is mainly driven by the pixel sum of the elemental image array. We generate the elemental image seven times faster than the original algorithm. The larger total quantity of the scene pixels further demonstrates the advantages of our algorithm. The experiments show that the update rate can reach more than 30 fps according to the VCD standard (352×240 resolution) using four parallel calculation threads with a four-core Intel® Core(TM) i7-7500U 2.70 GHz CPU despite various delay factors. The results verify the correctness of the theoretical analysis.

Performance enhancement of integral imaging based Fresnel hologram capturing by the intermediate view reconstruction.

A method aiming at improving the performance of integral imaging (II) based Fresnel hologram is proposed, which is generated by using the intermediate view reconstruction (IVR). The conventional integral holograms are generally generated through Fourier transforming the elemental images (EI) of II into hogels. However, a trade-off between the angular resolution and the spatial resolution of II is inevitable within the generation of integral hologram. The IVR is introduced to enhance the angular spectrum of II-based Fresnel hologram while keeping a compact image size and being free from moving the lenslet array. Multiple elemental image array (EIA) sequences are generated with the IVR and transformed to the corresponding holograms. All the generated hologram sequences shift depending on the relative position of the virtual lens array and are added together to synthesize the Fresnel hologram with a high angular spectrum. The synthesized hologram can reconstruct the 3D image with the combined light fields of all the integral hologram sequences. Finally, both the simulation with multiple objects and experiments of real 3D object are numerically and optically conducted. The high matching results among them confirm this work a better performance over the conventional methods.

Backward ray tracing based rectification for real-time integral imaging display system

In an integral imaging (InI) display system, the deviation of the lens position results in blur and distortion of the reconstructed three-dimensional (3D) image. To achieve undistorted and real-time 3D images, an image correction method based on the backward ray tracing (BRT) technology is presented. By finding the correct pixel-to-ray correspondence, the origin and the direction of the light ray for every pixel in elemental image array (EIA) are reset, and the corrected 3D image is obtained based on the BRT technique. Experimental results demonstrate the feasibility and effectiveness of the proposed method, and the structural similarity (SSIM) index of the reconstructed 3D image increased by 20∼30% compared with that without calibrating. The frame rate of EIA generation is over 40 fps, which can realize real-time generation.

Resolution-enhanced holographic stereogram based on integral imaging using an intermediate-view synthesis technique

We proposed a resolution-enhanced integral imaging (II) based holographic stereogram (HS) using an intermediate-view synthesis (IVS) technique. The HS can be generated fast by Fourier transforming the elemental images of II into hogels. However, the low sampling rate of II is inherited in HS. In this paper, a simple IVS technique is proposed to synthesize intermediate-view elemental images and then adopted to improve the sampling rate of II-based HS. The proposed method avoids multiple captures which are usually realized through the moving array lenslet technique (MALT). Multiple intermediate-view elemental image arrays (IEIAs) are digitally synthesized from an original elemental image array (EIA) with IVS. The original EIA and synthesized IEIAs are transformed to the corresponding HS sequences. All the HS sequences are shifted and added together in one hologram pattern to synthesize the HS with a high sampling rate. Superior to the MALT in II, this method does not need the time-multiplexing technique in both capture stage and display stage. The 3D image quality improvement was verified in numerical and optical reconstruction.

Dual-View Integral Imaging 3D Display Based on Multiplexed Lens-Array Holographic Optical Element

We propose a dual-view integral imaging 3D display based on a multiplexed lens-array holographic optical element (MHOE). A MHOE is a volume holographic optical element obtained by multiplexing technology, which can be used for dual-view integral imaging 3D display due to the angle selectivity of the volume HOE. In the fabrication of the MHOE, two spherical wavefront arrays with different incident angles are recorded using photopolymer material. In the reconstruction, two projectors are used to project the elemental image arrays (EIA) with corresponding angles for two viewing zones. We have developed a prototype of the dual-view integral imaging display. The experimental results demonstrate the correctness of the theory.

Fast-switching laterally virtual-moving microlens array for enhancing spatial resolution in light-field imaging system without degradation of angular sampling resolution

We present an electrically controllable fast-switching virtual-moving microlens array (MLA) consisting of a stacked structure of two polarization-dependent microlens arrays (PDMLAs) with optical orthogonality, where the position of the two stacked PDMLAs is shifted by half the elemental pitch in the diagonal direction. By controlling the polarization of the incident light without the physical movement of the molecules comprising the virtual-moving MLA, the periodic sampling position of the MLA can be switched fast using a polarization-switching layer based on a fast-switching liquid crystal cell. Using the fast-switching virtual-moving MLA, the spatial-resolution-enhanced light-field (LF) imaging system was demonstrated without a decrease in the angular sampling resolution as compared to the conventional LF imaging system comprising a passive MLA; two sets of elemental image arrays were captured quickly owing to the short switching time of the virtual-moving MLA of 450 μs. From the two captured sets of the elemental array image, four-times resolution-enhanced reconstruction images of the directional-view and depth-slice images could be obtained.

Real-time optical 3D reconstruction based on Monte Carlo integration and recurrent CNNs denoising with the 3D light field display.

A general integral imaging generation method based on the path-traced Monte Carlo (MC) method and recurrent convolutional neural networks denoising is presented. According to the optical layer structure of the three-dimensional (3D) light field display, screen pixels are encoded to specific viewpoints, then the directional rays are cast from viewpoints to screen pixels to preform the path integral. In the process of the integral, advanced illumination is used for high-quality elemental image array (EIA) generation. Recurrent convolutional neural networks are implemented as an auxiliary post-processing for the EIA to eliminate the noise of the 3D image in MC integration. 4K (3840 × 2160) resolution, 2 sample/pixel and the ray path tracing method are realized in the experiment. Experimental results demonstrate that the structural similarity metric (SSIM) value and peak signal-to-noise ratio (PSNR) gain of the reconstructed 3D image and target 3D image exceed 90% and 10 dB within 10 frames, respectively. Besides, real-time frame rate is more than 30 fps, showing the super efficiency and quality in optical 3D reconstruction.

Dual-View Three-Dimensional Display Based on Direct-Projection Integral Imaging with Convex Mirror Arrays

Dual three-dimensional (3-D) view displays have been attracting much attention in many practical application fields since they can provide two kinds of realistic 3-D images with different perspectives to the viewer. Thus, in this paper, a new type of the dual-view 3-D display system based on direct-projection integral imaging using a convex-mirror-array (CMA) is proposed. Two elemental image arrays (EIAs) captured from each of the two 3-D objects are synthesized into a single dual-view EIA (DV-EIA) with a selective sub-image mapping scheme. The divergent beam of the projector containing the information of the DV-EIA is projected onto the CMA. On each convex mirror of the CMA, left and right-view components of the DV-EIA are separated and reflected back into their viewing directions. Two different 3-D scene images are then integrated and displayed on their respective viewing zones. Ray-optical analysis with the parallel-ray-approximation method and experiments with the test 3-D objects on the implemented 22″ DV 3-D display prototype confirm the feasibility of the proposed system in the practical application

Crosstalk‐free dual‐view integral imaging display

AbstractWe propose a crosstalk‐free dual‐view integral imaging display. It is composed of a display panel, a barrier array, and a micro‐lens array. The central barrier is located at the vertical central axes of the display panel and the micro‐lens array to split the element image array and the viewing zone. Moreover, other barriers are located at the margins of the elemental images and corresponding micro‐lenses to eliminate the crosstalk. The lights emitting from the left and right half of the element image array are modulated by the left and right half of the micro‐lens array to reconstruct the right and left viewing zones, respectively. A prototype of the proposed dual‐view integral imaging display is developed, and good experimental results agree well with the theory.

Resolution-Enhancement for an Integral Imaging Microscopy Using Deep Learning

A novel resolution-enhancement method for an integral imaging microscopy that applies interpolation and deep learning is proposed, and the complete system with both hardware and software components is implemented. The resolution of the captured elemental image array is increased by generating intermediate-view elemental images between each neighboring elemental image, and an orthographic-view visualization of the specimen is reconstructed. Then, a deep learning algorithm is used to generate maximum possible resolution for each reconstructed directional-view image with improved quality. Since a pretrained model is applied, the proposed system processes the images directly without data training. The experimental results indicate that the proposed system produces resolution-enhanced directional-view images, and quantitative evaluation methods for reconstructed images such as the peak signal-to-noise ratio and the power spectral density confirm that the proposed system provides improvements in image quality.

Super-multiview integral imaging scheme based on sparse camera array and CNN super-resolution.

In this paper, we propose a scheme based on sparse camera array and convolution neural network super-resolution for super-multiview integral imaging. In particular, the proposed scheme is adequate to not only the virtual-world three-dimensional scene with high performance and efficiency, but also the real-world 3D scene with higher availability than the traditional methods. In the proposed scheme, we first adopt the sparse camera array strategy to capture the sparse viewpoint images and use these images to synthesize the low-resolution elemental image array, then the convolution neural network super-resolution scheme is used to restore the high-resolution elemental image array from the low-resolution elemental image array for super-multiview integral image display. Experimental results verify the feasibility of the proposed scheme.

360-degree viewable tabletop 3D display system based on integral imaging by using perspective-oriented layer

We propose a 360-degree viewable tabletop three-dimensional (3D) display system based on integral imaging by using a perspective-oriented layer (POL). The proposed system is composed of a two-dimensional (2D) display device, a POL, a lens array with large pitch and a light shaping diffuser screen. The 2D display device with high refresh rate is utilized to present the adaptive element image array (AEIA) sequentially. The AEIA is generated based on partially-overlapping algorithm to ensure that the 3D image corresponding to adjacent viewing zone is reconstructed without flipping. The 2D display device, the POL and the lens array rotate simultaneously to create a 360-degree continuous viewing zone by time division multiplexing. In our proposed tabletop 3D display system, the best viewing zone can be modulated to the lateral and the crosstalk caused by the narrow viewing angle can be eliminated. In addition, the proposed tabletop 3D display system is capable of reconstructing 3D images with both vertical and horizontal parallaxes. The experimental results agree well with the basic principle of the proposed method.

Resolution-enhanced holographic stereogram based on integral imaging using moving array lenslet technique

We proposed a resolution-enhanced integral imaging (II) based holographic stereogram (HS) using the moving array lenslet technique (MALT). The HS can be generated fast by Fourier transforming the elemental images of II into hogels. However, the low sampling rate of II is inherited in HS. The MALT was introduced to increase the sampling rate of II-based HS. Multiple elemental image array sequences are captured with MALT and transformed to the corresponding HS sequences. All the HS sequences are shifted depending on the movement of the lens array and added together to synthesize the HS with a high sampling rate. The synthesized HS can reconstruct the 3D image with the combined light fields of all the HS sequences. The 3D image quality improvement was verified in numerical and optical reconstruction. Superior to the MALT in II, this method does not need the time-multiplexing technique in the display stage because different HS sequences are displayed in one hologram pattern simultaneously.

3-D spatial floating display using multi-wavelength integral photography

Three-dimensional (3-D) autostereoscopic display with dedicated multiple spatial information under corresponding illumination is critical, especially for anti-counterfeiting, entertainment, etc. In this paper, we propose a 3-D spatial floating display using multi-wavelength integral photography (IP). Using dedicated inkjet printer and refraction-based IP algorithm, a complex two-dimensional (2-D) elemental image array (EIA) can be printed for both fluorescent and normal 3-D autostereoscopic display. With a micro-convex lens array (MLA) and a medium attached on the EIA, normal 3-D images are reconstructed under visible light, while fluorescent 3-D images can be reconstructed under ultraviolet (UV) light. Moreover, to provide comfortable 3-D images with multiple information in space, a feasible 3-D spatial floating display system is also proposed considering the spatial position of the observer with less UV radiation. The proposed method takes the wavelength of 3-D display into consideration to provide spatial multi-information, and can be applied for media, entertainment, etc. Experimental results verified the availability of the proposed method.

Optical arbitrary-depth refocusing for large-depth scene in integral imaging display based on reprojected parallax image

Abstract In this paper, we propose an approach to optically refocus a large-depth three-dimensional (3D) scene on its real depth based on the reprojected parallax images. In the refocusing approach, the parallax images only related to the specific depth can be generated without black zone by reprojecting the captured parallax images. The elemental image array for the same depth can be obtained simply. Therefore, a high-quality refocusing of the 3D object at arbitrary depth can be achieved. The homography matrix used for reprojection contains both the calibration parameters of the camera array and the pixel shifting value related to the specific depth. Experiments on real-world scene are carried out and the results validate the feasibility of our proposed method.