Elemental Image Array Research Articles

Naked-eye 3D visualization of cultural relics based on integrated imaging

AbstractDigital technology has become crucial for the protection of cultural relics, enabling their acquisition, preservation, exhibition, and dissemination in a non-contact manner.At present, the three-dimensional(3D) visualization technology of cultural relics focuses on the research of non-true 3D visualization, which cannot provide continuous viewing angle and takes a long time to render. This paper focuses on the acquisition and reproduction of digital information related to cultural relics, and proposes to achieve the true 3D display of cultural relics from a continuous perspective based on integrated imaging(InIm). Building up on traditional InIm, an Element image array (EIA) generation algorithm based on local depth template matching is designed by using spatial geometry relation. Experimental results show that the algorithm not only enables naked-eye 3D visualization of cultural relics, but also the generation speed is more than twice that of the compared 3D display data source generation algorithm.

Elemental image array generation based on BVH structure combined with spatial partition and display optimization

Integral imaging display has been widely used because of its features such as full parallax viewing, high comfort without glasses, simple structure and easy implementation. This paper enhances the speed of EIA generation by optimizing the acceleration structure of the ray tracing algorithm. By considering the characteristics of segmental objects captured by the camera during integral image rendering, a novel accelerating structure is constructed by combining BVH with the camera space. The BVH traversal is expanded into a 4-tree based on depth priority order to reduce hierarchy and expedite hit point detection. Additionally, the parameters of the camera array are constrained according to the reconstructed three-dimensional (3D) image range, ensuring optimal object coverage on screen. Experimental results demonstrate that this algorithm reduces ray tracing time for hitting triangle grid of collision objects while automatically determining display range for stereo images and adjusting camera parameters accordingly, thereby maximizing utilization of integrated imaging display resources.

Design of elemental-image-predistortion and splicing seamless integral near-eye display based on aspherical microlens arrays

The near-eye display (NED) has long been a challenging and intriguing research focus in the fields of augmented reality (AR) and virtual reality (VR). In this paper, we propose a design method of elemental-image (EI) predistortion and splicing seamless integral NED based on microlens arrays (MLAs), which combines rigorous optical design with image preprocessing. The construction of MLA is based on sub-channel optimization and splicing, enabling the realization of both VR-NED and AR-NED. The calibration of each sub-channel in NED is conducted using a perfect lens that possesses an effective focal length equivalent to that of the human eye. To achieve a high-precision seamless integral NED, the precise location and acquisition of initial EIs are accomplished through chief ray tracing and image clipping. The predistortion EI array (EIA) is further obtained by combining chief ray tracing and radial basis function (RBF) image warping techniques. A seamless MLA-based integral AR-NED with a full field of view (FOV) of 60° is realized and verified. A prototype of a seamless MLA-based integral VR-NED is further implemented by integrating multiple discrete single-focal predistortion EIAs. The effectiveness of the proposed method has been validated through illumination and luminance simulations as well as experiments.

High-luminance one-dimensional integral imaging display based on gradient-width line light sources

A high-luminance one-dimensional integral imaging display is presented in this paper. A gradient-width line light source array behind an elemental image array is proposed. The widths of the line light sources with equal pitches gradually increased from both sides to center. The viewing ranges generated by the line light sources were enlarged from both sides to the center. The light illuminating the left half of the elemental image array was superimposed on the right side, and the light illuminating the right half of the elemental image array was superimposed on the left side. The viewing zone was related to the leftmost and rightmost line light sources, and the luminance was improved by enlarging the other line light sources.

Viewing resolution and depth-of-field enhancement for a digital 3D display based on neural network-enabled multilayer view perspective fitting

The traditional digital three-dimensional (3D) display suffers from low resolution and a narrow depth of field (DoF) due to the lack of planar pixels transformed into view perspectives and the limitation of the diffraction effect of the lens, respectively, which are the main drawbacks to restrict the commercial application of this display technology. Here, the neural network-enabled multilayer view perspective fitting between the reconstructed and original view perspectives across the desired viewing depth range is proposed to render the optimal elemental image array (EIA) for enhancing the viewing resolution as well as the DoF of the digital 3D display. Actually, it is an end-to-end result-oriented coding method to render the fusion EIA with optimal multidepth fusion and resolution enhancement with high registration accuracies for both view perspective and depth reconstructions by using a depth-distributed fitting neural network paradigm. The 3D images presented in the simulations and optical experiments with improved viewing resolution and extended viewing depth range are demonstrated, verifying the feasibility of the proposed method.

Large viewing angle integral imaging 3D display system based on a symmetrical compound lens array.

We propose a large viewing angle integral imaging 3D display system based on a symmetrical compound lens array (SCLA). The display system comprises a high-resolution 2D display panel, an SCLA, and a light shaping diffuser. The high-resolution 2D display panel presents an elemental image array, the SCLA modulates the light rays emitted from the 2D display panel to form 3D images in space, and the light shaping diffuser eliminates the gaps between 3D pixels of the 3D images. We find that the lateral aberration is a crucial factor that affects the resolution of the reconstructed 3D image. The symmetrical structure of the SCLA enables a reduced focal length and the elimination of lateral aberration, improving the viewing angle and the 3D image resolution simultaneously. The experimental results confirm that the proposed display system increases the viewing angle to 68.6°, achieving a comparable resolution of the full field of view while maintaining a simple structure.

Integral imaging near-eye 3D display using a nanoimprint metalens array

Integral imaging (II) display, one of the most critical true-3D display technologies, has received increasing research recently. Significantly, an achromatic metalens array has realized a broadband metalens-array-based II (meta-II). However, the past micro-scale metalens arrays were incompatible with commercial micro-displays; furthermore, the elemental image array (EIA) rendering is always slow. The two hinders in device and algorithm prevent meta-II from being used for practical video-rate near-eye displays (NEDs). This research demonstrates a meta-II NED combining a commercial micro-display and a metalens array. The large-area nanoimprint technology fabricates the metalens array, and a novel real-time rendering algorithm is proposed to generate the EIA. The hardware and software efforts solve the bottlenecks of video-rate meta-II displays. We also build a see-through prototype based on our meta-II NED, demonstrating the feasibility of augmented reality. Our work explores the potential of video-rate meta-II displays, which we expect can be valuable for future virtual and augmented reality.

Computational Integral Imaging Reconstruction Based on Generative Adversarial Network Super-Resolution

To improve acquisition efficiency and achieve super high-resolution reconstruction, a computational integral imaging reconstruction (CIIR) method based on the generative adversarial network (GAN) network is proposed. Firstly, a sparse camera array is used to generate an elemental image array of the 3D object. Then, the elemental image array is mapped to a low-resolution sparse view image. Finally, a lite GAN super-resolution network is presented to up-sample the low-resolution 3D images to high-resolution 3D images with realistic image quality. By removing batch normalization (BN) layers, reducing basic blocks, and adding intra-block operations, better image details and faster generation of super high-resolution images can be achieved. Experimental results demonstrate that the proposed method can effectively enhance the image quality, with the structural similarity (SSIM) reaching over 0.90, and can also reduce the training time by about 20%.

Depth of field extended integral imaging based on multi-depth fitting fusion

The traditional lens-based imaging display suffers from narrow depth of field (DoF), which is one of the primary drawbacks to restrict the commercial application of integral imaging (InIm). The limitation of DoF exists because locations of the reference plane and reconstructed 3D imagery are not identical. Herein, a DoF-extended method of rendering the elemental image array (EIA) based on multi-depth fitting fusion is proposed. In this method, depth stratification across the desired viewing depth range is implemented with depth maps for accurate extracting depth information and realizing high registration accuracy for image fusion, and then the multilayer parallax images (PIs) of different depth layers are obtained. Furthermore, an adaptive depth-fusion coding is proposed to establish effective iteration of depth-feature fitting for rendering the fusion EIA with the multilayer PIs, which uses the adaptive Gaussian-based weight matrix and the least squares method. As a result, the fusion EIA with multi-depth information, namely the iteratively weighted least squares solution of the EIA, is rendered to overcome the DoF limitation for InIm. The simulation and experimental presented 3D images with extended viewing depth range are demonstrated, verifying the feasibility of the proposed method.

Enhancing the spatial resolution of light-field displays without losing angular resolution by a computational subpixel realignment.

Low spatial resolution is an urgent problem in integral imaging light-field displays (LFDs). This study proposes a computational method to enhance the spatial resolution without losing angular resolution. How rays reconstruct voxels through lenslets is changed so that every ray through a lenslet merely provides a subpixel. The three subpixels of a pixel no longer form one voxel but three independent voxels. We further demonstrate imperfect integration of subpixels, called the sampling error, can be eliminated on specific image depths, including the central depth plane. By realigning subpixels in the above manner under no sampling error, the sampling rate of voxels is three times the conventional pixel-based LFDs. Moreover, the ray number of every voxel is preserved for an unaffected angular resolution. With unavoidable component alignment errors, resolution gains of 2.52 and 2.0 are verified in simulation and experiment by computationally updating the elemental image array. The proposed computational method further reveals that LFDs intrinsically have a higher space-bandwidth product than presumed.

Three-dimensional optical watermarking scheme using compressed sensing in integral imaging system

The elemental image array (EIA) is a special form of the integral imaging three-dimensional (3D) display system for recording, transmitting and displaying 3D image information, which consists of a series of small image units taken from different perspectives. There is an urgent need for protecting the copyright of EIA to improve security and anti-aggressive capability. In this paper, we propose a 3D optical watermarking scheme for integral imaging system based on compressed sensing (CS) theory. The proposed watermarking scheme consists of two subsystems, one is the EIA watermark compression and embedding subsystem, and the other is the EIA watermark extraction and display subsystem. CS theory using by phase coding mask is applied for realizing the EIA encryption and compression simultaneously, which can reduce the total amount of embedded watermark information, expand the space of secret keys and make the system more flexible and compact. The experimental results are presented to demonstrate the feasibility and effectiveness of the proposed method. The parameters of the integral imaging watermark encryption system, including the focal length and aperture of the lenslet array, the object distance, the wavelength, the diffraction distance, and the measurement matrix of CS, can all be used as the secret keys. The method proposed in this paper can effectively broaden the keys space and enhance the security of the integral image 3D display system.

Real-time computer-generated integral imaging light field displays: revisiting the point retracing rendering method from a signal processing perspective.

Integral imaging light field displays (InIm-LFDs) can provide realistic 3D images by showing an elemental image array (EIA) under a lens array. However, it is always challenging to computationally generate an EIA in real-time with entry-level computing hardware because the current practice that projects many viewpoints to the EIA induces heavy computations. This study discards the viewpoint-based strategy, revisits the early point retracing rendering method, and proposes that InIm-LFDs and regular 2D displays share two similar signal processing phases: sampling and reconstructing. An InIm-LFD is demonstrated to create a finite number of static voxels for signal sampling. Each voxel is invariantly formed by homogeneous pixels for signal reconstructing. We obtain the static voxel-pixel mapping through arbitrarily accurate raytracing in advance and store it as a lookup table (LUT). Our EIA rendering method first resamples input 3D data with the pre-defined voxels and then assigns every voxel's value to its homogeneous pixels through the LUT. As a result, the proposed method reduces the computational complexity by several orders of magnitude. The experimental rendering speed is as fast as 7 to 10 ms for a full-HD EIA frame on an entry-level laptop. Finally, considering a voxel may not be perfectly integrated by its homogeneous pixels, called the sampling error, the proposed and conventional viewpoint-based methods are analyzed in the Fourier domain. We prove that even with severe sampling errors, the two methods negligibly differ in the output signal's frequency spectrum. We expect the proposed method to break the long-standing tradeoff between rendering speed, accuracy, and system complexity for computer-generated integral imaging.

Deep neural network-enabled resolution enhancement for the digital light field display based on holographic functional screen

The digital light-field display suffers from low resolution due to lack of planar pixels transformed into spatial information, which limits three-dimensional (3D) visual effect obviously. Herein, a pre-coding optimization of rendering the elemental image array (EIA) using a resolution-enhanced deep neural network (RE-DNN) is proposed, and it is used to enhance each reconstructed view perspective with high clarity besides using the holographic functional screen (HFS). The RE-DNN is used to cooperate with the HFS to form additional effective visual pixels/sub-pixels without increasement of physical planar resolution. The simulation and experimental presented 3D images with high clarity are demonstrated, verifying the feasibility and superiority of the proposed resolution enhancement method.

Dual convolutional neural network for aberration pre-correction and image quality enhancement in integral imaging display.

This paper proposes a method that utilizes a dual neural network model to address the challenges posed by aberration in the integral imaging microlens array (MLA) and the degradation of 3D image quality. The approach involves a cascaded dual convolutional neural network (CNN) model designed to handle aberration pre-correction and image quality restoration tasks. By training these models end-to-end, the MLA aberration is corrected effectively and the image quality of integral imaging is enhanced. The feasibility of the proposed method is validated through simulations and optical experiments, using an optimized, high-quality pre-corrected element image array (EIA) as the image source for 3D display. The proposed method achieves high-quality integral imaging 3D display by alleviating the contradiction between MLA aberration and 3D image resolution reduction caused by system noise without introducing additional complexity to the display system.

3D image steganography using cellular automata transform and depth estimation network

Traditional 3D image steganography is cumbersome and difficult owing to the special structure of 3D image, imposing a practical limit for its applications. Herein, we propose a 3D image steganography scheme utilizing cellular automata transformation (CAT). By using CAT with various rule numbers to hide images, the weakness of previous algorithms with only one transformation plane is overcome, and the hidden image quality can be improved by reducing energy loss. In the 3D reconstruction process, we apply a depth estimation algorithm based on a convolutional network to obtain corresponding depth maps from a single RGB image, simplifying the elemental image array (EIA) pickup process and even overcoming the phase difference problem caused by the traditional optical pickup process using lens arrays. Before hiding, we use the maximum length cellular automata (MLCA) to encrypt a single RGB image instead of encrypting EIA images with huge data, which greatly reduces the complexity of the encryption algorithm and improves efficiency. Finally, we reconstruct the decrypted image optically and obtain a full color and full parallax 3D image through the display device.

Optical steganography with RVNCA-based keys for 3D object

Traditional 3D image steganography is cumbersome and difficult owing to the special structure of 3D image, imposing a practical limit for its applications. Herein, we propose a 3D image steganography scheme utilizing computational ghost imaging (CGI). Furthermore, triple keys are introduced in the steganography to increase the difficulty of being cracked greatly. With the help of backward ray tracing (BRT) technology, the element image array (EIA) of the 3D object is obtained as the target of ghost imaging. For encryption, we utilize restricted vertical neighborhood cellular automata (RVNCA) to encode random masks of the CGI system as the key, which greatly improves the security of the steganography scheme and facilitates the management and transmission of the key. Finally, we reconstruct the decrypted image optically and obtain a full color and full parallax 3D image through the display device.

Comprehensive High-Quality Three-Dimensional Display System Based on a Simplified Light-Field Image Acquisition Method and a Full-Connected Deep Neural Network.

We propose a high-quality, three-dimensional display system based on a simplified light field image acquisition method, and a custom-trained full-connected deep neural network is proposed. The ultimate goal of the proposed system is to acquire and reconstruct the light field images with possibly the most elevated quality from the real-world objects in a general environment. A simplified light field image acquisition method acquires the three-dimensional information of natural objects in a simple way, with high-resolution/high-quality like multicamera-based methods. We trained a full-connected deep neural network model to output desired viewpoints of the object with the same quality. The custom-trained instant neural graphics primitives model with hash encoding output the overall desired viewpoints of the object within the acquired viewing angle in the same quality, based on the input perspectives, according to the pixel density of a display device and lens array specifications within the significantly short processing time. Finally, the elemental image array was rendered through the pixel re-arrangement from the entire viewpoints to visualize the entire field-of-view and re-constructed as a high-quality three-dimensional visualization on the integral imaging display. The system was implemented successfully, and the displayed visualizations and corresponding evaluated results confirmed that the proposed system offers a simple and effective way to acquire light field images from real objects with high-resolution and present high-quality three-dimensional visualization on the integral imaging display system.

The pre-coding optimization for the digital horizontal glass-free 3D display based on holographic functional screen

Each reconstructed view perspective suffers low display quality due to the shortage of planar pixels for mapping light rays with designated intensity and directions to integrate the 3D image, which limits the 3D visual effect of the digital glass-free three-dimensional (3D) display. Here, a pre-coding optimization method of rendering the elemental image array (EIA) is further proposed and used to enhance each reconstructed view perspective with high clarity and rich details besides using the holographic functional screen (HFS). The pre-coding is realized by reassigning designated sub-pixels of the parallax images (PIs), which are used to synthesize the EIA, by linearly weighting their adjacent sub-pixels with the normalized Gaussian function before the synthetization of the EIA with the PIs. The normalized Gaussian function based reassignment of sub-pixels is consistent with the visual property of the human eye. The simulation and experimental presented 3D images with high clarity and rich details are demonstrated, verifying the feasibility and superiority of the proposed method.

Viewing angle enhancement for integral imaging display using two overlapped panels.

Integral imaging three-dimensional (3D) display relies on display panel to provide visual information, but the intrinsic trade-off between the wide viewing angle and high resolution refrains its application in high-throughput 3D display. We propose a method to enhance the viewing angle without sacrificing the resolution by using two overlapped panels. The additionally introduced display panel is composed of two parts: the information area and the transparent area. The transparent area loaded with blank information enables light passing through without any modulation, while the opaque information area is loaded with element image array (EIA) for 3D display. The configuration of the introduced panel can block crosstalk from the original 3D display and establish a new and viewable perspective. Experimental results show that the horizontal viewing angle can be effectively extended from 8° to 16°, demonstrating the feasibility and effectiveness of our proposed method. This method provides the 3D display system with a higher space-bandwidth product, making it a potential technique to be applied for high information-capacity display, including integral imaging and holography.

Fast Numerical Reconstruction of Integral Imaging Based on a Determined Interval Mapping

In this paper, a fast numerical reconstruction of the integral imaging based on a determined interval mapping is proposed. To reduce the computation time, the proposed method employs the determined interval mapping instead of the use of magnification. In the numerical reconstruction procedure, the acquired elemental image array (EIA) from the 3D object is displayed. The flipped elemental image (EI)s are numerically formed by the virtual pinhole array. Then, the determined interval depending on the reconstruction plane is calculated and applied to each flipped EI. These flipped EIs are shifted to match the determined interval at the reconstruction plane and superimposed together. After this superimposed image is divided by the number of the superposition, the position error between the location of the shifted EI and the pixel position of the reconstruction plane is corrected by interpolation. As a result, the refocused image depending on the reconstruction plane can be reconstructed rapidly. From the experimental result, we confirmed that the proposed method largely decreased the computation time compared with the conventional method. In addition, we verified that the quality of the reconstruction by the proposed method is higher than the conventional method by the use of the structural similarity index method.