3D Integral Imaging Research Articles

Feasibility study for super-resolution 3D integral imaging using time-multiplexed compressive coding.

We propose a novel super-resolution 3D II method using a time-multiplexed coding mask for improving the resolution of 3D imaging based on compressive sensing (CS) theory. Instead of sensing raw pixel data, the recording device measures the compressive samples of the observed 3D scene through a coding mask placed in the aerial pickup plane in a 3D II system. With the aid of CS framework, we design an optimum coding mask pattern and use the time-multiplexed scheme to achieve a sequence of low-resolution elemental images (EIs), which contain the subpixel details of the observed 3D scene. The super-resolution EIs array is further recovered by an optimization algorithm. Both computational reconstruction and optical experimental results show the validity of the proposed method.

Integral Imaging Pickup Method With Extended Depth-of-Field by Gradient-Amplitude Modulation

One of the major drawbacks of the three-dimensional (3D) integral imaging (II) is the limited depth-of-field (DOF). One approach to enhance the DOF is applying the amplitude-modulating method on the microlens arrays. However, the conventional amplitude-modulating methods usually suffer some collateral effects such as lateral resolution reductions or contrast reversals. In this paper, we propose a gradient-amplitude-modulating (GAM) method to enhance the DOF of the II pickup system when recording large-depth 3D scenes. This method is practical and not accompanied by any obvious lateral resolution deteriorations or contrast reversals. Experimental results obtained from the optical pickup process and computational reconstruction process finally demonstrate the feasibility of the proposed method.

Head Tracking Three-Dimensional Integral Imaging Display Using Smart Pseudoscopic-to-Orthoscopic Conversion

A novel head tracking three-dimensional (3D) integral imaging display is presented. By means of proper application of the smart pseudoscopic-to-orthoscopic conversion (SPOC) method, our display allows an extended viewing angle accommodated to the viewer's position which is obtained by a head/eye tracking system. Using the SPOC, new sets of elemental images are calculated and adapted to any specific viewing position. Additionally, the crosstalk which is typical in conventional integral imaging, is eliminated for a large viewing angle. By performing the rotation transformation in the simulated display, viewing a 3D scene with head rotation can be realized for robust display. Experimental results verify the feasibility of our proposed method.

Regeneration of elemental images in integral imaging for occluded objects using a plenoptic camera

In this Letter, we propose an elemental image regeneration method of three-dimensional (3D) integral imaging for occluded objects using a plenoptic camera. In conventional occlusion removal techniques, the information of the occlusion layers may be lost. Thus, elemental images have cracked parts, so the visual quality of the reconstructed 3D image is degraded. However, these cracked parts can be interpolated from adjacent elemental images. Therefore, in this Letter, we try to improve the visual quality of reconstructed 3D images by interpolating and regenerating virtual elemental images with adjacent elemental images after removing the occlusion layers. To prove our proposed method, we carry out optical experiments and calculate performance metrics such as the mean square error (MSE) and the peak signal-to-noise ratio (PSNR).

用于集成成像3D显示的曲面微透镜阵列设计与仿真

用于集成成像3D显示的曲面微透镜阵列设计与仿真

Augmented Reality 3D Displays With Micro Integral Imaging

In this paper, we present a 3D augmented reality micro integral imaging display system by combining conventional integral imaging and an augmented reality technique. Compared with conventional integral imaging, our proposed system has two advantages: 1) it provides 3D augmented reality display capability and 2) it has a compact design. To validate the feasibility of our proposed method, we experimented with a 3D scene and used two computer-generated objects for augmented reality. By combining the captured 2D elemental images of the 3D object and the computer generated virtual objects, we reconstruct 3D images for the augmented reality micro integral imaging display system. To the best of our knowledge, the first report on a video see through 3D augmented reality display has been experimentally demonstrated with a micro integral imaging display system. The proposed 3D system has potential to be applied to the head mounted display system due to its small form factor.

Multiple-Planes Pseudoscopic-to-Orthoscopic Conversion for 3D Integral Imaging Display

In this paper, we propose an advanced pseudoscopic-to-orthoscopic conversion method for 3D integral imaging display systems. Compared to previous works, this method can generate more accurate images for orthoscopic 3D display using multiple reference planes and estimated depth information. 3D display results show the superiority of the proposed pseudoscopic-to-orthoscopic conversion method.

Synthetic Aperture Integral Imaging Display With Moving Array Lenslet Technique

We propose a method to improve the visual resolution of integral imaging by synchronously moving both a lenslet array and a display device to overcome the limited sampling rate using a stationary lenslet array. Compared to the previous moving array lenslet technique approach, the proposed approach can achieve more accurate sub-pixel movements. Experimental 3D reconstruction results using a non-stationary smart phone are presented to show the visual resolution improvement for the proposed integral imaging 3D display.

Extended depth-of-field 3D endoscopy with synthetic aperture integral imaging using an electrically tunable focal-length liquid-crystal lens.

Conventional synthetic-aperture integral imaging uses a lens array to sense the three-dimensional (3D) object or scene that can then be reconstructed digitally or optically. However, integral imaging generally suffers from a fixed and limited range of depth of field (DOF). In this Letter, we experimentally demonstrate a 3D integral-imaging endoscopy with tunable DOF by using a single large-aperture focal-length-tunable liquid crystal (LC) lens. The proposed system can provide high spatial resolution and an extended DOF in synthetic-aperture integral imaging 3D endoscope. In our experiments, the image plane in the integral imaging pickup process can be tuned from 18 to 38 mm continuously using a large-aperture LC lens, and the total DOF is extended from 12 to 51 mm. To the best of our knowledge, this is the first report on synthetic aperture integral imaging 3D endoscopy with a large-aperture LC lens that can provide high spatial resolution 3D imaging with an extend DOF.

Cross-talk-free integral imaging three-dimensional display based on a pyramid pinhole array

We propose a cross-talk-free integral imaging 3D display based on a pyramid pinhole array. The pyramid pinhole array is used to provide a point light source array. Since the generated point light source array is behind a transmission-type display panel that displays an elemental image array, the pseudoscopic problem can be resolved. By setting the appropriate parameters for the pyramid pinhole array, the cross talk can be eliminated. We experimentally verify the reconstruction of the orthoscopic and cross-talk-free 3D images using the proposed 3D display.

Interferometry based multispectral photon-limited 2D and 3D integral image encryption employing the Hartley transform.

We present a method of securing multispectral 3D photon-counted integral imaging (PCII) using classical Hartley Transform (HT) based encryption by employing optical interferometry. This method has the simultaneous advantages of minimizing complexity by eliminating the need for holography recording and addresses the phase sensitivity problem encountered when using digital cameras. These together with single-channel multispectral 3D data compactness, the inherent properties of the classical photon counting detection model, i.e. sparse sensing and the capability for nonlinear transformation, permits better authentication of the retrieved 3D scene at various depth cues. Furthermore, the proposed technique works for both spatially and temporally incoherent illumination. To validate the proposed technique simulations were carried out for both the 2D and 3D cases. Experimental data is processed and the results support the feasibility of the encryption method.

Double depth-enhanced 3D integral imaging in projection-type system without diffuser

Integral imaging is a three dimensional (3D) display technology without any additional equipment. A new system is proposed in this paper which consists of the elemental images of real images in real mode (RIRM) and the ones of virtual images in real mode (VIRM). The real images in real mode are the same as the conventional integral images. The virtual images in real mode are obtained by changing the coordinates of the corresponding points in elemental images which can be reconstructed by the lens array in virtual space. In order to reduce the spot size of the reconstructed images, the diffuser in conventional integral imaging is given up in the proposed method. Then the spot size is nearly 1/20 of that in the conventional system. And an optical integral imaging system is constructed to confirm that our proposed method opens a new way for the application of the passive 3D display technology.

Polarimetric 3D integral imaging in photon-starved conditions

We develop a method for obtaining 3D polarimetric integral images from elemental images recorded in low light illumination conditions. Since photon-counting images are very sparse, calculation of the Stokes parameters and the degree of polarization should be handled carefully. In our approach, polarimetric 3D integral images are generated using the Maximum Likelihood Estimation and subsequently reconstructed by means of a Total Variation Denoising filter. In this way, polarimetric results are comparable to those obtained in conventional illumination conditions. We also show that polarimetric information retrieved from photon starved images can be used in 3D object recognition problems. To the best of our knowledge, this is the first report on 3D polarimetric photon counting integral imaging.

Viewing Angle-Improved 3D Integral Imaging Display with Eye Tracking Sensor

In this paper, in order to solve the problems of a narrow viewing angle and the flip effect in a three-dimensional (3D) integral imaging display, we propose an improved system by using an eye tracking method based on the Kinect sensor. In the proposed method, we introduce two types of calibration processes. First process is to perform the calibration between two cameras within Kinect sensor to collect specific 3D information. Second process is to use a space calibration for the coordinate conversion between the Kinect sensor and the coordinate system of the display panel. Our calibration processes can provide the improved performance of estimation for 3D position of the observer's eyes and generate elemental images in real-time speed based on the estimated position. To show the usefulness of the proposed method, we implement an integral imaging display system using the eye tracking process based on our calibration processes and carry out the preliminary experiments by measuring the viewing angle and flipping effect for the reconstructed 3D images. The experimental results reveal that the proposed method extended the viewing angles and removed the flipping images compared with the conventional system.

Enhanced field-of-view integral imaging display using multi-Köhler illumination.

A common drawback in 3D integral imaging displays is the appearance of pseudoimages beyond the viewing angle. These pseudoimages appear when the light rays coming from each elemental image are not passing through the corresponding microlens, and a set of barriers must be used to avoid this flipping effect. We present a pure optical arrangement based on Köhler illumination to generate these barriers thus avoiding the pseudoimages. The proposed system does not use additional lenses to project the elemental images, so no optical aberrations are introduced. As an added benefit, Köhler illumination provides a higher contrast 3D display.

Three-dimensional integral imaging with flexible sensing.

We present to the best of our knowledge the first report on three-dimensional (3D) integral imaging capture and reconstruction method with unknown poses of sensors placed on a flexible surface. Compared to a conventional integral imaging system, where a lenslet or sensor array is commonly located on a planar surface, the flexible sensing integral imaging system allows sensors to be placed on a nonplanar surface that can increase the field of view of the 3D imaging system. To obtain the poses of the sensor array on a flexible surface, an estimation algorithm is developed based on two-view geometry theory and the camera projective model. In addition, a super-resolution image is generated from a sequence of low-resolution 2D images with sub-pixel shifts. Super-resolution 3D reconstruction results at different depths are presented to validate the proposed approach.

Real-time computer-generated integral imaging and 3D image calibration for augmented reality surgical navigation

Autostereoscopic 3D image overlay for augmented reality (AR) based surgical navigation has been studied and reported many times. For the purpose of surgical overlay, the 3D image is expected to have the same geometric shape as the original organ, and can be transformed to a specified location for image overlay. However, how to generate a 3D image with high geometric fidelity and quantitative evaluation of 3D image's geometric accuracy have not been addressed. This paper proposes a graphics processing unit (GPU) based computer-generated integral imaging pipeline for real-time autostereoscopic 3D display, and an automatic closed-loop 3D image calibration paradigm for displaying undistorted 3D images. Based on the proposed methods, a novel AR device for 3D image surgical overlay is presented, which mainly consists of a 3D display, an AR window, a stereo camera for 3D measurement, and a workstation for information processing. The evaluation on the 3D image rendering performance with 2560×1600 elemental image resolution shows the rendering speeds of 50–60 frames per second (fps) for surface models, and 5–8 fps for large medical volumes. The evaluation of the undistorted 3D image after the calibration yields sub-millimeter geometric accuracy. A phantom experiment simulating oral and maxillofacial surgery was also performed to evaluate the proposed AR overlay device in terms of the image registration accuracy, 3D image overlay accuracy, and the visual effects of the overlay. The experimental results show satisfactory image registration and image overlay accuracy, and confirm the system usability.

Three-dimensional integral imaging displays using a quick-response encoded elemental image array

Mobile devices are a ubiquitous technology, and many researchers are trying to implement three-dimensional (3D) displays on mobile devices for a variety of applications. We investigate a method to store compressed and encrypted elemental images (EIs) used for 3D integral imaging displays in multiple quick-response (QR) codes. This approach allows user friendly access, readout, and 3D display with mobile devices. We first compress the EIs and then use double-random-phase encryption to encrypt the compressed image and store this information in multiple QR codes. The QR codes are then scanned using a commercial Smartphone to reveal the encrypted information, which can be decrypted and decompressed. We also introduce an alternative scheme by applying photon counting to each color channel of the EIs prior to the aforementioned compression and encryption scheme to generate sparsity and nonlinearity for improved compression and security. Experimental results are presented to demonstrate both 3D computational reconstruction and optical 3D integral imaging display with a Smartphone using EIs from the QR codes. This work utilizing compressed QR encoded EIs for secure integral imaging displays using mobile devices may enable secure 3D displays with mobile devices.

렌즈분할 기반의 시간다중화 3D 집적영상 디스플레이를 위한 컴퓨터적인 요소영상 생성방법

In this paper, we propose a computational generation method of elemental images for time-multiplexed 3D integral imaging display based lens division. In the proposed method, we analyze the image formation between 3D object and elemental images based on ray optics. Based on the analyzed formation, we generate the elemental image set for time-multiplexed display. Positions of an object point picked up in proposed method is shifted for half size of lens divided from those in conventional method when generating elemental images. To show the usefulness of the proposed method, we carry out the preliminary experiments and present the results.

Human gesture recognition using three-dimensional integral imaging.

Three-dimensional (3D) integral imaging allows one to reconstruct a 3D scene, including range information, and provides sectional refocused imaging of 3D objects at different ranges. This paper explores the potential use of 3D passive sensing integral imaging for human gesture recognition tasks from sequences of reconstructed 3D video scenes. As a preliminary testbed, the 3D integral imaging sensing is implemented using an array of cameras with the appropriate algorithms for 3D scene reconstruction. Recognition experiments are performed by acquiring 3D video scenes of multiple hand gestures performed by ten people. We analyze the capability and performance of gesture recognition using 3D integral imaging representations at given distances and compare its performance with the use of standard two-dimensional (2D) single-camera videos. To the best of our knowledge, this is the first report on using 3D integral imaging for human gesture recognition.