3D Integral Imaging Research Articles

High-precision integral imaging 3D salient object detection and reconstruction with texture features based on E2E-TransGAN

Under the framework of integral imaging, a high-precision 3D salient object detection and high-quality texture features reconstruction method is proposed by using the element to element-transformer and generative adversarial network (E2E-TransGAN). The proposed method fully exploits the global salient clues of the element image array to establish cross-regional element image dependency, thereby the accuracy of the salient prediction improved. Meanwhile, our proposed E2E-TransGAN algorithm stratifies the binary salient object area and the grayscale salient object edge, effectively addressing salient region prediction errors, obvious background noise, and color loss. The experimental results show that the detection accuracy of the proposed method improves to an average of 90% across various evaluation metrics. Compared with other methods, the proposed method can reconstruct 3D images with texture features, which have high values of PSNR and SSIM, at 30 and 0.95 on average, respectively. We have experimentally verified that the proposed method achieves precise 3D object salient detection and restores the detailed textures and depth information of 3D salient objects. The proposed method provides strong support for efficient and accurate medical image analysis and simplifies subsequent computer vision tasks.

High-speed 3D Integral Imaging for Sensing and Visualization of Dynamic Underwater Events

High-speed 3D Integral Imaging for Sensing and Visualization of Dynamic Underwater Events

A 2D/3D convertible integral imaging display with enhanced depth of field

Depth of field (DOF) enhancement and two-dimensional (2D)/three-dimensional (3D) convertible display are important research areas of integral imaging 3D display. However, there is no accurate definition of DOF in existing integral imaging 3D displays, and it is difficult to realize DOF enhancement and 2D/3D convertible display simultaneously. In this paper, we propose a DOF analysis method for integral imaging 3D display, and derive an accurate definition of DOF by analyzing the voxel formation mechanism and the minimum resolution angle of the human eyes. Then a 2D/3D convertible integral imaging display with an enhanced DOF is proposed based on the definition of the DOF. The DOF enhancement and different display modes of 2D and 3D are realized by using two sets of dual-state optical components, including polymer dispersed liquid crystal (PDLC) films and liquid crystal micro-lens arrays (LCMLAs). When PDLC film in one set is scattering and LCMLA has the optical properties of a micro-lens array, while PDLC film in the other set is transparent and LCMLA has the optical properties of a flat glass at the same time, the system operates in 3D display modes. Two sets of DOFs are formed in turns and spliced to enhance the whole DOF based on the persistence of vision. When only one PDLC film is scattering and two LCMLAs both have the optical properties of a flat glass, the system realizes 2D display. The experiments on the 2D/3D convertible integral imaging display prototype verify that the DOF is enhanced to 1.71 times and 2D/3D convertible display is realized simultaneously compared with the display based on a conventional micro-lens array. The proposed display has potential for applications in areas such as education and medical care.

Fabrication of microlens arrays on curved substrates for large viewing angle integral imaging 3D display

Curved integral imaging for three-dimensional (3D) displays is well-known for offering an immersive experience and a wider display field of view. Presently, numerous techniques for fabricating curved microlens arrays (curved MLA) have been documented. Nevertheless, the majority of prior research relied on indirect approaches, like utilizing flexible substrates or creating molds. This paper introduces a method for directly fabricating microlens arrays (MLAs) on curved quartz glass. Well-designed arrays of controlled adhesive droplets were created on a hydrophilic-hydrophobic patterned surface. The resulting MLAs, manufactured on a curved substrate with a radius of 10 cm and dimensions of 5 cm × 5 cm, exhibit an 826.5 μm diameter, 56 μm inter-lens spacing, and a 4.5 mm focal length. The effective area of the curved MLA measures 45.8 mm × 45.8 mm. The study encompasses the design of a 2D elemental image and its integration with the curved MLA to achieve a compelling 3D display effect. The lateral viewing range is 2.2 cm. Notably, the field of view angle for the MLA, fabricated on a curved substrate, reached 11.2°, surpassing that of planar MLAs. The imaging system delivers an extensive field of view and attains a realistic 3D stereoscopic display effect. This method, employing various masks, facilitates the rapid fabrication of MLAs on substrates with diverse curvatures and sizes.

Viewing-Angle-Enhanced and Dual-View Compatible Integral Imaging 3D Display Based on a Dual Pinhole Array.

Conventional integral imaging (InIm) three-dimensional (3D) display has the defect of a small viewing angle and usually presents a single 3D image. In this paper, we propose a viewing-angle-enhanced and dual-view compatible InIm 3D display system. The crosstalk pixel areas within the conventional elemental images (EIs) that result in image crosstalk were effectively utilized either for viewing angle enhancement or for dual-view 3D display. In the viewing-angle-enhanced 3D display mode, a composite elemental image (CEI) that consisted of a normal EI and two view-enhanced EIs was imaged by a dual pinhole array and formed an extended 3D viewing area. A precisely designed mask array was introduced to block the overlapped rays between adjacent viewing areas to eliminate image crosstalk. While in the dual-view 3D display mode, a CEI was composed of image information of two different 3D scenes. With the help of the dual pinhole array and mask array, two different 3D images were reconstructed for the left and right perspectives. Experiments demonstrated that both the left and right sides were increased by 6 degrees from the conventional 3D viewing angle, and also, a dual-view 3D display effect that retains the same viewing angle as the conventional system was achieved. The proposed system has a compact structure and can be freely switched between two display modes.

الاقتراب من الزمن الحقيقي لتكوين الصور المتكاملة ثلاثية الابعاد

Generating a truly photo-realistic 3D real-time view capability in an ergonomic and cost effective manner is a crucial engineering challenge. In this paper, the advanced integral imaging methodology is proposed that is based on microlens arrays to generate 3D Holoscopic images by the intersection ray bundle that defines each image point in the associated scene at the correct spatial place. The core idea of the novel algorithm reported in this paper is to accelerate the computer generation of photo-realistic still 3D Holoscopic images based on multiprocessor ray-traced system. Consequently a significantly reduce in running time is successfully completed. Multiple subjective and objective images quality assessments are addressed and as a result there was not any observable distortion in images quality. The results obtained are very satisfactory and for the first time that the new hybrid algorithm has achieved the saving in terms of execution rendering time approximately (60 % - 78 %) that a near to real-time computer generation of 3D Holoscopic images. Keywords: Real-time Computer Generation content of 3D Holoscopic Images, Interpolation 3D integral images, Computer Graphics, Spatial Coherence, Ray Tracing, 3DTV.

3D object tracking using integral imaging with mutual information and Bayesian optimization

Integral imaging has proven useful for three-dimensional (3D) object visualization in adverse environmental conditions such as partial occlusion and low light. This paper considers the problem of 3D object tracking. Two-dimensional (2D) object tracking within a scene is an active research area. Several recent algorithms use object detection methods to obtain 2D bounding boxes around objects of interest in each frame. Then, one bounding box can be selected out of many for each object of interest using motion prediction algorithms. Many of these algorithms rely on images obtained using traditional 2D imaging systems. A growing literature demonstrates the advantage of using 3D integral imaging instead of traditional 2D imaging for object detection and visualization in adverse environmental conditions. Integral imaging’s depth sectioning ability has also proven beneficial for object detection and visualization. Integral imaging captures an object’s depth in addition to its 2D spatial position in each frame. A recent study uses integral imaging for the 3D reconstruction of the scene for object classification and utilizes the mutual information between the object’s bounding box in this 3D reconstructed scene and the 2D central perspective to achieve passive depth estimation. We build over this method by using Bayesian optimization to track the object’s depth in as few 3D reconstructions as possible. We study the performance of our approach on laboratory scenes with occluded objects moving in 3D and show that the proposed approach outperforms 2D object tracking. In our experimental setup, mutual information-based depth estimation with Bayesian optimization achieves depth tracking with as few as two 3D reconstructions per frame which corresponds to the theoretical minimum number of 3D reconstructions required for depth estimation. To the best of our knowledge, this is the first report on 3D object tracking using the proposed approach.

Large viewing angle integral imaging 3D display system based on a symmetrical compound lens array.

We propose a large viewing angle integral imaging 3D display system based on a symmetrical compound lens array (SCLA). The display system comprises a high-resolution 2D display panel, an SCLA, and a light shaping diffuser. The high-resolution 2D display panel presents an elemental image array, the SCLA modulates the light rays emitted from the 2D display panel to form 3D images in space, and the light shaping diffuser eliminates the gaps between 3D pixels of the 3D images. We find that the lateral aberration is a crucial factor that affects the resolution of the reconstructed 3D image. The symmetrical structure of the SCLA enables a reduced focal length and the elimination of lateral aberration, improving the viewing angle and the 3D image resolution simultaneously. The experimental results confirm that the proposed display system increases the viewing angle to 68.6°, achieving a comparable resolution of the full field of view while maintaining a simple structure.

Dual-mode optical see-through integral imaging 3D display with large depth of field

Conventional optical see-through integral imaging display system using holographic optical element (HOE) is actually in focused display mode whose central depth plane (CDP) is very close to the HOE plane, and the depth of field (DOF) of the system is limited to several centimeters around the HOE plane. In this paper, we proposed a dual-mode optical see-through integral imaging display system to reconstruct 3D images in both real and virtual display modes simultaneously and greatly extend the DOF. A dual-mode lens array HOE (DM-LAHOE) was specially designed by using two sets of lens arrays, and it featured as a lens array with positive focal length and negative focal length under the illumination of different probe beams. The designed DM-LAHOE can generate two CDPs away from the HOE plane. Hence, the real mode and virtual mode of the integral imaging display can be achieved simultaneously, and the DOF of the system was greatly improved by increasing the focal length of the recorded lens array and the number of CDPs. In the experiment, the DM-LAHOE with large focal lengths of ±14mm was fabricated, and the DOF of the proposed system was increased from 24.5mm to 190mm. The proposed dual-mode optical see-through integral imaging display system is expected to be applied in the fields of augmented reality display, vehicle display, window display, etc.

Underwater object detection and temporal signal detection in turbid water using 3D-integral imaging and deep learning.

Underwater scattering caused by suspended particles in the water severely degrades signal detection performance and poses significant challenges to the problem of object detection. This paper introduces an integrated dual-function deep learning-based underwater object detection and classification and temporal signal detection algorithm using three-dimensional (3D) integral imaging (InIm) under degraded conditions. The proposed system is an efficient object classification and temporal signal detection system for degraded environments such as turbidity and partial occlusion and also provides the object range in the scene. A camera array captures the underwater objects in the scene and the temporally encoded binary signals transmitted for the purpose of communication. The network is trained using a clear underwater scene without occlusion, whereas test data is collected in turbid water with partial occlusion. Reconstructed 3D data is the input to a You Look Only Once (YOLOv4) neural network for object detection and a convolutional neural network-based bidirectional long short-term memory network (CNN-BiLSTM) is used for temporal optical signal detection. Finally, the transmitted signal is decoded. In our experiments, 3D InIm provides better image reconstruction in a degraded environment over 2D sensing-based methods. Also, reconstructed 3D images segment out the object of interest from occlusions and background which improves the detection accuracy of the network with 3D InIm. To the best of our knowledge, this is the first report that combines deep learning with 3D InIm for simultaneous and integrated underwater object detection and optical signal detection in degraded environments.

3D integral imaging of acoustically trapped objects

3D imaging provides crucial details about the objects and scenes that may not be obtained via 2D imaging methods. However, there are several applications in which the object to be 3D-imaged requires to be immobilized. The integrated digital holographic microscopy (DHM) and optical trapping (OT) system is a useful solution for such a task, but both DHM and OT are mostly suitable for microscopic specimens. Here, for the first time to the best of our knowledge and as an analogy to the DHM-OT system, we introduce integral imaging (InIm) and acoustic trapping (AT) integrated system for 3D imaging of immobilized mesoscopic and macroscopic objects. Post-processing of InIm data enables reconstructing the scene at any arbitrary plane, therefore, it re-focuses any particular depth of the object, which is a curtail task, especially when the object is trapped by AT. We demonstrate the capability of our system by simultaneous trapping and 3D imaging of single and multiple irregularly shaped objects with mm sizes.

Dual convolutional neural network for aberration pre-correction and image quality enhancement in integral imaging display.

This paper proposes a method that utilizes a dual neural network model to address the challenges posed by aberration in the integral imaging microlens array (MLA) and the degradation of 3D image quality. The approach involves a cascaded dual convolutional neural network (CNN) model designed to handle aberration pre-correction and image quality restoration tasks. By training these models end-to-end, the MLA aberration is corrected effectively and the image quality of integral imaging is enhanced. The feasibility of the proposed method is validated through simulations and optical experiments, using an optimized, high-quality pre-corrected element image array (EIA) as the image source for 3D display. The proposed method achieves high-quality integral imaging 3D display by alleviating the contradiction between MLA aberration and 3D image resolution reduction caused by system noise without introducing additional complexity to the display system.

3D integral imaging depth estimation of partially occluded objects using mutual information and Bayesian optimization.

Integral imaging (InIm) is useful for passive ranging and 3D visualization of partially-occluded objects. We consider 3D object localization within a scene and in occlusions. 2D localization can be achieved using machine learning and non-machine learning-based techniques. These techniques aim to provide a 2D bounding box around each one of the objects of interest. A recent study uses InIm for the 3D reconstruction of the scene with occlusions and utilizes mutual information (MI) between the bounding box in this 3D reconstructed scene and the corresponding bounding box in the central elemental image to achieve passive depth estimation of partially occluded objects. Here, we improve upon this InIm method by using Bayesian optimization to minimize the number of required 3D scene reconstructions. We evaluate the performance of the proposed approach by analyzing different kernel functions, acquisition functions, and parameter estimation algorithms for Bayesian optimization-based inference for simultaneous depth estimation of objects and occlusion. In our optical experiments, mutual-information-based depth estimation with Bayesian optimization achieves depth estimation with a handful of 3D reconstructions. To the best of our knowledge, this is the first report to use Bayesian optimization for mutual information-based InIm depth estimation.

P‐145: Late‐News Poster: Integral Imaging 3D Display Based on Top‐hat Diffusing Screen

We propose an integral imaging 3D display based on top‐hat diffusing screen. The integral imaging 3D display consists of a 2D display, a micro‐lens array, and a top‐hat diffusing screen. The top‐hat diffusing screen is composed of multiple elemental lenses with different pitches. The light rays pass through the top‐hat diffusing screen to form a regular light spot, and the light intensity distribution of the light spot is top‐hat type. This can make the 3D image smooth and natural. The experimental results verify the feasibility of the proposed integral imaging 3D display.

Enhanced depth of field of integral imaging display using bifocal microlens array fabricated by two‐step lithography

AbstractDue to the limitation of traditional microlens arrays (MLAs) in integral imaging display, the depth of field (DOF) is restricted in space and the center depth plane is difficult to extend in a large range. Here, we propose a microfabrication method based on bifocal MLAs to improve DOF. The bifocal MLAs for extended DOF were fabricated by using two‐step photolithography and thermal reflow. This method allows diverse microlenses of high to low numerical aperture to achieve high spatial resolution as well as accurate depth estimation. Microlenses of different focal lengths were simultaneously deposited on a substrate by repeated photolithography with multiple photomasks with alignment mark to define micro‐posts of different thicknesses. Hexagonally packaged bifocal MLAs clearly show the DOF extended from 0.004 to 4.908 mm for 57.6 μm in lens diameter, and their corresponding object distance ranges from 0.125 to 0.165 mm. Based on the proposed scheme, this method provides potential applications in integral imaging 3D display or light field display.

PO-02-212 TWISTED FIBROTICALLY-INSULATED MYOBUNDLES CREATE THE COMMON TRACK AND PIVOTING POINTS FOR REENTRANT DRIVERS MAINTAINING PERSISTENT AF IN-VIVO

Localized reentrant drivers have been suggested to maintain persistent atrial fibrillation (perAF). However, their existence, electrophysiological mechanisms, and structural substrates in-vivo are controversial. Define AF driving mechanism in-vivo and ex-vivo in a perAF canine model. A clinically relevant canine model (n=5) of self-sustained (4 months) perAF was developed to study AF maintaining mechanisms both in-vivo and ex-vivo by integrated 3D imaging, including high-resolution near-infrared optical mapping (NIOM, 0.3-0.5mm resolution), contrast-enhanced magnetic resonance imaging (CE-MRI, 0.15mm resolution). Transmural fibrosis was validated with histology. CE-MRI was used to quantitatively analyze 3D atrial structure, including transmural fiber orientation and fibrosis content in the sub-epi, intramural, and sub-endocardial layers. In all canine models (n=5), sequential in-vivo to ex-vivo sub-surface NIOM and 3D CE-MRI revealed that the self-sustained perAF was primarily maintained by localized reentrant drivers with temporal stability 53±17% (n=10), which recurrent on the same arrhythmogenic hubs. The 3D structure of the arrhythmogenic driver hubs includes a common intramural reentry track and multiple return paths formed by fibrotically-insulated bundles. The in-vivo spatio-temporal activation patterns of reentrant drivers were preserved ex-vivo (Figure A), including driver AFCL (91±7 ms vs 96±12 ms, p>0.05) and temporal driver stability (51±8% vs 52±6%, n=4, p>0.05). Targeted ablation of the driver reentrant tracks terminated or converted AF to slower tachycardia. Quantitative CE-MRI analysis of 3D myofiber orientations revealed that the transmural fiber twist along the common reentry tracks and pivoting points is greater than in non-driver regions (Figure B-C). The driver reentrant tracks had increased interstitial fibrotic strands and significantly higher intramural fibrosis than the neighboring non-driver regions (Figure D-E). Our results provide in-vivo proof-of-concept that spatially stable reentrant drivers are a mechanism of perAF maintenance. Intramural fibrotic strands and fibrotically-insulated myobundles in regions with the highest transmural myofiber twist which may form the common tracks and pivoting points for return paths of the spatially-stable reentrant AF driver.

Flexible liquid crystal micro-lens arrays for curved integral imaging 2D/3D convertible display

ABSTRACT Glasses-free three-dimensional (3D) displays are being regarded as the key to the future of display that will redefine the display industry. Inspired by the curved screen, liquid crystal micro-lens arrays (LC MLAs) on flexible PET substrate with a radius of curvature of 5 cm was achieved. For demonstration, an integral imaging-based 2D/3D convertible display system is proposed by using the flexible LC MLAs (FLC MLAs) to switch its operation mode between 2D and 3D modes. When the FLC MLAs with an applied voltage are in the system, the prototype renders a matched 3D image and works in 3D mode. When the FLC MLAs do not have an applied voltage, the FLC MLAs are equivalent to glass, and the display system works in 2D mode. In addition, we also demonstrated that the FLC MLAs have a wider viewing angle than the flat integral imaging 3D display system. The depth of field ranges from 6.78 to 210.12 mm under low operating voltages of 3.4 to 3.1 Vrms, respectively. With its low voltage, thin structure, and adjustable focal length form factors, the developed display system can be integrated with off-the-shelf purchased flat panels, making it a promising option for portable electronics.

4: Wide‐Viewing Angle Integral Imaging 3D Display System Based on Aspherical Compound Lens Array

4: Wide‐Viewing Angle Integral Imaging 3D Display System Based on Aspherical Compound Lens Array

Novel diffraction limit implied geometric lens imaging model based on a finite number of pixels for integral imaging display

We propose a novel diffraction limit implied geometric lens imaging (DLI-GLI) model considering diffraction effect on a finite number of pixels in elemental image (EI)and 3D depth priority-oriented integral imaging (DPO-II) showing variations of the lateral resolution (LR) within enhanced depth of focus (DoF) range graphically. We assigned 3 representative gaps between lens array and display panel including gap at conventional DPII to find the optimized DoF and enhanced LR. We show that the number of pixels in an elemental lens (EL), and focal length and width of the lens determine practical LR and DoF at the focal plane, and the numerically induced DOP-II with two specified DPO-gaps which are larger than focal length have the enhanced LR and DoF comparing with those of the conventional DPII presenting the optimized LR and DoF near to two DPO-gaps. We show that one of the DPO-gaps has largest DoF and another has largest resolution within some part of DoF in the on-axial DPO-spot size diagram (DPO-SD). Though this DPO-SD, it is expected that the various 3D geometric imaging configurations according to the application fields of 3D imaging could be designed as well. Finally, we verified that the numerically induced 3 geometrical structures based on the proposed model show the similar results with measurements of the optical experiments on the built-up DPO-asymmetric II (DPO-AII) system.

Generalized Hough transform for 3D object recognition and visualization in integral imaging.

Object recognition is an automated image processing application of great interest in areas ranging from defect inspection to robot vision. In this regard, the generalized Hough transform is a well-established technique for the recognition of geometrical features even when they are partially occluded or corrupted by noise. To extend the original algorithm-aimed at detecting 2D geometrical features out of single images-we propose the robust integral generalized Hough transform, which corresponds to transformation under the generalized Hough transform of an elemental image array obtained from a 3D scene under integral imaging capture. The proposed algorithm constitutes a robust approach to pattern recognition in 3D scenes that takes into account information obtained not only from the individual processing of each image of the array but also from the spatial restrictions arising from perspective shifts between images. The problem of global detection of a 3D object of given size, position, and orientation is then exchanged under the robust integral generalized Hough transform for a more easily solved maximum detection in an accumulation (Hough) space dual to the elemental image array of the scene. Detected objects can then be visualized following refocusing schemes of integral imaging. Validation experiments for the detection and visualization of partially occluded 3D objects are presented. To the best of our knowledge, this is the first implementation of the generalized Hough transform for 3D object detection in integral imaging.