Light Field Imaging Research Articles

Hybrid calibration procedure for structured light field system

A typical structured light field system consists of a light field camera and a pattern projector. By combining Fringe Projection Profilometry and light field imaging technology, the phase distribution characteristics of each position and each angle of the scene under test could be obtained, thus providing more information dimensions, and therefore improving the robustness and dynamic range of three-dimensional (3D) measurement. System calibration is a decisive factor for high-efficiency and high precision measurement. However, the complicated structure and working method make the calibration of structured light field system usually costly and cumbersome. In this paper, we propose a hybrid calibration approach combining the flexibility of the stereo vision method and high precision characteristics of the phase-coordinate mapping method. The feasibility of the proposed method has been verified in actual experiments with typical 3D targets.

End-to-End Residual Network for Light Field Reconstruction on Raw Images and View Image Stacks

Light field (LF) technology has become a focus of great interest (due to its use in many applications), especially since the introduction of the consumer LF camera, which facilitated the acquisition of dense LF images. Obtaining densely sampled LF images is costly due to the trade-off between spatial and angular resolutions. Accordingly, in this research, we suggest a learning-based solution to this challenging problem, reconstructing dense, high-quality LF images. Instead of training our model with several images of the same scene, we used raw LF images (lenslet images). The raw LF format enables the encoding of several images of the same scene into one image. Consequently, it helps the network to understand and simulate the relationship between different images, resulting in higher quality images. We divided our model into two successive modules: LFR and LF augmentation (LFA). Each module is represented using a convolutional neural network-based residual network (CNN). We trained our network to lessen the absolute error between the novel and reference views. Experimental findings on real-world datasets show that our suggested method has excellent performance and superiority over state-of-the-art approaches.

Axial refocusing precision model with light fields

In light field imaging, axial refocusing precision corresponds to the minimum distance in the axial direction between two distinguishable refocusing planes. The refocusing precision can be essential for applications like light field microscopy. In this paper, we introduce a refocusing precision model based on a geometrical analysis of the flow of rays within the virtual camera. The model establishes the relationship between the feature separability of refocusing and different camera settings. As extending numerical aperture (NA) in classical imaging, the baseline extension of light field also gives more accurate refocusing results. To test the axial refocus precision, we conduct experiments with 1st generation Lytro camera as well as a Blender light field simulation. The results is basically consistent with our prediction. Then, we show that computationally extending the light field baseline increases the axial refocusing precision on real plenoptic camera and light field microscopy datasets.

Preview the Products on Display at Display Week 2022

Preview the Products on Display at Display Week 2022

A Wide Field-of-View Light-Field Camera with Adjustable Multiplicity for Practical Applications.

The long-fascinated idea of creating 3D images that depict depth information along with color and brightness has been realized with the advent of a light-field camera (LFC). Recently advanced LFCs mainly utilize micro-lens arrays (MLAs) as a key component to acquire rich 3D information, including depth, encoded color, reflectivity, refraction, occlusion, and transparency. The wide field-of-view (FOV) capability of LFCs, which is expected to be of great benefit for extended applications, is obstructed by the fundamental limitations of LFCs. Here, we present a practical strategy for the wide FOV-LFC by adjusting the spacing factor. Multiplicity (M) is the inverse magnification of the MLA located between the image plane and the sensor, which was introduced as the overlap ratio between the micro-images. M was adopted as a design parameter in several factors of the LFC, and a commercial lens with adjustable FOV was used as the main lens for practicality. The light-field (LF) information was evaluated by considering the pixel resolution and overlapping area in narrow and wide FOV. The M was optimized for narrow and wide FOV, respectively, by the trade-off relationship between pixel resolution and geometric resolution. Customized wide FOV-LFCs with different M were compared by spatial resolution test and depth information test, and the wide FOV-LFC with optimized M provides LF images with high accuracy.

Trilobite-inspired neural nanophotonic light-field camera with extreme depth-of-field

A unique bifocal compound eye visual system found in the now extinct trilobite, Dalmanitina socialis, may enable them to be sensitive to the light-field information and simultaneously perceive both close and distant objects in the environment. Here, inspired by the optical structure of their eyes, we demonstrate a nanophotonic light-field camera incorporating a spin-multiplexed bifocal metalens array capable of capturing high-resolution light-field images over a record depth-of-field ranging from centimeter to kilometer scale, simultaneously enabling macro and telephoto modes in a snapshot imaging. By leveraging a multi-scale convolutional neural network-based reconstruction algorithm, optical aberrations induced by the metalens are eliminated, thereby significantly relaxing the design and performance limitations on metasurface optics. The elegant integration of nanophotonic technology with computational photography achieved here is expected to aid development of future high-performance imaging systems.

F-number and focal length of light field systems: a comparative study of field of view, light efficiency, signal to noise ratio, and depth of field

The paper discusses the light efficiency and signal-to-noise ratio (SNR) of light field imaging systems in comparison to classical 2D imaging, which necessitates the definition of focal length and f-number. A comparison framework between 2D imaging and arbitrary light field imaging systems is developed and exemplified for the kaleidoscopic and the afocal light field imaging architectures. Since the f-number, in addition to the light efficiency of the system, is conceptually linked to the depth-of-field, an appropriate depth-of-field interpretation for light field systems is discussed as well.

Robust depth estimation on real-world light field images using Gaussian belief propagation

Depth estimation on light field images has been well researched on synthetic datasets. However, existing methods often fail on real-world light field images because of the real-world noises. Due to the lack of real-world light field image datasets for depth estimation, the learning-based methods, e.g., convolutional neural networks, are not capable of performing well on real-world light field images with only synthetic datasets available. In this paper, we adopt the optical flow estimation method to estimate depth from light field images since (i) the existing optical flow methods have shown robustness with significant real-world noises; (ii) the adopted optical flow estimation method does not require any training data, thus not limited to the synthetic datasets. The depth can be solved via the approach of optical flow estimation because (i) a sub-aperture image array can be converted from a light field image; (ii) the optical flow between every two adjacent sub-aperture images of the image array is exactly the disparity between the two images. Furthermore, since a sub-aperture array contains numerous adjacent image pairs, numerous optical flow maps can be generated from the sub-aperture array. We show the depth can be well refined from these numerous optical flow maps through modeling these maps in a single graphical model (a pairwise Gaussian MRF to be precise) and inferencing this graphical model by Gaussian belief propagation (GaBP). Experiments show that i) the proposed method achieves better depth estimation than the state-of-the-art methods on real-world light field images; ii) the proposed method can estimate the depth of objects at a far distance (3.5–9.5 m) much more accurately than the state-of-the-art convolutional neural networks.

Effect of Finite-Sized Optical Components and Pixels on Light-Field Imaging through Correlated Light.

Diffraction-limited light-field imaging has been recently achieved by exploiting light spatial correlations measured on two high-resolution detectors. As in conventional light-field imaging, the typical operations of refocusing and 3D reconstruction are based on ray tracing in a geometrical optics context, and are thus well defined in the ideal case, both conceptually and theoretically. However, some properties of the measured correlation function are influenced by experimental features such as the finite size of apertures, detectors, and pixels. In this work, we take into account realistic experimental conditions and analyze the resulting correlation function through theory and simulation. We also provide an expression to evaluate the pixel-limited resolution of the refocused images, as well as a strategy for eliminating artifacts introduced by the finite size of the optical elements.

Deep-learning-based 3D object salient detection via light-field integral imaging.

This Letter proposes an effective light-field 3D saliency object detection (SOD) method, which is inspired by the idea that the spatial and angular information inherent in a light-field implicitly contains the geometry and reflection characteristics of the observed scene. These characteristics can provide effective background clues and depth information for 3D saliency reconstruction, which can greatly improve the accuracy of object detection and recognition. We use convolutional neural networks (CNNs) to detect the saliency of each elemental image (EI) with different viewpoints in an elemental image array (EIA) and the salient EIA is reconstructed by using a micro-lens array, forming a 3D salient map in the reconstructed space. Experimental results show that our method can generate high-quality 3D saliency maps and can be observed simultaneously from different angles and positions.

Fourier light-field imaging of human organoids with a hybrid point-spread function

Volumetric interrogation of the cellular morphology and dynamic processes of organoid systems with a high spatiotemporal resolution provides critical insights for understanding organogenesis, tissue homeostasis, and organ function. Fluorescence microscopy has emerged as one of the most vital and informative driving forces for probing the cellular complexity in organoid research. However, the underlying scanning mechanism of conventional imaging methods inevitably compromises the time resolution of volumetric acquisition, leading to increased photodamage and inability to capture fast cellular and tissue dynamic processes. Here, we report Fourier light-field microscopy using a hybrid point-spread function (hPSF-FLFM) for fast, volumetric, and high-resolution imaging of entire organoids. hPSF-FLFM transforms conventional 3D microscopy and enables exploration of less accessible spatiotemporally-challenging regimes for organoid research. To validate hPSF-FLFM, we demonstrate 3D imaging of rapid responses to extracellular physical cues such as osmotic and mechanical stresses on human induced pluripotent stem cells-derived colon organoids (hCOs). The system offers cellular (2–3 μm and 5–6 μm in x-y and z, respectively) and millisecond-scale spatiotemporal characterization of whole-organoid dynamic changes that span large imaging volumes (>900 μm × 900 μm × 200 μm in x, y, z, respectively). The hPSF-FLFM method provides a promising avenue to explore spatiotemporal-challenging cellular responses in a wide variety of organoid research.

Efficient Light Field Image Compression with Enhanced Random Access

In light field image compression, facilitating random access to individual views plays a significant role in decoding views quickly, reducing memory footprint, and decreasing the bandwidth requirement for transmission. Highly efficient light field image compression methods mainly use inter view prediction. Therefore, they typically do not provide random access to individual views. On the other hand, methods that provide full random access usually reduce compression efficiency. To address this trade-off, a light field image encoding method that favors random access is proposed in this paper. Light field image views are grouped into independent (3× 3) views, which are called Macro View Images (MVIs) . To encode MVIs, the central view is used as a reference to compress its adjacent neighboring views using a hierarchical reference structure. To encode the central view of each MVI, the most central view along with the center of a maximum of three MVIs, are used as reference images for the disparity estimation. In addition, the proposed method allows the use of parallel processing to reduce the maximum encoding/decoding time-complexity in multi-core processors. Tile partitioning can also be used to randomly access different regions of the light field images. The simulation results show that the proposed method outperforms other state-of-the-art methods in terms of compression efficiency while providing random access to both views and regions of interest.

Comparing synthetic refocusing to deconvolution for the extraction of neuronal calcium transients from light fields.

.SignificanceLight-field microscopy (LFM) enables fast, light-efficient, volumetric imaging of neuronal activity with calcium indicators. Calcium transients differ in temporal signal-to-noise ratio (tSNR) and spatial confinement when extracted from volumes reconstructed by different algorithms.AimWe evaluated the capabilities and limitations of two light-field reconstruction algorithms for calcium fluorescence imaging.ApproachWe acquired light-field image series from neurons either bulk-labeled or filled intracellularly with the red-emitting calcium dye CaSiR-1 in acute mouse brain slices. We compared the tSNR and spatial confinement of calcium signals extracted from volumes reconstructed with synthetic refocusing and Richardson–Lucy three-dimensional deconvolution with and without total variation regularization.ResultsBoth synthetic refocusing and Richardson–Lucy deconvolution resolved calcium signals from single cells and neuronal dendrites in three dimensions. Increasing deconvolution iteration number improved spatial confinement but reduced tSNR compared with synthetic refocusing. Volumetric light-field imaging did not decrease calcium signal tSNR compared with interleaved, widefield image series acquired in matched planes.ConclusionsLFM enables high-volume rate, volumetric imaging of calcium transients in single cell somata (bulk-labeled) and dendrites (intracellularly loaded). The trade-offs identified for tSNR, spatial confinement, and computational cost indicate which of synthetic refocusing or deconvolution can better realize the scientific requirements of future LFM calcium imaging applications.

Weight coefficient calculation through equivalent ray tracing method for light field particle image velocimetry

Light field particle image velocimetry (LF-PIV) can measure three-dimensional (3D) flow velocity from a single snapshot of a light field camera based on the 3D reconstruction of tracer particles of light field images. However, it requires light field intensity calibrations to calculate accurate weight coefficients. Conventionally, the weight coefficients are calculated through in-situ calibration approaches whereas the translation of the calibration board within the entire measurement area is required. Therefore, these approaches are inapplicable for internal industrial flows and space-constrained applications. This study presents an equivalent ray tracing method for the weight coefficients calculation. With a light field snapshot of a smart calibration board, a mapping relationship is established to relate the target points sampled in the flow field with their equivalent points in the air. The weight coefficients are then calculated through the ray tracing method by changing the starting points of ray tracing from the target points to their equivalent points. The calculated weight coefficients are employed to reconstruct the feature marks of the calibration board. Results show that the spatial locations of the marks can be reconstructed accurately with a mean lateral and depth error of 0.63% and 5.6%, respectively. Experiments were also carried out on a low-speed laminar flow. The result indicates that the equivalent ray tracing method provides a similar measurement accuracy with the in-situ calibration method. The overall error of 6.77% is achieved for the velocity measurement. It is demonstrated that the proposed method is capable of measuring the 3D flow velocity for internal industrial and space-constrained applications without translating the calibration board within the entire measurement area.

Light-sheet light-field fluorescence microscopy

Light sheet fluorescence microscopy (LSFM) is a competitive 3D fluorescence imaging method for live biological specimens owing to its simultaneously fast volumetric imaging speed, high contrast, and low phototoxicity. Usually, the axial resolution is hard to be improved in an optical microscopy, especially in a LSFM with a light sheet illuminating. Here, the axial resolution is improved by using the light field imaging that can treat the light sheet with certain thickness as a volumetric imaging but not simply two-dimensional wide-field imaging. By using the light field reconstruction with backprojection (LFBP) approach and depth estimation algorithm, the axial resolution of the modified LSFM can be improved without loss of spatial resolution. In the proof-of-principle experiment, the axial resolution is improved to less than 0.5μm under a 6-μm-thick light sheet illumination. With the proposed method, it provides a potential for realizing both high axial resolution and large field of view in a LSFM.

Calculation model and analysis of fragment position with light field imaging mechanism under considering engineering uncertainty error factors

The spatial positions of multiple fragments formed by projectile explosion have random uncertainty dispersion state, so it is difficult to recognize each fragment and measure its spatial position when multiple fragments reach a certain area at the same time. In order to solve the problem, we propose a test method for measuring the spatial positions of multiple fragments based on dual light field intersection imaging mechanism, study refocusing principle of micro-lens array and four-dimensional light field imaging information, and establish the analytical model of spatial positions of multiple fragments. We set up the uncertainty error evaluation function of two light field cameras intersection test method from the perspective of engineering design, and give the uncertainty analysis method under engineering model. The influence of various factors on the uncertainty of test results under the engineering model is analyzed by calculation. Combined with the actual test data of the wooden target, the results show that the fragment spatial position test method based on dual light field intersection imaging mechanism is feasible and scientific.

Noise-Resilient Depth Estimation for Light Field Images Using Focal Stack and FFT Analysis.

Depth estimation for light field images is essential for applications such as light field image compression, reconstructing perspective views and 3D reconstruction. Previous depth map estimation approaches do not capture sharp transitions around object boundaries due to occlusions, making many of the current approaches unreliable at depth discontinuities. This is especially the case for light field images because the pixels do not exhibit photo-consistency in the presence of occlusions. In this paper, we propose an algorithm to estimate the depth map for light field images using depth from defocus. Our approach uses a small patch size of pixels in each focal stack image for comparing defocus cues, allowing the algorithm to generate sharper depth boundaries. Then, in contrast to existing approaches that use defocus cues for depth estimation, we use frequency domain analysis image similarity checking to generate the depth map. Processing in the frequency domain reduces the individual pixel errors that occur while directly comparing RGB images, making the algorithm more resilient to noise. The algorithm has been evaluated on both a synthetic image dataset and real-world images in the JPEG dataset. Experimental results demonstrate that our proposed algorithm outperforms state-of-the-art depth estimation techniques for light field images, particularly in case of noisy images.

Light Field Reconstruction Using Residual Networks on Raw Images.

Although Light-Field (LF) technology attracts attention due to its large number of applications, especially with the introduction of consumer LF cameras and its frequent use, reconstructing densely sampled LF images represents a great challenge to the use and development of LF technology. Our paper proposes a learning-based method to reconstruct densely sampled LF images from a sparse set of input images. We trained our model with raw LF images rather than using multiple images of the same scene. Raw LF can represent the two-dimensional array of images captured in a single image. Therefore, it enables the network to understand and model the relationship between different images of the same scene well and thus restore more texture details and provide better quality. Using raw images has transformed the task from image reconstruction into image-to-image translation. The feature of small-baseline LF was used to define the images to be reconstructed using the nearest input view to initialize input images. Our network was trained end-to-end to minimize the sum of absolute errors between the reconstructed and ground-truth images. Experimental results on three challenging real-world datasets demonstrate the high performance of our proposed method and its outperformance over the state-of-the-art methods.

Objective Quality Assessment Metrics for Light Field Image Based on Textural Features

Light Field (LF) imaging is a plenoptic data collection method enabling a wide variety of image post-processing such as 3D extraction, viewpoint change and digital refocusing. Moreover, LF provides the capability to capture rich information about a scene, e.g., texture, geometric information, etc. Therefore, a quality assessment model for LF images is needed and poses significant challenges. Many LF Image Quality Assessment (LF-IQA) metrics have been recently presented based on the unique characteristics of LF images. The state-of-the-art objective assessment metrics have taken into account the image content and human visual system such as SSIM and IW-SSIM. However, most of these metrics are designed for images and video with natural content. Additionally, other models based on the LF characteristics (e.g., depth information, angle information) trade high performance for high computational complexity, along with them possessing difficulties of implementation for LF applications due to the immense data requirements of LF images. Hence, this paper presents a novel content-adaptive LF-IQA metric to improve the conventional LF-IQA performance that is also low in computational complexity. The experimental results clearly show improved performance compared to conventional objective IQA metrics, and we also identify metrics that are well-suited for LF image assessment. In addition, we present a comprehensive content-based feature analysis to determine the most appropriate feature that influences human visual perception among the widely used conventional objective IQA metrics. Finally, a rich LF dataset is selected from the EPFL dataset, allowing for the study of light field quality by qualitative factors such as depth (wide and narrow), focus (background or foreground) and complexity (simple and complex).

COLF‐GAN: Learning to axial super‐resolve focal stacks

In order to generate a denser focal stack, a cooperative generative adversarial network is proposed to learn the refocusing ability from light field imaging. To keep the axial continuity, the proposed framework is designed to learn features of a focal stack in both axial directions. Different from the classic generative adversarial network, our generative module consists of a forward prediction sub-network and a backward prediction sub-network, taking the forward-and-backward focal stacks as their inputs, respectively. The bi-directional predictions are then fused by a weighting process, which is guided by an adversarial module. The proposed network is trained on light field focal stacks conducted via digital refocusing. Without loss of the refocus continuity, one can axial super-resolve a focal stack by using the trained model. The effectiveness of the proposed algorithm on different types of focal stacks produced by both light fields and traditional camera shootings is validated. The experimental results indicate that the refocus variation of a focal stack can be well learned and predicted without a complete light field. Therefore, the proposed algorithm outperforms the traditional digital refocusing in terms of run-time.