Light Field Imaging Research Articles

Combining independent and joint spatial-angular information learning for light field image super-resolution

Removal of occlusions in light-field (LF) images is strongly influenced by the receptive field of the neural network. Existing methods often suffer from limited receptive fields, restricting their ability to capture long-range dependencies and recover occluded regions effectively. To overcome this, we propose LF-PyrNet, a novel end-to-end deep learning model that enhances occlusion removal through multi-scale receptive field learning and hierarchical feature pyramid-based refinement. Our model consists of three key components. First, the feature extractor expands the receptive field by integrating Residual Atrous Spatial Pyramid Pooling (ResASPP) and a modified receptive field block (RFB). These components allow the model to capture broader context and multi-scale spatial dependencies. Next, the core occlusion reconstruction network consists of three cascaded Residual Dense Blocks (RDBs). Each block contains four densely connected layers. A Feature Pyramid Network (FPN) then performs multi-scale feature fusion and refines the representations effectively. Finally, the refinement module, which incorporates both separable and standard convolutions, enhances detailed structural consistency and improves texture restoration in occluded regions. Experimental results show that expanding the receptive field significantly enhances the occlusion removal performance, making LF-PyrNet a reliable solution for reconstructing occluded regions in LF images.

We propose an attention-based back-projection network that enhances light field reconstruction quality by modeling inter-view dependencies. The network uses pixel shuffle to efficiently extract initial features. Spatial attention focuses on important regions while capturing inter-view dependencies. Skip connections in the refinement network improve stability and reconstruction performance. In addition, channel attention within the projection blocks enhances structural representation across views. The proposed method reconstructs high-quality light field images not only in general scenes but also in complex scenes containing occlusions and reflections. The experimental results show that the proposed method outperforms existing approaches.

Special Section Guest Editorial: Light Field Imaging

RealSLF and FlexiDim: Towards Practical Spectral Light Field Imaging

Due to sensor limitations, the light field (LF) images captured by the LF camera suffer from low dynamic range and are prone to poor exposure. To solve this problem, combining multi-exposure technology with LF camera imaging can achieve high dynamic range (HDR) LF imaging. However, for dynamic scenes, this approach tends to produce disturbing ghosting artifacts and destroy the parallax structure of the generated results. To this end, this paper proposes a novel ghost-free HDR LF imaging method using multi-attention learning and exposure guidance. Specifically, the proposed method first designs a multi-scale cross-attention module to achieve efficient multi-exposure LF feature alignment. After that, a dual self-attention-driven Transformer block is constructed to excavate the geometric information of LF and fuse the aligned LF features. In particular, exposure masks derived from middle-exposure are introduced in the feature fusion to guide the network to focus on information recovery in low- and high-brightness regions. Besides, a local compensation module is integrated to cope with local alignment errors and refine details. Finally, a multi-objective reconstruction strategy combined with exposure masks is employed to restore high-quality HDR LF images. Extensive experimental results on the benchmark dataset show that the proposed method generates HDR LF results with high spatial-angular quality consistency and outperforms the state-of-the-art methods in quantitative and qualitative comparisons. Furthermore, the proposed method can enhance the performance of existing LF applications, such as depth estimation.

The light field camera has significantly advanced conventional imaging methods and microscopy over the past decades, providing high-dimensional information in 2D images and enabling a variety of applications. However, inherent shortcomings persist, mainly due to the complex optical setup and the trade-off between resolution. In this work, we propose a Neural Defocus Light Field (NDLF) rendering method, which constructs the light field without a micro-lens array but achieves the same resolution as the original image. The basic unit of NDLF is the 3D point spread function (3D-PSF), which extends the 2D-PSF by incorporating the focus depth axis. NDLF can directly solve the distribution of PSFs in 3D space, enabling direct manipulation of the PSF in 3D and enhancing our understanding of the defocus process. NDLF achieves the focused images rendering by redefining the focus images as slices of the NDLF, which are superpositions of cross-sections of the 3D-PSFs. NDLF modulates the 3D-PSFs using three multilayer perceptron modules, corresponding to three Gaussian-based models from coarse to fine. NDLF is trained on 20 highresolution (1024 × 1024) images at different focus depths, enabling it to render focused images at any given focus depth. The structural similarity index between the predicted and measured focused images is 0.9794. Moreover, we developed a hardware system to collect the high resolution focused images with corresponding focus depth, and depth maps. NDLF achieves high-resolution light field imaging with a single-lens camera and also resolves the distribution of 3D-PSFs in 3D space, paving the way for novel lightfield synthesis techniques and deeper insights into defocus blur.

Light field imaging captures a deeper representation of a given scene and this makes it widely applicable across various fields. It helps in exploring more dimensions for image analysis. In this paper, a multi-stream Convolutional Neural Network (CNN) architecture for light field saliency detection is proposed and this architecture is customized for salient object detection from 4D light field images. The architecture utilizes the defocus stack and all-in-focus light field images to extract salient regions. It also uses manually extracted saliency cues for salient region discrimination. Integrating these manual cues into the CNN framework enhances the learning capability of the model. A data augmentation scheme based on the fusion of defocus stack images is also implemented in the proposed work. This helps in the generation of a greater number of all-in-focus images which aids in improving the model training efficacy. Comparing the proposed saliency detection model to the state-of-the-art algorithms mentioned here, the F-measure increased by an average of 18.877%.

The growing adoption of light field imaging in computational photography, autonomous driving, and immersive display systems has created an urgent need for accurate quality assessment. While Gabor-based methods can effectively analyze texture through multi-scale representations, their conventional fixed-scale implementations lack adaptability to varying distortion levels, resulting in compromised accuracy and efficiency. To address this limitation, we propose an adaptive-scale Gabor framework for light field image quality assessment (LFIQA) that dynamically adjusts feature extraction according to distortion severity. Specifically, we construct an axially symmetric dual-fan filter that first quantifies high-frequency distortion in the Fourier domain. Then, we map the quantified high-frequency distortion to optimal Gabor scales through an exponential transform, enabling distortion-adaptive feature extraction. By computing structural similarity of these adaptive Gabor features, our method achieves state-of-the-art performance across three benchmark datasets. Notably,our distortion-adaptive mechanism reduces computational redundancy compared to conventional multi-scale approaches. To our knowledge, this is the first LFIQA method that successfully integrates frequency-domain distortion quantification with adaptive Gabor scaling, offering both superior accuracy and practical efficiency for real-word applications.

Assessing the Quality of Light Field Images: A Graph-based Approach

To address the viewing angle limitations of traditional compressive light field (LF) display techniques, a composite LF display technique that combines compressive LF and integral imaging (II) display techniques was proposed. The technique first utilizes a partitioned display structure to construct multiple sub-viewing angles and uses compressive LFs within each sub-viewing angle to get continuous disparity. Then, it uses a large-aperture II system to concatenate the various sub-viewing angles to get a composite LF display with a large viewing angle. A partitioned decomposition algorithm using perspective correlation was designed for the composite LF display, significantly reducing the number of iterations. Meanwhile, the non-negative matrix factorization (NMF) algorithm using the Nesterov accelerated gradient (NAG) solver was designed to increase the decomposition efficiency further. A rendering pipeline from 3D model data to composite LF was built using a GPU. For a composite LF with 15×27 viewpoints and a resolution of 2439 × 1355 × 2, only 2-3 iterations are needed to approach the optimal solution. The decomposition rate is 8.94 times faster than the traditional NMF algorithm. A composite LF display system with a viewing angle of horizontal 43.6° and vertical 22.6° respectively, was built, verifying the feasibility of the technology.

Fringe projection profilometry (FPP) is widely used in industrial metrology and 3D scanning. However, relying on traditional lens-based imaging mechanism, existing FPP systems face challenges in achieving high-precision measurement over large depth range. To overcome this limitation, this paper presents an approach integrating micro-electro-mechanical system (MEMS) projection with light field imaging (LFI). To accurately characterize the projection-imaging relationship across a large depth of field (DOF), an extended depth ray fusion model (EDRFM) is proposed, utilizing ray tracing to unify the representation of both projection and imaging processes. Furthermore, phase consistency ensures spatial alignment across multiple viewpoints, and a fusion method incorporating the angle between projection-camera rays and phase modulation quality is introduced to reduce errors caused by view variations. Experimental results demonstrate that the proposed method achieves a measurement accuracy of 0.12 mm within a 650 mm depth range.

The introduction of the metalens array brings technological innovation and performance enhancements to light field imaging systems. While end-to-end design methods overcome the limitations imposed by the manual design of metalens array functionalities, exploration of their potential in achieving an extended depth of field remains unexplored, to the best of our knowledge. This paper proposes a framework for the end-to-end design of a metalens array imaging system tailored for extended depth of field imaging tasks, comprising an image formation model, an image reconstruction model and a specialized training algorithm. This framework effectively covers the extended depth range by leveraging depth importance sampling. The proposed image enhancement network utilizes a multi-scale residual convolution network incorporating channel aggregation and channel attention modules to obtain the final reconstruction results. The end-to-end training process is guided by a composite loss function and employs an alternating training algorithm enhanced with a stop-gradient strategy. Experiments conducted under two depth of field settings demonstrate, through both quantitative metrics and qualitative visual evaluations, that the proposed framework achieves superior imaging performance. Specifically, it demonstrates PSNR improvements of 1.42 dB and 10.03 dB, respectively, when different metalens parameterizations are applied, compared to the conventional design in the 11-times depth of field setting. Ablation studies further validate the effectiveness of the individual techniques proposed. To the best of our knowledge, this is the first work to integrate end-to-end optimization with metalens array design for extended depth of field imaging, addressing the limitations of traditional approaches and improving performance over a broad depth range.

To solve the problem of poor depth estimation due to the influence of occlusion in light-field imaging systems, an embeddable adaptive occlusion-aware module (AOAM) is proposed to effectively compensate for the deficiencies of most existing frameworks. Considering the low computational resource consumption, an adaptive occlusion optimization mode is built that introduces a voting strategy. The beam propagation characteristics are analyzed to filter the disparity values, and the adaptive voting cost is utilized to achieve regional partitioning and noise reduction in the global domain. The superiority of the proposed method is validated on a common light-field dataset.

Joint Super-Resolution and Nonuniformity Correction Model for Infrared Light Field Images Based on Frequency Correlation Learning

An End-to-End Spatially Scalable Light Field Image Compression Method

Three-dimensional flame temperature reconstruction through adaptive segmentation-weighted non-negative least squares and light field imaging

Light field cameras can capture comprehensive light information within a scene. Their core architecture incorporates a micro-lens array (MLA) positioned in front of the imaging sensor, mimicking the compound eye structure of insects. By processing the acquired data, these cameras enable the reconstruction of 3D spatial information. Despite their numerous advantages, light field cameras suffer from a critical drawback: low spatial resolution. To address this limitation, this study proposes a high-spatial-resolution light field imaging approach based on unidirectional object motion (UOMLF). A non-focused light field camera equipped with a hexagonally arranged MLA was developed as the foundation for a unidirectional object motion platform. Additionally, a 3D reconstruction algorithm tailored to high-resolution light field imaging was designed to complement this system. Experimental evaluations were conducted using the USAF resolution test chart and detector chip leads. Results demonstrate that the proposed method significantly enhances xy-plane resolution (approximately four times improvement) while maintaining z-axis resolution. This high-spatial-resolution light field technique based on unidirectional object motion offers broad potential for applications in both industrial and scientific fields.

ABSTRACTLight field imaging has increasingly gained attention in the research community for its ability to provide a richer and more complete representation of visual scenes by capturing both spatial and angular information. However, the massive data volume generated by light field images poses significant challenges in terms of storage and transmission, particularly in bandwidth-limited or real-time environments. This work proposes a novel approach for compressing light field data by integrating Sparse Sampling techniques with the Versatile Video Coding (VVC) standard. The proposed method leverages the Discrete Cosine Transform (DCT) to generate a sparse representation of light field sub-aperture images, followed by random sparse sampling to reduce the dimensionality of the data. A reconstruction process using orthogonal matching pursuit ensures high-fidelity recovery of the light field prior to VVC encoding. The performance of the proposed solution is evaluated using samples of the HCI Dataset, and results demonstrate significant reductions in data size while maintaining visual quality in up to 38db (PSNR) and 0,95 SSIM. The integration of sparse sampling and VVC coding demonstrates promising results for light field compression, enabling its deployment in high-speed communication networks.

Optically transparent photodetectors are becoming essential components in next-generation photonic technologies such as augmented reality and light-field imaging. While transparent photodetectors have been extensively developed for the visible spectrum, extending this capability to the short-wavelength infrared (SWIR) regime remains a significant challenge. This is primarily due to the lack of suitable transparent electrodes and the difficulty in minimizing the thickness of light-absorbing layers. In this work, we demonstrate an SWIR-transparent silicon hot-carrier photodetector, enabled by an ultrathin silver film topped with a high-refractive-index overlayer, serving as a transparent electrode. The electrode design exploits destructive interference to minimize reflection, achieving an 86% transmittance at 1300 nm and a normalized transmittance of 123% relative to a silicon substrate. Integrating this electrode into a silicon substrate forms a metal–silicon Schottky junction for SWIR photon detection through hot-carrier injection, with photon absorption confined to a sub-10 nm metal layer. By leveraging the optical transparency of our photodetector, we demonstrate a laser power monitoring strategy that enables real-time optical power measurements without compromising the spatial profile of the laser beam and altering its optical path. This work paves the way for compact, streamlined designs in applications such as optical data transmission and light detection and ranging (LiDAR), where continuous laser power monitoring is crucial.