Epipolar Plane Image Research Articles

Beyond Photometric Consistency: Geometry-Based Occlusion-Aware Unsupervised Light Field Disparity Estimation.

Although learning-based light field disparity estimation has achieved great progress in the most recent years, the performance of unsupervised light field learning is still hindered by occlusions and noises. By analyzing the overall strategy underlying the unsupervised methodology and the light field geometry implied in epipolar plane images (EPIs), we look beyond the photometric consistency assumption, and design an occlusion-aware unsupervised framework to deal with the situations of photometric consistency conflict. Specifically, we present a geometry-based light field occlusion modeling, which predicts a group of visibility masks and occlusion maps, respectively, by forward warping and backward EPI-line tracing. In order to learn better the noise- and occlusion-invariant representations of the light field, we propose two occlusion-aware unsupervised losses: occlusion-aware SSIM and statistics-based EPI loss. Experiment results demonstrate that our method can improve the estimation accuracy of light field depth over the occluded and noisy regions, and preserve the occlusion boundaries better.

Lensless light-field imaging using LMI

Light-field imaging is widely used in many fields, such as computer vision, graphics, and microscopy imaging, to record high-dimensional light information for abundant visual perception. However, light-field imaging systems generally have high system complexity and limited resolution. Over the last decades, lensless imaging systems have attracted tremendous attention to alleviate the restrictions of lens-based architectures. Despite their advantages, lensless light-field imaging introduces significant errors in light-field reconstruction. This paper introduces a novel, to our knowledge, light field moment imaging-based lensless imaging system (LMI-LIS) aiming to improve the quality of light-field reconstruction. The proposed approach first uses light field moment imaging (LMI) with a sinc angular distribution model of the light field to extract the encoded information of the scene for each sub-aperture area. Meanwhile, the corresponding sub-aperture point spread function is segmented from the system point spread function. Finally, sub-aperture images of the scene are reconstructed separately for each sub-aperture area. To evaluate the light-field reconstruction performance, the imaging quality and angular consistency of different lensless light-filed imaging methods are compared through digital refocusing, epipolar plane image, peak signal-to-noise ratio, and structural similarity index. Furthermore, the effectiveness of the proposed methodology is verified using experimental results and theoretical analysis. It is demonstrated that lensless light-field imaging using LMI and the sinc model of the angular distribution achieves high-quality sub-aperture images.

Spatial–angular–epipolar transformer for light field spatial and angular super-resolution

Transformer-based light field (LF) super-resolution (SR) methods have recently achieved significant performance improvements due to global feature modeling by self-attention mechanisms. However, as a method designed for natural language processing, 4D LFs are reshaped into 1D sequences with an immense set of tokens, which results in a quadratic computational complexity cost. In this paper, a spatial–angular–epipolar swin transformer (SAEST) is proposed for spatial and angular SR (SASR), which sufficiently extracts SR information in the spatial, angular, and epipolar domains using local self-attention with shifted windows. Specifically, in SAEST, a spatial swin transformer and an angular standard transformer are firstly cascaded to extract spatial and angular SR features, separately. Then, the extracted SR feature is reshaped into the epipolar plane image pattern and fed into an epipolar swin transformer to extract the spatial–angular correlation information. Finally, several SAEST blocks are cascaded in a Unet framework to extract multi-scale SR features for SASR. Experiment results indicate that SAEST is a fast transformer-based SASR method with less running time and GPU consumption and has outstanding performance on simulated and real-world public datasets.

Quality assessment for multi-exposure fusion light field images with dynamic region segmentation

The luminance dynamic range of real scenes can reach up to nine orders, whereas the existing light field camera captures light field images (LFIs) with a limited luminance dynamic range of about two orders. Although LFIs can be enhanced using multi-exposure fusion (MEF) technologies, this process introduces various distortions that affect human visual perception. Therefore, it is crucial to develop an effective quality assessment tool for multi-exposure fusion light field images (MEF-LFIs). Starting from the complex distortion characteristics generated by MEF with dynamic scenes, this paper proposes an MEF-LFI quality metric with dynamic region segmentation. The proposed metric consists of three main modules. The first module focuses on spatial feature extraction and incorporates a dynamic region segmentation scheme to address artifacts in MEF-LFIs. The second module utilizes Shearlet transform for spectrum feature extraction to characterize detail loss. Additionally, the third module addresses distortion in the angular domain of MEF-LFIs through angular feature extraction using epipolar plane images and tensor decomposition. Finally, the quality of the MEF-LFIs is evaluated by integrating spatial features, spectrum features and angular features into feature vectors which are then inputted into a regression model. The experimental results show that the proposed metric is superior to the representative quality metrics and has better consistency with the human visual perception.

Inertial Algorithm with Dry Fraction and Convolutional Sparse Coding for 3D Localization with Light Field Microscopy

Light field microscopy is a high-speed 3D imaging technique that records the light field from multiple angles by the microlens array(MLA), thus allowing us to obtain information about the light source from a single image only. For the fundamental problem of neuron localization, we improve the method of combining depth-dependent dictionary with sparse coding in this paper. In order to obtain higher localization accuracy and good noise immunity, we propose an inertial proximal gradient acceleration algorithm with dry friction, Fast-IPGDF. By preventing falling into a local minimum, our algorithm achieves better convergence and converges quite fast, which improves the speed and accuracy of obtaining the locolization of the light source based on the matching depth of epipolar plane images (EPI). We demonstrate the effectiveness of the algorithm for localizing non-scattered fluorescent beads in both noisy and non-noisy environments. The experimental results show that our method can achieve simultaneous localization of multiple point sources and effective localization in noisy environments. Compared to existing studies, our method shows significant improvements in both localization accuracy and speed.

End-to-End Light Field Image Compression with Multi-Domain Feature Learning

Recently, end-to-end light field image compression methods have been explored to improve compression efficiency. However, these methods have difficulty in efficiently utilizing multi-domain features and their correlation, resulting in limited improvement in compression performance. To address this problem, a novel multi-domain feature learning-based light field image compression network (MFLFIC-Net) is proposed to improve compression efficiency. Specifically, an EPI-based angle completion module (E-ACM) is developed to obtain a complete angle feature by fully exploring the angle information with a large disparity contained in the epipolar plane image (EPI) domain. Furthermore, in order to effectively reduce redundant information in the light field image, a spatial-angle joint transform module (SAJTM) is proposed to reduce redundancy by modeling the intrinsic correlation between spatial and angle features. Experimental results demonstrate that MFLFIC-Net achieves superior performance on MS-SSIM and PSNR metrics compared to public state-of-the-art methods.

GLGNet: light field angular superresolution with arbitrary interpolation rates

Acquiring high-resolution light fields (LFs) is expensive. LF angular superresolution aims to synthesize the required number of views from a given sparse set of spatially high-resolution images. Existing methods struggle with sparsely sampled LFs captured with large baselines. Some methods rely on depth estimation and view reprojection, and are sensitive to textureless and occluded regions. Other non-depth based methods suffer from aliasing or blurring effects due to the large disparity. In addition, most methods require specific models for different interpolation rates, which reduces their flexibility in practice. In this paper, we propose a learning framework that overcomes these challenges by exploiting the global and local structures of LFs. Our framework includes aggregation across both the angular and spatial dimensions to fully exploit the input data and a novel bilateral upsampling module that upsamples each epipolar plane image while better preserving its local parallax structure. Furthermore, our method predicts the weights of the interpolation filters based on both subpixel offset and range difference, allowing angular superresolution at different rates with a single model. We show that our non-depth based method outperforms the state-of-the-art methods in terms of handling large disparities and flexibility on both real-world and synthetic LF images.

In situ three-dimensional reconstruction of laser powder bed fusion parts by light field camera

The assessment of output quality in the laser powder bed fusion process entails the analysis of the three-dimensional morphology of the manufactured parts, which plays a crucial role in quality control to prevent metallurgical defects. This study proposes a novel approach of three-dimensional reconstruction for laser powder bed fusion parts using a light field camera. Initially, a consistency method for high-precision camera calibration was explored to determine the virtual camera pixel size and coefficient of effective focal length of light field camera. Subsequently, the model based light field epipolar-plane image Unet is designed to obtain disparity information without any data augmentation techniques. By leveraging both binocular and light field optical paths, a robust mapping relationship of the disparity, depth, and visually coherent three-dimensional contour was established. Finally, the three-dimensional contours of laser powder bed fusion parts were reconstructed by the established mapping with a single exposure of the light field camera. The proposed method enables three-dimensional contour visualization of laser powder bed fusion parts from multi-view perspectives, which facilitates in situ monitoring of the spattering distance, three-dimensional spatial points and lines of the printed layers and contours of printed parts. With a calibration error of 50 μm and resolution of 180 μm for the light field camera (Lytro illum camera), the maximum relative and absolute errors between reconstructed three-dimensional spatial points were 4.33 % and 260 μm respectively when compared with theoretical dimension, and 0.5 % and 210 μm respectively when compared with manufactured dimension. The maximum root-mean-square error of reconstructed three-dimensional contour surfaces was found to be 1.55 mm. This study represents the first application of a light field camera for in situ three-dimensional reconstruction of laser powder bed fusion parts, which notably can be obtained from only a single round of calibration and exposure, enabling seamless monitoring of their surface morphology and facilitating online quality control. These advancements have vast potential for enhancing high-end additive manufacturing.

Light Field Depth Estimation via Stitched Epipolar Plane Images.

Depth estimation is a fundamental problem in light field processing. Epipolar-plane image (EPI)-based methods often encounter challenges such as low accuracy in slope computation due to discretization errors and limited angular resolution. Besides, existing methods perform well in most regions but struggle to produce sharp edges in occluded regions and resolve ambiguities in texture-less regions. To address these issues, we propose the concept of stitched-EPI (SEPI) to enhance slope computation. SEPI achieves this by shifting and concatenating lines from different EPIs that correspond to the same 3D point. Moreover, we introduce the half-SEPI algorithm, which focuses exclusively on the non-occluded portion of lines to handle occlusion. Additionally, we present a depth propagation strategy aimed at improving depth estimation in texture-less regions. This strategy involves determining the depth of such regions by progressing from the edges towards the interior, prioritizing accurate regions over coarse regions. Through extensive experimental evaluations and ablation studies, we validate the effectiveness of our proposed method. The results demonstrate its superior ability to generate more accurate and robust depth maps across all regions compared to state-of-the-art methods.

Blind Quality Evaluator of Light Field Images by Group-Based Representations and Multiple Plane-Oriented Perceptual Characteristics

Due to the emergency of multi-view cameras and commercial Light Field (LF) cameras, the demand of high-performance LF quality evaluator is of great significance for guiding LF acquisition, processing and application and further promoting the visual perceived quality of LF visualizations. However, LF Images (LFIs), as high-dimensional data, suffer from various quality degradations not only in the spatial domain but also in the angular domain. Therefore, it is of great challenge to predict LF quality accurately. An effective LF evaluator should be able to represent these heterogeneous artifacts. In this paper, we provide a novel No-Reference LF Quality Assessment Evaluator (NR LF-QAE) to tackle this problem. Firstly, to measure angular consistency among viewports, we utilize group-based representations to character information similarity of aligned view stacks. Secondly, to better describe the texture information of LFIs, unifying spatial-angular texture statistic measurement is performed via Local Binary Patterns from Three Orthogonal Planes (LBP-TOP). Thirdly, we design 3D Log-Gabor filters to extract LF global structure information in Sub-Aperture Images (SAIs) as spatial feature characterizations and 2D Log-Gabor filters are adopted to characterize ray direction/depth information in Epipolar Plane Images (EPIs) as angular feature characterizations. By comprehensive LF information analyses in angular consistency and spatial-angular feature extraction with texture and structure descriptors, experimental results demonstrate the superiority of the proposed NR LF-QAE over the state-of-the-art comparative models in predicting the quality of LFIs on three available benchmark databases. The code will be released at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/zerosola/NR-LF-QAE</uri> .

Learning Reliable Gradients From Undersampled Circular Light Field for 3D Reconstruction.

The paper presents a 3D reconstruction algorithm from an undersampled circular light field (LF). With an ultra-dense angular sampling rate, every scene point captured by a circular LF corresponds to a smooth trajectory in the circular epipolar plane volume (CEPV). Thus per-pixel disparities can be calculated by retrieving the local gradients of the CEPV-trajectories. However, the continuous curve will be broken up into discrete segments in an undersampled circular LF, which leads to a noticeable deterioration of the 3D reconstruction accuracy. We observe that the coherent structure is still embedded in the discrete segments. With less noise and ambiguity, the scene points can be reconstructed using gradients from reliable epipolar plane image (EPI) regions. By analyzing the geometric characteristics of the coherent structure in the CEPV, both the trajectory itself and its gradients could be modeled as 3D predictable series. Thus a mask-guided CNN+LSTM network is proposed to learn the mapping from the CEPV with a lower angular sampling rate to the gradients under a higher angular sampling rate. To segment the reliable regions, the reliable-mask-based loss that assesses the difference between learned gradients and ground truth gradients is added to the loss function. We construct a synthetic circular LF dataset with ground truth for depth and foreground/background segmentation to train the network. Moreover, a real-scene circular LF dataset is collected for performance evaluation. Experimental results on both public and self-constructed datasets demonstrate the superiority of the proposed method over existing state-of-the-art methods.

Visual Security Index Combining CNN and Filter for Perceptually Encrypted Light Field Images

Visual security index (VSI) represents a quantitative index for the visual security evaluation of perceptually encrypted images. Recently, the research on visual security of encrypted light field (LF) images faces two challenges. One is that the existing perceptually encrypted image databases are often too small, which is easy to cause overfitting in convolutional neural network (CNN). The other is that existing VSI models did not take a full account the intrinsic characteristics of the LF images and highly relied on handcrafted feature extraction. In this article, we construct a new database of perceptually encrypted LF images, called the PE-SLF, which is 2.6 times as big as the existing largest perceptual encrypted image database. Moreover, a novel visual security index (VSI) model is proposed by taking into full consideration the intrinsic spatial-angular characteristics of the LF images and the outstanding capabilities of CNN in feature extraction. First, we exploit CNN to detect the texture and structure features of encrypted sub-aperture images in the spatial domain. Second, we apply the Gabor filter to detect the Gabor feature over the epi-polar plane images in angular domain. Last, the spatial and angular similarity measurements are subsequently calculated for jointly yielding the final visual security score. Experimental results on the constructed PE-SLF demonstrate that the proposed VSI model is closer to the perception of HVS in visual security evaluation of encrypted LF images compared to other classical and state-of-the-art models.

Multi‐cues underwater image restoration algorithm combined with light field technology

AbstractUnderwater images suffer from color distortion, low clarity and halos problems due to light absorption, particle scattering and non‐uniform illumination. To address these degradation issues, a multi‐cues underwater image restoration algorithm combined with light field technology is proposed. First, based on Epipolar Plane Image, the light field cue transmittance containing depth information is calculated. Then, according to the turbidity of the underwater image, the light field cue transmittance and the polarization cue transmittance are fused to obtain the multi‐cues transmittance, which can effectively reduce the effect of particle scattering and color bias. Finally, the background light is estimated through the all‐focus operation, which can effectively overcome the distortion of an underwater single image and simultaneously reduce the halo phenomenon. Experimental results show that the method achieves the best results evaluated by UCIQE, UIQM, PSNR, and SSIM, and the restored color under the method is closer to the actual image than other underwater restoration methods.

No-Reference Light Field Image Quality Assessment Exploiting Saliency

In the near future, the broadcasting scenario will be characterized by immersive content. One of the systems for capturing the 3D content of a scene is the Light Field imaging. The huge amount of data and the specific transmission scenario impose strong constraints on services and applications. Among others, the evaluation of the quality of the received media cannot rely on the original signal but should be based only on the received data. In this direction, we propose a no-reference quality metric for light field images which is based on spatial and angular characteristics. In more details, the estimated saliency and cyclopean maps of light field images are exploited to extract the spatial features. The angular consistency features are, instead, measured with the use of the Global Luminance Distribution knowledge and the Weighted Local Binary Patterns operator on Epipolar Plane Images. The effectiveness of the proposed metric is assessed by comparing its performance with state-of-the-art quality metrics using 4 datasets: SMART, Win5-LID, VALID 10-bit, and VALID 8-bit. Furthermore, the performance is analyzed in cross-datasets, with different distortions, and for different saliency maps. The achieved results show that the performance of the proposed model outperforms state-of-the-art approaches and perform well for different distortion types and with various saliency models.

Learning light field macro-GEPI representation for semantic segmentation

2D optical imaging systems cause irreparable stereo visual feature loss in the pixel-level semantic segmentation task, especially in challenging scenes. To overcome this limitation, light field imaging with microlens arrays can acquire the 4D multi angle subaperture images, providing a viable alternative to semantic segmentation. However, these images are redundant and difficult to process. Therefore, we propose a macro generalized epipolar-plane image (macro-GEPI) representation. The macro-GEPI is obtained by slicing the subaperture images, stacking the slice images in sequence, and finally aligning them. This new image representation can clearly highlight the spatial-angular interactive information of different objects in one image. Based on this, we propose a convolutional neural network (CNN) that can independently learn the light field spatial-angular information for semantic segmentation. We also present a light field dataset composed of 1840 macro-GEPIs with annotations. We have studied the proposed network and compared it to state-of-the-art (SOTA) algorithms. The experiments showed that our network outperformed SOTA algorithms on the macro-GEPI dataset and a benchmark light field dataset, especially in challenging scenes. Moreover, our algorithm achieved a high mean intersection over union of 90.83%, demonstrating the reliability of the light field macro-GEPI representation and the effectiveness of our CNN algorithm.

Dense light field reconstruction based on epipolar focus spectrum

Existing light field (LF) representations, such as epipolar plane image (EPI) and sub-aperture images, do not consider the structural characteristics across the views, so they usually require additional disparity and spatial structure cues for follow-up tasks. Besides, they have difficulties dealing with occlusions or large disparity scenes. To this end, this paper proposes a novel Epipolar Focus Spectrum (EFS) representation by rearranging the EPI spectrum. Different from the classical EPI representation where an EPI line corresponds to a specific depth, there is a one-to-one mapping from the EFS line to the view. By exploring the EFS sampling task, the analytical function is derived for constructing a non-aliasing EFS. To demonstrate its effectiveness, we develop a trainable EFS-based pipeline for light field reconstruction, where a dense light field can be reconstructed by compensating the missing EFS lines given a sparse light field, yielding promising results with cross-view consistency, especially in the presence of severe occlusion and large disparity. Experimental results on both synthetic and real-world datasets demonstrate the validity and superiority of the proposed method over SOTA methods.

MFSR: Light Field Images Spatial Super Resolution Model Integrated with Multiple Features

Light Field (LF) cameras can capture angular and spatial information simultaneously, making them suitable for a wide range of applications such as refocusing, disparity estimation, and virtual reality. However, the limited spatial resolution of the LF images hinders their applicability. In order to address this issue, we propose an end-to-end learning-based light field super-resolution (LFSR) model called MFSR, which integrates multiple features, including spatial, angular, epipolar plane images (EPI), and global features. These features are extracted separately from the LF image and then fused together to obtain a comprehensive feature using the Feature Extract Block (FE Block) iteratively. Gradient loss is added into the loss function to ensure that the MFSR has good performance for LF images with rich texture. Experimental results on synthetic and real-world datasets demonstrate that the proposed method outperforms other state-of-the-art methods, with a peak signal-to-noise ratio (PSNR) improvement of 0.208 dB and 0.274 dB on average for the 2× and 4× super-resolution tasks, and structural similarity (SSIM) of both improvements of 0.01 on average.

Accurate light field depth estimation under occlusion

Epipolar plane images (EPIs) have advantages in light field depth estimation, but the current EPI-based methods only use one aspect of the information related to the line or its surroundings in EPIs. Moreover, most current methods merely extract the depth map of the target view, ignoring the depth information from other views. To fully utilize the available information, we first introduce a novel data cost by comprehensively utilizing the characteristics of pixel consistency on the line and region difference around the line in EPIs to improve the robustness against occlusion and noise. Then, we put forward a multi-view depth integration strategy that copes with weak texture and occlusion areas. Finally, an edging preserving filter is applied to further refine the depth map. Experiments on synthetic and real light field datasets show that the proposed method outperforms the state-of-the-art light field depth estimation algorithms, especially in the presence of occluded pixels.

Light Field Image Super-Resolution Network via Joint Spatial-Angular and Epipolar Information

This paper proposes a novel efficient convolutional neural network (CNN) for light field (LF) spatial super-resolution (SR). Due to the high dimensionality of LF data, it is important to decompose the original 4D LF data structure into a lower data subspace helping to reduce memory and computational costs. In this context, we propose a novel hybrid CNN structure working on different 2D data representations to fully handle the 4D structure. Our proposed network can simultaneously exploit information from three different types of 4D LF representations: spatial, angular, and epipolar plane image (EPI) information. Based on pixel similarities of each data subspace, we propose an inter-intra spatial feature extractor (II-SFE) and an inter-intra angular feature extractor (II-AFE) that can exploit the pixel correlations among the spatial-angular domain. Moreover, based on the correlated epipolar geometry among pixels, we also propose a multi-orientation epipolar feature extractor (MO-EFE) by decomposing an LF image into four orientation stacks of views and feeding them into different network branches to learn rich sub-pixel information in the EPI domain. The extracted features of each 2D subspace are integrated by our proposed attention fusion module (AFM), which adaptively rescales the embedding weights based on their correlations. Experimental results on both small- and wide-baseline benchmark LF datasets confirm the effectiveness of the proposed method compared to the well-known state-of-the-art methods in terms of objective and subjective performance.

Light-field angular super-resolution reconstruction via fusing 3D epipolar plane images

Light-field angular super-resolution reconstruction via fusing 3D epipolar plane images