Light Field Depth Estimation Research Articles

Enhanced light field depth estimation through occlusion refinement and feature fusion

Light field depth estimation is crucial for various applications, but current algorithms often falter when dealing with complex textures and edges. To address this, we propose a light field depth estimation network based on multi-scale fusion and channel attention (LFMCNet). It incorporates a convolutional multi-scale fusion module to enhance feature extraction and utilizes a channel attention mechanism to refine depth map accuracy. Additionally, LFMCNet integrates the Transformer Feature Fusion Module (TFFM) and Channel Attention-Based Perspective Fusion (CAPF) module for improved occlusion refinement, effectively handling challenges in occluded regions. Testing on the 4D HCI and real-world datasets demonstrates that LFMCNet significantly reduces the Bad Pixel (BP) rate and Mean Square Error (MSE).

Edge enhancement and feature modulation based network for light field depth estimation

Estimating depth from light field images is a critical issue in light field applications. While learning-based methods have made significant strides in light field depth estimation, achieving high accuracy and speed simultaneously remains a major challenge. This paper proposes a light field depth estimation network based on edge enhancement and feature modulation, which significantly improves depth estimation results by emphasizing inter-view correlations while preserving image edge features. Specifically, to prioritize edge details, we introduce an Edge-Enhanced Cost Constructor (EECC) that integrates edge information with existing cost constructors to improve depth estimation performance in complex areas. Furthermore, most light field depth estimation networks utilize only sub-aperture images (SAIs) without considering the inherent angular information in macro-pixel image (MacPI). To address this limitation, we propose the MacPI-Guided Feature Modulation (MGFM) module, which leverages angular information between different views in MacPI to modulate features at each view. Experimental results show that our method not only performs excellently on synthetic datasets but also demonstrates outstanding generalization on real-world datasets, achieving a better balance between accuracy and computation speed.

Occlusion-Aware Unsupervised Light Field Depth Estimation Based on Multi-Scale GANs

Occlusion-Aware Unsupervised Light Field Depth Estimation Based on Multi-Scale GANs

Unsupervised Light Field Depth Estimation via Multi-View Feature Matching With Occlusion Prediction

Unsupervised Light Field Depth Estimation via Multi-View Feature Matching With Occlusion Prediction

Triple Consistency for Transparent Cheating Problem in Light Field Depth Estimation

Triple Consistency for Transparent Cheating Problem in Light Field Depth Estimation

Multi-scale foreground-background separation for light field depth estimation with deep convolutional networks

There are three viewpoints to divide the depth from light field (DFLF) methods: light field (LF) representation, signal processing, and learning viewpoints. In the signal processing viewpoint in details, the DFLF methods can be divided into three groups: cost-based, foreground-background separation (FBS)-based, and depth model-based methods. Although the deep learning techniques are widely implemented in the cost-based and depth model-based methods for the DFLF, they are not implemented in the FBS-based method, yet. In this paper, we propose a DFLF method based on the FBS with the deep learning techniques. Based on an observation that the angular aliasing artifacts are reduced in the resulting disparity maps obtained from spatially low-scale LF images, multi-scale (MS) FBS score maps are used in the proposed methods. To combine the MS FBS score maps into a single FBS score maps, four directional LF gradients, features from the 4-D LF, and features from the center view images are utilized. Based on the sensitivity and specificity loss, the gradient matching loss between the predicted and ground truth disparity maps is used to improve the performance around the occlusion boundary in the resulting disparity maps. The experimental results show that the proposed method performs reasonably in both the synthetic and real LF datasets. Especially, in the texture-less region in the real dataset, the resulting disparity maps obtained by the proposed method show that the artifacts are reduced compared to those by the existing DFLF methods.

Dynamic depth estimation of weakly textured objects based on light field speckle projection and adaptive step length of optical flow method

Repetitive textures in the depth estimation of weakly textured surfaces lead to numerous errors. The existing technology does not provide a complete method for light-field depth estimation in weak-texture scenes. Therefore we developed an algorithm that estimates the cost volume using speckle stereo correspondences. The subpixel-matching result is obtained using the adaptive label step length displacement theorem. The proposed method can adaptively select an appropriate step length for the sub-pixel phase-shift theorem without depending on the dataset category. The build cost volume can be adapted for weakly textured scenes. A common dataset was used to evaluate the effectiveness of the optical flow method for the adaptive selection of step length. Finally, high-precision step gauge blocks with a step difference of 10 mm were used to verify the depth-direction errors, which were less than 0.35 mm.

P‐2.29: Depth Estimation of Light Field Image Based on Compressed Sensing and Multi‐clue Fusion

In the current research field of light field depth estimation, the occlusion of complex scenes and a large amount of computational data are the problems that every researcher must face. For complex occlusion scenes, this paper proposes a depth estimation method based on the fusion of adaptive defocus cues and constrained angular entropy cues, which is more robust to occlusion. At the same time, the compressed sensing theory is used to compress and reconstruct the light field image to solve the problem of a large amount of data in the process of light field image acquisition, transmission, and processing. The experimental results show that the proposed method has a good overall effect in dealing with the depth estimation of the occlusion scene, and the correct depth information can be obtained. The light field image reconstructed by compressed sensing can not only obtain good depth estimation results but also reduce the amount of data effectively.

Attention-based encoder–decoder network for depth estimation from color-coded light fields

Compressive light field cameras have attracted notable attention over the past few years because they can efficiently determine redundancy from light fields. However, much of the research has only concentrated on reconstructing the entire light field from compressed sampling, which ignores the possibility of directly extracting information such as depth from it. In this paper, we introduce a light field camera configuration with a random color-coded microlens array. Considering the color-coded light fields, we propose a novel attention-based encoder–decoder network. Specifically, the encoder part compresses the coded measurement into a low-dimensional representation that removes most redundancy, and the decoder part constructs the depth map directly from the latent representation. The attention mechanism enables the network to process spatial and angular features dynamically and effectively, thus significantly improving performance. Extensive experiments on synthetic and real-world datasets show that our method outperforms the state-of-the-art light field depth estimation method designed for non-coded light fields. To our knowledge, this is the first study that combines the color-coded light field with the attention-based deep learning approach, which provides a crucial insight into the design of enhanced light field photography systems.

Accurate light field depth estimation under occlusion

Epipolar plane images (EPIs) have advantages in light field depth estimation, but the current EPI-based methods only use one aspect of the information related to the line or its surroundings in EPIs. Moreover, most current methods merely extract the depth map of the target view, ignoring the depth information from other views. To fully utilize the available information, we first introduce a novel data cost by comprehensively utilizing the characteristics of pixel consistency on the line and region difference around the line in EPIs to improve the robustness against occlusion and noise. Then, we put forward a multi-view depth integration strategy that copes with weak texture and occlusion areas. Finally, an edging preserving filter is applied to further refine the depth map. Experiments on synthetic and real light field datasets show that the proposed method outperforms the state-of-the-art light field depth estimation algorithms, especially in the presence of occluded pixels.

Decoupled and Reparameterized Compound Attention-Based Light Field Depth Estimation Network

Decoupled and Reparameterized Compound Attention-Based Light Field Depth Estimation Network

Exploiting Sequence Analysis for Accurate Light-Field Depth Estimation

Exploiting Sequence Analysis for Accurate Light-Field Depth Estimation

Occlusion-aware light field depth estimation with view attention

Depth estimation for light field images is crucial in light field applications such as image-based rendering and refocusing. Previous learning-based methods com bining neural network with cost volume can achieve accurate depth estimation but fail in regions with occlusion. In this paper, a two-stage attention-based occlusion-aware light field depth estimation network is proposed. In the initial depth estimation stage, the sub-aperture images are divided into four groups based on different view directions and four initial cost volumes are constructed using the feature maps from each group to aggregate initial depth maps. Then in the refined depth estimation stage, the four aggregated volumes from the initial stage are fused into one based on the view attention, where features of views with less occlusion are highly weighted to provide more effective information. Experiment results demonstrate that the proposed method can accomplish robust and accurate depth estimation in the presence of occlusion, which ranks the first place on 4D light field benchmark in terms of most accuracy metrics.

EPI Light Field Depth Estimation Based on a Directional Relationship Model and Multiviewpoint Attention Mechanism.

Light field (LF) image depth estimation is a critical technique for LF-related applications such as 3D reconstruction, target detection, and tracking. The refocusing property of LF images provide rich information for depth estimations; however, it is still challenging in cases of occlusion regions, edge regions, noise interference, etc. The epipolar plane image (EPI) of LF can effectively deal with the depth estimation because of its characteristics of multidirectionality and pixel consistency—in which the LF depth estimations are converted to calculate the EPI slope. This paper proposed an EPI LF depth estimation algorithm based on a directional relationship model and attention mechanism. Unlike the subaperture LF depth estimation method, the proposed method takes EPIs as input images. Specifically, a directional relationship model was used to extract direction features of the horizontal and vertical EPIs, respectively. Then, a multiviewpoint attention mechanism combining channel attention and spatial attention is used to give more weight to the EPI slope information. Subsequently, multiple residual modules are used to eliminate the redundant features that interfere with the EPI slope information—in which a small stride convolution operation is used to avoid losing key EPI slope information. The experimental results revealed that the proposed algorithm outperformed the compared algorithms in terms of accuracy.

Unsupervised learning of light field depth estimation with spatial and angular consistencies

Learning-based depth estimation from light fields has made significant advances in recent years, however, most of these work abandon the traditional non-learning based formulations and start over with an end-to-end deep network framework. Actually traditional methods have also presented many effective and evident depth cues or constraints for light field depth estimation. In this paper, we try to combine the advantages of the learning and non-learning strategies, and incorporate the traditional light field constraints into an unsupervised framework by using a learnable approximation scheme to yield differentiable unsupervised loss functions. Specifically, we first propose an unsupervised coarse-to-fine network architecture for light field depth estimation, and then design an adaptive spatio-angular consistency loss combined with the differentiable versions of modified traditional constraints. Comparative experiments demonstrate that our method is superior to the state-of-the-art unsupervised methods.

Nonlinear Optimization of Light Field Point Cloud.

The problem of accurate three-dimensional reconstruction is important for many research and industrial applications. Light field depth estimation utilizes many observations of the scene and hence can provide accurate reconstruction. We present a method, which enhances existing reconstruction algorithm with per-layer disparity filtering and consistency-based holes filling. Together with that we reformulate the reconstruction result to a form of point cloud from different light field viewpoints and propose a non-linear optimization of it. The capability of our method to reconstruct scenes with acceptable quality was verified by evaluation on a publicly available dataset.

Occlusion-Aware Unsupervised Learning of Depth From 4-D Light Fields.

Depth estimation is a fundamental issue in 4-D light field processing and analysis. Although recent supervised learning-based light field depth estimation methods have significantly improved the accuracy and efficiency of traditional optimization-based ones, these methods rely on the training over light field data with ground-truth depth maps which are challenging to obtain or even unavailable for real-world light field data. Besides, due to the inevitable gap (or domain difference) between real-world and synthetic data, they may suffer from serious performance degradation when generalizing the models trained with synthetic data to real-world data. By contrast, we propose an unsupervised learning-based method, which does not require ground-truth depth as supervision during training. Specifically, based on the basic knowledge of the unique geometry structure of light field data, we present an occlusion-aware strategy to improve the accuracy on occlusion areas, in which we explore the angular coherence among subsets of the light field views to estimate initial depth maps, and utilize a constrained unsupervised loss to learn their corresponding reliability for final depth prediction. Additionally, we adopt a multi-scale network with a weighted smoothness loss to handle the textureless areas. Experimental results on synthetic data show that our method can significantly shrink the performance gap between the previous unsupervised method and supervised ones, and produce depth maps with comparable accuracy to traditional methods with obviously reduced computational cost. Moreover, experiments on real-world datasets show that our method can avoid the domain shift problem presented in supervised methods, demonstrating the great potential of our method. The code will be publicly available at https://github.com/jingjin25/LFDE-OccUnNet.

Depth Estimation from Light Field Geometry Using Convolutional Neural Networks.

Depth estimation based on light field imaging is a new methodology that has succeeded the traditional binocular stereo matching and depth from monocular images. Significant progress has been made in light-field depth estimation. Nevertheless, the balance between computational time and the accuracy of depth estimation is still worth exploring. The geometry in light field imaging is the basis of depth estimation, and the abundant light-field data provides convenience for applying deep learning algorithms. The Epipolar Plane Image (EPI) generated from the light-field data has a line texture containing geometric information. The slope of the line is proportional to the depth of the corresponding object. Considering the light field depth estimation as a spatial density prediction task, we design a convolutional neural network (ESTNet) to estimate the accurate depth quickly. Inspired by the strong image feature extraction ability of convolutional neural networks, especially for texture images, we propose to generate EPI synthetic images from light field data as the input of ESTNet to improve the effect of feature extraction and depth estimation. The architecture of ESTNet is characterized by three input streams, encoding-decoding structure, and skipconnections. The three input streams receive horizontal EPI synthetic image (EPIh), vertical EPI synthetic image (EPIv), and central view image (CV), respectively. EPIh and EPIv contain rich texture and depth cues, while CV provides pixel position association information. ESTNet consists of two stages: encoding and decoding. The encoding stage includes several convolution modules, and correspondingly, the decoding stage embodies some transposed convolution modules. In addition to the forward propagation of the network ESTNet, some skip-connections are added between the convolution module and the corresponding transposed convolution module to fuse the shallow local and deep semantic features. ESTNet is trained on one part of a synthetic light-field dataset and then tested on another part of the synthetic light-field dataset and real light-field dataset. Ablation experiments show that our ESTNet structure is reasonable. Experiments on the synthetic light-field dataset and real light-field dataset show that our ESTNet can balance the accuracy of depth estimation and computational time.

Occlusion-aware light field depth estimation using side window angular coherence.

Depth estimation is crucial in many light field applications. However, the accuracy of light field depth estimation is prone to be affected by occlusions. In this paper, a method of side window angular coherence is proposed to handle different types of occlusions, and the ability of the proposed method to resist occlusions is theoretically analyzed. The angular patch is divided into several discrete side window subsets. These subsets are a pure occluder-type subset, a pure object point-type subset, and a hybrid-type subset. The photo-consistency of the pure object point-type subset can reflect the true depth. Meanwhile, the occlusion edges can be detected to identify occluded points and nonoccluded points so the robustness of the algorithm can be further enhanced by processing the two types of points. Moreover, fast guided filtering is applied to cost volume for improving the accuracy of depth estimation. Experimental results demonstrate that our method outperforms the state-of-the-art depth estimation methods on both synthetic and real scenes, especially near occlusion boundaries.

A Novel Occlusion-Aware Vote Cost for Light Field Depth Estimation.

Capturing the directions of light by light field cameras powers next-generation immersive multimedia applications. A critical problem in taking advantage of the rich visual information in light field images is depth estimation. Conventional light field depth estimation methods build a cost volume that measures the photo-consistency of pixels refocused to a range of depths, and the highest consistency indicates the correct depth. This strategy works well in most regions but usually generates blurry edges in the estimated depth map due to occlusions. Recent work shows that integrating occlusion models to light field depth estimation can largely reduce blurry edges. However, existing occlusion handling methods rely on complex edge-aided processing and post-refinement, and this reliance limits the resultant depth accuracy and impacts on the computational performance. In this paper, we propose a novel occlusion-aware vote cost (OAVC) which is able to accurately preserve edges in the depth map. Instead of using photo-consistency as an indicator of the correct depth, we construct a novel cost from a new perspective that counts the number of refocused pixels whose deviations from the central-view pixel are less than a small threshold, and utilizes that number to select the correct depth. The pixels from occluders are thus excluded in determining the correct depth. Without the use of any explicit occlusion handling methods, the proposed method can inherently preserve edges and produces high-quality depth estimates. Experimental results show that the proposed OAVC outperforms state-of-the-art light field depth estimation methods in terms of depth estimation accuracy and computational complexity.