Light Field Depth Estimation Research Articles

Light field depth estimation guided by 3D occlusion model

Light field depth estimation guided by 3D occlusion model

Light field depth estimation of fusing consistency and difference constraints

Light field depth estimation of fusing consistency and difference constraints

Occlusion Handling by Successively Excluding Foregrounds for Light Field Depth Estimation Based on Foreground-Background Separation

This paper proposes a depth from light field (DFLF) method specifically to deal with occlusion based on the foreground-background separation (FBS). The FBS-based methods infer the disparity maps by accumulating the binary maps which divide whether each pixel is the foreground or background. Although there have been widely studied to handle the occlusion problem with the cost-based method, there are not enough researches to handle the occlusion problem with the FBS-based methods yet. We found that errors around the occlusion boundary in the resulting disparity maps of the FBS-based methods arise from the fattened foreground by the light field reprameterization. To avoid fattened foregrounds, the inferred foreground maps in the front region with respect to the disparity axis could be utilized in the back region in the three-dimensional volume construction, which corresponds to the cost volume construction in the cost-based methods. With the front-to-back scanning manner of the FBS-based method, by successively excluding inferred foreground maps, errors around occlusion boundary could be effectively reduced in the resulting disparity maps. With synthetic and real LF images, the proposed method shows reasonable performance compared to the existing methods and better performance than existing FBS-based methods.

A Lightweight Depth Estimation Network for Wide-Baseline Light Fields.

Existing traditional and ConvNet-based methods for light field depth estimation mainly work on the narrow-baseline scenario. This paper explores the feasibility and capability of ConvNets to estimate depth in another promising scenario: wide-baseline light fields. Due to the deficiency of training samples, a large-scale and diverse synthetic wide-baseline dataset with labelled data is introduced for depth prediction tasks. Considering the practical goal for real-world applications, we design an end-to-end trained lightweight convolutional network to infer depths from light fields, called LLF-Net. The proposed LLF-Net is built by incorporating a cost volume which allows variable angular light field inputs and an attention module that enables to recover details at occlusion areas. Evaluations are made on the synthetic and real-world wide-baseline light fields, and experimental results show that the proposed network achieves the best performance when compared to recent state-of-the-art methods. We also evaluate our LLF-Net on narrow-baseline datasets, and it consequently improves the performance of previous methods.

Light-field-depth-estimation network based on epipolar geometry and image segmentation.

In this paper, we propose a convolutional neural network based on epipolar geometry and image segmentation for light-field depth estimation. Epipolar geometry is utilized to estimate the initial disparity map. Multi-orientation epipolar images are selected as input data, and the convolutional blocks are adopted based on the disparity of different-direction epipolar images. Image segmentation is used to obtain the edge information of the central sub-aperture image. By concatenating the output of the two parts, an accurate depth map could be generated with fast speed. Our method achieves a high rank on most quality assessment metrics in the HCI 4D Light Field Benchmark and also shows effectiveness in estimating accurate depth on real-world light-field images.

Structured-light-field 3D imaging without phase unwrapping

Directly using wrapped phases may cause a problem of phase ambiguity in fringe-projection-based three-dimensional (3D) imaging. In our previous work, the phase ambiguity was addressed by introducing light-field imaging to associate a wrapped phase-encoded field in the image space with several candidates of spatial points in the object space to identify the correct fringe orders for absolute phase unwrapping. The candidate spatial points with absolute phases can be reconstructed by satisfying the mandatory requirements of system fixation and pre-calibration, which may lack flexibility and applicability. Since we also illustrated that the correspondence cue in an unwrapped phase-encoded field could be used for accurate light-field depth estimation, an instructive question is whether or not depth cues of the light-field imaging can provide constraint conditions to deal with the phase ambiguity without these systematic requirements. To this end, a method for structured-light-field 3D imaging is proposed. The wrapped phase-encoded field was processed to be locally continuous with respect to the phases in specific angular coordinates. After that, the correspondence cue focusing on the local angular variance of the processed wrapped phase-encoded field was used to estimate depths correctly without the requirement of system pre-calibration. Furthermore, comparative experiments under different measurement conditions, including the changes of the fringe period of projected patterns, the focal length of the projector, and the relative orientation among system components, produced the consistent measurement results. Consequently, the proposed method is proved to adapt to the changes in the measurement environment and be a flexible way for structured-light-field 3D imaging without the need for phase unwrapping.

Light-field depth estimation considering plenoptic imaging distortion.

Light-field imaging can simultaneously record spatio-angular information of light rays to carry out depth estimation via depth cues which reflect a coupling of the angular information and the scene depth. However, the unavoidable imaging distortion in a light-field imaging system has a side effect on the spatio-angular coordinate computation, leading to incorrectly estimated depth maps. Based on the previously established unfocused plenoptic metric model, this paper reports a study on the effect of the plenoptic imaging distortion on the light-field depth estimation. A method of light-field depth estimation considering the plenoptic imaging distortion is proposed. Besides, the accuracy analysis of the light-field depth estimation was performed by using standard components. Experimental results demonstrate that efficiently compensating the plenoptic imaging distortion results in a six-fold improvement in measuring accuracy and more consistency across the measuring depth range. Consequently, the proposed method is proved to be suitable for light-field depth estimation and three-dimensional measurement with high quality, enabling unfocused plenoptic cameras to be metrological tools in the potential application scenarios such as industry, biomedicine, entertainment, and many others.

结合光场深度估计和大气散射模型的图像去雾方法

结合光场深度估计和大气散射模型的图像去雾方法

Zero-Shot Depth Estimation From Light Field Using A Convolutional Neural Network

This article proposes a zero-shot learning-based framework for light field depth estimation, which learns an end-to-end mapping solely from an input light field to the corresponding disparity map with neither extra training data nor supervision of groundtruth depth. The proposed method overcomes two major difficulties posed in existing learning-based methods and is thus much more feasible in practice. First, it saves the huge burden of obtaining groundtruth depth of a variety of scenes to serve as labels during training. Second, it avoids the severe domain shift effect when applied to light fields with drastically different content or captured under different camera configurations from the training data. On the other hand, compared with conventional non-learning-based methods, the proposed method better exploits the correlations in the 4D light field and generates much superior depth results. Moreover, we extend this zero-shot learning framework to depth estimation from light field videos. For the first time, we demonstrate that more accurate and robust depth can be estimated from light field videos by jointly exploiting the correlations across spatial, angular, and temporal dimensions. We conduct comprehensive experiments on both synthetic and real-world light field image datasets, as well as a self collected light field video dataset. Quantitative and qualitative results validate the superior performance of our method over the state-of-the-arts, especially for the challenging real-world scenes.

Fast Depth Estimation for Light Field Cameras

Fast depth estimation for light field images is an important task for multiple applications such as image-based rendering and refocusing. Most previous approaches to light field depth estimation involve high computational costs. Therefore, in this study, we propose a fast depth estimation method based on multi-view stereo matching for light field images. Similar to other conventional methods, our method consists of initial depth estimation and refinement. For the initial estimation, we use a one-bit feature for each pixel and calculate matching costs by summing all combinations of viewpoints with a fast algorithm. To reduce computational time, we introduce an offline viewpoint selection strategy and cost volume interpolation. Our refinement process solves the minimization problem in which the objective function consists of ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> data and smoothness terms. Although this problem can be solved via a graph cuts algorithm, it is computationally expensive; therefore, we propose an approximate solver based on a fast-weighted median filter. Experiments on synthetic and real-world data show that our method achieves competitive accuracy with the shortest computational time of all methods.

Anti-specular light-field depth estimation algorithm

Anti-specular light-field depth estimation algorithm

Depth Estimation From Light Field Using Graph-Based Structure-Aware Analysis

Existing light field depth map estimation approaches only utilize partial angular views in occlusion areas and local spatial dependencies in the optimization. This paper proposes a novel two-stage light field depth estimation method via graph spectral analysis to exploit the complete correlations and dependencies within angular patches and spatial images. The initial depth map estimation leverages the undirected graph to jointly consider occluded and unoccluded views within each angular patch. The estimated depth minimizes the structural incoherence of its corresponding angular patch with the focused one by evaluating the highest graph frequency component. Subsequently, depth map refinement optimizes the initial depth map with the color consistency and smoothness formulated by weighted adjacency matrix. The structural constraints are efficiently employed using low-pass graph filtering with Chebyshev polynomial approximation. Experimental results demonstrate that the proposed method improves the depth map estimation, especially in the edge regions.

A hybrid learning of multimodal cues for light field depth estimation

Learning based light field depth estimation methods have been proven to be successful, however, most of them focus on the depth cues extraction from a single representation of light field. In our opinion, existing methods are difficult to simultaneously learn various cues with different modalities, which are implied in different representations of light field. In this paper, we formulate the light field depth estimation as a pixel-wise classification task, and introduce a hybrid learning architecture to combine multimodal cues from multiple representations of light field. Three convolutional pathways are applied to predict the disparity label of each pixel from three modality representations of light field. Performing discrete disparity label classification instead of continuous disparity value regression allows us to easily fuse the prediction results of three pathways in the form of probability distribution. Evaluation experiments are carried out on the HCI 4D Light Field Benchmark. Our method, named as “FusionNet” on the benchmark website, ranks first among the published methods on high accuracy metrics of Bad Pixel Ratio 0.03 and 0.01.

Robust depth estimation for multi-occlusion in light-field images.

Occlusion is one of the most important issues in light-field depth estimation. In this paper, we propose a light-field multi-occlusion model with the analysis of light transmission. By the model, occlusions in different views are discussed separately. An adaptive algorithm of anti-occlusion in the central view is proposed to obtain more precise consistency regions (unoccluded views) in the angular domain and a subpatch approach of anti-occlusion in other views is presented to optimize the initial depth maps, where depth boundaries are better preserved. Then we propose a curvature confidence analysis approach to make depth evaluation more accurate and it is designed in an energy model to regularize the depth maps. Experimental results demonstrate that the proposed algorithm achieves better subjective and objective quality in depth maps compared with state-of-the-art algorithms.

Accurate depth estimation in structured light fields.

Passive light field imaging generally uses depth cues that depend on the image structure to perform depth estimation, causing robustness and accuracy problems in complex scenes. In this study, the commonly used depth cues, defocus and correspondence, were analyzed by using phase encoding instead of the image structure. The defocus cue obtained by spatial variance is insensitive to the global spatial monotonicity of the phase-encoded field. In contrast, the correspondence cue is sensitive to the angular variance of the phase-encoded field, and the correspondence responses across the depth range have single-peak distributions. Based on this analysis, a novel active light field depth estimation method is proposed by directly using the correspondence cue in the structured light field to search for non-ambiguous depths, and thus no optimization is required. Furthermore, the angular variance can be weighted to reduce the depth estimation uncertainty according to the phase encoding information. The depth estimation of an experimental scene with rich colors demonstrated that the proposed method could distinguish different depth regions in each color segment more clearly, and was substantially improved in terms of phase consistency compared to the passive method, thus verifying its robustness and accuracy.

Accurate Light Field Depth Estimation Using Multi-Orientation Partial Angular Coherence

Occlusion is a critical issue that affects the accuracy for light field depth estimation. In the presence of occlusions, the photo-consistency property is broken, making ambiguity near occlusion areas. In this paper, we proposed using multi-orientation partial angular coherence to achieve accurate depth estimation by which the occlusions are explicitly treated. Unlike previous approaches, the correspondence is partially measured along several lines with different directions. We show that depending on the occlusion edge orientation, the pixels along the occlusion orientation still preserve photo-consistency. By computing the partial angular coherence in four potential directions, we can obtain a sharp initial depth map. Since occlusions are precisely tackled, the outliers can be mostly removed by fast guided filtering on the cost volume. As a result, the depth accuracy at occlusion boundaries is greatly improved. The proposed method can obtain sharp transition edges at occlusion boundaries and has no requirement for extra edge information. Experimental results on a recent benchmark demonstrate that the proposed method outperforms the state-of-the-art algorithms in the accuracy metrics. Further experiments on real-world scenes also show the superior performance of the proposed method.

Reduction of Aliasing Artifacts by Sign Function Approximation in Light Field Depth Estimation Based on Foreground–Background Separation

A sign function approximation method for depth from light field (DFLF) based on the foreground–background separation (FBS) is proposed. From a signal processing viewpoint, the FBS-based method can be considered as a bridge between the cost-based DFLF methods and depth model-based ones. The proposed sign function approximation corresponds to the winner-takes-all (WTA) approaches as the cost-based methods do. Experimental results on the synthetic images show that the proposed method reasonably performs in terms of the mean squared error as the state-of-the-art methods do. Especially, by using a suitable WTA method in the framework of our previous FBS-based DFLF work, the proposed method effectively reduces the angular aliasing artifacts in the resulting disparity maps of both the synthetic and real images.

Cost aggregation benchmark for light field depth estimation

Light field depth estimation has become a mature research topic and there are numerous algorithms introduced by various research groups. However, comprehensive and fair benchmark is difficult to apply because there are large step variances of the introduced algorithms. It is essential to analyze each step in the light field depth estimation so that it could help design better and more robust algorithms. Thus, a thorough analysis of cost aggregation is conducted in this paper to analyze the performance of various cost aggregation methods on light field depth estimation. A study on the parameter setting for each cost aggregation method is performed. Then, each cost aggregation with its optimal parameters is evaluated individually. Instead of using the standard rank system, this paper utilizes the weighted rank system based on the score difference on each criterion. Experimental results confirm that the guided-filter based method outperforms other methods in most evaluation criteria.

Guided filtering based data fusion for light field depth estimation with [formula omitted] gradient minimization

In this paper, we propose guided filtering based data fusion for light field depth estimation with L0 gradient minimization. Stereo disparity produces good depth edge, while defocus response yields good depth information in homogeneous regions. We fuse stereo disparity and defocus response from light filed data in a guided filtering framework. In the guided filtering framework, we adopt L0 gradient minimization as the regularization term instead of penalizing linear coefficients to consider depth characteristics that have similar depth in the same object. Moreover, we utilize edge direction in stereo matching to prevent the confusion caused by occlusion. Experimental results on both synthetic and real light field datasets show that the proposed method achieves clearer edge and less error in depth than state-of-the-arts.

Occlusion Aware Reduced Angular Candidates based Light Field Depth Estimation from an Epipolar Plane Image

Occlusion Aware Reduced Angular Candidates based Light Field Depth Estimation from an Epipolar Plane Image