Dense Disparity Estimation Research Articles

EnhancedNet, an End-to-End Network for Dense Disparity Estimation and its Application to Aerial Images

Recent developments in deep learning technology have boosted the performance of dense stereo reconstruction. However, the state-of-the-art deep learning-based stereo matching methods are mainly trained using close-range synthetic images. Consequently, the application of these methods in aerial photogrammetry and remote sensing is currently far from straightforward. In this paper, we propose a new disparity estimation network for stereo matching and investigate its generalization abilities in regard to aerial images. First, we propose an end-to-end deep learning network for stereo matching, regularized by disparity gradients, which includes a residual cost volume and a reconstruction error volume in a refinement module, and multiple losses. In order to investigate the influence of the multiple losses, a comprehensive analysis is presented. Second, based on this network trained with synthetic close-range data, we propose a new pipeline for matching high-resolution aerial imagery. The experimental results show that the proposed network improves the disparity accuracy by up to 40% in terms of errors larger than 1 px compared to results when not including the refinement network, especially in areas containing detailed small objects. In addition, in qualitative and quantitative experiments, we are able to show that our model, pre-trained on a synthetic stereo dataset, achieves very competitive sub-pixel geometric accuracy on aerial images. These results confirm that the domain gap between synthetic close-range and real aerial images can be satisfactorily bridged using the proposed new deep learning method for dense image matching.

Active Disparity Sampling for Stereo Matching With Adjoint Network.

The sparse signals provided by external sources have been leveraged as guidance for improving dense disparity estimation. However, previous methods assume depth measurements to be randomly sampled, which restricts performance improvements due to under-sampling in challenging regions and over-sampling in well-estimated areas. In this work, we introduce an Active Disparity Sampling problem that selects suitable sampling patterns to enhance the utility of depth measurements given arbitrary sampling budgets. We achieve this goal by learning an Adjoint Network for a deep stereo model to measure its pixel-wise disparity quality. Specifically, we design a hard-soft prior supervision mechanism to provide hierarchical supervision for learning the quality map. A Bayesian optimized disparity sampling policy is further proposed to sample depth measurements with the guidance of the disparity quality. Extensive experiments on standard datasets with various stereo models demonstrate that our method is suited and effective in different stereo architectures and outperforms existing fixed and adaptive sampling methods under different sampling rates. Remarkably, the proposed method makes substantial improvements when generalized to heterogeneous unseen domains.

Two-stage cross-fusion network for stereo event-based depth estimation

Drawing inspiration from biological retina, event camera uses dynamic vision sensor to asynchronously capture pixel intensity changes. Event camera provides a series of advantages, including high dynamic range, high temporal resolution, lack of motion blur, and low power consumption, which make it suitable candidate for depth estimation, specially in challenging scenarios with fast motion and low-lighting conditions. However, due to the sparse nature of event data, it is troublesome to estimate dense disparity map from them. Although it is plausible to estimate disparity map from edges of a structure from where most events are captured, estimating dense disparity is challenging in the areas where events rarely occur. In this study, we develop a stereo matching network for dense disparity estimation, which effectively fuses reconstructed image features with event to estimate disparity. To this end, we propose a two-stage cross-fusion network (TCFNet) to estimate disparity by fusing event and image features with two cascaded fusion mechanisms. First, we propose cross-SPADE fusion, based on the spatially-adaptive denormalization mechanism (SPADE), which modulates event features with image features, and image features with event features. In the second stage, the modulated features are further fused using the cross-gating mechanism, which uses the multi-axis gated MLP (gMLP). Experiments on both real and synthetic datasets indicate that the proposed two-stage cross-fusion network outperforms the state-of-the-art event-based stereo matching networks empirically and visually.

Light-field spectral decomposition with a spatial–angular consistency prior for disparity estimation

Modeling the spatial–angular relationship is the theoretical basis for accurate disparity estimation from the light-field data. In this paper, the spatial–angular coupling relationship of light-field is established under the assumption of constant radiance along rays, and the spatial–angular consistency model of light-field is established under the assumption of Lambertian radiance. Based on the spatial–angular consistency prior of light-field, the light-field spectral decomposition based on disparity layers can be modeled as the ill-posed inverse problem. Based on the spectral decomposition, the disparity estimation is transformed into the process of searching for the spatial domain support set in the scene disparity hierarchy. The normalized cross-correlation function is utilized to estimate the disparity based on spectral decomposition under the spatial–angular consistency of the light-field. The performance on both simulated and real captured data show that the proposed algorithm is effective at high-precision and dense disparity estimation.

IMPROVING DISPARITY ESTIMATION BASED ON RESIDUAL COST VOLUME AND RECONSTRUCTION ERROR VOLUME

Abstract. Recently, great progress has been made in formulating dense disparity estimation as a pixel-wise learning task to be solved by deep convolutional neural networks. However, most resulting pixel-wise disparity maps only show little detail for small structures. In this paper, we propose a two-stage architecture: we first learn initial disparities using an initial network, and then employ a disparity refinement network, guided by the initial results, which directly learns disparity corrections. Based on the initial disparities, we construct a residual cost volume between shared left and right feature maps in a potential disparity residual interval, which can capture more detailed context information. Then, the right feature map is warped with the initial disparity and a reconstruction error volume is constructed between the warped right feature map and the original left feature map, which provides a measure of correctness of the initial disparities. The main contribution of this paper is to combine the residual cost volume and the reconstruction error volume to guide training of the refinement network. We use a shallow encoder-decoder module in the refinement network and do learning from coarse to fine, which simplifies the learning problem. We evaluate our method on several challenging stereo datasets. Experimental results demonstrate that our refinement network can significantly improve the overall accuracy by reducing the estimation error by 30% compared with our initial network. Moreover, our network also achieves competitive performance compared with other CNN-based methods.

Parallel framework for dense disparity map estimation using Hamming distance

A novel framework for sparse and dense disparity estimation was designed, and the proposed framework has been implemented in CPU and GPU for a parallel processing capability. The Census transform is applied in the first stage, and then, the Hamming distance is later used as similarity measure in the stereo matching stage followed by a matching consistency check. Next, a disparity refinement is performed on the sparse disparity map via weighted median filtering and color K-means segmentation, in addition to clustered median filtering to obtain the dense disparity map. The results are compared with state-of-the-art frameworks, demonstrating this process to be competitive and robust. The quality criteria used are structural similarity index measure and percentage of bad pixels (B) for objective results and subjective perception via human visual system demonstrating better performance in maintaining fine features in disparity maps. The comparisons include processing times and running environments, to place each process into context.

A learned sparseness and IGMRF-based regularization framework for dense disparity estimation using unsupervised feature learning

In this work, we propose a new approach for dense disparity estimation in a global energy minimization framework. We propose to use a feature matching cost which is defined using the learned hierarchical features of given left and right stereo images and we combine it with the pixel-based intensity matching cost in our energy function. Hierarchical features are learned using the deep deconvolutional network which is trained in an unsupervised way using a database consisting of large number of stereo images. In order to perform the regularization, we propose to use the inhomogeneous Gaussian Markov random field (IGMRF) and sparsity priors in our energy function. A sparse autoencoder-based approach is proposed for learning and inferring the sparse representation of disparities. The IGMRF prior captures the smoothness as well as preserves sharp discontinuities while the sparsity prior captures the sparseness in the disparity map. Finally, an iterative two-phase algorithm is proposed to estimate the dense disparity map where in phase one, sparse representation of disparities are inferred from the trained sparse autoencoder, and IGMRF parameters are computed, keeping the disparity map fixed and in phase two, the disparity map is refined by minimizing the energy function using graph cuts, with other parameters fixed. Experimental results on the Middlebury stereo benchmarks demonstrate the effectiveness of the proposed approach.

Robust and dense depth estimation for light field images.

We propose a depth estimation method for light field images. Light field images can be considered as a collection of 2D images taken from different viewpoints arranged in a regular grid. We exploit this configuration and compute the disparity maps between specific pairs of views. This computation is carried out by a state of the art two-view stereo method providing a non dense disparity estimation. We propose a disparity interpolation method increasing the density and improving the accuracy of this initial estimate. Disparities obtained from several pairs of views are fused to obtain a unique and robust estimation. Finally, different experiments on synthetic and real images show how the proposed method outperforms state of the art results.

User assisted disparity remapping for stereo images

This study concentrates on user assisted disparity remapping for stereo image footage, i.e. the disparity of an object of interest is altered while leaving the remaining scene unattended. This application is useful in the sense that it provides a method for emphasizing/de-emphasizing an object on the scene by adjusting its depth with respect to the camera. The proposed technique can also be used as a post-processing step for retargeting stereoscopic footage on different display sizes and resolutions. The proposed technique involves an MRF-based energy minimization step for interactive stereo image segmentation, for which user assistance on only one of the stereo pairs is required for determining the location of stereo object pair. A key contribution of the proposed study is elimination of dense disparity estimation step from the pipeline. This step is realized through a sparse feature matching technique between the stereo pairs. Moreover, by the help of the proposed technique, novel disparity adjusted views are synthesized using the produced stereo object segments and background information for the images. Qualitative and quantitative evaluation of the generated segments and the disparity adjusted images prove the functionality and superiority of the proposed technique.

Stereo image reconstruction using regularized adaptive disparity estimation

In this paper, stereo image reconstruction using regularized adaptive disparity estimation is proposed. That is, by adaptively predicting the mutual correlation between stereo images pair using the proposed algorithm, the bandwidth of stereo input images pair can be compressed to the level of a conventional 2D image and a predicted image also can be effectively reconstructed using a reference image and disparity vectors. Especially, in the proposed algorithm, the first feature values are extracted from the input stereo images pair. Then, a matching window for stereo matching is adaptively selected depending on the magnitude of these feature values. That is, for the region having larger feature values, a smaller matching window is selected, while, for the opposite case, a larger matching window is selected by comparing predetermined threshold values. This approach is not only able to reduce a mismatching of disparity vectors, which occurs in the conventional dense disparity estimation with a small matching window, but is also able to reduce blocking effects that occur in the coarse disparity estimation with a large matching window. In addition, in this paper, a new regularized adaptive disparity estimation technique is proposed. That is, by regularizing the estimated disparity vector with the neighboring disparity vectors, problems of the conventional adaptive disparity estimation scheme might be solved, and the predicted stereo image can be more effectively reconstructed. From experiments using stereo sequences of “Man”, “Fichier”, “Manege”, and “Tunnel”, it is shown that the proposed algorithm improves the PSNRs of a reconstructed image to about 6.90 dB on average at ±30 search ranges as compared to those of conventional algorithms. Also, it is found that there is almost no difference between an original image and a reconstructed image through the proposed algorithm by comparison to that of conventional algorithms.

Regularized Stereo Matching Scheme using Adaptive Disparity Estimation

In this paper, a regularized stereo matching scheme using adaptive disparity estimation is proposed. That is, by adaptively predicting the mutual correlation between stereo image pair using the proposed algorithm, the bandwidth of an input stereo image pair can be compressed to the level of a conventional two-dimensional (2D) image and a predicted image can also be effectively reconstructed using a reference image and disparity vectors. In the adaptive disparity estimation method, feature values are extracted from an input stereo image pair and a matching window for stereo matching is adaptively selected depending on the magnitude of the feature values. Accordingly, this adaptive window matching algorithm (AMA) could improve the overall performance of stereo matching by reducing the mismatch of disparity vectors, which occurs in the conventional dense disparity estimation with a small matching window, and also by reducing a blocking effect, which occurs in the coarse disparity estimation with a large matching window. However, it still has problems of overlap and disallocation of matching windows. Therefore, to alleviate those problems, a new regularized adaptive disparity estimation method is proposed in this paper. That is, the estimated disparity vector is regularized with the neighboring disparity vectors; as a result, the predicted stereo image is found to be more effectively reconstructed. Some experiments with stereo sequences of “Man”, “Hoon”, “Yong”, and “Car” reveal that the proposed algorithm improves the peak signal-to-noise ratios (PSNRs) of the reconstructed images by 8.47 and 1.53 dB, on average, compared with those in the cases of feature and pixel-based window matching algorithm (FPMA) and AMA, respectively.

Real-Time Dense Stereo for Intelligent Vehicles

Stereo vision is an attractive passive sensing technique for obtaining three-dimensional (3-D) measurements. Recent hardware advances have given rise to a new class of real-time dense disparity estimation algorithms. This paper examines their suitability for intelligent vehicle (IV) applications. In order to gain a better understanding of the performance and the computational-cost tradeoff, the authors created a framework of real-time implementations. This consists of different methodical components based on single instruction multiple data (SIMD) techniques. Furthermore, the resulting algorithmic variations are compared with other publicly available algorithms. The authors argue that existing publicly available stereo data sets are not very suitable for the IV domain. Therefore, the authors' evaluation of stereo algorithms is based on novel realistically looking simulated data as well as real data from complex urban traffic scenes. In order to facilitate future benchmarks, all data used in this paper is made publicly available. The results from this study reveal that there is a considerable influence of scene conditions on the performance of all tested algorithms. Approaches that aim for (global) search optimization are more affected by this than other approaches. The best overall performance is achieved by the proposed multiple-window algorithm, which uses local matching and a left-right check for a robust error rejection. Timing results show that the simplest of the proposed SIMD variants are more than twice as fast than the most complex one. Nevertheless, the latter still achieves real-time processing speeds, while their average accuracy is at least equal to that of publicly available non-SIMD algorithms

One-dimensional dense disparity estimation for three-dimensional reconstruction

We present a method for fully automatic three-dimensional (3D) reconstruction from a pair of weakly calibrated images in order to deal with the modeling of complex rigid scenes. A two-dimensional (2D) triangular mesh model of the scene is calculated using a two-step algorithm mixing sparse matching and dense motion estimation approaches. The 2D mesh is iteratively refined to fit any arbitrary 3D surface. At convergence, each triangular patch corresponds to the projection of a 3D plane. The proposed algorithm relies first on a dense disparity field. The dense field estimation modelized within a robust framework is constrained by the epipolar geometry. The resulting field is then segmented according to homographic models using iterative Delaunay triangulation. In association with a weak calibration and camera motion estimation algorithm, this 2D planar model is used to obtain a VRML-compatible 3D model of the scene.

Computer Vision for a 3-D Visualisation and Telepresence Collaborative Working Environment

An integrated media system is a computer-based environment that supports the creation, sharing, distribution and effective communication of multimodal information across the boundaries of space and time. The EU Information Societies Technology (IST) project — VIRTUE (Virtual Team User Environment) is working steadily towards the realisation of most aspects and properties of such a system, with particular emphasis on a three-way semi-immersive telepresence videoconferencing scenario. In contrast with the traditional videoconferencing system that we know now, the outcome of the project is expected to demonstrate distinctive presence features and experience for the conference participants. These include views of full-body-size realistically rendered images, eye-to-eye contact, gaze awareness, normal hand gesturing and direct body language. The purpose of this paper is to describe the current work in its related technical field, and the main objectives and scope of this project. One optional software system framework is outlined, and also illustrated are some component technologies in 3-D computer vision analysis that are being developed. The application of these component technologies, notably the dense-disparity estimation and the novel view synthesis, in 3-D interactive video manipulation and visualisation, are widely expected.