Accurate Disparity Map Research Articles

Stereo matching algorithm using deep learning and edge-preserving filter for machine vision

Machine vision research began with a single-camera system, but these systems had various limitations from having just one point-of-view of the environment and no depth information, therefore stereo cameras were invented. This paper proposes a hybrid method of a stereo matching algorithm with the goal of generating an accurate disparity map critical for applications such as 3D surface reconstruction and robot navigation to name a few. Convolutional neural network (CNN) is utilised to generate the matching cost, which is then input into cost aggregation to increase accuracy with the help of a bilateral filter (BF). Winner-take-all (WTA) is used to generate the preliminary disparity map. An edge-preserving filter (EPF) is applied to that output based on a transform that defines an isometry between curves on the 2D image manifold in 5D and the real line to eliminate these artefacts. The transform warps the input signal adaptively to allow linear 1D filtering. Due to the filter's resistance to high contrast and brightness, it is effective in refining and removing noise from the output image. Based on experimental research employing a Middlebury standard validation benchmark, this approach gives high accuracy with an average non-occluded error of 6.71% comparable to other published methods.

Event-Image Fusion Stereo Using Cross-Modality Feature Propagation

Event cameras asynchronously output the polarity values of pixel-level log intensity alterations. They are robust against motion blur and can be adopted in challenging light conditions. Owing to these advantages, event cameras have been employed in various vision tasks such as depth estimation, visual odometry, and object detection. In particular, event cameras are effective in stereo depth estimation to find correspondence points between two cameras under challenging illumination conditions and/or fast motion. However, because event cameras provide spatially sparse event stream data, it is difficult to obtain a dense disparity map. Although it is possible to estimate disparity from event data at the edge of a structure where intensity changes are likely to occur, estimating the disparity in a region where event occurs rarely is challenging. In this study, we propose a deep network that combines the features of an image with the features of an event to generate a dense disparity map. The proposed network uses images to obtain spatially dense features that are lacking in events. In addition, we propose a spatial multi-scale correlation between two fused feature maps for an accurate disparity map. To validate our method, we conducted experiments using synthetic and real-world datasets.

A Comparison and Evaluation of Stereo Matching on Active Stereo Images.

The relationship between the disparity and depth information of corresponding pixels is inversely proportional. Thus, in order to accurately estimate depth from stereo vision, it is important to obtain accurate disparity maps, which encode the difference between horizontal coordinates of corresponding image points. Stereo vision can be classified as either passive or active. Active stereo vision generates pattern texture, which passive stereo vision does not have, on the image to fill the textureless regions. In passive stereo vision, many surveys have discovered that disparity accuracy is heavily reliant on attributes, such as radiometric variation and color variation, and have found the best-performing conditions. However, in active stereo matching, the accuracy of the disparity map is influenced not only by those affecting the passive stereo technique, but also by the attributes of the generated pattern textures. Therefore, in this paper, we analyze and evaluate the relationship between the performance of the active stereo technique and the attributes of pattern texture. When evaluating, experiments are conducted under various settings, such as changing the pattern intensity, pattern contrast, number of pattern dots, and global gain, that may affect the overall performance of the active stereo matching technique. Through this evaluation, our discovery can act as a noteworthy reference for constructing an active stereo system.

Robust Scale-Aware Stereo Matching Network

Recently, deep convolutional neural networks (CNNs) have emerged as powerful tools for the correspondence problem in stereo matching task. However, the existence of multiscale objects and inevitable ill-conditioned regions, such as textureless regions, in real-world scene images continue to challenge current CNN architectures. In this article, we present a robust scale-aware stereo matching network, which aims to predict multiscale disparity maps and fuse them to achieve a more accurate disparity map. To this end, powerful feature representations are extracted from stereo images and are concatenated into a 4-D feature volume. The feature volume is then fed into a series of connected encoder–decoder cost aggregation structures for the construction of multiscale cost volumes. Following this, we regress multiscale disparity maps from the multiscale cost volumes and feed them into a fusion module to predict final disparity map. However, uncertainty estimations at each scale and complex disparity relationships among neighboring pixels pose a challenge on the disparity fusion. To overcome this challenge, we design a robust learning-based scale-aware disparity map fusion model, which seeks to map multiscale disparity maps onto the ground truth disparity map by leveraging their complementary strengths. Experimental results show that the proposed network is more robust and outperforms recent methods on standard stereo evaluation benchmarks.

A novel cell structure‐based disparity estimation for unsupervised stereo matching

It is well known that preserving depth edges is an effective solution for achieving the accurate disparity map in stereo matching, but many state-of-the-art methods do not preserve depth edges well. In order to solve it efficiently, the cell structure containing irregular and regular shape regions is designed to preserve depth edges. Based on the well-designed cell structure, a novel disparity estimation method for stereo matching is proposed, in which a two-layer disparity optimization method is proposed to refine the disparity plane; it includes the front-parallel disparities computation and slanted-surfaces disparity plane refinement. In the framework of front-parallel disparities computation, a tree-based cost aggregation method is presented to make full use of the segmentation information of cells and then performing semi-global cost aggregation. In the framework of slanted-surfaces disparity plane refinement, a new probability model is proposed that employs Bayesian inference for refining disparities in textureless, weak texture and occluded regions. Experimental results show that higher accuracy could be achieved via the proposed method compared with some known state-of-the-art stereo methods on KITTI 2015 and Middlebury dataset, which are the standard benchmarks for testing the stereo matching methods. It can also be indicated that the proposed method can produce accurate disparity map and have good generalization performance.

LESC: Superpixel cut‐based local expansion for accurate stereo matching

The rapid estimation of the accurate disparity between pixels is the goal of stereo matching. However, it is very difficult for the 3D labels-based methods due to huge search space of 3D labels, especially for high-resolution images. In this paper, a novel superpixel cut-based method is proposed, in an attempt to get the accurate disparity map efficiently, including the multi-layer superpixel optimization and iteractive local α-expansion in parallel. As for the multi-layer superpixel optimization, feature point optimization is designed to get accurate candidate labels that are set for most pixels using non-local cost aggregation strategy and update per-pixel labels of the corresponding superpixels from the candidate label sets on the small-size superpixel layer, and then update the middle to large-size superpixel layers progressively using non-local cost aggregation strategy. In order to provide more prior information to identify weak texture and textureless regions in non-local cost aggregation, the weight combination of “intensity + gradient + binary image” is proposed for constructing an optimal minimum spanning tree (MST) to calculate the aggregated matching cost and obtain the labels of minimum aggregated matching cost. Moreover, the local patch surrounding the corresponding superpixels is designed to accelerate superpixel optimization in parallel, and a neighborhood structure is presented to optimize the algorithm in this study, including superpixel neighborhood and patch neighborhood. As for the iteractive local α-expansion, three layers of patch structure corresponding to the superpixel neighborhood structure is proposed for optimizing the algorithm in this study. The experimental results show that higher accuracy could be achieved via the method in this study compared with some known state-of-the-art stereo methods on KITTI 2015 and Middlebury benchmark V3, which are the standard benchmarks for testing the stereo matching methods.

Hybrid cost aggregation for dense stereo matching

Matching cost initialization and aggregation are two major steps in the stereo matching framework. For dense stereo matching, a matching cost needs to be computed at each pixel for all disparities within the search range so that it can be used to evaluate pixel-to-pixel correspondence. Cost aggregation connects the matching cost with a certain neighbourhood to reduce mismatches by a supporting smoothness term. This paper presents a hybrid cost aggregation method to overcome mismatches caused by textureless surface, depth-discontinuity areas, inconsistent lightings in an image. The steps taken to aggregate costs for an energy function include adaptive support regions, multi-path aggregation, and adaptive penalties to generate a more accurate disparity map. Compared with two top-ranked stereo matching algorithms, the proposed algorithm yielded the disparity maps of the dataset in Middlebury benchmark V2 with smaller error ratios in depth-discontinuity regions.

Unsupervised deep learning for depth estimation with offset pixels.

Offset Pixel Aperture (OPA) camera has been recently proposed to estimate disparity of a scene with a single shot. Disparity is obtained in the image by offsetting the pixels by a fixed distance. Previously, correspondence matching schemes have been used for disparity estimation with OPA. To improve disparity estimation we use a data-oriented approach. Specifically, we use unsupervised deep learning to estimate the disparity in OPA images. We propose a simple modification to the training strategy which solves the vanishing gradients problem with the very small baseline of the OPA camera. Training degenerates to poor disparity maps if the OPA images are used directly for left-right consistency check. By using images obtained from displaced cameras at training, accurate disparity maps are obtained. The performance of the OPA camera is significantly improved compared to previously proposed single-shot cameras and unsupervised disparity estimation methods. The approach provides 8 frames per second on a single Nvidia 1080 GPU with 1024×512 OPA images. Unlike conventional approaches, which are evaluated in controlled environments, our paper shows the utility of deep learning for disparity estimation with real life sensors and low quality images. By combining OPA with deep learning, we obtain a small depth sensor capable of providing accurate disparity at usable frame rates. Also the ideas in this work can be used in small-baseline stereo systems for short-range depth estimation and multi-baseline stereo to increase the depth range.

Stereo matching for infrared images using guided filtering weighted by exponential moving average

Infrared imaging is less susceptible to illumination conditions and haze than visible light imaging. The advantage makes infrared sensing suitable for providing remote visibility with reduced distortion. However, infrared images tend to have low resolution and lack rich textures that facilitate stereo matching. To enhance the applicability of infrared stereo imaging, the authors re-examine the guided-image techniques to include advanced edge-aware filters for aggregation and propose a novel guided-image filtering scheme here. Based on the exponential moving average, the weights are recursively calculated such that all pixels on the infrared image pair can contribute to a discrepancy cost. The arrangement allows additional pixels to be involved in the cost aggregation to reduce the demand for rich texture. Experimental results using the colour and thermal stereo (CATS) benchmark testbed demonstrate that the proposed approach outperforms several state-of-art approaches in generating accurate disparity maps.

An end-to-end stereo matching algorithm based on improved convolutional neural network.

Deep end-to-end learning based stereo matching methods have achieved great success as witnessed by the leaderboards across different benchmarking datasets. Depth information in stereo vision systems are obtained by a dense and accurate disparity map, which is computed by a robust stereo matching algorithm. However, previous works adopt network layer with the same size to train the feature parameters and get an unsatisfactory efficiency, which cannot be satisfied for the real scenarios by existing methods. In this paper, we present an end-to-end stereo matching algorithm based on "downsize" convolutional neural network (CNN) for autonomous driving scenarios. Firstly, the road images are feed into the designed CNN to get the depth information. And then the "downsize" full-connection layer combined with subsequent network optimization is employed to improve the accuracy of the algorithm. Finally, the improved loss function is utilized to approximate the similarity of positive and negative samples in a more relaxed constraint to improve the matching effect of the output. The loss function error of the proposed method for KITTI 2012 and KITTI 2015 datasets are reduced to 2.62 and 3.26% respectively, which also reduces the runtime of the proposed algorithm. Experimental results illustrate that the proposed end-to-end algorithm can obtain a dense disparity map and the corresponding depth information can be used for the binocular vision system in autonomous driving scenarios. In addition， our method also achieves better performance when the size of the network is compressed compared with previous methods.

Multiple data cost-based stereo matching method to generate dense disparity maps from images under radiometric variations

Stereo matching algorithms are capable of providing dense 3D information of the environment through two images taken simultaneously from a pair of cameras placed parallel to each other. Obtaining an accurate disparity map from a stereo image pair is a challenging task and also computationally expensive. If we take into consideration the environmental effect, then the difficulty of the task increases drastically. The authors try to overcome this problem by combining multiple stereo cost functions in the form of a linear equation. To reduce the computation time, a segmentation based cost aggregation method is followed in an attempt to produce an accurate disparity map even in the presence of radiometric variations in the images. The performance of the proposed algorithm is observed while varying the parameter 'α' between the cost functions and the number of segments in the images. Different image pairs from the Middlebury stereo dataset were considered here.

Multiple data cost-based stereo matching method to generate dense disparity maps from images under radiometric variations

Stereo matching algorithms are capable of providing dense 3D information of the environment through two images taken simultaneously from a pair of cameras placed parallel to each other. Obtaining an accurate disparity map from a stereo image pair is a challenging task and also computationally expensive. If we take into consideration the environmental effect, then the difficulty of the task increases drastically. The authors try to overcome this problem by combining multiple stereo cost functions in the form of a linear equation. To reduce the computation time, a segmentation based cost aggregation method is followed in an attempt to produce an accurate disparity map even in the presence of radiometric variations in the images. The performance of the proposed algorithm is observed while varying the parameter ‘’ between the cost functions and the number of segments in the images. Different image pairs from the Middlebury stereo dataset were considered her

Segment-Based Disparity Refinement With Occlusion Handling for Stereo Matching.

In this paper, we propose a disparity refinement method that directly refines the winner-take-all (WTA) disparity map by exploring its statistical significance. According to the primary steps of the segment-based stereo matching, the reference image is over-segmented into superpixels and a disparity plane is fitted for each superpixel by an improved random sample consensus (RANSAC). We design a two-layer optimization to refine the disparity plane. In the global optimization, mean disparities of superpixels are estimated by Markov random field (MRF) inference, and then, a 3D neighborhood system is derived from the mean disparities for occlusion handling. In the local optimization, a probability model exploiting Bayesian inference and Bayesian prediction is adopted and achieves second-order smoothness implicitly among 3D neighbors. The two-layer optimization is a pure disparity refinement method because no correlation information between stereo image pairs is demanded during the refinement. Experimental results on the Middlebury and KITTI datasets demonstrate that the proposed method can perform accurate stereo matching with a faster speed and handle the occlusion effectively. It can be indicated that the "matching cost computation + disparity refinement" framework is a possible solution to produce accurate disparity map at low computational cost.

An Efficient Dense Stereo Matching Method for Planetary Rover

Stereo matching is one of the most important and challenging subjects in the field of planetary rover with a stereo vision system. The generated disparity map can make rover to avoid the obstacle and explore the planetary surface automatically. In this paper, we propose an efficient dense stereo matching method to generate disparity maps for a planetary rover, which relies on 3-D plane fitting, adaptive penalties, and coarse-to-fine disparity constraint. In order to achieve efficient stereo matching at the coarsest level of the pyramid, we present a 3-D plane fitting to reduce the disparity search range and propose adaptive penalties in the more-global matching method for obtaining an accurate disparity map. At the finer level, our method then estimates the disparity search range based on coarse-to-fine disparity constraint and utilizes adaptive penalties to obtain an accurate disparity map. The extensive experiments with stereo images of Chang'e-3 rover demonstrate that our approach can generate disparity maps efficiently and accurately compared with the most state-of-the-art semi-global matching methods, especially at low texture regions, depth discontinuities, and occlusion regions.

Depth Estimation From a Light Field Image Pair With a Generative Model

In this paper, we propose a novel method to estimate the disparity maps from a light field image pair captured by a pair of light field cameras. Our method integrates two types of critical depth cues, which are separately inferred from the epipolar plane images and binocular stereo vision into a global solution. At the same time, in order to produce highly accurate disparity maps, we adopt a generative model, which can estimate a light field image only with the central subaperture view and corresponding hypothesized disparity map. The objective function of our method is formulated to minimize two energy terms/differences. One is the difference between the two types of previously extracted disparity maps and the target disparity maps, directly optimized in the gray-scale disparity space. The other indicates the difference between the estimated light field images and the input light field images, optimized in the RGB color space. Comprehensive experiments conducted on real and virtual scene light field image pairs demonstrate the effectiveness of our method.

Segmentation-Based Color Channel Registration for Disparity Estimation of Dual Color-Filtered Aperture Camera.

Single-lens-based optical range finding systems were developed as an efficient, compact alternative for conventional stereo camera systems. Among various single-lens-based approaches, a multiple color-filtered aperture (MCA) system can generate disparity information among color channels, as well as normal color information. In this paper, we consider a dual color-filtered aperture (DCA) system as the most minimal version of the MCA system and present a novel inter-color image registration algorithm for disparity estimation. This proposed registration algorithm consists of three steps: (i) color channel independent feature extraction; (ii) feature-based adaptive weight disparity estimation; and (iii) color mapping matrix (CMM)-based cross-channel image registration. Experimental results show that the proposed method can not only generate an accurate disparity map, but also realize high quality cross-channel registration with a disparity prior for DCA-based range finding and color image enhancement.

Learning a convolutional neural network for propagation-based stereo image segmentation

Stereo image segmentation is the key technology in stereo image editing with the population of stereoscopic 3D media. Most previous methods perform stereo image segmentation on both views relying primarily on per-pixel disparities, which results in the segmentation quality closely connected to the accuracy of the disparities. Therefore, a mechanism to remove the errors of the disparities are highly demanded. To date, there’s no such a method yet that can produce accurate disparity maps. In this paper, we propose a novel convolutional neural network (CNN)-based framework, which will automatically propagate the segmentation result from one view to the other. The key problem of accurate stereo image segmentation is the missing of occluded regions. To solve this problem, the CNN architecture is proposed to improve the stereo segmentation performance. In order to address the inevitable inaccuracies problem of the disparities computed from a stereo pair of images, we utilize the coherent disparity propagation that propagates segment result via those pixels with coherent disparities. The pixels by coherent disparity propagation and the high confidence pixels of the object probability map produced by the CNN architecture are then used to generate the initial reliable pixels to perform an energy minimization framework-based segmentation. A comprehensive evaluations and comparisons on Middlebury and Adobe benchmark datasets show the effectiveness of our proposed method in terms of high-quality results, and the robustness against various types of inputs.

Deep self-guided cost aggregation for stereo matching

In this paper, we present a deep self-guided cost aggregation method used to obtain an accurate disparity map from a pair of stereo images. Conventional cost aggregation methods typically perform joint image filtering on each cost volume slice. Thus, a guidance image is necessary for the conventional methods to work effectively. However, a guidance image might be unreliable due to several distortions, such as noise, blur, radiometric variation. Based on our observations, each cost volume slice can guide itself based on the internal features. However, finding a direct mapping function from the initial and filtered cost volume slice without any guidance image is difficult. To solve this problem, we use an advanced deep learning technique to perform self-guided cost aggregation. Because of the absence of ground truth cost volume, we offer the solution for the dataset generation. Our proposed deep learning network consists of two sub-networks: dynamic weight network and descending filtering network. We integrate the feature reconstruction loss and the pixelwise mean square loss function to preserve the edge property. Experimental results show that the proposed method achieves better results even though it does not employ a guidance image.

An efficient stereo matching based on fragment matching

We propose a stereo matching method based on image fragments. Unlike traditional pixel-based stereos matching methods, we use edge information in the reference image to divide it into small fragments, and we then use the segments to find the best matching fragments in another reference image from the horizontal and vertical directions. We obtain two disparity maps, and using the match confidence value for each disparity map, we can produce a more accurate disparity map. Next, we calculate the exact disparity value for each pixel within the fragment. Finally, the disparity map is filled and smoothed to obtain the final disparity result. Experiments demonstrated that the proposed method has low computation complexity, high matching accuracy, and the disparity of object edge is clear, and it achieved good performance with the Middlebury and KITTI benchmark.

Stereo Matching via Dual Fusion

We present a two-fold fusion framework that constructs comprehensive cost volumes for stereo matching. To this end, we develop fusion schemes at two key steps, i.e., a) the raw cost computation and b) the cost aggregation. Specifically, we commence by fusing both structure- and data-oriented features as raw costs. We then incorporate the guided filtered costs into the cross-based cost aggregation and obtain the fused aggregated costs. The fusion schemes at the two steps effectively complement each other and result in an accurate disparity map. Experiments on the Middlebury benchmark v3 demonstrate the state-of-the-art performance of our framework in terms of various metrics.