Dense Stereo Matching Algorithm Research Articles

Matching cost function analysis and disparity optimization for low-quality binocular images

State-of-the-art dense stereo matching algorithms have achieved excellent performance, demonstrating a capability to attain precise matching in most areas. However, current such methods rarely achieve this when images are captured under poor conditions. To improve the accuracy of the algorithm in such cases, this paper introduces a post-optimization algorithm to rectify matching errors and enhance outcomes. The main research areas of this paper include three aspects. (1) Disparities are classified into reliable and unreliable results based on the analysis of geometric matching relationships, local features in the images, and components within the matching cost function; (2) Subsequent analysis of horizontal image features identifies local characteristic indices calculated through integration along the horizontal axis, which establish specific matching criteria, forming the foundation for a cost volume that encompasses these distinct matches; (3) A redefined matching cost function is applied to previously classified unreliable results to rectify matching errors. This energy function is based on the cost volume above. Experimental results validate the efficacy of the proposed post-optimization algorithm, reducing the average matching errors from 8.66% to 5.85%.

CONTINUOUS 3D-LABEL SEMI-GLOBAL MATCHING FOR SATELLITE STEREO

Abstract. We present Continuous 3D-Label Semi-Global Matching (CoSGM), a new dense stereo matching algorithm for satellite stereo. CoSGM overparameterizes the disparity map as an array of local planes, so it can regularize the first-order smoothness of the estimated disparity. Furthermore, the algorithm can produce a dense normal map beside the disparity map. We show experimentally that CoSGM could achieve denser depth maps with comparable accuracy to traditional semi-global matching algorithms.

Salient object detection via joint perception of region-level spatial distribution and color contrast

Nowadays, many RGB-D saliency detection models have been proposed, and they are conducted based on the pixel-level depth data obtained by active depth sensors or dense stereo matching algorithms in advance. However, the pixel-level depth maps may have a negative impact on saliency ranking, especially when they are inconsistent or invalid within the object area. It can be found that people tend to observe an object as a whole and ignore the details of the target surface. Inspired by this characteristic, we propose a salient object detection method via joint perception of region-level spatial distribution and color contrast. First, a twice segmentation strategy based on multifeature fusion is used to compute the region-level information. Then, the region-level spatial distribution maps are constructed instead of pixel-level depth maps, which is helpful for avoiding the interference of dense depth information. To improve the integrity of object detection, color saliency maps are also computed based on regional segmentation information. After that, we adopt a fusion strategy to achieve the effective complementarity of the two kinds of information. Two optimization strategies are employed to further improve the results of saliency ranking. Experimental results on two benchmark datasets demonstrate that the proposed method has better performance than most of the state-of-the-art methods, and it also shows competitive capability compared with deep learning-based methods.

An automatized method to parameterize embedded stereo matching algorithms

Many applications rely on 3D information as a depth map. Stereo Matching algorithms reconstruct a depth map from a pair of stereoscopic images. Stereo Matching algorithms are computationally intensive, that is why implementing efficient stereo matching algorithms on embedded systems is very challenging for real-time applications. Indeed, like many vision algorithms, stereo matching algorithms have to set a lot of parameters and thresholds to work efficiently. When optimizing a stereo-matching algorithm, or changing algorithms parts, all those parameters have to be set manually. Finding the most efficient solution for a stereo-matching algorithm on a specific platform then becomes troublesome. This paper proposes an automatized method to find the optimal parameters of a dense stereo matching algorithm by learning from ground truth on a database in order to compare it with respect to any other alternative. Finally, for the C6678 platform, a map of the best compromise between quality and execution time is obtained, with execution times that are between 42 ms and 382 ms and output errors that are between 6% and 9.8%.

A comparison of stereo matching algorithms on multi-core Digital Signal Processor platform

Stereo Matching algorithms reconstruct a depth map from a pair of stereoscopic images. Stereo Matching algorithms are computationally intensive. Implementing efficient stereo matching algorithms on embedded systems is very challenging. This paper compares implementation efficiency and output quality of the state of the art dense stereo matching algorithms on the same multicore embedded system. The three different classes of stereo matching algorithms are local methods, semi-global methods and global methods. This paper compares three algorithms of the literature with a good trade-off between complexity and accuracy : Bilateral Filtering Aggregation (BFA, Local Method), One Dimension Belief Propagation (BP-1D, Semi Global Methods) and Semi Global Matching (SGM, Semi Global Methods). For the same input data the BFA, BP-1D and SGM were fully optimized and parallelized on the C6678 platform and run at respectively 10.7 ms, 4.1 ms and 47.1 ms.

A DETAILED STUDY ABOUT DIGITAL SURFACE MODEL GENERATION USING HIGH RESOLUTION SATELLITE STEREO IMAGERY

Photogrammetry is currently in a process of renaissance, caused by the development of dense stereo matching algorithms to provide very dense Digital Surface Models (DSMs). Moreover, satellite sensors have improved to provide sub-meter or even better Ground Sampling Distances (GSD) in recent years. Therefore, the generation of DSM from spaceborne stereo imagery becomes a vivid research area. This paper presents a comprehensive study about the DSM generation of high resolution satellite data and proposes several methods to implement the approach. The bias-compensated Rational Polynomial Coefficients (RPCs) Bundle Block Adjustment is applied to image orientation and the rectification of stereo scenes is realized based on the Project-Trajectory-Based Epipolarity (PTE) Model. Very dense DSMs are generated from WorldView-2 satellite stereo imagery using the dense image matching module of the C/C++ library LibTsgm. We carry out various tests to evaluate the quality of generated DSMs regarding robustness and precision. The results have verified that the presented pipeline of DSM generation from high resolution satellite imagery is applicable, reliable and very promising.

A DETAILED STUDY ABOUT DIGITAL SURFACE MODEL GENERATION USING HIGH RESOLUTION SATELLITE STEREO IMAGERY

Abstract. Photogrammetry is currently in a process of renaissance, caused by the development of dense stereo matching algorithms to provide very dense Digital Surface Models (DSMs). Moreover, satellite sensors have improved to provide sub-meter or even better Ground Sampling Distances (GSD) in recent years. Therefore, the generation of DSM from spaceborne stereo imagery becomes a vivid research area. This paper presents a comprehensive study about the DSM generation of high resolution satellite data and proposes several methods to implement the approach. The bias-compensated Rational Polynomial Coefficients (RPCs) Bundle Block Adjustment is applied to image orientation and the rectification of stereo scenes is realized based on the Project-Trajectory-Based Epipolarity (PTE) Model. Very dense DSMs are generated from WorldView-2 satellite stereo imagery using the dense image matching module of the C/C++ library LibTsgm. We carry out various tests to evaluate the quality of generated DSMs regarding robustness and precision. The results have verified that the presented pipeline of DSM generation from high resolution satellite imagery is applicable, reliable and very promising.

Automatic registration of airborne LiDAR point cloud data and optical imagery depth map based on line and points features

Airborne light detection and ranging (LiDAR) technology draws increasing interest in large scale 3D urban modeling in recent years. In this paper, we propose a novel automatic registration method based on depth map using point-feature based and line-feature based registration ways to carry on the processing. First of all, the paper applies the mathematical morphology filter to preprocess point cloud data. Secondarily the depth maps of optical imagery and point cloud data are generated from adaptive support weight dense stereo matching algorithm and Delaunay triangulation algorithm, respectively. After that, point-feature based registration and line-feature based registration method are carried on. The point-feature based registration employs scale-invariant feature transform (SIFT) to generate feature correspondence between the two depth maps. Thereafter the line-feature based registration extracts line features thus to choose the angle and the length ratio of the intersecting line segments as similaritymeasure for coarse registration. In addition, outliers are removed with a two-level random sample consensus (RANSAC) algorithm to improve robustness and efficiency. And we can get accuracy estimation of camera position parameters. Texture and color mapping are achieved in the last step. The registration methods both can be completed automatically, and do not require GPS/INS prior knowledge. The proposed methods are efficient, which are validated by the experimental results tested with real data.

A Novel Image Representation via Local Frequency Analysis for Illumination Invariant Stereo Matching.

In this paper, we propose a novel image representation approach to tackle illumination variations in stereo matching problems. Images are mapped using their Fourier transforms which are convolved with a set of monogenic filters. Frequency analysis is carried out at different scales to account for most image content. The phase congruency and the local weighted mean phase angle are then computed over all the scales. The original image is transformed into a new representation using these two mappings. This representation is invariant to illumination and contrast variations. More importantly, it is generic and can be used with most sparse as well as dense stereo matching algorithms. In addition, sequential feature matching or tracking can also benefit from our approach in varying radiometric conditions. We demonstrate the improvements introduced with our image mappings on well-established data sets in the literature as well as on our own experimental scenarios that include high dynamic range imagery. The experiments include both dense and sparse stereo and sequential matching algorithms where the latter is considered in the very challenging visual odometry framework.

用于弱纹理场景三维重建的机器人视觉系统

To realize the obstacle avoidance and automatic navigation of a robot in a low texture environment,an active stereo visual system consisting a binocular camera and a compact laser projector was established.The dense stereo matching algorithm was investigated.Firstly,the compact laser projector generated the spot patterns with excellent uniqueness and anti-noise performance for increasing the texture information.Then,an adaptive-window matching algorithm was proposed based on Integral Grayscale Variance(IGSV)and Integral Gradient Variance(IGV).The algorithm was used calculate the integral variance in a matching window using the integral image obtained by the left image.If it was greater than the variance threshold,the correlation between the left and right image pixelswas calculated to get the dense disparity maps.Experimental results show that the vision system accurately gets the 3Ddense scene around the robot and the 3D reconstruction accuracy is 0.16 mm,which is suitable for the obstacle avoidance and automatic navigation.As compared with the traditional methods,the computation cost of dense matching has at least decreased by 93%since the computation used for image variance could not increase with the size of the matching window.

Optimized stereo matching in binocular three-dimensional measurement system using structured light.

In this paper, we develop an optimized stereo-matching method used in an active binocular three-dimensional measurement system. A traditional dense stereo-matching algorithm is time consuming due to a long search range and the high complexity of a similarity evaluation. We project a binary fringe pattern in combination with a series of N binary band limited patterns. In order to prune the search range, we execute an initial matching before exhaustive matching and evaluate a similarity measure using logical comparison instead of a complicated floating-point operation. Finally, an accurate point cloud can be obtained by triangulation methods and subpixel interpolation. The experiment results verify the computational efficiency and matching accuracy of the method.

Efficient interscanline consistency enforcing method for dynamic programming-based dense stereo matching algorithms

The dynamic programming-based dense stereo matching algorithms are faster than other global ones, but they usually suffer from the tangible horizontal “streaks” in their generated disparity maps. This problem takes place due to the independent processing of scanlines, ignoring vertical consistency between them. In this paper the path remembering method, which uses the information of the best paths of previous rows, is employed with new developments to establish more depth similarity between rows, especially along the slanted edges of objects in the scene. Also, a new way for estimating matching costs in the left border of the image is presented based on the information of the right-hand neighboring pixels. Mentioned improvements were applied to a well known and standard dynamic programming-based algorithm called “DP.” Evaluation of the new proposed algorithm on standard stereo images shows considerable improvements in the depth estimation along the slanted edges and in the left border of the final results. As a more precise report, the average error rate reduces from 14.2% to 10.4%.

Accurate Real-Time Disparity Map Computation Based on Variable Support Window

We propose in this paper a real-time dense stereo matching algorithm using variable support window for disparity map computing. Basic real-time local algorithms relying on a fixed and rectangular correlation window suffer from the difficulty for window-based methods lies in determining the best window shape and size for each pixel. This work proposes a novel local approach using a combination of the DSI (Disparity Space Image) structure and gradient information. Two improvements are introduced so that an accurate and fast result will be reached. The first one concerns the proposition of a new strategy in order to optimize the computation time of the initial disparity map. The second one, concerns the pixel similarity measure for matching score computation and it consists to use in addition to the traditional pixel intensities, the magnitude and orientation of the gradients providing more accuracy. Experimental results on real data sets are conducted and a comparative evaluation of the obtained results relatively to the state-of-art methods is presented.

Real-time stereo on GPGPU using progressive multi-resolution adaptive windows

We introduce a new GPGPU-based real-time dense stereo matching algorithm. The algorithm is based on a progressive multi-resolution pipeline which includes background modeling and dense matching with adaptive windows. For applications in which only moving objects are of interest, this approach effectively reduces the overall computation cost quite significantly, and preserves the high definition details. Running on an off-the-shelf commodity graphics card, our implementation achieves a 36 fps stereo matching on 1024 × 768 stereo video with a fine 256 pixel disparity range. This is effectively same as 7200 M disparity evaluations per second. For scenes where the static background assumption holds, our approach outperforms all published alternative algorithms in terms of the speed performance, by a large margin. We envision a number of potential applications such as real-time motion capture, as well as tracking, recognition and identification of moving objects in multi-camera networks.

Modeling Dynamic 3D Environments by Means of The Stixel World

Correlation-based stereo vision has proven its power in commercially available driver assistance systems. Recently, real-time dense stereo matching algorithms have become available on inexpensive low-power FPGA hardware. In order to manage that huge amount of data, a medium-level representation named “Stixel World” has been proposed for further analysis. In this representation the free space in front of the vehicle is limited by adjacent rectangular sticks of a certain width. Distance and height of each so called Stixel are determined by those parts of the obstacle it represents. This Stixel World is a compact but flexible representation of the three-dimensional traffic situation. The underlying model assumption is that objects stand on the ground and have a vertical pose with a flat surface.

Color to gray conversions in the context of stereo matching algorithms

This study tackles the image color to gray conversion problem. The aim was to understand the conversion qualities that can improve the accuracy of results when the grayscale conversion is applied as a pre-processing step in the context of vision algorithms, and in particular dense stereo matching. We evaluated many different state of the art color to grayscale conversion algorithms. We also propose an ad-hoc adaptation of the most theoretically promising algorithm, which we call Multi-Image Decolorize (MID). This algorithm comes from an in-depth analysis of the existing conversion solutions and consists of a multi-image extension of the algorithm by Grundland and Dodgson (The decolorize algorithm for contrast enhancing, color to grayscale conversion, Tech. Rep. UCAM-CL-TR-649, University of Cambridge, 2005) which is based on predominant component analysis. In addition, two variants of this algorithm have been proposed and analyzed: one with standard unsharp masking and another with a chromatic weighted unsharp masking technique (Nowak and Baraniuk in IEEE Trans Image Process 7(7):1068–1074, 1998) which enhances the local contrast as shown in the approach by Smith et al. (Comput Graph Forum 27(2), 2008). We tested the relative performances of this conversion with respect to many other solutions, using the StereoMatcher test suite (Scharstein and Szeliski in Int J Comput Vis 47(1–3):7–42, 2002) with a variety of different datasets and different dense stereo matching algorithms. The results show that the overall performance of the proposed MID conversion are good and the reported tests provided useful information and insights on how to design color to gray conversion to improve matching performance. We also show some interesting secondary results such as the role of standard unsharp masking vs. chromatic unsharp masking in improving correspondence matching.

Efficient Sequential Correspondence Selection by Cosegmentation

In many retrieval, object recognition, and wide-baseline stereo methods, correspondences of interest points (distinguished regions) are commonly established by matching compact descriptors such as SIFTs. We show that a subsequent cosegmentation process coupled with a quasi-optimal sequential decision process leads to a correspondence verification procedure that 1) has high precision (is highly discriminative), 2) has good recall, and 3) is fast. The sequential decision on the correctness of a correspondence is based on simple statistics of a modified dense stereo matching algorithm. The statistics are projected on a prominent discriminative direction by SVM. Wald's sequential probability ratio test is performed on the SVM projection computed on progressively larger cosegmented regions. We show experimentally that the proposed sequential correspondence verification (SCV) algorithm significantly outperforms the standard correspondence selection method based on SIFT distance ratios on challenging matching problems.

An optimal time–space algorithm for dense stereo matching

The increasing demand for higher resolution images and higher frame rate videos will always pose a challenge to computational power when performance is required to solve the stereo-matching problem in 3D reconstruction applications. Therefore, the use of asymptotic analysis is necessary to measure the time and space performance of stereo-matching algorithms regardless of the size of the input and of the computational power available. In this paper, we survey several classic stereo-matching algorithms with regard to time---space complexity. We also report running time experiments for several algorithms that are consistent with our complexity analysis. We present a new dense stereo-matching algorithm based on a greedy heuristic path computation in disparity space. A procedure which improves disparity maps in depth discontinuity regions is introduced. This procedure works as a post-processing step for any technique that solves the dense stereo-matching problem. We prove that our algorithm and post-processing procedure have optimal O(n) time---space complexity, where n is the size of a stereo image. Our algorithm performs only a constant number of computations per pixel since it avoids a brute force search over the disparity range. Hence, our algorithm is faster than real-time techniques while producing comparable results when evaluated with ground-truth benchmarks. The correctness of our algorithm is demonstrated with experiments in real and synthetic data.

A portable stereo vision system for whole body surface imaging

This paper presents a whole body surface imaging system based on stereo vision technology. We have adopted a compact and economical configuration which involves only four stereo units to image the frontal and rear sides of the body. The success of the system depends on a stereo matching process that can effectively segment the body from the background in addition to recovering sufficient geometric details. For this purpose, we have developed a novel sub-pixel, dense stereo matching algorithm which includes two major phases. In the first phase, the foreground is accurately segmented with the help of a predefined virtual interface in the disparity space image, and a coarse disparity map is generated with block matching. In the second phase, local least squares matching is performed in combination with global optimization within a regularization framework, so as to ensure both accuracy and reliability. Our experimental results show that the system can realistically capture smooth and complete whole body shapes with high accuracy.

Graph-cut-based stereo matching using image segmentation with symmetrical treatment of occlusions

This paper describes a dense stereo matching algorithm for epipolar rectified images. The method applies colour segmentation on the reference image. Our basic assumptions are that disparity varies smoothly inside a segment, while disparity boundaries coincide with the segment borders. The use of these assumptions makes the algorithm capable of handling large untextured regions, estimating precise depth boundaries and propagating disparity information to occluded regions, which are challenging tasks for conventional stereo methods. We model disparity inside a segment by a planar equation. Initial disparity segments are clustered to form a set of disparity layers, which are planar surfaces that are likely to occur in the scene. Assignments of segments to disparity layers are then derived by minimization of a global cost function. This cost function is based on the observation that occlusions cannot be dealt with in the domain of segments. Therefore, we propose a novel cost function that is defined on two levels, one representing the segments and the other corresponding to pixels. The basic idea is that a pixel has to be assigned to the same disparity layer as its segment, but can as well be occluded. The cost function is then effectively minimized via graph-cuts. In the experimental results, we show that our method produces good-quality results, especially in regions of low texture and close to disparity boundaries. Results obtained for the Middlebury test set indicate that the proposed method is able to compete with the best-performing state-of-the-art algorithms.