Stereo Matching Method Research Articles

Event-based stereo matching using semiglobal matching

In this article, we focus on the problem of depth estimation from a stereo pair of event-based sensors. These sensors asynchronously capture pixel-level brightness changes information (events) instead of standard intensity images at a specified frame rate. So, these sensors provide sparse data at low latency and high temporal resolution over a wide intrascene dynamic range. However, new asynchronous, event-based processing algorithms are required to process the event streams. We propose a fully event-based stereo three-dimensional depth estimation algorithm inspired by semiglobal matching. Our algorithm considers the smoothness constraints between the nearby events to remove the ambiguous and wrong matches when only using the properties of a single event or local features. Experimental validation and comparison with several state-of-the-art, event-based stereo matching methods are provided on five different scenes of event-based stereo data sets. The results show that our method can operate well in an event-driven way and has higher estimation accuracy.

Stereo matching method for non-coding circular reference points based on motion consistency

Reference point-based fusion of multi-view measurement data is a popular method of object digitization that relies on the accurate matching of reference points in the images. However, due to the lack of coding information in the linear-structured light measurement, establishing a correspondence with the reference points becomes a challenging task. As the pose of a handheld linear-structured light measurement device undergoes very minor changes across neighboring views, this paper exploits this property and proposes a novel method of stereoscopic matching of non-coded reference circle points for a binocular linear-structured light measurement system. First, the polar line constraint is used to determine the initial set of matches, and these matches are used to compute the distance and direction of motion of the camera across neighboring views. The motion information is then used to eliminate the instances of mismatches from the initial results. Experiments on a self-locating linear-structured light measurement system show that the proposed algorithm can achieve an accurate and stable stereoscopic matching of binocular non-coded reference circle points.

Stereo matching using cost volume fusion for high dynamic range scenes

A method is proposed to obtain accurate disparity maps for high dynamic range (HDR) scenes using stereo image pairs acquired under different exposure times and viewpoints. In HDR scenes, saturated pixels such as too-dark or too-bright pixels are inevitable because of the limited bit-depth representations of images. These saturated regions include little information for matching images and harm the performance of conventional stereo matching methods. To address this problem, a method is proposed that generates a new cost volume by fusing cost volumes obtained using different stereo image pairs taken under different exposure times as well as different points of view. In addition, the directional semi-global matching method is proposed to efficiently optimise the cost volume. Experimental results show that the proposed method generates more accurate results for saturated regions than existing methods on various datasets.

Continuous 3D Label Stereo Matching Using Local Expansion Moves.

We present an accurate stereo matching method using local expansion moves based on graph cuts. This new move-making scheme is used to efficiently infer per-pixel 3D plane labels on a pairwise Markov random field (MRF) that effectively combines recently proposed slanted patch matching and curvature regularization terms. The local expansion moves are presented as many -expansions defined for small grid regions. The local expansion moves extend traditional expansion moves by two ways: localization and spatial propagation. By localization, we use different candidate -labels according to the locations of local -expansions. By spatial propagation, we design our local -expansions to propagate currently assigned labels for nearby regions. With this localization and spatial propagation, our method can efficiently infer MRF models with a continuous label space using randomized search. Our method has several advantages over previous approaches that are based on fusion moves or belief propagation; it produces submodular moves deriving a subproblem optimality; it helps find good, smooth, piecewise linear disparity maps; it is suitable for parallelization; it can use cost-volume filtering techniques for accelerating the matching cost computations. Even using a simple pairwise MRF, our method is shown to have best performance in the Middlebury stereo benchmark V2 and V3.

Event-Based Stereo Depth Estimation Using Belief Propagation.

Compared to standard frame-based cameras, biologically-inspired event-based sensors capture visual information with low latency and minimal redundancy. These event-based sensors are also far less prone to motion blur than traditional cameras, and still operate effectively in high dynamic range scenes. However, classical framed-based algorithms are not typically suitable for these event-based data and new processing algorithms are required. This paper focuses on the problem of depth estimation from a stereo pair of event-based sensors. A fully event-based stereo depth estimation algorithm which relies on message passing is proposed. The algorithm not only considers the properties of a single event but also uses a Markov Random Field (MRF) to consider the constraints between the nearby events, such as disparity uniqueness and depth continuity. The method is tested on five different scenes and compared to other state-of-art event-based stereo matching methods. The results show that the method detects more stereo matches than other methods, with each match having a higher accuracy. The method can operate in an event-driven manner where depths are reported for individual events as they are received, or the network can be queried at any time to generate a sparse depth frame which represents the current state of the network.

Depth Estimation Using an Infrared Dot Projector and an Infrared Color Stereo Camera

This paper proposes a method of estimating depth from two kinds of stereo images: color stereo images and infrared stereo images. An infrared dot pattern is projected on a scene by a projector so that infrared cameras can capture the scene textured by the dots and the depth can be estimated even where the surface is not textured. The cost volumes are calculated for the infrared and color stereo images for each frame and are extended in the time direction to define a spatiotemporal cost volume (st-cost volume). We also extend the cost volume filter in the time direction by modifying the cross-based local multipoint filter (CLMF) and applying it to the st-cost volumes in order to restrain flicker on the time-varying depth maps. To get a reliable cost volume, the infrared and color st-cost volumes are integrated into a single cost volume by selecting the cost of either the infrared or the color st-cost volumes according to the size of the adaptive kernel used for the CLMF. Then, a graphcut is executed on the cost volume in order to estimate the disparity robustly even when the baselines of the stereo cameras are set wide enough to ensure spatially high resolution in the depth direction and the shapes of blocks are deformed by the affine transformation. A 2D graphcut is executed on each scan line to reduce the processing time and memory consumption. We experimented with the proposed method using infrared color stereo data sets of scenes in the real world and evaluated its effectiveness by comparing it with other recent stereo matching methods and depth cameras.

Real-time geometry-aware augmented reality in minimally invasive surgery.

The potential of augmented reality (AR) technology to assist minimally invasive surgery (MIS) lies in its computational performance and accuracy in dealing with challenging MIS scenes. Even with the latest hardware and software technologies, achieving both real-time and accurate augmented information overlay in MIS is still a formidable task. In this Letter, the authors present a novel real-time AR framework for MIS that achieves interactive geometric aware AR in endoscopic surgery with stereo views. The authors’ framework tracks the movement of the endoscopic camera and simultaneously reconstructs a dense geometric mesh of the MIS scene. The movement of the camera is predicted by minimising the re-projection error to achieve a fast tracking performance, while the three-dimensional mesh is incrementally built by a dense zero mean normalised cross-correlation stereo-matching method to improve the accuracy of the surface reconstruction. The proposed system does not require any prior template or pre-operative scan and can infer the geometric information intra-operatively in real time. With the geometric information available, the proposed AR framework is able to interactively add annotations, localisation of tumours and vessels, and measurement labelling with greater precision and accuracy compared with the state-of-the-art approaches.

Improving stereo matching by incorporating geometry prior into ConvNet

Deep learning-based methods for stereo matching have shown superior performance over traditional ones. However, most of them ignore the inherent geometry prior of stereo matching when training, i.e. the reference image can be reconstructed from the second image in the visible regions. The reconstruction can be achieved by backward warping the second image using the disparity map of the reference image, while the visible regions can be calculated by left-right consistency check. This prior is useful especially when the ground truth disparity is sparse (e.g. the outdoor scene such as KITTI 2015). This prior incorporated into a two-stage end-to-end training process, both of which try to minimise the end-point-error with respect to the sparse ground truth disparity (supervised learning), and the reconstruction error (self-supervised learning). The predicted disparity and the reconstruction error of the first stage act as additional information, and are fed to the second stage to make further use of this prior knowledge to improve performance. Experiments on the challenging KITTI 2015 dataset show that the method improves the results in the foreground region, and ranks first among all the published methods on the D1-fg metric.

Hierarchical Stereo Matching in Two-Scale Space for Cyber-Physical System.

Dense disparity map estimation from a high-resolution stereo image is a very difficult problem in terms of both matching accuracy and computation efficiency. Thus, an exhaustive disparity search at full resolution is required. In general, examining more pixels in the stereo view results in more ambiguous correspondences. When a high-resolution image is down-sampled, the high-frequency components of the fine-scaled image are at risk of disappearing in the coarse-resolution image. Furthermore, if erroneous disparity estimates caused by missing high-frequency components are propagated across scale space, ultimately, false disparity estimates are obtained. To solve these problems, we introduce an efficient hierarchical stereo matching method in two-scale space. This method applies disparity estimation to the reduced-resolution image, and the disparity result is then up-sampled to the original resolution. The disparity estimation values of the high-frequency (or edge component) regions of the full-resolution image are combined with the up-sampled disparity results. In this study, we extracted the high-frequency areas from the scale-space representation by using difference of Gaussian (DoG) or found edge components, using a Canny operator. Then, edge-aware disparity propagation was used to refine the disparity map. The experimental results show that the proposed algorithm outperforms previous methods.

An efficient approach for stereo matching of planar objects in stereo-digital image correlation

In many standard mechanical tests and in all two-dimensional digital image correlation (2D-DIC) applications, the surfaces of specimens to be measured are planar. For the special case of planar surfaces, in this paper an efficient approach for stereo matching was proposed to further improve the computation efficiency of stereo-DIC. The proposed stereo matching method utilizes the characteristic of planar objects that the projection transformation functions between left and right images are the same for all matched subsets and can be expressed by eight parameters. In each pair of images, four discrete points were matched to calculate the projection transformation matrix, and then the stereo matching of all other points could be accomplished based on the calculated projection transformation matrix in an easy and efficient way. Both simulations and experimental results demonstrated that the proposed method is feasible and effective and the computational speed is about 20 times faster than that of traditional method. Based on the proposed method, real-time stereo-DIC with a higher frame rate and more points should be achieved.

An Improved Multi-Image Matching Method in Stereo-Radargrammetry

The new generation of synthetic aperture radar (SAR) sensors provides us with an opportunity to match multiple high-resolution SAR images. Moreover, the multiple SAR image matching methods have recently gained a lot of attention due to the fact that they can obtain more accurate, better distributed, and more reliable matches than the stereo matching methods. In this letter, we present an improved multi-image matching method to simultaneously identify matches from multiple SAR amplitude images. The proposed method makes better use of the relationships between the pixels in the deformed correlation window and integrates geometric and radiometric information from multiple SAR images. Experiments on Chinese Academy of Surveying and Mapping Synthetic Aperture Radar (CASMSAR) data sets demonstrate that the improved multi-image matching method is capable of providing more accurate and better distributed matches, as well as offering a better multi-image matching solution in stereo-radargrammetry under the conditions of geometric and radiometric distortions, especially in low-texture areas.

Light-Field Depth Estimation via Epipolar Plane Image Analysis and Locally Linear Embedding

In this paper, we propose a novel method for 4D light-field (LF) depth estimation exploiting the special linear structure of an epipolar plane image (EPI) and locally linear embedding (LLE). Without high computational complexity, depth maps are locally estimated by locating the optimal slope of each line segmentation on the EPIs, which are projected by the corresponding scene points. For each pixel to be processed, we build and then minimize the matching cost that aggregates the intensity pixel value, gradient pixel value, spatial consistency, as well as reliability measure to select the optimal slope from a predefined set of directions. Next, a subangle estimation method is proposed to further refine the obtained optimal slope of each pixel. Furthermore, based on a local reliability measure, all the pixels are classified into reliable and unreliable pixels. For the unreliable pixels, LLE is employed to propagate the missing pixels by the reliable pixels based on the assumption of manifold preserving property maintained by natural images. We demonstrate the effectiveness of our approach on a number of synthetic LF examples and real-world LF data sets, and show that our experimental results can achieve higher performance than the typical and recent state-of-the-art LF stereo matching methods.

A Mismatch Detection Method Based on Affine Transformation for Stereo Light Microscopy Stereo Matching

For the light microscopy images that have the characteristics of shallow depth of field, serious distortion and poor resolution, mismatch is a ubiquitous phenomenon. The paper presents a mismatch detection method for the stereo light microscopy stereo matching. Affine transformation matrix and matching constraint condition are calibrated by the calibration board which has the precision solid dots and the motorized stage. Bias vector of affine transformation of each matching pair is taken as the criteria to apply mismatch detection. The experimental results show that the method can detect more mismatching pairs and preserve more matching pairs than the traditional RANSAC method and the epipolar rectification method.

A Nonlocal Method with Modified Initial Cost and Multiple Weight for Stereo Matching

This paper presents a new nonlocal cost aggregation method for stereo matching. The minimum spanning tree (MST) employs color difference as the sole component to build the weight function, which often leads to failure in achieving satisfactory results in some boundary regions with similar color distributions. In this paper, a modified initial cost is used. The erroneous pixels are often caused by two pixels from object and background, which have similar color distribution. And then inner color correlation is employed as a new component of the weight function, which is determined to effectively eliminate them. Besides, the segmentation method of the tree structure is also improved. Thus, a more robust and reasonable tree structure is developed. The proposed method was tested on Middlebury datasets. As can be expected, experimental results show that the proposed method outperforms the classical nonlocal methods.

Stereo Matching Method Using Slant-based Cost Aggregation

Stereo matching methods based on color segmentation usually obtain the initial matching cost for each pixel and interpolate the disparity value with the local gradient of image patches. It is assumed that the image regions may be approximated with local surfaces in the initial matching cost aggregation. However, color segmentation and disparity plane estimation have high computational loads. This paper presents an efficient real-time stereo matching method to improve initial cost aggregation. The initial matching costs are computed using the direction of arbitrary slopes in matching cost volume without color segmentation and plane estimation procedures.

An Evaluation and Improved Matching Cost of Stereo Matching Method

An Evaluation and Improved Matching Cost of Stereo Matching Method

Spatial Detection of Vehicles in Images using Convolutional Neural Networks and Stereo Matching

Convolutional Neural Networks combined with a state of the artstereo-matching method are used to find and estimate the 3D positionof vehicles in pairs of stereo images. Pixel positions of vehiclesare first estimated separately in pairs of stereo images usinga Convolutional Neural Network for regression. These coordinatesare then combined with a state-of-art stereo-matching method todetermine the depth, and thus the 3D location, of the vehicles. Weshow in this paper that cars can be detected with a combined accuracyof approximately 90% with a tolerated radius error of 5%,and a Mean Absolute Error of 5.25m on depth estimation for carsup to 50m away.

Reliability-Based Discontinuity-Preserving Stereo Matching

In stereo matching, it is a challengeable task to effectively deal with occlusions between objects, which often cause disparity errors along object boundaries. In this paper, we propose discontinuity-preserving stereo matching based on reliability. We first estimate an initial disparity map by segmentation-based adaptive weights. Then, we obtain reliability for each pixel in an image, which indicates the correctness of the estimated disparity based on the matching cost. Finally, we minimize disparity errors caused by occlusions between objects using reliability and produce discontinuity-preserving disparity maps. Experiments on the Middlebury data set show that the proposed method successfully preserves disparity discontinuities along object boundaries and outperforms the state-of-the-art stereo matching methods.

Mobile Robot Obstacle Avoidance Algorithms Based on Information Fusion of Vision and Sonar

In view of the problem of AS-R autonomous wheeled mobile robot obstacle avoidance, a rapid convergence of sonar and binocular stereovision sensor distance information in order to detect and avoid obstacle algorithm is proposed in this paper. The algorithm first uses binocular camera (CCD) to get three-dimensional image of the real environment, through the stereo matching and V-disparity method which is used to calculate disparity map, then obstacles is extracted by Hough lines detection algorithm, finally we will get information about obstacles and sonar return information with T-S fuzzy neural network fusion, then it will output walking controlled decisions. Experimental results proved that the algorithm is effective and practical.

Multi-crop-row detection algorithm based on binocular vision

Pathway determination is an important process in vision-based navigation. The pathway is very difficult to determine simply using 2D image processing, because fields are often infested with weeds, and images contain shadows, illumination variation, irregular backgrounds and other unexpected noise. Stereo vision techniques can be used to locate the spatial positions of crop rows for pathway determination. However, the stereo matching of field images is generally time-consuming and insufficiently accurate. To solve this problem, a multi-crop-row detection algorithm based on binocular vision is proposed in this paper. The algorithm is composed of the modules of image preprocessing, stereo matching and centreline detection of multiple crop rows. An accurate stereo matching method was put forward to locate the 3D position of crop rows based on the rank transformation, Harris detector and random sample consensus methods. A new method for detecting the centrelines of multiple crop rows was proposed according to their spatial distribution. The proposed algorithm was validated by comparative experiments. Regarding the proposed algorithm in situations without turnrows, the correct detection rate is greater than 92.78%; for the average deviation angle, the absolute average value is less than 1.05°, and the average standard deviation is less than 3.66°; for the processing time, the average value is less than 634 ms, and the average standard deviation is less than 101 ms. The results indicate that the proposed algorithm can satisfy the requirements of accuracy and real-time execution in field operation.