Local Stereo Matching Research Articles

A new stereo matching algorithm based on improved four-moded census transform and adaptive cross pyramid model

<p>Stereo matching is still very challenging in terms of depth discontinuity, occlusions, weak texture regions, and noise resistance. To address the problems of poor noise immunity of local stereo matching and low matching accuracy in weak texture regions, a stereo matching algorithm (iFCTACP) based on improved four-moded census transform (iFCT) and a novel adaptive cross pyramid (ACP) structure were proposed. The algorithm combines the improved four-moded census transform matching cost with traditional measurement methods, which allows better anti-interference performance. The cost aggregation is performed on the adaptive cross pyramid structure, a unique structure that improves the traditional single mode of the cross. This structure not only enables regions with similar color and depth to be connected but also achieves cost smoothing across regions, significantly reducing the possibility of mismatch due to inadequate corresponding matching information and providing stronger robustness to weak texture regions. Experimental results show that the iFCTACP algorithm can effectively suppress noise interference, especially in illumination and exposure. Furthermore, it can markedly improve the error matching rate in weak texture regions with better generalization. Compared with some typical algorithms, the iFCTACP algorithm exhibits better performance whose average mismatching rate is only 3.33$ \% $.</p>

Local stereo matching with side window

Although local stereo matching methods have attracted much attention for their promising performance in disparity estimation, the issue of edge-blurring remains a challenge for existing local methods. To address this issue, we propose a novel local stereo matching method based on side window filtering technique, namely LSMSW. In the proposed method, the multiple-window aggregation strategy is adopted, which can adaptively select the window type according to the different texture information. Compared to traditional local methods, our cost aggregation model shows the significant performance in terms of accuracy and computational efficiency, especially for images with complex edges. The experimental results on the Middlebury benchmark confirm our achievements.

Implementation of a Human Motion Capture System Based on the Internet of Things Machine Vision

The classification of the stereo matching comprehensive analysis related algorithm model can be subdivided into local stereo matching based on the entire acquisition and global stereo matching based on the entire local. But it can have a higher capture efficiency because the log-likelihood variance cost calculation function can have a faster feature convergence capture speed than the ordinary log-mean-square error cost function. Through the combination of gray channel and frame difference channel, a better network structure and parameters on the KTH data set are obtained, which can ensure the classification effect while greatly reducing the number of parameters, improving training efficiency and improving classification accuracy. The article uses dual-channel 3D convolutional human neural network technology to achieve 92.5% accuracy of human feature capture, which is significantly better than many traditional feature extraction techniques proposed in the literature.

Adaptive Deconvolution-Based Stereo Matching Net for Local Stereo Matching

In deep learning-based local stereo matching methods, larger image patches usually bring better stereo matching accuracy. However, it is unrealistic to increase the size of the image patch size without restriction. Arbitrarily extending the patch size will change the local stereo matching method into the global stereo matching method, and the matching accuracy will be saturated. We simplified the existing Siamese convolutional network by reducing the number of network parameters and propose an efficient CNN based structure, namely adaptive deconvolution-based disparity matching net (ADSM net) by adding deconvolution layers to learn how to enlarge the size of input feature map for the following convolution layers. Experimental results on the KITTI2012 and 2015 datasets demonstrate that the proposed method can achieve a good trade-off between accuracy and complexity.

Depth measurement based on a convolutional neural network and structured light

Depth measurement methods based on structured light are popular due to their advantages of low cost, good portability and ease of implementation. Depth information for the object is obtained based on the geometric relationship of the imaging system and triangulation theory, which usually requires local stereo matching operations. However, this is computationally intensive, resulting in reduced depth accuracy and worse depth maps. To address these problems, this paper proposes a novel depth measurement method based on a convolutional neural network (DMCNN), which is cast as a pixel-wise classification–regression task without matching. Firstly, the DMCNN network is designed as an encoder–decoder structure. A feature pyramid is adopted in the encoder to extract multi-scale fusion features, and parallel classification and regression branches are constructed at the end of the decoder to achieve depth prediction from coarse to fine. Secondly, we use a four-step phase shift algorithm to generate ground truth depth maps and build a dataset containing a large number of speckle distortion images and their corresponding depth maps to train our network. The algorithm runs on an RTX 2080Ti graphics processing unit (GPU) using 20 000 training images. Experimental results show that our method can achieve higher accuracy than alternative depth measurement methods.

Improved stereo matching algorithm based on multi-scale fusion

Aiming at the low matching accuracy of local stereo matching algorithm in weak texture or discontinuous disparity areas, a stereo matching algorithm combining multi-scale fusion of convolutional neural network (CNN) and feature pyramid structure (FPN) is proposed. The feature pyramid is applied on the basis of the convolutional neural network to realize the multi-scale feature extraction and fusion of the image, which improves the matching similarity of the image blocks. The guide graph filter is used to quickly and effectively complete the cost aggregation. The disparity selection stage adapts the improvement dynamic programming algorithm to obtain the initial disparity map. The initial disparity map is refined so as to obtain the final disparity map. The algorithm is trained and tested on the image provided by Middlebury data set, and the result shows that the disparity map obtained by the algorithm has good effect.

Binary Descriptors for Dense Stereo Matching

Dense local stereo matching is traditionally based on initial cost evaluation using a simple metric Dense local stereo matching is traditionally based on initial cost evaluation using a simple metric followed by sophisticated support aggregation. There is a high potential of replacing these simple metrics by robust binary descriptors. However, the available studies focus on comparing descriptors for sparse matching rather than the dense case of extracting a descriptor per each pixel. Therefore, this paper studies the design decisions of well-established binary descriptors such as BRIEF (Binary Robust Independent Elementary Features), ORB (Oriented FAST and rotated BRIEF), BRISK (Binary Robust Invariant Scalable Keypoints) and FREAK (Fast Retina Keypoint) to decide which one is more suitable for the dense matching case. The expremental results shows that agregation is required for use with binary descriptors to handle edges. Also, BRIEF produced the smoothnest disparity map if geometric transformations is not present. Whereas, FREAK and BRISK achieved the least overall error percentage across all regions. The lastest Middlebury Stereo benchmark is utilized in the experiments.

Local Stereo Matching Using Adaptive Cross-Region-Based Guided Image Filtering with Orthogonal Weights

Adaptive cross-region-based guided image filtering (ACR-GIF) is a commonly used cost aggregation method. However, the weights of points in the adaptive cross-region (ACR) are generally not considered, which affects the accuracy of disparity results. In this study, we propose an improved cost aggregation method to address this issue. First, the orthogonal weight is proposed according to the structural feature of the ACR, and then the orthogonal weight of each point in the ACR is computed. Second, the matching cost volume is filtered using ACR-GIF with orthogonal weights (ACR-GIF-OW). In order to reduce the computing time of the proposed method, an efficient weighted aggregation computing method based on orthogonal weights is proposed. Additionally, by combining ACR-GIF-OW with our recently proposed matching cost computation method and disparity refinement method, a local stereo matching algorithm is proposed as well. The results of Middlebury evaluation platform show that, compared with ACR-GIF, the proposed cost aggregation method can significantly improve the disparity accuracy with less additional time overhead, and the performance of the proposed stereo matching algorithm outperforms other state-of-the-art local and nonlocal algorithms.

Efficient local stereo matching algorithm based on fast gradient domain guided image filtering

Guided image filtering (GIF) based cost aggregation or disparity refinement stereo matching algorithms are studied extensively owing to the edge-aware preserved smoothing property. However, GIF suffers from halo artifacts in sharp edges and shows high computational costs on high-resolution images. The performance of GIF in stereo matching would be limited by the above two defects. To solve these problems, a novel fast gradient domain guided image filtering (F-GDGIF) is proposed. To be specific, halo artifacts are effectively alleviated by incorporating an efficient multi-scale edge-aware weighting into GIF. With this multi-scale weighting, edges can be preserved much better. In addition, high computational costs are cut down by sub-sampling strategy, which decreases the computational complexity from O(N) to O(N/s2) (s: sub-sampling ratio) To verify the effectiveness of the algorithm, F-GDGIF is applied to cost aggregation and disparity refinement in stereo matching algorithms respectively. Experiments on the Middlebury evaluation benchmark demonstrate that F-GDGIF based stereo matching method can generate more accuracy disparity maps with low computational cost compared to other GIF based methods.

Hardware accelerator for an accurate local stereo matching algorithm using binary neural network

Convolutional neural networks (CNNs) have shown appealing performance on stereo matching tasks in recent years. However, existing deep neural networks (DNNs) based matching algorithms use semi-global matching (SGM) to aggregate the matching costs, which limit the processing speed. In this study, we present a novel binarized CNN stereo matching hardware acceleration using local methods on an FPGA, which can provide high accuracy stereo estimation and achieve high-throughput at the same time. To reduce the consumption of hardware resource and improve the processing speed, we propose pipelined architecture for BNN, and sparse local aggregation to optimize the implementation on an FPGA. We evaluated the proposed implementation on a challenging stereo dataset with a Stratix V FPGA. From the experimental results, our binarized CNN stereo matching implementation shows significant improvement in the real-time performance and the energy efficiency over other computing platforms with a 6.95% error rate on the KITTI2015 dataset.

Stereo Matching with Spatiotemporal Disparity Refinement Using Simple Linear Iterative Clustering Segmentation

Stereo matching is a challenging problem, especially for computer vision, e.g., three-dimensional television (3DTV) or 3D visualization. The disparity maps from the video streams must be estimated. However, the estimated disparity sequences may cause undesirable flickering errors. These errors result in poor visual quality for the synthesized video and reduce the video coding information. In order to solve this problem, we here propose a spatiotemporal disparity refinement method for local stereo matching using the simple linear iterative clustering (SLIC) segmentation strategy, outlier detection, and refinements of the temporal and spatial domains. In the outlier detection, the segmented region in the initial disparity is used to distinguish errors in the binocular disparity. Based on the color similarity and disparity difference, we recalculate the aggregated cost to determine adaptive disparities to recover the disparity errors in disparity sequences. The flickering errors are also effectively removed, and the object boundaries are well preserved. Experiments using public datasets demonstrated that our proposed method creates high-quality disparity maps and obtains a high peak signal-to-noise ratio compared to state-of-the-art methods.

An edge-aware based adaptive multi-feature set extraction for stereo matching of binocular images

Object Stereo Vision has conventionally been one of the deeply examined areas in computer vision. Stereo matching is employed in numerous modern applications, including robot navigation, augmented reality, and automotive applications. Even though it has a long research history, it is still challenging for the edges of textureless, discontinues, and occluded regions under radiometric variation. This research article proposes a modified histogram equalization, a novel feature extraction, a spatial gradient model, and matching cost, which is robust and stable to images taken in different radiometric variations. The proposed method reduced the average percentage of bad pixels to 3.35 and reduced the relative mean square error (RMSE) up to 30.08 on the Middlebury dataset for different illumination and exposure values. Quantitative and qualitative evaluation of the proposed method demonstrates significant improvement in increasing PSNR and decreasing bad pixel percentage against radiometric variation and state-of-the-art local stereo matching algorithms.

An efficient local stereo matching method based on an adaptive exponentially weighted moving average filter in SLIC space

Rapidly obtaining accurate dense disparity maps has been the focus of stereo matching research. At present, approaches that achieve superior disparity maps require a large amount of computation, which is not suitable for practical applications. To address this issue, this paper proposes an efficient local matching method based on an adaptive exponentially weighted moving average filter and simple linear iterative clustering segmentation algorithm. First, an effective matching cost is introduced to adaptively integrate absolute intensity difference with Census transform, which is robust against texture free and luminance variate. Following this, during the cost aggregation, the exponentially weighted moving average filter and the SLIC segmentation are combined to handle the problems of computing consumption and adaptive expansion of the cost aggregation window. Finally, the dense disparity map is obtained by a winner-takes-all approach and disparity refinement. To demonstrate its efficiency and validity, the method is quantitatively tested and compared to existing approaches on the Middlebury benchmark. The results show that it has a non-occlusion accuracy of 90.66% and an average runtime of 7.01 s on the 2014 Middlebury dataset. Compared with existing competitive methods, the proposed method achieves superior matching results with a lower time cost.

Local Stereo Matching: An Adaptive Weighted Guided Image Filtering-Based Approach

Innovative applications in rapidly evolving domains such as robotic navigation and autonomous (driverless) vehicles rely on binocular computer vision systems that meet stringent response time and accuracy requirements. A key problem in these vision systems is stereo matching, which involves matching pixels from two input images in order to construct the output, a 3D map. Building upon the existing local stereo matching algorithms, this paper proposes a novel stereo matching algorithm that is based on a weighted guided filtering foundation. The proposed algorithm consists of three main steps; each step is designed with the goal of improving accuracy. First, the matching costs are computed using a unique combination of complementary methods (absolute difference, Census, and gradient algorithms) to reduce errors. Second, the costs are aggregated using an adaptive weighted guided image filtering method. Here, the regularization parameters are adjusted adaptively using the Canny method, further reducing errors. Third, a disparity map is generated using the winner-take-all strategy; this map is subsequently refined using a densification method to reduce errors. Our experimental results indicate that the proposed algorithm provides a higher level of accuracy in comparison to a collection of the existing state-of-the-art local algorithms.

Stereo Matching with Multi‐scale Based on Anisotropic Match Cost

AbstractBinocular vision is a popular method for 3d measurement, but the precision and efficiency is still a limitation. In order to solve the problem of low precision and high mismatching rate of current stereo match methods in weak texture region, a local stereo matching algorithm with multi‐scale based on anisotropic match cost was proposed. Not only including the gray value between the adjacent pixels as the absolute difference (AD) method, the matching cost function also involved gradient and phase information to make an anisotropic evaluation and eliminate the outliers of the flat area. Meanwhile, a multi‐scale strategy with the image pyramid was referenced, and the dynamic disparity search range of variable window was used on the original cost aggregation framework by improving the stereo matching algorithm of cross‐scale cost aggregation. The matching cost volume in each scale space used the dynamic support window to guide the filter to aggregate the matching cost. Finally, to overcome the problems of disparity selection ambiguity of WTA (winner‐take‐all) strategy and the horizontal fringe introduced by left‐right consistency (LRC) detection, the weighted median filter based on guided filter weight was used to carry out disparity refinement. According to the experiments, the method got more high matching accuracy, and the average error on the Middlebury test platform reached 5.25%.

Texture category-based matching cost and adaptive support window for local stereo matching

We present a local stereo matching method with a robust texture category-based matching cost and adaptive support window to deal with the disparity errors caused by repetitive patterns, occlusions, and nontextured cases. First, we decompose an input reference image into textured regions and nontextured regions. Then, different cost computation strategies are adopted for these two regions. For textured regions, we use the common absolute intensity difference and gradient similarity. For nontextured regions, we propose a matching cost computation method that is a combination of gradient and epipolar distance transform (EDT). In the cost aggregation step, we introduce an adaptive support window based on a modified linearly expanded cross skeleton. To obtain the cross skeleton, a depth edge detection technique and a triple expansion strategy are presented. The experimental results demonstrate that the proposed algorithm achieves outstanding matching performance compared with other existing local algorithms on the Middlebury stereo benchmark, especially in repetitive patterns, occlusions, and nontextured regions.

Improved Cost Computation and Adaptive Shape Guided Filter for Local Stereo Matching of Low Texture Stereo Images

Dense stereo matching has been widely used in photogrammetry and computer vision applications. Even though it has a long research history, dense stereo matching is still challenging for occluded, textureless and discontinuous regions. This paper proposed an efficient and effective matching cost measurement and an adaptive shape guided filter-based matching cost aggregation method to improve the stereo matching performance for large textureless regions. At first, an efficient matching cost function combining enhanced image gradient-based matching cost and improved census transform-based matching cost is introduced. This proposed matching cost function is robust against radiometric variations and textureless regions. Following this, an adaptive shape cross-based window is constructed for each pixel and a modified guided filter based on this adaptive shape window is implemented for cost aggregation. The final disparity map is obtained after disparity selection and multiple steps disparity refinement. Experiments were conducted on the Middlebury benchmark dataset to evaluate the effectiveness of the proposed cost measurement and cost aggregation strategy. The experimental results demonstrated that the average matching error rate on Middlebury standard image pairs is 9.40%. Compared with the traditional guided filter-based stereo matching method, the proposed method achieved a better matching result in textureless regions.

A Disparity Refinement in Stereo Matching based on Mean-shift Segmentation and Spatiotemporal Domain

Abstract A stereo matching algorithm is used to find the best match between a pair of images. To compute the cost of the matching points from the sequence of images, the disparity maps from video streams are estimated. However, the estimated disparity sequences may cause undesirable flickering errors. These errors result in low visibility of the synthesized video and reduce video coding. In order to solve this problem, in this article, the authors propose a spatiotemporal disparity refinement on local stereo matching based on the segmentation strategy. Based on segmentation information, matching point searching, and color similarity, adaptive disparity values to recover the disparity errors in disparity sequences can be obtained. The flickering errors are also effectively removed, and the boundaries of objects are well preserved. The procedures of the proposed approach consist of a segmentation process, matching point searching, and refinement in the temporal and spatial domains. Experimental results verify that the proposed approach can yield a high quantitative evaluation and a high-quality disparity map compared with other methods.

Three-dimensional rapid registration and reconstruction of multi-view rigid objects based on end-to-end deep surface model

Three-dimensional object reconstruction from multi-view images is an important topic in computer vision, which has attracted enormous attention during the past decades. With the further study in deep learning, remarkable progress of three-dimensional object reconstruct has been obtained in recent years. In this paper, we proposed three-dimensional rapid registration and reconstruction of multi-view rigid objects based on end-to-end deep surface model in the field of three-dimensional object reconstruction. Firstly, we introduce a matching algorithm called local stereo matching algorithm based on improved census transform and multi-scale spatial, aiming to improve the matching results for those regions. In cost aggregation step, guided map filtering algorithm with excellent gradient preserving property is introduced into Gaussian pyramid structure and regularization is added to strengthen cost volume consistency. Secondly, the improved inception RESNET module is added to improve the feature extraction ability of the network, and multiple features are extracted by using multiple network structures, and finally multiple features are sequentially input into the VRNN module to enhance the reconstruction effect of multi-view images. The experimental results show that our proposed method can not only achieve better reconstruction results, but also reconstruct more details and spend less time in training.

3D Vision System for a Robotic Arm Based on Equal Baseline Camera Array

This paper presents a lightweight 3D vision system called Equal Baseline Camera Array (EBCA). EBCA can work in different light conditions and it can be applied for measuring large range of distances. The system is a useful alternative to other known distance measuring devices such as structured-light 3D scanners, time-of-flight cameras, Light Detection and Ranging (LIDAR) devices and structure from motion techniques. EBCA can be mounted on a robotic arm without putting significant load on its construction. EBCA consists of a central camera and a ring of side cameras. The system uses stereo matching algorithms to acquire disparity maps and depth maps similarly as in case of using stereo cameras. This paper introduces methods of adapting stereo matching algorithms designed for stereo cameras to EBCA. The paper also presents the analysis of local, semi-global and global stereo matching algorithms in the context of the EBCA usage. Experiments show that, on average, results obtained from EBCA contain 37.49% less errors than the results acquired from a single stereo camera used in the same conditions.