Middlebury Dataset Research Articles

Disparity Map Computation from Stereo Images Using Hill-Climbing Segmentation

Stereo matching is one of the most active research areas in computer vision for decades. The task of stereo matching is to find the point correspondence between two images of the same scene taken from different viewpoints. This paper presents a segment-based stereo matching algorithm. Firstly, the reference image is segmented using hill-climbing algorithm and local stereo matching is performed Scale Invariant Feature Transform (SIFT) feature points with Sum of Absolute Differences (SAD) block matching. Secondly, a set of reliable pixels is constructed by comparing the matching cost and the mutual cross-checking consistent between the left and right initial disparity maps, which can lead to an actual disparity plane. Thirdly, a set of all possible disparity planes are extracted and then plane fitting and neighboring segment merging are performed. Finally, the disparity planes are set in each region using graph cuts to obtain final disparity map. The evaluation of proposed algorithm on the Middlebury data set result shows that the proposed algorithm is competitive with state-of-the-art stereo matching algorithms.

Disparity map computation of tree using stereo vision system and effects of canopy shapes and foliage density

Monitoring the growth of trees, plants, and crops is an important work in precision agriculture. Tree canopy geometric characteristics are related to tree growth and productivity. Computer vision techniques can be used to map tree canopy volume, which is useful for planning management. This study investigates the potential of using stereo vision system for obtaining tree disparity map for the analysis of geometric attributes. Experiments were conducted to examine the effects of the canopy shapes and foliage density on the performance of stereo vision system in disparity map computation. Two canopy shapes (conic and ellipse) and three foliage density levels were evaluated using two algorithms (algorithm based on local methods (ABLM) and algorithm based on global methods (ABGM)) to match pair stereo images. The well-known Middlebury dataset was considered and the performance of algorithms was evaluated on that. The results showed that the ABGM studied algorithm succeeded in computation disparity map on both Middlebury and trees images because it aggregated matching cost from several directions. The tree canopy shapes and foliage density did not affect the results of algorithms. Also, noises were numerous and more dispersed in ABLM matching algorithm. It was observed that maximum disparity limits search space. Best results were obtained in real value. For smaller value of maximum disparity than that of real value, disparity map had missed disparities. Window size was affected on maps and noise level and best results were obtained when this parameter was set to 15. Smaller value obtained more detailed map but with noise and un-smooth. The algorithm was robust for real trees in natural conditions.

Accurate, dense and shading-aware multi-view stereo reconstruction using metaheuritic optimization

Among the modern means of 3D geometry creation that exist in the literature, there are the Multi-View Stereo (MVS) reconstruction methods that received much attention from the research community and the multimedia industry. In fact, several methods showed that it is possible to recover geometry only from images with reconstruction accuracies paralleling that of excessively expensive laser scanners. The majority of these methods perform on images such as online community photo collection and estimate the surface position with its orientation by minimizing a matching cost function defined over a small local region. However, these datasets not only they are large but also contain more challenging scenes setups with different photometric effects; therefore fine-grained details of an object’s surface cannot be captured. This paper presents a robust multi-view stereo method based on metaheuristic optimization namely the Particle Swarm Optimization (PSO) in order to find the optimal depth, orientation, and surface roughness. To deal with the various shading and stereo mismatch problems caused by rough surfaces, shadows, and interreflections, we propose to use a robust matching/energy function which is a combination of two similarity measurements. Finally, our method computes individual depth maps that can be merged into compelling scene reconstructions. The proposed method is evaluated quantitatively using well-known Middlebury datasets and the obtained results show a high completeness score and comparable accuracy to those of the current top performing algorithms.

Geodesic distance framework for contour-based consistent stereo image segmentation

This study concentrates on consistent object contour extraction method for stereo image segmentation after the object regions in the left image have been obtained. By taking advantage of the epipolar geometry, our approach introduces an energy optimization framework that incorporates both the stereo correspondence term and patch-based object contour probability term. The contour map is generated by integrating the terms of stereo correspondence and patch-based object contour probability; then, the optimal contours are obtained using geodesic distance technology. The core of the proposed method is to build upon an energy optimization framework with two key contributions: first, it incorporates the patch-based object contour probability term that introduces two search strategies to efficiently find the joint nearest neighbor patch pairs for the stereo image pair. The patch-based object contour probability term provides consistent and reliable priors for the contour extraction. Second, previous methods encounter missing pixels in the extracted contour in the occluded regions. Our approach overcomes this limit by introducing the geodesic distance technology to search the optimal contours. Experimental evaluation on Middlebury dataset and Adobe open dataset indicates that the results of our stereo image segmentation are comparable with or of higher quality than state-of-the-art methods.

Methodology for filtering of depth maps based on the MCbR algorithm supported by color, shape and neighboring features

Depth maps from Time-of-Flight sensors are often noisy with regions of unknown depth denoted as black pixels. Although acceptable inpainting of depth maps using single images can be achieved with non-linear techniques, there are regions whose information is inevitably lost. Modified closing by reconstruction algorithm is a clear example of this. Because of the nature of the holes’ regions, a series of rules to classify if a hole was correctly filled based on its contours pixels is proposed. Then, a method for total filtering based on shape, color, and neighboring characteristics to infer the missing depth values is presented, evaluated and compared qualitatively and quantitatively with other state-of-the-art methods using images from the Middlebury dataset with good results.

Cost and power efficient FPGA based stereo vision system using directional graph transform

3D information of an environment using stereo cameras is important information for navigation of intelligent systems. The cost, power, accuracy, and speed are four important parameters in these systems. In this article, an accurate, real-time, low-power and low-cost system is provided to extract disparity maps in a stereo vision, using FPGA hardware platform. First, a new transform based on directional graphs is proposed. Then, benefiting from this graph transform and cross-based matching method, disparity map is computed. By using optimized hardware for the proposed transform and algorithm, we have obtained an accurate, low-cost, low-power and fast stereo vision system. The proposed system is fully implemented on relatively low cost FPGA platform, XC7K160t, in order to operate as a Standalone system. This system uses 40 K registers, 31 K LUTs, 215 memory blocks, and 258 DSP blocks of this FPGA. The proposed system is tested and evaluated in Middlebury dataset. The results show that the proposed stereo system can process a HD quality video at 60 frames per second for 64 disparity levels with only 7.1% error in the final disparity map. The total power consumption of the proposed stereo vision core is about 1 W.

Full-image guided method based on confidence coefficient for fast stereo matching

Stereo matching is a challenging problem and is widely used in many applications, including autonomous navigation, 3D-reconstruction and etc. Therefore, we propose a new non-local method for stereo matching in this paper, which is mainly improved in two aspects. On the one hand, all the elements are employed during the step of cost aggregation aiming to handle the textureless regions. Besides, the calculation of aggregation weight is decreased greatly as a result of summing up the penalty terms to propagate weight. On the other hand, different aggregated costs are combined by defining the confidence coefficient instead of assigning weight in conventional methods. We evaluate the performance of our algorithm on the Middlebury dataset. Our model shows excellent results in particular for repetitive textures.

Efficient graph cut optimization for shape from focus

Shape From Focus refers to the inverse problem of recovering the depth in every point of a scene from a set of differently focused 2D images. Recently, some authors stated it in the variational framework and solved it by minimizing a non-convex functional. However, the global optimality on the solution is not guaranteed and evaluations are often application-specific. To overcome these limits, we propose to globally and efficiently minimize a convex functional by decomposing it into a sequence of binary problems using graph cuts. To illustrate the genericity of such a decomposition-based approach, data-driven strategies are considered, allowing us to optimize (in terms of reconstruction error) the choice of the depth values for a given number of possible depths. We provide qualitative and quantitative evaluation on Middlebury datasets and we show that, according to classic statistics on error values, the proposed approach exhibits high performance and robustness against corrupted data.

Integrated cosparse analysis model with explicit edge inconsistency measurement for guided depth map upsampling

A low-resolution depth map can be upsampled through the guidance from the registered high-resolution color image. This type of method is so-called guided depth map upsampling. Among the existing methods based on Markov random field (MRF), either data-driven or model-based prior is adopted to construct the regularization term. The data-driven prior can implicitly reveal the relation between color-depth image pair by training on external data. The model-based prior provides the anisotropic smoothness constraint guided by high-resolution color image. These types of priors can complement each other to solve the ambiguity in guided depth map upsampling. An MRF-based approach is proposed that takes both of them into account to regularize the depth map. Based on analysis sparse coding, the data-driven prior is defined by joint cosparsity on the vectors transformed from color-depth patches using the pair of learned operators. It is based on the assumption that the cosupports of such bimodal image structures computed by the operators are aligned. The edge inconsistency measurement is explicitly calculated, which is embedded into the model-based prior. It can significantly mitigate texture-copying artifacts. The experimental results on Middlebury datasets demonstrate the validity of the proposed method that outperforms seven state-of-the-art approaches.

Adaptive disparity computation using local and non-local cost aggregations

A new method is proposed to adaptively compute the disparity of stereo matching by choosing one of the alternative disparities from local and non-local disparity maps. The initial two disparity maps can be obtained from state-of-the-art local and non-local stereo algorithms. Then, the more reasonable disparity is selected. We propose two strategies to select the disparity. One is based on the magnitude of the gradient in the left image, which is simple and fast. The other utilizes the fusion move to combine the two proposal labelings (disparity maps) in a theoretically sound manner, which is more accurate. Finally, we propose a texture-based sub-pixel refinement to refine the disparity map. Experimental results using Middlebury datasets demonstrate that the two proposed selection strategies both perform better than individual local or non-local algorithms. Moreover, the proposed method is compatible with many local and non-local algorithms that are widely used in stereo matching.

Learning and Selecting Confidence Measures for Robust Stereo Matching.

We present a robust approach for computing disparity maps with a supervised learning-based confidence prediction. This approach takes into consideration following features. First, we analyze the characteristics of various confidence measures in the random forest framework to select effective confidence measures depending on the characteristics of the training data and matching strategies, such as similarity measures and parameters. We then train a random forest using the selected confidence measures to improve the efficiency of confidence prediction and to build a better prediction model. Second, we present a confidence-based matching cost modulation scheme, based on predicted confidence values, to improve the robustness and accuracy of the (semi-) global stereo matching algorithms. Finally, we apply the proposed modulation scheme to popularly used algorithms to make them robust against unexpected difficulties that could occur in an uncontrolled environment using challenging outdoor datasets. The proposed confidence measure selection and cost modulation schemes are experimentally verified from various perspectives using the KITTI and Middlebury datasets.

Object Detection from a Range Image Using Sparse Keypoint Detector Technique

This proposal presents the object detection using Sparse Keypoint Detector technique, in which is computed the interesting points from directional surface. This surface is built with the normal vectors corresponding to the homogeneous surface, and this in turn, is obtained from range image. A probability dense function (pdf) analysis applied to the normal vectors contained in the directional surface, allows us to select the highlight points to obtain the contour of the scene. Furthermore, a probability mass function (pmf) analysis applied to the information contained in the surrounding of these points, aim to select a region of interest, in which is found the object 3D to be detected. Finally, it is applied a Chess distance to the interesting points contained in the region of interest to detect the object. The presented experimental tests involve a qualitative and quantitative analysis using the Middlebury and DSPLab dataset.

Fast Optical Flow Estimation Based on the Split Bregman Method

Fast and accurate optical flow estimation is a challenging problem in computer vision. In this paper, we present a novel model to solve the optical flow problem by combining the strengths of the Split Bregman method with the advantages of an efficient variational framework. It allows us to employ different regularization tensors for the Split Bregman regularizer to preserve motion discontinuities. Simultaneously, a novel nonlocal Split Bregman method with adaptive support weights is also developed for the smoothness regularization to preserve motion details, repel outliers, enhance contrast, and reduce motion blurring and staircase effect. The proposed algorithm shows significant improvements in preserving sharp flow edges and important motion details. Most of all, it requires only a few iterations to achieve fast convergence and runs faster than the classic TV-L1 flow method. Our method significantly outperforms the current state-of-the-art methods on the challenging MPI-Sintel dataset and shows good performance on the Kitti flow 2015 and Middlebury datasets.

New Keypoint Matching Method Using Local Convolutional Features for Power Transmission Line Icing Monitoring.

Power transmission line icing (PTLI) problems, which cause tremendous damage to the power grids, has drawn much attention. Existing three-dimensional measurement methods based on binocular stereo vision was recently introduced to measure the ice thickness in PTLI, but failed to meet requirements of practical applications due to inefficient keypoint matching in the complex PTLI scene. In this paper, a new keypoint matching method is proposed based on the local multi-layer convolutional neural network (CNN) features, termed Local Convolutional Features (LCFs). LCFs are deployed to extract more discriminative features than the conventional CNNs. Particularly in LCFs, a multi-layer features fusion scheme is exploited to boost the matching performance. Together with a location constraint method, the correspondence of neighboring keypoints is further refined. Our approach achieves 1.5%, 5.3%, 13.1%, 27.3% improvement in the average matching precision compared with SIFT, SURF, ORB and MatchNet on the public Middlebury dataset, and the measurement accuracy of ice thickness can reach 90.9% compared with manual measurement on the collected PTLI dataset.

Local convolutional features and metric learning for SAR image registration

The conventional synthetic aperture radar (SAR) image registration focuses on designing diverse hand-crafted features and distance metrics. It is still challenging to obtain accurate feature correspondence when influenced by speckle noise and geometric distortion. This paper proposes local convolutional neural network (CNN) features based method to solve the keypoint matching problem, whose contributions are threefold. (1) a feature descriptor based on local convolutional features of image patches (LCFs-P) is deployed to extract more discriminative features than the conventional CNNs. (2) A new feature correspondence scheme based on metric learning is proposed to boost the feature matching performance. (3) A local geometric similarity method is employed to remove the mismatches of the tentative matches. The experimental results on the real SAR dataset and Middlebury dataset demonstrate that the proposed model outperforms the existing state-of-the-art methods in terms of matching accuracy and efficiency.

A Combined Back and Foreground-Based Stereo Matching Algorithm Using Belief Propagation and Self-Adapting Dissimilarity Measure

Belief propagation (BP) algorithm still exists some shortages, such as inaccurate edge preservation and ambiguous detail information in the foreground, while self-adapting dissimilarity measure (SDM) also exists some shortages, such as ill textureless and occluded information in the background. To address these problems, we present a novel stereo matching algorithm fusing BP and SDM with an excellent background and foreground information. Lots of experiments show that BP and SDM can complement each other. BP algorithm can hold the better background information due to message propagation inference, whereas SDM can possess the better foreground information due to detail treatment. Therefore, a piecewise function is proposed, which can combine BP algorithm in an excellent background information and SDM in the foreground information, and greatly improve the disparity effect as a whole. We also expect that this work can attract more attention on combination of local methods and global methods, due to its simplicity, efficiency, and accuracy. Experimental results show that the proposed method can keep the superior performance and hold better background and foreground on the Middlebury datasets, compared to BP and SDM.

Intensity guided cost metric for fast stereo matching under radiometric variations.

Reliable and efficient stereo matching is a challenging task due to the presence of multiple radiometric variations. In stereo matching, correspondence between left and right images can become hard owing to low correlation between radiometric changes in left and right images. Previously presented cost metrics are not robust enough against intensive radiometric variations and/or are computationally expensive. In this work, we propose a new similarity metric coined as Intensity Guided Cost Metric (IGCM). IGCM turns out to significantly contribute to the depth accuracy by rejecting outliers and reducing the edge-fattening effect in object boundaries. IGCM is further combined explicitly with a color formation model to handle various radiometric changes that occur between stereo images. Experimental results on Middlebury dataset show 13.8%, 22.8%, 20.9%, 19.5 % and 9.1% decrease in average error rate compared to Adaptive Normalized Cross-Correlation (ANCC), Dense Adaptive Self-Correlation (DASC), Adaptive Descriptor(AD), Fast Cost Volume Filtering (FCVF) and Iterative Guided Filter (IGF)-based methods, respectively. Moreover, using integral images IGCM can achieve a speedup of 20x, 6x, 41x, 25x and 45x compared to the aforementioned methods.

Disparity refinement process based on RANSAC plane fitting for machine vision applications

This paper presents a new disparity map refinement process for stereo matching algorithm and the refinement stage that will be implemented by partitioning the place or mask image and re-projected to the preliminary disparity images. This process is to refine the noise and sparse of initial disparity map from weakly textured. The plane fitting algorithm is using Random Sample Consensus. Two well-known stereo matching algorithms have been tested on this framework with different filtering techniques applied at disparity refinement stage. The framework is evaluated on three Middlebury datasets. The experimental results show that the proposed framework produces better-quality and more accurate than normal flow state-of-the-art stereo matching algorithms. The performance evaluations are based on standard image quality metrics i.e. structural similarity index measure, peak signal-to-noise ratio and mean square error.

Scene Flow Estimation Based on 3D Local Rigidity Assumption and Depth Map Driven Anisotropic Smoothness

In this paper, we present a novel method to estimate the dense scene flow from the aligned depth map and color image using variational framework. For scene flow estimation, most scenes can be seen as scenes composed by independent 3-D rigid parts, so we apply 3-D local rigidity assumption to data term as a fidelity measure for each pixel. Meanwhile, in order to improve the accuracy of scene flow estimation at the boundaries of motion, we assume that depth map and color image are aligned and utilize the boundaries information of depth map to yield smoothness term which is weighted by a depth map driven anisotropic diffusion tensor. In addition, an efficient numerical algorithm named primal-dual algorithm is implemented for the variational formulation of scene flow estimation. Our method is tested on the Middlebury data sets, and the real-world scene data set captured by KINECT. Experimental results show that our method can receive dense and accurate scene flow and preserve motion boundaries well.

An Improved Fractional-Order Optical Flow Model for Motion Estimation

The Horn and Schunck (HS) optical flow model cannot preserve discontinuity of motion estimation and has low accuracy especially for the image sequence, which includes complex texture. To address this problem, an improved fractional-order optical flow model is proposed. In particular, the fractional-order Taylor series expansion is applied in the brightness constraint equation of the HS model. The fractional-order flow field derivative is also used in the smoothing constraint equation. The Euler-Lagrange equation is utilized for the minimization of the energy function of the fractional-order optical flow model. Two-dimensional fractional differential masks are proposed and applied to the calculation of the model simplification. Considering the spatiotemporal memory property of fractional-order, the algorithm preserves the edge discontinuity of the optical flow field while improving the accuracy of the estimation of the dense optical flow field. Experiments on Middlebury datasets demonstrate the predominance of our proposed algorithm.