Stereo Algorithm Research Articles

Spatiotemporal Data Fusion using Dictionary Learning and Temporal Edge Primitives

The main target of stereo matching algorithms is to find out the three dimensional (3D) distance, or depth of objects from a stereo pair of images. Depth information can be derived from images using disparity map of the same scene. There are many applications of computer vision like People tracking, Gesture recognition, Industrial automation and inspection, Security and Biometrics, Three-dimensional modeling, Web and Cloud, Aerial surveys etc. There are large categories of stereo algorithms which are used for finding the disparity or depth. This paper presents a proposed stereo matching algorithm to obtain depth map, enhance and measure. The hybrid mathematical process of the algorithm are color conversion, block matching, guided filtering, Minimum disparity assignment design, mathematical perimeter, zero depth assignment, combination of hole filling and permutation of morphological operator and last non linear spatial filtering. Our algorithm is produce noise less, reliable, smooth and efficient depth map. We obtained the results with ground truth image using Structural Similarity Index Map (SSIM) and Peak Signal to Noise Ratio (PSNR). Index Terms—Stereo Matching; Disparity; Depth; Morphological operator; guided filter, Zero depth.

EVALUATION OF STEREO ALGORITHMS FOR OBSTACLE DETECTION WITH FISHEYE LENSES

Abstract. For autonomous navigation of micro aerial vehicles (MAVs), a robust detection of obstacles with onboard sensors is necessary in order to avoid collisions. Cameras have the potential to perceive the surroundings of MAVs for the reconstruction of their 3D structure. We equipped our MAV with two fisheye stereo camera pairs to achieve an omnidirectional field-of-view. Most stereo algorithms are designed for the standard pinhole camera model, though. Hence, the distortion effects of the fisheye lenses must be properly modeled and model parameters must be identified by suitable calibration procedures. In this work, we evaluate the use of real-time stereo algorithms for depth reconstruction from fisheye cameras together with different methods for calibration. In our experiments, we focus on obstacles occurring in urban environments that are hard to detect due to their low diameter or homogeneous texture.

Reconstructing building mass models from UAV images

We present an automatic reconstruction pipeline for large scale urban scenes from aerial images captured by a camera mounted on an unmanned aerial vehicle. Using state-of-the-art Structure from Motion and Multi-View Stereo algorithms, we first generate a dense point cloud from the aerial images. Based on the statistical analysis of the footprint grid of the buildings, the point cloud is classified into different categories (i.e., buildings, ground, trees, and others). Roof structures are extracted for each individual building using Markov random field optimization. Then, a contour refinement algorithm based on pivot point detection is utilized to refine the contour of patches. Finally, polygonal mesh models are extracted from the refined contours. Experiments on various scenes as well as comparisons with state-of-the-art reconstruction methods demonstrate the effectiveness and robustness of the proposed method.

结构化场景重建中一种融合直线信息的平面拟合方法

Structured scenes are an important element of 3D reconstruction. Characteristics of structured scenes are analyzed, based on which a modified PEaRL algorithm is proposed to perform plane fitting on points provided by multiple view stereo algorithms. In this method, candidate planes are generated from both 3D lines and 3D points. The number of candidate planes is reduced using information provided by the 3D lines; furthermore, the reliability of the planes is improved. Subsequently, 3D lines are employed to obtain a more reliable neighborhood system, which leads to more accurate plane fitting results at plane intersections. For energy optimization, the smoothness term and label cost term are improved to better manage the irregularity of multi-view stereo(MVS)point clouds. Experiments on synthesized and actual data show that the proposed method provides satisfactory time efficiency, and obtains more reliable plane fitting results than other multi-model fitting methods, including PEaRL.

Vegetation Height Estimation Near Power transmission poles Via satellite Stereo Images using 3D Depth Estimation Algorithms

Abstract. Monitoring vegetation encroachment under overhead high voltage power line is a challenging problem for electricity distribution companies. Absence of proper monitoring could result in damage to the power lines and consequently cause blackout. This will affect electric power supply to industries, businesses, and daily life. Therefore, to avoid the blackouts, it is mandatory to monitor the vegetation/trees near power transmission lines. Unfortunately, the existing approaches are more time consuming and expensive. In this paper, we have proposed a novel approach to monitor the vegetation/trees near or under the power transmission poles using satellite stereo images, which were acquired using Pleiades satellites. The 3D depth of vegetation has been measured near power transmission lines using stereo algorithms. The area of interest scanned by Pleiades satellite sensors is 100 square kilometer. Our dataset covers power transmission poles in a state called Sabah in East Malaysia, encompassing a total of 52 poles in the area of 100 km. We have compared the results of Pleiades satellite stereo images using dynamic programming and Graph-Cut algorithms, consequently comparing satellites’ imaging sensors and Depth-estimation Algorithms. Our results show that Graph-Cut Algorithm performs better than dynamic programming (DP) in terms of accuracy and speed.

Adaptive stereo similarity fusion using confidence measures

In most stereo-matching algorithms, stereo similarity measures are used to determine which image patches in a left–right image pair correspond to each other. Different similarity measures may behave very differently on different kinds of image structures, for instance, some may be more robust to noise whilst others are more susceptible to small texture variations. As a result, it may be beneficial to use different similarity measures in different image regions. We present an adaptive stereo similarity measure that achieves this via a weighted combination of measures, in which the weights depend on the local image structure. Specifically, the weights are defined as a function of a confidence measure on the stereo similarities: similarity measures with a higher confidence at a particular image location are given higher weight. We evaluate the performance of our adaptive stereo similarity measure in both local and global stereo algorithms on standard benchmarks such as the Middlebury and KITTI data sets. The results of our experiments demonstrate the potential merits of our adaptive stereo similarity measure.

Accurate Normal and Reflectance Recovery Using Energy Optimization

In this paper, we propose a novel energy optimization framework to accurately estimate surface normal and reflectance of an object from an input image sequence. Input images are captured from a fixed viewpoint under varying lighting conditions. In the proposed approach we combine photometric stereo and Retinex constraints into our energy function. To formulate inter-image constraints, shading information is added to the Lambertian model to account for shadows. For intra-image constraints, we moderate the strength of shading smoothness according to shadow mask and normal variations. By minimizing this energy function we are able to recover accurate surface normals and reflectance. Experimental results show that our approach yields more realistic normal map and accurate albedo map than the state-of-the-art uncalibrated photometric stereo algorithms. As for intrinsic image decomposition, results on the real and synthetic scenes show that the proposed approach outperforms previous ones.

Automatic registration of UAV-borne sequent images and LiDAR data

Use of direct geo-referencing data leads to registration failure between sequent images and LiDAR data captured by mini-UAV platforms because of low-cost sensors. This paper therefore proposes a novel automatic registration method for sequent images and LiDAR data captured by mini-UAVs. First, the proposed method extracts building outlines from LiDAR data and images and estimates the exterior orientation parameters (EoPs) of the images with building objects in the LiDAR data coordinate framework based on corresponding corner points derived indirectly by using linear features. Second, the EoPs of the sequent images in the image coordinate framework are recovered using a structure from motion (SfM) technique, and the transformation matrices between the LiDAR coordinate and image coordinate frameworks are calculated using corresponding EoPs, resulting in a coarse registration between the images and the LiDAR data. Finally, 3D points are generated from sequent images by multi-view stereo (MVS) algorithms. Then the EoPs of the sequent images are further refined by registering the LiDAR data and the 3D points using an iterative closest-point (ICP) algorithm with the initial results from coarse registration, resulting in a fine registration between sequent images and LiDAR data. Experiments were performed to check the validity and effectiveness of the proposed method. The results show that the proposed method achieves high-precision robust co-registration of sequent images and LiDAR data captured by mini-UAVs.

Stochastic Optimization Based 3D Dense Reconstruction from Multiple Views with High Accuracy and Completeness

This paper presents a stochastic optimization based 3D dense reconstruction from multiple views. Accuracy and completeness are two major measure indices for performance evaluation of various multi-view stereo (MVS) algorithms. First, the reconstruction accuracy is highly related to the stereo mismatches over the multiple views. Stereo mismatches occur in the image regions involving the lack of texture, depth discontinuity, or repeated texture patterns. Second, an insufficient number of views or occlusions between objects also lead to the difficulty in matching so that the reconstruction completeness degrades. In pursuit of high accuracy and completeness we present the appropriate techniques to solve the above problems in the reconstruction task. To deal with the various stereo mismatch problems we propose to apply adaptive matching functions and allow partial matching. We shall model the object to be reconstructed by a set of 3D oriented planar patches covering the visible object surface. The adopted multi-view reconstruction is formulated as a patch expansion process under a tree hierarchy. In order to find the optimal patches via multi-view stereo matching we shall employ a PSO (Particle Swarm Optimization) method for the sake of implementation simplicity and avoidance of possible local traps as found in the derivative based optimization methods. The success in the PSO method relies on imposing proper constraints on ranges of the patch parameters including the patch depth and patch normal vector which are involved in the PSO objective function (i.e., the stereo matching function). Furthermore, we use a varying patch size to obtain the reliable patches in the areas containing less texture, repeated texture pattern, or depth discontinuity. To secure a high reconstruction quality we advocate a patch priority queue to select the best patch during the patch expansion. All of the above mentioned techniques are also effective in the situations when the number of views is sparse or the camera baseline width is wide. The proposed method is tested on synthetic and real image data sets. The experimental results indicate that the proposed method is superior or comparable to the top ranked reconstruction methods reported in the public Middlebury MVS evaluation website.

Phase-Based Window Matching with Geometric Correction for Multi-View Stereo

SUMMARY Methods of window matching to estimate 3D points are the most serious factors affecting the accuracy, robustness, and computational cost of Multi-View Stereo (MVS) algorithms. Most existing MVS algorithms employ window matching based on Normalized CrossCorrelation (NCC) to estimate the depth of a 3D point. NCC-based window matching estimates the displacement between matching windows with sub-pixel accuracy by linear/ cubic interpolation, which does not represent accurate sub-pixel values of matching windows. This paper proposes a technique of window matching that is very accurate using Phase-Only Correlation (POC) with geometric correction for MVS. The accurate sub-pixel displacement between two matching windows can be estimated by fitting the analytical correlation peak model of the POC function. The proposed method also corrects the geometric transformations of matching windows by taking into consideration the 3D shape of a target object. The use of the proposed geometric correction approach makes it possible to achieve accurate 3D reconstruction from multi-view images even for images with large transformations. The proposed method demonstrates more accurate 3D reconstruction from multi-view images than the conventional methods

A plane tting approach using line information to reconstruct structured scenes

Structured scenes are an important element of 3D reconstruction. Characteristics of structured scenes are analyzed, based on which a modified PEaRL algorithm is proposed to perform plane fitting on points provided by multiple view stereo algorithms. In this method, candidate planes are generated from both 3D lines and 3D points. The number of candidate planes is reduced using information provided by the 3D lines; furthermore, the reliability of the planes is improved. Subsequently, 3D lines are employed to obtain a more reliable neighborhood system, which leads to more accurate plane fitting results at plane intersections. For energy optimization, the smoothness term and label cost term are improved to better manage the irregularity of multi-view stereo（MVS）point clouds. Experiments on synthesized and actual data show that the proposed method provides satisfactory time efficiency, and obtains more reliable plane fitting results than other multi-model fitting methods, including PEaRL.

DSM Accuracy Evaluation for the ISPRS Commission I Image Matching Benchmark

Abstract. To improve the quality of algorithms for automatic generation of Digital Surface Models (DSM) from optical stereo data in the remote sensing community, the Working Group 4 of Commission I: Geometric and Radiometric Modeling of Optical Airborne and Spaceborne Sensors provides on its website http://www2.isprs.org/commissions/comm1/wg4/benchmark-test.html a benchmark dataset for measuring and comparing the accuracy of dense stereo algorithms. The data provided consists of several optical spaceborne stereo images together with ground truth data produced by aerial laser scanning. In this paper we present our latest work on this benchmark, based upon previous work. As a first point, we noticed that providing the abovementioned test data as geo-referenced satellite images together with their corresponding RPC camera model seems too high a burden for being used widely by other researchers, as a considerable effort still has to be made to integrate the test datas camera model into the researchers local stereo reconstruction framework. To bypass this problem, we now also provide additional rectified input images, which enable stereo algorithms to work out of the box without the need for implementing special camera models. Care was taken to minimize the errors resulting from the rectification transformation and the involved image resampling. We further improved the robustness of the evaluation method against errors in the orientation of the satellite images (with respect to the LiDAR ground truth). To this end we implemented a point cloud alignment of the DSM and the LiDAR reference points using an Iterative Closest Point (ICP) algorithm and an estimation of the best fitting transformation. This way, we concentrate on the errors from the stereo reconstruction and make sure that the result is not biased by errors in the absolute orientation of the satellite images. The evaluation of the stereo algorithms is done by triangulating the resulting (filled) DSMs and computing for each LiDAR point the nearest Euclidean distance to the DSM surface. We implemented an adaptive triangulation method minimizing the second order derivative of the surface in a local neighborhood, which captures the real surface more accurate than a fixed triangulation. As a further advantage, using our point-to-surface evaluation, we are also able to evaluate non-uniformly sampled DSMs or triangulated 3D models in general. The latter is for example needed when evaluating building extraction and data reduction algorithms. As practical example we compare results from three different matching methods applied to the data available within the benchmark data sets. These results are analyzed using the above mentioned methodology and show advantages and disadvantages of the different methods, also depending on the land cover classes.

Automated recovery of three-dimensional models of plant shoots from multiple color images.

Increased adoption of the systems approach to biological research has focused attention on the use of quantitative models of biological objects. This includes a need for realistic three-dimensional (3D) representations of plant shoots for quantification and modeling. Previous limitations in single-view or multiple-view stereo algorithms have led to a reliance on volumetric methods or expensive hardware to record plant structure. We present a fully automatic approach to image-based 3D plant reconstruction that can be achieved using a single low-cost camera. The reconstructed plants are represented as a series of small planar sections that together model the more complex architecture of the leaf surfaces. The boundary of each leaf patch is refined using the level-set method, optimizing the model based on image information, curvature constraints, and the position of neighboring surfaces. The reconstruction process makes few assumptions about the nature of the plant material being reconstructed and, as such, is applicable to a wide variety of plant species and topologies and can be extended to canopy-scale imaging. We demonstrate the effectiveness of our approach on data sets of wheat (Triticum aestivum) and rice (Oryza sativa) plants as well as a unique virtual data set that allows us to compute quantitative measures of reconstruction accuracy. The output is a 3D mesh structure that is suitable for modeling applications in a format that can be imported in the majority of 3D graphics and software packages.

Evaluation of feature-based 3-d registration of probabilistic volumetric scenes

Automatic estimation of the world surfaces from aerial images has seen much attention and progress in recent years. Among current modeling technologies, probabilistic volumetric models (PVMs) have evolved as an alternative representation that can learn geometry and appearance in a dense and probabilistic manner. Recent progress, in terms of storage and speed, achieved in the area of volumetric modeling, opens the opportunity to develop new frameworks that make use of the PVM to pursue the ultimate goal of creating an entire map of the earth, where one can reason about the semantics and dynamics of the 3-d world. Aligning 3-d models collected at different time-instances constitutes an important step for successful fusion of large spatio-temporal information. This paper evaluates how effectively probabilistic volumetric models can be aligned using robust feature-matching techniques, while considering different scenarios that reflect the kind of variability observed across aerial video collections from different time instances. More precisely, this work investigates variability in terms of discretization, resolution and sampling density, errors in the camera orientation, and changes in illumination and geographic characteristics. All results are given for large-scale, outdoor sites. In order to facilitate the comparison of the registration performance of PVMs to that of other 3-d reconstruction techniques, the registration pipeline is also carried out using Patch-based Multi-View Stereo (PMVS) algorithm. Registration performance is similar for scenes that have favorable geometry and the appearance characteristics necessary for high quality reconstruction. In scenes containing trees, such as a park, or many buildings, such as a city center, registration performance is significantly more accurate when using the PVM.

Local stereo matching with adaptive shape support window based cost aggregation.

Cost aggregation is the most important step in a local stereo algorithm. In this work, a novel local stereo-matching algorithm with a cost-aggregation method based on adaptive shape support window (ASSW) is proposed. First, we compute the initial cost volume, which uses both absolute intensity difference and gradient similarity to measure dissimilarity. Second, we apply an ASSW-based cost-aggregation method to get the aggregated cost within the support window. There are two main parts: at first we construct a local support skeleton anchoring each pixel with four varying arm lengths decided on color similarity; as a result, the support window integral of multiple horizontal segments spanned by pixels in the neighboring vertical is established. Then we utilize extended implementation of guided filter to aggregate cost volume within the ASSW, which has better edge-preserving smoothing property than bilateral filter independent of the filtering kernel size. In this way, the number of bad pixels located in the incorrect depth regions can be effectively reduced through finding optimal support windows with an arbitrary shape and size adaptively. Finally, the initial disparity value of each pixel is selected using winner takes all optimization and post processing symmetrically, considering both the reference and the target image, is adopted. The experimental results demonstrate that the proposed algorithm achieves outstanding matching performance compared with other existing local algorithms on the Middlebury stereo benchmark, especially in depth discontinuities and piecewise smooth regions.

Photometric Stereo Using Sparse Bayesian Regression for General Diffuse Surfaces.

Most conventional algorithms for non-Lambertian photometric stereo can be partitioned into two categories. The first category is built upon stable outlier rejection techniques while assuming a dense Lambertian structure for the inliers, and thus performance degrades when general diffuse regions are present. The second utilizes complex reflectance representations and non-linear optimization over pixels to handle non-Lambertian surfaces, but does not explicitly account for shadows or other forms of corrupting outliers. In this paper, we present a purely pixel-wise photometric stereo method that stably and efficiently handles various non-Lambertian effects by assuming that appearances can be decomposed into a sparse, non-diffuse component (e.g., shadows, specularities, etc.) and a diffuse component represented by a monotonic function of the surface normal and lighting dot-product. This function is constructed using a piecewise linear approximation to the inverse diffuse model, leading to closed-form estimates of the surface normals and model parameters in the absence of non-diffuse corruptions. The latter are modeled as latent variables embedded within a hierarchical Bayesian model such that we may accurately compute the unknown surface normals while simultaneously separating diffuse from non-diffuse components. Extensive evaluations are performed that show state-of-the-art performance using both synthetic and real-world images.

Stereo Model Selection and Point Cloud Filtering using an Out-of-Core Octree

Abstract. Dense image matching methods enable the retrieval of dense surface information using any kind of imagery. The best quality can be achieved for highly overlapping datasets, which avoids occlusions and provides highly redundant observations. Thus, images are acquired close to each other. This leads to datasets with increasing size – especially when large scenes are captured. While image acquisition can be performed in relatively short time, more time is required for data processing due to the computational complexity of the involved algorithms. For the dense surface reconstruction task, Multi-View Stereo algorithms can be used – which are typically beneficial due to the efficiency of image matching on stereo models. Our dense image matching solution SURE uses such an approach, where the result of stereo matching is fused using a multi-stereo triangulation in order to exploit the available redundancy. One key challenge of such Multi-View Stereo methods is the selection of suitable stereo models, where object space information should be considered to avoid unnecessary processing. Subsequently, the dense image matching step provides up to one 3D point for each pixel, which leads to massive point clouds. This large amount of 3D data needs to be filtered and integrated efficiently in object space. Within this paper, we present an out-of-core octree, which enables neighborhood and overlap analysis between point clouds. It is used on low-resolution point clouds to support the stereo model selection. Also, this tree is designed for the processing of massive point clouds with low memory requirements and thus can be used to perform outlier rejection, redundancy removal and resampling.

Line Matching Across Views Based on Multiple View Stereo

A graph-based multiple view line matching method is proposed based on results of multiple view stereo (MVS) algorithms. With the 3D points and their visibility information provided by MVS, point-line correspondences are firstly established through 3D-to-2D re-projection. Each image line detected in different views is described using a 3D point set as well as a unit vector representing its coarse 3D direction. From such a description, pairwise similarity and consistency are evaluated. Then, a graph is constructed to contain all image lines as nodes. To get a unified node distance measure, a spectral graph analysis method is employed. Finally, a modified DBSCAN algorithm is introduced to obtain reliable line matches from the graph. Experiments show that our method is more robust and exhibits better accuracy than the existing methods.

Comparing Two Photo-Reconstruction Methods to Produce High Density Point Clouds and DEMs in the Corral del Veleta Rock Glacier (Sierra Nevada, Spain)

In this paper, two methods based on computer vision are presented in order to produce dense point clouds and high resolution DEMs (digital elevation models) of the Corral del Veleta rock glacier in Sierra Nevada (Spain). The first one is a semi-automatic 3D photo-reconstruction method (SA-3D-PR) based on the Scale-Invariant Feature Transform algorithm and the epipolar geometry theory that uses oblique photographs and camera calibration parameters as input. The second method is fully automatic (FA-3D-PR) and is based on the recently released software 123D-Catch that uses the Structure from Motion and MultiView Stereo algorithms and needs as input oblique photographs and some measurements in order to scale and geo-reference the resulting model. The accuracy of the models was tested using as benchmark a 3D model registered by means of a Terrestrial Laser Scanner (TLS). The results indicate that both methods can be applied to micro-scale study of rock glacier morphologies and processes with average distances to the TLS point cloud of 0.28 m and 0.21 m, for the SA-3D-PR and the FA-3D-PR methods, respectively. The performance of the models was also tested by means of the dimensionless relative precision ratio parameter resulting in figures of 1:1071 and 1:1429 for the SA-3D-PR and the FA-3D-PR methods, respectively. Finally, Digital Elevation Models (DEMs) of the study area were produced and compared with the TLS-derived DEM. The results showed average absolute differences with the TLS-derived DEM of 0.52 m and 0.51 m for the SA-3D-PR and the FA-3D-PR methods, respectively.

Multimodal Stereo from Thermal Infrared and Visible Spectrum

Recent advances in thermal infrared imaging (LWIR) has allowed its use in applications beyond of the military domain. Nowadays, this new family of sensors is included in different technical and scientific applications. They offer features that facilitate tasks, such as detection of pedestrians, hot spots, differences in temperature, among others, which can significantly improve the performance of a system where the persons are expected to play the principal role. For instance, video surveillance applications, monitoring, and pedestrian detection.During the dissertation the next question is stated: \textit{Could a couple of sensors measuring different bands of the electromagnetic spectrum, as the visible and thermal infrared, be used to extract depth information?} Although it is a complex question, we shows that a system of these characteristics is possible as well as their advantages, drawbacks, and potential opportunities.In this research an experimental study that compares different cost functions and matching approaches is performed, in order to build a multimodalstereovision system. Furthermore, the common problems in infrared/visible stereo, specially in the outdoor scenes are identified. Our framework summarizes the architecture of a generic stereo algorithm, at different levels: computational, functional, and structural, which can be extended toward high-level fusion (semantic) and high-order (prior).The proposed framework is intended to explore novel multimodal stereo matching approaches, going from sparse to dense representations (both disparity and depth maps). Moreover, context information is added in form of priors and assumptions. Finally, the dissertation shows a promissory way toward the integration of multiple sensors for recovering three-dimensional information.