Stereo Method Research Articles

A Benchmark Dataset and Evaluation for Non-Lambertian and Uncalibrated Photometric Stereo

Classic photometric stereo is often extended to deal with real-world materials and work with unknown lighting conditions for practicability. To quantitatively evaluate non-Lambertian and uncalibrated photometric stereo, a photometric stereo image dataset containing objects of various shapes with complex reflectance properties and high-quality ground truth normals is still missing. In this paper, we introduce the 'DiLiGenT' dataset with calibrated Directional Lightings, objects of General reflectance with different shininess, and 'ground Truth' normals from high-precision laser scanning. We use our dataset to quantitatively evaluate state-of-the-art photometric stereo methods for general materials and unknown lighting conditions, selected from a newly proposed photometric stereo taxonomy emphasizing non-Lambertian and uncalibrated methods. The dataset and evaluation results are made publicly available, and we hope it can serve as a benchmark platform that inspires future research.

OSSIM: An Object-Based Multiview Stereo Algorithm Using SSIM Index Matching Cost

Multiview stereo (MVS) is a crucial process in image-based automatic 3-D reconstruction and mapping applications. In a dense matching process, the matching cost is generally computed between image pairs, making the efficiency low due to the large number of stereo pairs. This paper presents a novel object-based MVS algorithm using structural similarity (SSIM) index matching cost in a coarse-to-fine workflow. As far as we know, this is the first time SSIM index is introduced to calculate the matching cost of MVS applications. In contrast to classical stereo methods, the proposed object-based structural similarity (OSSIM) method computes only a depth map for each image. Thus, the efficiency can be greatly improved when the overlap between images is large. To obtain an optimized depth map, the winner-take-all and semi-global matching strategies are implemented. Moreover, an object-based multiview consistency checking strategy is also proposed to eliminate wrong matches and perform pixelwise view selection. The proposed method was successfully applied on a close-range Fountain-P11 data set provided by EPFL and aerial data sets of Vaihingen and Zurich by the ISPRS. Experimental results demonstrate that the proposed method can deliver matches at high completeness and accuracy. For the Vaihingen data set, the correctness and completeness rate were 71.12% and 95.99% with an RMSE of 2.8 GSD. For the Foutain-P11 data set, the proposed method outperformed the other existing methods with the ratio of pixels less than 2 cm. Extensive comparison using Zurich data set shows that it can derive results comparable to the state-of-the-art software (PhotoScan, Pix4d, and Smart3D) in urban buildings areas.

SOFT‐SLAM: Computationally efficient stereo visual simultaneous localization and mapping for autonomous unmanned aerial vehicles

AbstractAutonomous navigation of unmanned aerial vehicles (UAVs) in GPS‐denied environments is a challenging problem, especially for small‐scale UAVs characterized by a small payload and limited battery autonomy. A possible solution to the aforementioned problem is vision‐based simultaneous localization and mapping (SLAM), since cameras, due to their dimensions, low weight, availability, and large information bandwidth, circumvent all the constraints of UAVs. In this paper, we propose a stereo vision SLAM yielding very accurate localization and a dense map of the environment developed with the aim to compete in the European Robotics Challenges (EuRoC) targeting airborne inspection of industrial facilities with small‐scale UAVs. The proposed approach consists of a novel stereo odometry algorithm relying on feature tracking (SOFT), which currently ranks first among all stereo methods on the KITTI dataset. Relying on SOFT for pose estimation, we build a feature‐based pose graph SLAM solution, which we dub SOFT‐SLAM. SOFT‐SLAM has a completely separate odometry and mapping threads supporting large loop‐closing and global consistency. It also achieves a constant‐time execution rate of 20 Hz with deterministic results using only two threads of an onboard computer used in the challenge. The UAV running our SLAM algorithm obtained the highest localization score in the EuRoC Challenge 3, Stage IIa–Benchmarking, Task 2. Furthermore, we also present an exhaustive evaluation of SOFT‐SLAM on two popular public datasets, and we compare it to other state‐of‐the‐art approaches, namely ORB‐SLAM2 and LSD‐SLAM. The results show that SOFT‐SLAM obtains better localization accuracy on the majority of datasets sequences, while also having a lower runtime.

SVCV: segmentation volume combined with cost volume for stereo matching

Stereo matching between binocular stereo images is fundamental to many computer vision tasks, such as three‐dimensional (3D) reconstruction and robot navigation. Various structures of real 3D scenes lead stereo matching to be an old yet still challenging problem. In this study, the authors proposed a novel adaptive support weights technique which exploits the hierarchical information provided by multilevel segmentation to preserve the robustness to imaging conditions and spatial proximity in cost aggregation. Besides, a generalisable cost refinement strategy is designed to remove the matching ambiguity in large weakly textured regions. The proposed strategy utilises both the fluctuation of the filtered cost volume and the colour information to further improve the matching accuracy. Experimental results of 50 stereo images demonstrate the effectiveness and efficiency of the proposed method. Furthermore, a systematic evaluation is developed to assess the conventional steps in local stereo methods and then reliable suggestions are given to the beginners and researchers outside the stereo matching field.

A Photometric Stereo Using Re-Projected Images for Active Stereo Vision System

In optical 3D shape measurement, stereo vision with structured light can measure 3D scan data with high accuracy and is used in many applications, but fine surface detail is difficult to obtain. On the other hand, photometric stereo can capture surface details but has disadvantages, in that its 3D data accuracy drops and it requires multiple light sources. When the two measurement methods are combined, more accurate 3D scan data and detailed surface features can be obtained at the same time. In this paper, we present a 3D optical measurement technique that uses re-projection of images to implement photometric stereo without an external light source. 3D scan data is enhanced by combining normal vector from this photometric stereo method, and the result is evaluated with the ground truth.

Deviation correction method for close-range photometric stereo with nonuniform illumination

Classical photometric stereo requires uniform collimated light, but point light sources are usually employed in practical setups. This introduces errors to the recovered surface shape. We found that when the light sources are evenly placed around the object with the same slant angle, the main component of the errors is the low-frequency deformation, which can be approximately described by a quadratic function. We proposed a postprocessing method to correct the deviation caused by the nonuniform illumination. The method refines the surface shape with prior information from calibration using a flat plane or the object itself. And we further introduce an optimization scheme to improve the reconstruction accuracy when the three-dimensional information of some locations is available. Experiments were conducted using surfaces captured with our device and those from a public dataset. The results demonstrate the effectiveness of the proposed approach.

Stereo vision during adverse weather — Using priors to increase robustness in real-time stereo vision

Stereo vision can deliver a dense 3D reconstruction of the environment in real-time for driver assistance as well as autonomous driving. Semi-Global Matching (SGM) is a popular method of choice for solving this task which is already in use for production vehicles. Despite the enormous progress in the field and the high level of performance of modern stereo methods, one key challenge remains: robust stereo vision in automotive scenarios during rain, snow and darkness. Under these circumstances, current methods generate strong temporal noise, many disparity outliers and false positives on object level. These problems are addressed in this work by regularizing stereo vision via prior information. We formulate a temporal prior and a scene prior, which we apply to SGM in order to overcome the deficiencies. The temporal prior integrates knowledge from the previous disparity map to exploit the high temporal correlation, the scene prior exploits knowledge of a representative traffic scene. Using these priors, the object detection rate improves significantly on a driver assistance dataset of 3000 frames including bad weather while reducing the rate of erroneous object detections. We also outperform the ECCV Robust Vision Challenge 2012 winner, iSGM, on this dataset. In addition, results are presented for the KITTI dataset, even showing improvements under good weather conditions when exploiting the temporal prior.We also show that the temporal and scene priors are easy and efficient to implement on a hybrid CPU/reconfigurable hardware platform. The use of these priors can be extended to other application areas such as mobile robotics.

Photometric Stereo by Hemispherical Metric Embedding

Photometric Stereo methods seek to reconstruct the 3d shape of an object from motionless images obtained with varying illumination. Most existing methods solve a restricted problem where the physical reflectance model, such as Lambertian reflectance, is known in advance. In contrast, we do not restrict ourselves to a specific reflectance model. Instead, we offer a method that works on a wide variety of reflectances. Our approach uses a simple yet uncommonly used property of the problem - the sought after normals are points on a unit hemisphere. We present a novel embedding method that maps pixels to normals on the unit hemisphere. Our experiments demonstrate that this approach outperforms existing manifold learning methods for the task of hemisphere embedding. We further show successful reconstructions of objects from a wide variety of reflectances including smooth, rough, diffuse and specular surfaces, even in the presence of significant attached shadows. Finally, we empirically prove that under these challenging settings we obtain more accurate shape reconstructions than existing methods.

Sparse representation over discriminative dictionary for stereo matching

We propose a novel data-driven matching cost for dense correspondence based on sparse theory. The ability of sparse coding to selectively express the sources of influence on stereo images allows us to learn a discriminative dictionary. The dictionary learning process is incorporated with discriminative learning and weighted sparse coding to enhance the discrimination of sparse coefficients and weaken the influence of radiometric changes. Then, the sparse representations over the learned discriminative dictionary are utilized to measure the dissimilarity between image patches. Semi-global cost aggregation and postprocessings are finally enforced to further improve the matching accuracy. Extensive experimental comparisons demonstrate that: the proposed matching cost outperforms traditional matching costs, the discriminative dictionary learning model is more suitable than previous dictionary learning models for stereo matching, and the proposed stereo method ranks the third place on the Middlebury benchmark v3 in quarter resolution up to the submitting, and achieves the best accuracy on 30 classic stereo images.

METRIC EVALUATION PIPELINE FOR 3D MODELING OF URBAN SCENES

Abstract. Publicly available benchmark data and metric evaluation approaches have been instrumental in enabling research to advance state of the art methods for remote sensing applications in urban 3D modeling. Most publicly available benchmark datasets have consisted of high resolution airborne imagery and lidar suitable for 3D modeling on a relatively modest scale. To enable research in larger scale 3D mapping, we have recently released a public benchmark dataset with multi-view commercial satellite imagery and metrics to compare 3D point clouds with lidar ground truth. We now define a more complete metric evaluation pipeline developed as publicly available open source software to assess semantically labeled 3D models of complex urban scenes derived from multi-view commercial satellite imagery. Evaluation metrics in our pipeline include horizontal and vertical accuracy and completeness, volumetric completeness and correctness, perceptual quality, and model simplicity. Sources of ground truth include airborne lidar and overhead imagery, and we demonstrate a semi-automated process for producing accurate ground truth shape files to characterize building footprints. We validate our current metric evaluation pipeline using 3D models produced using open source multi-view stereo methods. Data and software is made publicly available to enable further research and planned benchmarking activities.

Hyperspectral photometric stereo for a single capture

We present a single-capture photometric stereo method using a hyperspectral camera. A spectrally and spatially designed illumination enables a point-wise estimation of reflectance spectra and surface normals from a single hyperspectral image. The illumination works as a reflectance probe in wide spectral regions where reflectance spectra are measured, and the full spectra are estimated by interpolation. It also works as the resource for shadings in other spectral regions. The accuracy of estimation is evaluated in a simulation. Also, we prepare an experimental setup and demonstrate a surface reconstruction against a real scene.

Robust and dense depth estimation for light field images.

We propose a depth estimation method for light field images. Light field images can be considered as a collection of 2D images taken from different viewpoints arranged in a regular grid. We exploit this configuration and compute the disparity maps between specific pairs of views. This computation is carried out by a state of the art two-view stereo method providing a non dense disparity estimation. We propose a disparity interpolation method increasing the density and improving the accuracy of this initial estimate. Disparities obtained from several pairs of views are fused to obtain a unique and robust estimation. Finally, different experiments on synthetic and real images show how the proposed method outperforms state of the art results.

SymPS: BRDF Symmetry Guided Photometric Stereo for Shape and Light Source Estimation.

We propose uncalibrated photometric stereo methods that address the problem due to unknown isotropic reflectance. At the core of our methods is the notion of "constrained half-vector symmetry" for general isotropic BRDFs. We show that such symmetry can be observed in various real-world materials, and it leads to new techniques for shape and light source estimation. Based on the 1D and 2D representations of the symmetry, we propose two methods for surface normal estimation; one focuses on accurate elevation angle recovery for surface normals when the light sources only cover the visible hemisphere, and the other for comprehensive surface normal optimization in the case that the light sources are also non-uniformly distributed. The proposed robust light source estimation method also plays an essential role to let our methods work in an uncalibrated manner with good accuracy. Quantitative evaluations are conducted with both synthetic and real-world scenes, which produce the state-of-the-art accuracy for all of the non-Lambertian materials in MERL database and the real-world datasets.

Efficient Stereo Matching Leveraging Deep Local and Context Information

Stereo matching is a challenging problem with respect to weak texture, discontinuities, illumination difference and occlusions. Therefore, a deep learning framework is presented in this paper, which focuses on the first and last stage of typical stereo methods: the matching cost computation and the disparity refinement. For matching cost computation, two patch-based network architectures are exploited to allow the trade-off between speed and accuracy, both of which leverage multi-size and multi-layer pooling unit with no strides to learn cross-scale feature representations. For disparity refinement, unlike traditional handcrafted refinement algorithms, we incorporate the initial optimal and sub-optimal disparity maps before outlier detection. Furthermore, diverse base learners are encouraged to focus on specific replacement tasks, corresponding to the smooth regions and details. Experiments on different datasets demonstrate the effectiveness of our approach, which is able to obtain sub-pixel accuracy and restore occlusions to a great extent. Specifically, our accurate framework attains near-peak accuracy both in non-occluded and occluded region and our fast framework achieves competitive performance against the fast algorithms on Middlebury benchmark.

Efficient Photometric Stereo Using Kernel Regression.

Photometric stereo estimates surface normals from multiple images captured under different light directions using a fixed camera. To deal with non-Lambertian reflections, the recent photometric stereo methods employ iterative or optimization frameworks that are computationally expensive. This paper proposes an efficient photometric stereo method using kernel regression, which can be transformed to an eigendecomposition problem. The kernel parameter is variable for each surface point so that it can cope with the variety of general reflectances. The best kernel parameter is automatically determined by leave-one-out cross validation. To improve computational efficiency, the leave-one-out process is accelerated by fast matrix computation and proper normal initialization. The proposed photometric stereo method is extensively evaluated on synthetic and real surfaces with various reflectances. Experimental results validate that the method is computationally efficient and achieves the state-of-the-art accuracy in surface normal estimation.

Effective energy-based multi-view piecewise planar stereo

For piecewise planar scene modeling, many challenging issues still persist, in particular, how to generate sufficient candidate planes and how to assign an optimal plane for each spatial patch. To address these issues, we present a novel multi-view piecewise planar stereo method for the complete reconstruction. In our method, reconstruction is formulated as an energy-based plane labeling problem, where photo-consistency and geometric constraints are incorporated to a unified superpixel-level MRF (Markov Random Field) framework. To enhance the efficacy of the plane inference and optimization, an effective multi-direction plane sweeping with much restricted search space is carried out to generate sufficient and reliable candidate planes. Experiments show that our method can effectively handle many challenging factors (e.g., slant surfaces, textureless regions) and achieve satisfactory results.

Computationally Efficient Two Stage Sequential Framework for Stereo Matching

Almost all the existing stereo algorithms fall under a common assumption that corresponding color or intensity values will be similar like one another. On the other hand, it is also not that true in practice where the image color or intensity values are regularly affected by different radiometric factors like illumination direction, change in image device, illuminant color and so on. For this issue, the information about the raw color of the images which is recorded by the camera should not depend on it totally, and also the common assumptions on color consistency doesn’t influence good (great) between the stereo images in real scenario. Therefore, most of the conventional stereo algorithms can be seriously degraded in terms of performance under radiometric variations. In this work, we intend to develop a new stereo matching algorithm which will be insensitive to change in radiometric conditions between stereo pairs i.e. left image as well as right image. Unlike the other stereo algorithms, we propose a computationally efficient two stage sequential framework for stereo matching which can handle the various radiometric variations between the stereo pairs.Experimental results proves that the proposed method outperforms extremely well compare to other state of the art stereo methods under change in various radiometric conditions for a given stereo pair and it is also found from the results that the execution time is less compare to existing methods.

Energy-based multi-view piecewise planar stereo

The piecewise planar model (PPM) is an effective means of approximating a complex scene by using planar patches to give a complete interpretation of the spatial points reconstructed from projected 2D images. The traditional piecewise planar stereo methods suffer from either a very restricted number of directions for plane detection or heavy reliance on the segmentation accuracy of superpixels. To address these issues, we propose a new multi-view piecewise planar stereo method in this paper. Our method formulates the problem of complete scene reconstruction as a multi-level energy minimization problem. To detect planes along principal directions, a novel energy formulation with pair-wise potentials is used to assign an optimal plane for each superpixel in an iterative manner, where reliable scene priors and geometric constraints are incorporated to enhance the modeling efficacy and inference efficiency. To detect non-principal-direction planes, we adopt a multi-direction plane sweeping with a restricted search space method to generate reliable candidate planes. To handle the multi-surface straddling problem of a single superpixel, a superpixel sub-segmenting scheme is proposed and a robust Pn Potts model-like higher-order potential is introduced to refine the resulting depth map. Our method is a natural integration of pixel- and superpixel-level multi-view stereos under a unified energy minimization framework. Experimental results for standard data sets and our own data sets show that our proposed method can satisfactorily handle many challenging factors (e.g., slanted surfaces and poorly textured regions) and can obtain accurate piecewise planar depth maps.

Robust Multiview Photometric Stereo Using Planar Mesh Parameterization.

We propose a robust uncalibrated multiview photometric stereo method for high quality 3D shape reconstruction. In our method, a coarse initial 3D mesh obtained using a multiview stereo method is projected onto a 2D planar domain using a planar mesh parameterization technique. We describe methods for surface normal estimation that work in the parameterized 2D space that jointly incorporates all geometric and photometric cues from multiple viewpoints. Using an estimated surface normal map, a refined 3D mesh is then recovered by computing an optimal displacement map in the same 2D planar domain. Our method avoids the need of merging view-dependent surface normal maps that is often required in conventional methods. We conduct evaluation on various real-world objects containing surfaces with specular reflections, multiple albedos, and complex topologies in both controlled and uncontrolled settings and demonstrate that accurate 3D meshes with fine geometric details can be recovered by our method.

A TV Prior for High-Quality Scalable Multi-View Stereo Reconstruction

We present a scalable multi-view stereo method able to reconstruct accurate 3D models from hundreds of high-resolution input images. Local fusion of disparity maps obtained with semi-global matchin...