Dense Stereo Reconstruction Research Articles

EnhancedNet, an End-to-End Network for Dense Disparity Estimation and its Application to Aerial Images

Recent developments in deep learning technology have boosted the performance of dense stereo reconstruction. However, the state-of-the-art deep learning-based stereo matching methods are mainly trained using close-range synthetic images. Consequently, the application of these methods in aerial photogrammetry and remote sensing is currently far from straightforward. In this paper, we propose a new disparity estimation network for stereo matching and investigate its generalization abilities in regard to aerial images. First, we propose an end-to-end deep learning network for stereo matching, regularized by disparity gradients, which includes a residual cost volume and a reconstruction error volume in a refinement module, and multiple losses. In order to investigate the influence of the multiple losses, a comprehensive analysis is presented. Second, based on this network trained with synthetic close-range data, we propose a new pipeline for matching high-resolution aerial imagery. The experimental results show that the proposed network improves the disparity accuracy by up to 40% in terms of errors larger than 1 px compared to results when not including the refinement network, especially in areas containing detailed small objects. In addition, in qualitative and quantitative experiments, we are able to show that our model, pre-trained on a synthetic stereo dataset, achieves very competitive sub-pixel geometric accuracy on aerial images. These results confirm that the domain gap between synthetic close-range and real aerial images can be satisfactorily bridged using the proposed new deep learning method for dense image matching.

Accurate, dense and shading-aware multi-view stereo reconstruction using metaheuritic optimization

Among the modern means of 3D geometry creation that exist in the literature, there are the Multi-View Stereo (MVS) reconstruction methods that received much attention from the research community and the multimedia industry. In fact, several methods showed that it is possible to recover geometry only from images with reconstruction accuracies paralleling that of excessively expensive laser scanners. The majority of these methods perform on images such as online community photo collection and estimate the surface position with its orientation by minimizing a matching cost function defined over a small local region. However, these datasets not only they are large but also contain more challenging scenes setups with different photometric effects; therefore fine-grained details of an object’s surface cannot be captured. This paper presents a robust multi-view stereo method based on metaheuristic optimization namely the Particle Swarm Optimization (PSO) in order to find the optimal depth, orientation, and surface roughness. To deal with the various shading and stereo mismatch problems caused by rough surfaces, shadows, and interreflections, we propose to use a robust matching/energy function which is a combination of two similarity measurements. Finally, our method computes individual depth maps that can be merged into compelling scene reconstructions. The proposed method is evaluated quantitatively using well-known Middlebury datasets and the obtained results show a high completeness score and comparable accuracy to those of the current top performing algorithms.

Casual 3D photography

We present an algorithm that enables casual 3D photography. Given a set of input photos captured with a hand-held cell phone or DSLR camera, our algorithm reconstructs a 3D photo , a central panoramic, textured, normal mapped, multi-layered geometric mesh representation. 3D photos can be stored compactly and are optimized for being rendered from viewpoints that are near the capture viewpoints. They can be rendered using a standard rasterization pipeline to produce perspective views with motion parallax. When viewed in VR, 3D photos provide geometrically consistent views for both eyes. Our geometric representation also allows interacting with the scene using 3D geometry-aware effects, such as adding new objects to the scene and artistic lighting effects. Our 3D photo reconstruction algorithm starts with a standard structure from motion and multi-view stereo reconstruction of the scene. The dense stereo reconstruction is made robust to the imperfect capture conditions using a novel near envelope cost volume prior that discards erroneous near depth hypotheses. We propose a novel parallax-tolerant stitching algorithm that warps the depth maps into the central panorama and stitches two color-and-depth panoramas for the front and back scene surfaces. The two panoramas are fused into a single non-redundant, well-connected geometric mesh. We provide videos demonstrating users interactively viewing and manipulating our 3D photos.

WASS: An open-source pipeline for 3D stereo reconstruction of ocean waves

Stereo 3D reconstruction of ocean waves is gaining more and more popularity in the oceanographic community and industry. Indeed, recent advances of both computer vision algorithms and computer processing power now allow the study of the spatio-temporal wave field with unprecedented accuracy, especially at small scales. Even if simple in theory, multiple details are difficult to be mastered for a practitioner, so that the implementation of a sea-waves 3D reconstruction pipeline is in general considered a complex task. For instance, camera calibration, reliable stereo feature matching and mean sea-plane estimation are all factors for which a well designed implementation can make the difference to obtain valuable results. For this reason, we believe that the open availability of a well tested software package that automates the reconstruction process from stereo images to a 3D point cloud would be a valuable addition for future researches in this area.We present WASS (http://www.dais.unive.it/wass), an Open-Source stereo processing pipeline for sea waves 3D reconstruction. Our tool completely automates all the steps required to estimate dense point clouds from stereo images. Namely, it computes the extrinsic parameters of the stereo rig so that no delicate calibration has to be performed on the field. It implements a fast 3D dense stereo reconstruction procedure based on the consolidated OpenCV library and, lastly, it includes set of filtering techniques both on the disparity map and the produced point cloud to remove the vast majority of erroneous points that can naturally arise while analyzing the optically complex nature of the water surface.In this paper, we describe the architecture of WASS and the internal algorithms involved. The pipeline workflow is shown step-by-step and demonstrated on real datasets acquired at sea.

Field phenotyping of grapevine growth using dense stereo reconstruction.

BackgroundThe demand for high-throughput and objective phenotyping in plant research has been increasing during the last years due to large experimental sites. Sensor-based, non-invasive and automated processes are needed to overcome the phenotypic bottleneck, which limits data volumes on account of manual evaluations. A major challenge for sensor-based phenotyping in vineyards is the distinction between the grapevine in the foreground and the field in the background – this is especially the case for red-green-blue (RGB) images, where similar color distributions occur both in the foreground plant and in the field and background plants. However, RGB cameras are a suitable tool in the field because they provide high-resolution data at fast acquisition rates with robustness to outdoor illumination.ResultsThis study presents a method to segment the phenotypic classes ‘leaf’, ‘stem’, ‘grape’ and ‘background’ in RGB images that were taken with a standard consumer camera in vineyards. Background subtraction is achieved by taking two images of each plant for depth reconstruction. The color information is furthermore used to distinguish the leaves from stem and grapes in the foreground. The presented approach allows for objective computation of phenotypic traits like 3D leaf surface areas and fruit-to-leaf ratios. The method has been successfully applied to objective assessment of growth habits of new breeding lines. To this end, leaf areas of two breeding lines were monitored and compared with traditional cultivars. A statistical analysis of the method shows a significant (p <0.001) determination coefficient R 2= 0.93 and root-mean-square error of 3.0%.ConclusionsThe presented approach allows for non-invasive, fast and objective assessment of plant growth. The main contributions of this study are 1) the robust segmentation of RGB images taken from a standard consumer camera directly in the field, 2) in particular, the robust background subtraction via reconstruction of dense depth maps, and 3) phenotypic applications to monitoring of plant growth and computation of fruit-to-leaf ratios in 3D. This advance provides a promising tool for high-throughput, automated image acquisition, e.g., for field robots.Electronic supplementary materialThe online version of this article (doi:10.1186/s12859-015-0560-x) contains supplementary material, which is available to authorized users.

Robust stereo matching using adaptive random walk with restart algorithm

In this paper, we propose a robust dense stereo reconstruction algorithm using a random walk with restart. The pixel-wise matching costs are aggregated into superpixels and the modified random walk with restart algorithm updates the matching cost for all possible disparities between the superpixels. In comparison to the majority of existing stereo methods using the graph cut, belief propagation, or semi-global matching, our proposed method computes the final reconstruction through the determination of the best disparity at each pixel in the matching cost update. In addition, our method also considers occlusion and depth discontinuities through the visibility and fidelity terms. These terms assist in the cost update procedure in the calculation of the standard smoothness constraint. The method results in minimal computational costs while achieving high accuracy in the reconstruction. We test our method on standard benchmark datasets and challenging real-world sequences. We also show that the processing time increases linearly in relation to an increase in the disparity search range.

Dense terrain stereoreconstruction using modified SGM

A technique for creating 3D terrain models based on digital aerial images, using an algorithm of semi-global matching, is presented. A method of hierarchical application of the algorithm is demonstrated. A number of modifications and improvements at all stages of algorithm operation are proposed. Among them the following ones are distinguished: due account of the gradient and color information of images, the interpolation of the pixel matching metric for increasing the algorithm resolution and improving the stability to small distortions of the input data, filtering the matching cost and post-processing of disparity map by a bilateral filter. Results of algorithm application and the advantages of the modification are demonstrated by an example of processing of pairs of high-resolution aerial images.

LARGE SCALE URBAN RECONSTRUCTION FROM REMOTE SENSING IMAGERY

Abstract. Automatic large-scale stereo reconstruction of urban areas is increasingly becoming a vital aspect for physical simulations as well as for rapid prototyping large scale 3D city models. In this paper we describe an easily reproducible workflow for obtaining an accurate and textured 3D model of the scene, with overlapping aerial images as input. Starting with the initial camera poses and their refinement via bundle adjustment, we create multiple heightmaps by dense stereo reconstruction and fuse them into one Digital Surface Model (DSM). This DSM is then triangulated, and to reduce the amount of data, mesh simplification methods are employed. The resulting 3D mesh is finally projected into each of the input images to obtain the best fitting texture for each triangle. As verification, we provide visual results as well as numerically evaluating the accuracy by comparing the resulting 3D model against ground truth generated by aerial laser scanning (LiDAR).

Guest Editorial: Scenes, Images and Objects

An important aspect of current computer vision research is the analysis of scenes, and particularly the extraction of their 3D structure and their segmentation into objects. Linked to that is the interpretation of the images and/or recognition of types of images, either from the output of the 3D reconstruction and segmentation or directly from the images. Solving such problems is often accomplished by combining a variety of methods and/or attributes of the images, as the papers in this special issue show. Ladicky et al. provide us with a framework to unify dense stereo reconstruction and object segmentation, where both are formulated as Random Field labelling which are jointly optimized. Evaluation is done on an enhanced Leuven data set, which is publicly available. The paper by Hwang and Grauman provides an enhanced image search methodology by incorporating high-level human scene perception aspects, such as the order of associated key-words, into the visual representation used. Sun et al. present a system that iteratively combines object detection, 3D scene layout estimation, and segmentation of the objects’ support region. As knowledge of the scene becomes available from the layout estimation, it can be used to improve the confidence in the object detection and so remove false detections. The results are demonstrated on the authors’ own dataset and two publicly available datasets.

Archeological excavation monitoring using dense stereo matching techniques

Abstract Several new tools to obtain three-dimensional information from unorganized image sets are now available for the public use. The main advantage of this software, which is based on dense stereo matching, is the possibility to generate 3D content without the need of high-cost hardware (e.g. 3D scanning devices). Nevertheless, their use in real-world application domains (like cultural heritage) is still not very diffused, due to the non-straightforward usability of the raw data produced. In this paper, we investigate the use of automatic dense stereo reconstruction tools for the monitoring of an excavation site. A methodology for the effective acquisition and processing of data is presented. In addition, the results of the data assessment demonstrate the repeatability of the data acquisition process, which is a key factor when qualitative analysis is performed. The use of three-dimensional data is integrated in an open source mesh processing tool, thus showing that a spatio-temporal analysis can be performed in a very intuitive way using off-the-shelf or free/open digital tools. Moreover, the use of peculiar rendering and the creation of snapshots from arbitrary points of view increase the amount of documentation data, and suggest a perfect integration of data produced with dense stereo matching in the future standard documentation for excavation monitoring.

Joint Optimization for Object Class Segmentation and Dense Stereo Reconstruction

The problems of dense stereo reconstruction and object class segmentation can both be formulated as Random Field labeling problems, in which every pixel in the image is assigned a label corresponding to either its disparity, or an object class such as road or building. While these two problems are mutually informative, no attempt has been made to jointly optimize their labelings. In this work we provide a flexible framework configured via cross-validation that unifies the two problems and demonstrate that, by resolving ambiguities, which would be present in real world data if the two problems were considered separately, joint optimization of the two problems substantially improves performance. To evaluate our method, we augment the Leuven data set ( http://cms.brookes.ac.uk/research/visiongroup/files/Leuven.zip ), which is a stereo video shot from a car driving around the streets of Leuven, with 70 hand labeled object class and disparity maps. We hope that the release of these annotations will stimulate further work in the challenging domain of street-view analysis. Complete source code is publicly available ( http://cms.brookes.ac.uk/staff/Philip-Torr/ale.htm ).

Immersion into visual media: new applications of image understanding

Using images to interface machines and humans offers an exciting new application domain for image understanding (IU). This article describes two such applications: digital libraries that bring visual data to the user, and virtualized reality that brings users to the visual data. The digital library calls on IU techniques such as scene segmentation and content analysis, and joins together natural language analysis and IU. Virtualized reality uses precise, dense, and video-rate stereo reconstruction techniques.