Stereo 3D Reconstruction Research Articles

EMVS: Event-Based Multi-View Stereo—3D Reconstruction with an Event Camera in Real-Time

Event cameras are bio-inspired vision sensors that output pixel-level brightness changes instead of standard intensity frames. They offer significant advantages over standard cameras, namely a very high dynamic range, no motion blur, and a latency in the order of microseconds. However, because the output is composed of a sequence of asynchronous events rather than actual intensity images, traditional vision algorithms cannot be applied, so that a paradigm shift is needed. We introduce the problem of event-based multi-view stereo (EMVS) for event cameras and propose a solution to it. Unlike traditional MVS methods, which address the problem of estimating dense 3D structure from a set of known viewpoints, EMVS estimates semi-dense 3D structure from an event camera with known trajectory. Our EMVS solution elegantly exploits two inherent properties of an event camera: (1) its ability to respond to scene edges—which naturally provide semi-dense geometric information without any pre-processing operation—and (2) the fact that it provides continuous measurements as the sensor moves. Despite its simplicity (it can be implemented in a few lines of code), our algorithm is able to produce accurate, semi-dense depth maps, without requiring any explicit data association or intensity estimation. We successfully validate our method on both synthetic and real data. Our method is computationally very efficient and runs in real-time on a CPU.

WASS: An open-source pipeline for 3D stereo reconstruction of ocean waves

Stereo 3D reconstruction of ocean waves is gaining more and more popularity in the oceanographic community and industry. Indeed, recent advances of both computer vision algorithms and computer processing power now allow the study of the spatio-temporal wave field with unprecedented accuracy, especially at small scales. Even if simple in theory, multiple details are difficult to be mastered for a practitioner, so that the implementation of a sea-waves 3D reconstruction pipeline is in general considered a complex task. For instance, camera calibration, reliable stereo feature matching and mean sea-plane estimation are all factors for which a well designed implementation can make the difference to obtain valuable results. For this reason, we believe that the open availability of a well tested software package that automates the reconstruction process from stereo images to a 3D point cloud would be a valuable addition for future researches in this area.We present WASS (http://www.dais.unive.it/wass), an Open-Source stereo processing pipeline for sea waves 3D reconstruction. Our tool completely automates all the steps required to estimate dense point clouds from stereo images. Namely, it computes the extrinsic parameters of the stereo rig so that no delicate calibration has to be performed on the field. It implements a fast 3D dense stereo reconstruction procedure based on the consolidated OpenCV library and, lastly, it includes set of filtering techniques both on the disparity map and the produced point cloud to remove the vast majority of erroneous points that can naturally arise while analyzing the optically complex nature of the water surface.In this paper, we describe the architecture of WASS and the internal algorithms involved. The pipeline workflow is shown step-by-step and demonstrated on real datasets acquired at sea.

On Structure-From-Motion Application Challenges

Numerous software solutions that implement the Structure-from-Motion/Multi-View Stereo (SfM/MVS) 3D reconstruction approach have been made available during the last decade. These allow the production of high quality in terms of geometry and colour information 3D models with the use of unordered image collections that depict a static scene or object from arbitrary viewpoints. Nowadays SfM/MVS-based 3D reconstruction approaches constitute a popular solution in a variety of applications within the cultural heritage domain. As with all 3D reconstruction approaches it has its limitations and application challenges. In this work, the authors attempt to compose a set of guidelines that are based on the important outcomes of important published works that propose solutions that overcome the various challenges introduced by non-friendly surface types.

A High-Throughput, Field-Based Phenotyping Technology for Tall Biomass Crops.

Recent advances in omics technologies have not been accompanied by equally efficient, cost-effective, and accurate phenotyping methods required to dissect the genetic architecture of complex traits. Even though high-throughput phenotyping platforms have been developed for controlled environments, field-based aerial and ground technologies have only been designed and deployed for short-stature crops. Therefore, we developed and tested Phenobot 1.0, an auto-steered and self-propelled field-based high-throughput phenotyping platform for tall dense canopy crops, such as sorghum (Sorghum bicolor). Phenobot 1.0 was equipped with laterally positioned and vertically stacked stereo RGB cameras. Images collected from 307 diverse sorghum lines were reconstructed in 3D for feature extraction. User interfaces were developed, and multiple algorithms were evaluated for their accuracy in estimating plant height and stem diameter. Tested feature extraction methods included the following: (1) User-interactive Individual Plant Height Extraction (UsIn-PHe) based on dense stereo three-dimensional reconstruction; (2) Automatic Hedge-based Plant Height Extraction (Auto-PHe) based on dense stereo 3D reconstruction; (3) User-interactive Dense Stereo Matching Stem Diameter Extraction; and (4) User-interactive Image Patch Stereo Matching Stem Diameter Extraction (IPaS-Di). Comparative genome-wide association analysis and ground-truth validation demonstrated that both UsIn-PHe and Auto-PHe were accurate methods to estimate plant height, while Auto-PHe had the additional advantage of being a completely automated process. For stem diameter, IPaS-Di generated the most accurate estimates of this biomass-related architectural trait. In summary, our technology was proven robust to obtain ground-based high-throughput plant architecture parameters of sorghum, a tall and densely planted crop species.

Metrological characterization of 3D biometric face recognition systems in actual operating conditions

<p>Nowadays, face recognition systems are going to widespread in many fields of application, from automatic user login for financial activities and access to restricted areas, to surveillance for improving security in airports and railway stations, to cite a few.<br />In such scenarios, the architectures based on stereo vision and 3D reconstruction of the face are going to assume a predominant role because they can generally assure a better reliability than solutions based on a single camera (which make use of a single image instead of a couple of images). To realize such systems, different architectures can be considered by varying the positioning of the pair of cameras with respect to the face of the subject to be identified, as well as both kind and resolution of camera considered. These parameters can affect the correct decision rate of the system in classifying the input face, especially in presence of image uncertainty.<br />In this paper, several 3D architectures differing in camera specifications and geometrical positioning of the camera pair (with respect to the input face) are realized and compared. The detection of facial features in the images is made by adopting a popular method based on the Active Appearance Model (AAM) algorithm. 3D position of facial features is then obtained by means of stereo triangulation. The performance of the realized systems has been compared in terms of sensitivity to the quantities of influence and related uncertainty, and of typical indexes for the analysis of classification systems. Main results of such comparison show that the best performance can be reached by reducing the distance between cameras and subject to be identified and by minimizing the horizontal angle between the plane containing the camera pair axis and the face to be identified.</p>

Multiview Differential Geometry of Curves

The field of multiple view geometry has seen tremendous progress in reconstruction and calibration due to methods for extracting reliable point features and key developments in projective geometry. Point features, however, are not available in certain applications and result in unstructured point cloud reconstructions. General image curves provide a complementary feature when keypoints are scarce, and result in 3D curve geometry, but face challenges not addressed by the usual projective geometry of points and algebraic curves. We address these challenges by laying the theoretical foundations of a framework based on the differential geometry of general curves, including stationary curves, occluding contours, and non-rigid curves, aiming at stereo correspondence, camera estimation (including calibration, pose, and multiview epipolar geometry), and 3D reconstruction given measured image curves. By gathering previous results into a cohesive theory, novel results were made possible, yielding three contributions. First, we derive the differential geometry of an image curve (tangent, curvature, curvature derivative) from that of the underlying space curve (tangent, curvature, curvature derivative, torsion). Second, we derive the differential geometry of a space curve from that of two corresponding image curves. Third, the differential motion of an image curve is derived from camera motion and the differential geometry and motion of the space curve. The availability of such a theory enables novel curve-based multiview reconstruction and camera estimation systems to augment existing point-based approaches. This theory has been used to reconstruct a “3D curve sketch”, to determine camera pose from local curve geometry, and tracking; other developments are underway.

Comprehensive and Practical Vision System for Self-Driving Vehicle Lane-Level Localization.

Vehicle lane-level localization is a fundamental technology in autonomous driving. To achieve accurate and consistent performance, a common approach is to use the LIDAR technology. However, it is expensive and computational demanding, and thus not a practical solution in many situations. This paper proposes a stereovision system, which is of low cost, yet also able to achieve high accuracy and consistency. It integrates a new lane line detection algorithm with other lane marking detectors to effectively identify the correct lane line markings. It also fits multiple road models to improve accuracy. An effective stereo 3D reconstruction method is proposed to estimate vehicle localization. The estimation consistency is further guaranteed by a new particle filter framework, which takes vehicle dynamics into account. Experiment results based on image sequences taken under different visual conditions showed that the proposed system can identify the lane line markings with 98.6% accuracy. The maximum estimation error of the vehicle distance to lane lines is 16 cm in daytime and 26 cm at night, and the maximum estimation error of its moving direction with respect to the road tangent is 0.06 rad in daytime and 0.12 rad at night. Due to its high accuracy and consistency, the proposed system can be implemented in autonomous driving vehicles as a practical solution to vehicle lane-level localization.

Accurate stereo matching using pixel normalized cross correlation in time domain

This paper proposes a novel method for stereo matching which is based on combination of active and passive stereo 3D reconstruction approaches. A laser line is used to scan the reconstructed scene and a stereo camera pair is used for the image acquisition. Each image pixel is scanned at a specific scan time so that the intensity time patterns of the correspondent pixels are highly correlated. This yield in highly confident and accurate disparity map calculation and also allows the reconstruction of poorly textured as well as the extremely textured surface which are very hard to deal with using the conventional passive stereo approaches. The occluded regions are also detected successfully. This method is not computationally intensive and can be used for turning the smartphone into the practical 3D scanner as presented in this work.

Three-dimensional combined pyrometric sizing and velocimetry of combusting coal particles. II: Pyrometry.

Knowledge of the in situ temperature, size, velocity, and number density of a population of burning coal particles yields insight into the chemical and aerodynamic behavior of a pulverized coal flame (e.g., through means of combustion model validation). Sophisticated and reasonably accurate methods are available for the simultaneous measurement of particle velocity and temperature; however, these methods typically produce single particle measurements in small analyzed volumes and require extensive instrumentation. We present a simple, inexpensive method for the simultaneous, in situ, three-dimensional (3D) measurement of particle velocity, number density, size, and temperature. The proposed method uses a combination of stereo imaging, 3D reconstruction, multicolor pyrometry, and digital image processing techniques. The details of theoretical and algorithmic backgrounds are presented, along with examples and validation experiments. Rigorous uncertainty quantification was performed using numerical simulations to estimate the accuracy of the method and explore how different parameters affect measurement uncertainty. This paper, Part II of two parts that discuss this method [Appl. Opt.54, 4049 (2015)], describes particle temperature and size measurement in overexposed emission images.

The Technology of Fast 3D Reconstruction Based on Stereo Vision

Reverse engineering has become an independent research area in CAD/CAM system with the development of modern manufacturing. The 3D reconstruction technology of generating a digit model from physical model is an important part of reverse engineering. The 3D reconstruction technology based on stereo vision has advantages such as non-contact measurement, simple system structure, being fast and efficient. In this paper, a fast 3D reconstruction technology based on stereo vision is presented. Firstly, BP neural network is used to simulate the mapping relationship between the 3D object and its images. Then, the epipole positions and adjusting coefficients for epipolar lines are calculated according to the epipolar line constraint. After the rectification transformations on the left and right image planes is conducted, the pairs of conjugate epipolar lines become collinear and parallel to the horizontal image axis. Therefore, when the stereo correspondences between the left and right images are being performed, the search is done only along the horizontal lines of the rectified images, which is highly advantageous to fast image matching for stereo vision and 3D reconstruction of free-form surface.

A COMPARISON BETWEEN ACTIVE AND PASSIVE TECHNIQUES FOR UNDERWATER 3D APPLICATIONS

Abstract. In the field of 3D scanning, there is an increasing need for more accurate technologies to acquire 3D models of close range objects. Underwater exploration, for example, is very hard to perform due to the hostile conditions and the bad visibility of the environment. Some application fields, like underwater archaeology, require to recover tridimensional data of objects that cannot be moved from their site or touched in order to avoid possible damages. Photogrammetry is widely used for underwater 3D acquisition, because it requires just one or two digital still or video cameras to acquire a sequence of images taken from different viewpoints. Stereo systems composed by a pair of cameras are often employed on underwater robots (i.e. ROVs, Remotely Operated Vehicles) and used by scuba divers, in order to survey archaeological sites, reconstruct complex 3D structures in aquatic environment, estimate in situ the length of marine organisms, etc. The stereo 3D reconstruction is based on the triangulation of corresponding points on the two views. This requires to find in both images common points and to match them (correspondence problem), determining a plane that contains the 3D point on the object. Another 3D technique, frequently used in air acquisition, solves this point-matching problem by projecting structured lighting patterns to codify the acquired scene. The corresponding points are identified associating a binary code in both images. In this work we have tested and compared two whole-field 3D imaging techniques (active and passive) based on stereo vision, in underwater environment. A 3D system has been designed, composed by a digital projector and two still cameras mounted in waterproof housing, so that it can perform the various acquisitions without changing the configuration of optical devices. The tests were conducted in a water tank in different turbidity conditions, on objects with different surface properties. In order to simulate a typical seafloor, we used various concentrations of clay. The performances of the two techniques are described and discussed. In particular, the point clouds obtained are compared in terms of number of acquired 3D points and geometrical deviation.

Three-dimensional feature parameters measurement of bubbles in gas–liquid two-phase flow based on virtual stereo vision

In this paper, three-dimensional (3D) feature parameters reconstruction and measurement methods of bubbles in gas–liquid two-phase flow based on virtual stereo vision have been developed. A mathematical model of virtual stereo vision is established, and the optimized design method of structure parameters is discussed in detail. In addition, the key technologies for the 3D measurement of bubbles, including the camera calibration, the virtual stereo vision sensor calibration, image processing, feature extraction and 3D reconstruction of bubble trajectories are introduced. All the intrinsic and extrinsic parameters of two virtual cameras, as well as coefficients of lens distortion, can be estimated by the complete mathematical model of the camera. With the sensor calibration, the relative position and orientation of two virtual cameras are determined. Several improved image processing algorithms are used to extract the primary modality parameters of bubbles, and then the 3D trajectories of bubbles can be reconstructed based on the measurement model of virtual stereo vision. The experiment is tested on the upstream bubble-producing appliance. It shows that the method is valid, and the 3D trajectories of bubbles can be reconstructed with high precision. The measured absolute error of bubble 3D displacement is smaller than 0.13 mm, and the relative error is smaller than 0.49%.

Research on Vision-based Autonomous Navigation Algorithm for RVD between Spacecrafts

In order to solve the autonomous navigation problem of RVD(Rendezvous and Docking) in near distance (<2m) for the on-orbit service of spacecraft, a vision-guided method based on geometry feature of the spacecraft was proposed to measure the relative pose (position and attitude) between two spacecrafts. Firstly, after smoothing the captured image, edge image was obtained by using fast self-adaptive edge detection. We use the line feature to segment the interesting area of the spacecraft and the geometry feature was used to recognize the interesting areas, and the intersection points of the object were gotten. Secondly, the coordinates of these points in the world coordinate were figured out by the proposed fast stereo matching algorithm and 3D reconstruction technical. Based on these coordinates, the pose with respect to the world coordinate was calculated. Finally, lines of the recognized region were extracted and tracked based on Hough transform. In order to verify the effectiveness of the proposed algorithm, a hardware system was established based on high performance DSP. The results of satellite model experiment demonstrate that the relative position errors are less than ±20mm, relative attitude errors are less than ±2°, and measuring speed is up to 8fps which satisfies the precision and speed requirement of the RVD system. The errors in this measurement system were analyzed.

Stereo Calibration and Rectification for Omnidirectional Multi-Camera Systems

Stereo vision has been studied for decades as a fundamental problem in the field of computer vision. In recent years, computer vision and image processing with a large field of view, especially using omnidirectional vision and panoramic images, has been receiving increasing attention. An important problem for stereo vision is calibration. Although various kinds of calibration methods for omnidirectional cameras are proposed, most of them are limited to calibrate catadioptric cameras or fish-eye cameras and cannot be applied directly to multi-camera systems. In this work, we propose an easy calibration method with closed-form initialization and iterative optimization for omnidirectional multi-camera systems. The method only requires image pairs of the 2D target plane in a few different views. A method based on the spherical camera model is also proposed for rectifying omnidirectional stereo pairs. Using real data captured by Ladybug3, we carry out some experiments, including stereo calibration, rectification and 3D reconstruction. Statistical analyses and comparisons of the experimental results are also presented. As the experimental results show, the calibration results are precise and the effect of rectification is promising.

Out-of-plane Testing Procedure for Inverse Identification Purpose: Application in Sheet Metal Plasticity

Many recent works in inverse identification of constitutive parameters have pointed to the need of tests which exhibit heterogeneous strain paths. The present study details a new testing procedure based on out-of-plane motion capture by Stereo-Image Correlation (SIC). With the original test proposed hereby, a unique sample is deformed on a tensile machine along two perpendicular tensile directions, two perpendicular shear directions and one expansion area. The choice of the sample shape is discussed along with the stereo imaging device, 3D reconstruction and measurement uncertainties. The test sample is made from a sheet of commercially pure titanium. A Finite-Element updating inverse method is applied in order to identify six parameters of an anisotropic plastic constitutive model. Results show that this new testing procedure allows every constitutive parameter of the model to be identified from one and only one test.

3D dense reconstruction from 2D video sequence via 3D geometric segmentation

3D reconstruction is a major problem in computer vision. This paper considers the problem of reconstructing 3D structures, given a 2D video sequence. This problem is challenging since it is difficult to identify the trajectory of each object point/pixel over time. Traditional stereo 3D reconstruction methods and volumetric 3D reconstruction methods suffer from the blank wall problem, and the estimated dense depth map is not smooth, resulting in loss of actual geometric structures such as planes. To retain geometric structures embedded in the 3D scene, this paper proposes a novel surface fitting approach for 3D dense reconstruction. Specifically, we develop an expanded deterministic annealing algorithm to decompose 3D point cloud to multiple geometric structures, and estimate the parameters of each geometric structure. In this paper, we only consider plane structure, but our methodology can be extended to other parametric geometric structures such as spheres, cylinders, and cones. The experimental results show that the new approach is able to segment 3D point cloud into appropriate geometric structures and generate accurate 3D dense depth map.

Disparity map refinement and 3D surface smoothing via directed anisotropic diffusion

We propose a new binocular stereo algorithm and 3D reconstruction method from multiple disparity images. First, we present an accurate binocular stereo algorithm. In our algorithm, we use neither color segmentation nor plane fitting methods, which are common techniques among many sophisticated binocular stereo algorithms. These methods assume that the 3D world consists of a collection of planes and that each segment of a disparity map obeys a plane equation. We exclude these assumptions and introduce a directed anisotropic diffusion technique for refining a disparity map. Second, we show a method to fill some holes in a distance map and smooth the reconstructed 3D surfaces by using another type of anisotropic diffusion technique. The evaluation results on the Middlebury datasets show that our stereo algorithm is competitive with other algorithms that adopt plane fitting methods. We present an experiment that shows the high accuracy of a reconstructed 3D model using our method. The results demonstrate the effectiveness and practicality of our proposed method in a real environment that consists of both curved and planar surfaces.

Matching free stereovision for detecting obstacles on a ground plane

This paper presents a method for detecting obstacles on a ground plane from a stereo pair of images. Although we use stereovision, the obstacle detection algorithm relies neither on stereo matching nor 3D reconstruction. The principle here is to apply the projective transformation constraining the left and right images to obtain a frame of superimposed features (e.g. edges). By analysing feature superimposition after the projective transformation, a free moving space or space occupied by obstacles/occluded features can be determined.

Remarks on Fever

Recent advances in omics technologies have not been accompanied by equally efficient, cost-effective, and accurate phenotyping methods required to dissect the genetic architecture of complex traits. Even though high-throughput phenotyping platforms have been developed for controlled environments, field-based aerial and ground technologies have only been designed and deployed for short-stature crops. Therefore, we developed and tested Phenobot 1.0, an auto-steered and self-propelled field-based high-throughput phenotyping platform for tall dense canopy crops, such as sorghum (Sorghum bicolor). Phenobot 1.0 was equipped with laterally positioned and vertically stacked stereo RGB cameras. Images collected from 307 diverse sorghum lines were reconstructed in 3D for feature extraction. User interfaces were developed, and multiple algorithms were evaluated for their accuracy in estimating plant height and stem diameter. Tested feature extraction methods included the following: (1) User-interactive Individual Plant Height Extraction (UsIn-PHe) based on dense stereo three-dimensional reconstruction; (2) Automatic Hedge-based Plant Height Extraction (Auto-PHe) based on dense stereo 3D reconstruction; (3) User-interactive Dense Stereo Matching Stem Diameter Extraction; and (4) User-interactive Image Patch Stereo Matching Stem Diameter Extraction (IPaS-Di). Comparative genome-wide association analysis and ground-truth validation demonstrated that both UsIn-PHe and Auto-PHe were accurate methods to estimate plant height, while Auto-PHe had the additional advantage of being a completely automated process. For stem diameter, IPaS-Di generated the most accurate estimates of this biomass-related architectural trait. In summary, our technology was proven robust to obtain ground-based high-throughput plant architecture parameters of sorghum, a tall and densely planted crop species.