Multiple Stereo Images Research Articles

Rotation alignment of a camera-IMU system using a single affine correspondence.

We propose an accurate and easy-to-implement method on rotation alignment of a camera-inertial measurement unit (IMU) system using only a single affine correspondence in the minimal case. The known initial rotation angles between the camera and IMU are utilized; thus, the alignment model can be formulated as a polynomial equation system based on homography constraints by expressing the rotation matrix in a first-order approximation. By solving the equation system, we can recover the rotation alignment parameters. Furthermore, more accurate alignment results can be achieved with the joint optimization of multiple stereo image pairs. The proposed method does not require additional auxiliary equipment or a camera's particular motion. The experimental results on synthetic data and two real-world data sets demonstrate that our method is efficient and precise for the camera-IMU system's rotation alignment.

Enhanced Lunar Topographic Mapping Using Multiple Stereo Images Taken by Yutu-2 Rover with Changing Illumination Conditions

In a planetary-rover exploration mission, stereovision-based 3D reconstruction has been widely applied to topographic mapping of the planetary surface using stereo cameras onboard the rover. In this study, we propose an enhanced topographic mapping method based on multiple stereo images taken at the same rover location with changing illumination conditions. Key steps of the method include dense matching of stereo images, 3D point-cloud generation, point-cloud co-registration, and fusion. The final point cloud has more complete coverage and more details of the terrain than that conventionally generated from a single stereo pair. The effectiveness of the proposed method is verified by experiments using the Yutu-2 rover, in which two data sets were acquired by the navigation cameras at two locations and under changing illumination conditions. This method, which does not involve complex operations, has great potential for application in planetary-rover and lander missions.

Self-calibration approach to stereo cameras with radial distortion based on epipolar constraint.

In this paper, we propose a self-calibration approach to stereo cameras with radial distortion from stereo image pairs of a common 3D scene. Based on the epipolar constraint in the stereo image pair, the intrinsic and extrinsic parameters of stereo cameras are estimated synchronously with a minimum number of nine image point correspondences. It is significant within a random sample consensus (RANSAC) scheme to cope with the outliers of feature matches efficiently and robustly. Then the inliers of the stereo image pair that have been determined after RANSAC are used to optimize the calibration parameters of stereo cameras. Furthermore, more accurate calibration results can be achieved with the joint optimization of multiple stereo image pairs. Both synthetic and real data are used to evaluate the performance of the proposed method, demonstrating that our method can calibrate stereo cameras with radial distortions efficiently and accurately.

Geology and Physical Properties Investigations by the InSight Lander

Although not the prime focus of the InSight mission, the near-surface geology and physical properties investigations provide critical information for both placing the instruments (seismometer and heat flow probe with mole) on the surface and for understanding the nature of the shallow subsurface and its effect on recorded seismic waves. Two color cameras on the lander will obtain multiple stereo images of the surface and its interaction with the spacecraft. Images will be used to identify the geologic materials and features present, quantify their areal coverage, help determine the basic geologic evolution of the area, and provide ground truth for orbital remote sensing data. A radiometer will measure the hourly temperature of the surface in two spots, which will determine the thermal inertia of the surface materials present and their particle size and/or cohesion. Continuous measurements of wind speed and direction offer a unique opportunity to correlate dust devils and high winds with eolian changes imaged at the surface and to determine the threshold friction wind stress for grain motion on Mars. During the first two weeks after landing, these investigations will support the selection of instrument placement locations that are relatively smooth, flat, free of small rocks and load bearing. Soil mechanics parameters and elastic properties of near surface materials will be determined from mole penetration and thermal conductivity measurements from the surface to 3–5 m depth, the measurement of seismic waves during mole hammering, passive monitoring of seismic waves, and experiments with the arm and scoop of the lander (indentations, scraping and trenching). These investigations will determine and test the presence and mechanical properties of the expected 3–17 m thick fragmented regolith (and underlying fractured material) built up by impact and eolian processes on top of Hesperian lava flows and determine its seismic properties for the seismic investigation of Mars’ interior.

Planar self-calibration for stereo cameras with radial distortion.

In this paper, we present a robust technique of stereo calibration using homography constraints. Our method is novel as stereo calibration is performed by solving a polynomial equation system including two radial distortion parameters, using a minimal number of five image point correspondences. This enables us to calibrate from only a pair of stereo images of a planar scene, and to provide the exact algebraic solution to the stereo calibration problem. The minimal case solution is useful to reduce the computation time and increase the calibration robustness when using random sample consensus (RANSAC) from the correspondences of the stereo image pair. Further, a non-linear parameter optimization for the intrinsic and extrinsic parameters of stereo cameras is performed using the inliers, which are determined after RANSAC. In addition, our method can achieve more robust calibration results with multiple stereo image pairs by performing joint optimization. In contrast to the previous stereo calibration methods, our method works without requiring any special hardware and has no problems with one stereo image pair, even corrupted by severe radial distortions. Finally, by evaluating our method on both synthetic and real scene data, we demonstrate that our method is both efficient and accurate for stereo calibration.

Construction of pixel-level resolution DEMs from monocular images by shape and albedo from shading constrained with low-resolution DEM

Lunar Digital Elevation Model (DEM) is important for lunar successful landing and exploration missions. Lunar DEMs are typically generated by photogrammetry or laser altimetry approaches. Photogrammetric methods require multiple stereo images of the region of interest and it may not be applicable in cases where stereo coverage is not available. In contrast, reflectance based shape reconstruction techniques, such as shape from shading (SfS) and shape and albedo from shading (SAfS), apply monocular images to generate DEMs with pixel-level resolution. We present a novel hierarchical SAfS method that refines a lower-resolution DEM to pixel-level resolution given a monocular image with known light source. We also estimate the corresponding pixel-wise albedo map in the process and based on that to regularize the shape reconstruction with pixel-level resolution based on the low-resolution DEM. In this study, a Lunar–Lambertian reflectance model is applied to estimate the albedo map. Experiments were carried out using monocular images from the Lunar Reconnaissance Orbiter Narrow Angle Camera (LRO NAC), with spatial resolution of 0.5–1.5m per pixel, constrained by the Selenological and Engineering Explorer and LRO Elevation Model (SLDEM), with spatial resolution of 60m. The results indicate that local details are well recovered by the proposed algorithm with plausible albedo estimation. The low-frequency topographic consistency depends on the quality of low-resolution DEM and the resolution difference between the image and the low-resolution DEM.

SHAPE AND ALBEDO FROM SHADING (SAfS) FOR PIXEL-LEVEL DEM GENERATION FROM MONOCULAR IMAGES CONSTRAINED BY LOW-RESOLUTION DEM

Lunar topographic information, e.g., lunar DEM (Digital Elevation Model), is very important for lunar exploration missions and scientific research. Lunar DEMs are typically generated from photogrammetric image processing or laser altimetry, of which photogrammetric methods require multiple stereo images of an area. DEMs generated from these methods are usually achieved by various interpolation techniques, leading to interpolation artifacts in the resulting DEM. On the other hand, photometric shape reconstruction, e.g., SfS (Shape from Shading), extensively studied in the field of Computer Vision has been introduced to pixel-level resolution DEM refinement. SfS methods have the ability to reconstruct pixel-wise terrain details that explain a given image of the terrain. If the terrain and its corresponding pixel-wise albedo were to be estimated simultaneously, this is a SAfS (Shape and Albedo from Shading) problem and it will be under-determined without additional information. Previous works show strong statistical regularities in albedo of natural objects, and this is even more logically valid in the case of lunar surface due to its lower surface albedo complexity than the Earth. In this paper we suggest a method that refines a lower-resolution DEM to pixel-level resolution given a monocular image of the coverage with known light source, at the same time we also estimate the corresponding pixel-wise albedo map. We regulate the behaviour of albedo and shape such that the optimized terrain and albedo are the likely solutions that explain the corresponding image. The parameters in the approach are optimized through a kernel-based relaxation framework to gain computational advantages. In this research we experimentally employ the Lunar-Lambertian model for reflectance modelling; the framework of the algorithm is expected to be independent of a specific reflectance model. Experiments are carried out using the monocular images from Lunar Reconnaissance Orbiter (LRO) Narrow Angle Camera (NAC) (0.5 m spatial resolution), constrained by the SELENE and LRO Elevation Model (SLDEM 2015) of 60 m spatial resolution. The results indicate that local details are largely recovered by the algorithm while low frequency topographic consistency is affected by the low-resolution DEM.

SHAPE AND ALBEDO FROM SHADING (SAfS) FOR PIXEL-LEVEL DEM GENERATION FROM MONOCULAR IMAGES CONSTRAINED BY LOW-RESOLUTION DEM

Abstract. Lunar topographic information, e.g., lunar DEM (Digital Elevation Model), is very important for lunar exploration missions and scientific research. Lunar DEMs are typically generated from photogrammetric image processing or laser altimetry, of which photogrammetric methods require multiple stereo images of an area. DEMs generated from these methods are usually achieved by various interpolation techniques, leading to interpolation artifacts in the resulting DEM. On the other hand, photometric shape reconstruction, e.g., SfS (Shape from Shading), extensively studied in the field of Computer Vision has been introduced to pixel-level resolution DEM refinement. SfS methods have the ability to reconstruct pixel-wise terrain details that explain a given image of the terrain. If the terrain and its corresponding pixel-wise albedo were to be estimated simultaneously, this is a SAfS (Shape and Albedo from Shading) problem and it will be under-determined without additional information. Previous works show strong statistical regularities in albedo of natural objects, and this is even more logically valid in the case of lunar surface due to its lower surface albedo complexity than the Earth. In this paper we suggest a method that refines a lower-resolution DEM to pixel-level resolution given a monocular image of the coverage with known light source, at the same time we also estimate the corresponding pixel-wise albedo map. We regulate the behaviour of albedo and shape such that the optimized terrain and albedo are the likely solutions that explain the corresponding image. The parameters in the approach are optimized through a kernel-based relaxation framework to gain computational advantages. In this research we experimentally employ the Lunar-Lambertian model for reflectance modelling; the framework of the algorithm is expected to be independent of a specific reflectance model. Experiments are carried out using the monocular images from Lunar Reconnaissance Orbiter (LRO) Narrow Angle Camera (NAC) (0.5 m spatial resolution), constrained by the SELENE and LRO Elevation Model (SLDEM 2015) of 60 m spatial resolution. The results indicate that local details are largely recovered by the algorithm while low frequency topographic consistency is affected by the low-resolution DEM.

POTENTIAL OF DENSE MATCHING FOR THE GENERATION OF HIGH QUALITY DIGITAL ELEVATION MODELS

Abstract. Until recently, the acquisition of high quality Digital Elevation Models was dominated by the use of airborne LiDAR. However, the increasing quality of digital airborne cameras in combination with recent improvements in matching algorithms meanwhile allow for the automatic image based collection as a suitable alternative. Within the paper, these progresses will be demonstrated on the example of photogrammetric DEM generation using the Semi-Global Matching (SGM) stereo method. Since this approach aims at a pixel-wise matching, dense 3D point clouds can be generated. The tests described in the paper are based on data collected from different digital airborne cameras at various flight scenarios during a recent test on photogrammetric 3D data capture. By these means, the impact of different stereo configurations on the quality of the final outcome can be evaluated and compared to already available test results. Special interest is also paid to the analysis and combination of multiple stereo image pairs with different base- to-height ratios, which can be used efficiently to increase the accuracy and reliability of the matching result.

Maximum likelihood estimation of cloud height from multi-angle satellite imagery

We develop a new estimation technique for recovering depth-of-field from multiple stereo images. Depth-of-field is estimated by determining the shift in image location resulting from different camera viewpoints. When this shift is not divisible by pixel width, the multiple stereo images can be combined to form a super-resolution image. By modeling this super-resolution image as a realization of a random field, one can view the recovery of depth as a likelihood estimation problem. We apply these modeling techniques to the recovery of cloud height from multiple viewing angles provided by the MISR instrument on the Terra Satellite. Our efforts are focused on a two layer cloud ensemble where both layers are relatively planar, the bottom layer is optically thick and textured, and the top layer is optically thin. Our results demonstrate that with relative ease, we get comparable estimates to the M2 stereo matcher which is the same algorithm used in the current MISR standard product (details can be found in [IEEE Transactions on Geoscience and Remote Sensing 40 (2002) 1547–1559]). Moreover, our techniques provide the possibility of modeling all of the MISR data in a unified way for cloud height estimation. Research is underway to extend this framework for fast, quality global estimates of cloud height.

Development of a virtual reality GIS using stereo vision

This paper presents an integrated virtual reality geographic information system (VRGIS) based on computational stereo. In this system, three-dimensional (3D) and 360° panoramic scenes are reconstructed from multiple stereo image pairs acquired with a mobile stereo machine vision system. The spatial coordinates of panoramic scenes are registered to electronic maps to obtain a real scale VRGIS that is capable of efficient retrieval of spatial information and allows for navigation in virtual reality. A software program was implemented for image processing, coordinate transformation, 3D surface model building, data fusion, digital mapping, and database management for the VRGIS. In setting up 3D models for the virtual reality environment, we focused on comparisons of pattern matching algorithms for creating disparity images from stereo pairs, and adopted the graph cut method and the non-parametric local transform method in our system. Following the computation of disparity images, they are transformed into cylindrical coordinates and stitched into a 3D and 360° panoramic scene. In order to allow for virtual reality navigation, the panoramic scene is further cast into 3D triangular surface model for fast texture mapping using memoryless simplification algorithm. The effects of reducing the number of edges in a 3D surface model on computation time and discrepancy from 3D surface model with no edge reduction were also analyzed. Experimental results show that the discrepancy was about 13% when 80% edge reduction was applied to create the approximate 3D surface model. This indicates a significant computational and memory reduction while the computer graphic display of the scene model was still visually satisfactory. Finally, multiple 3D scenes were integrated into a site on the electronic map permitting the user to navigate through location to location in a virtual reality environment. The developed system may serve as a versatile platform in establishing field information for many applications such as precision agriculture, vehicle automation, geographic visualization, archaeology site modelling, etc.

Improved motion stereo matching based on a modified dynamic programming

A new method for computing precise depth map estimates of 3D shape of a moving object is proposed. 3D shape recovery in motion stereo is formulated as a matching optimization problem of multiple stereo images. The proposed method is a heuristic modification of dynamic programming applied to two-dimensional optimization problem. 3D shape recovery using real motion stereo images demonstrates a good performance of the algorithm in terms of reconstruction accuracy.

ステレオ画像からの3次元立体の復元の一手法 (第1報)

ステレオ画像からの3次元立体の復元の一手法 (第1報)