Stereo Image Pairs Research Articles

Automatic relative RPC image model bias compensation through hierarchical image matching for improving DEM quality

The quality and efficiency of automated Digital Elevation Model (DEM) extraction from stereoscopic satellite imagery is critically dependent on the accuracy of the sensor model used for co-locating pixels between stereo-pair images. In the absence of ground control or manual tie point selection, errors in the sensor models must be compensated with increased matching search-spaces, increasing both the computation time and the likelihood of spurious matches. Here we present an algorithm for automatically determining and compensating the relative bias in Rational Polynomial Coefficients (RPCs) between stereo-pairs utilizing hierarchical, sub-pixel image matching in object space. We demonstrate the algorithm using a suite of image stereo-pairs from multiple satellites over a range stereo-photogrammetrically challenging polar terrains. Besides providing a validation of the effectiveness of the algorithm for improving DEM quality, experiments with prescribed sensor model errors yield insight into the dependence of DEM characteristics and quality on relative sensor model bias. This algorithm is included in the Surface Extraction through TIN-based Search-space Minimization (SETSM) DEM extraction software package, which is the primary software used for the U.S. National Science Foundation ArcticDEM and Reference Elevation Model of Antarctica (REMA) products.

High throughput hardware architecture for accurate semi-global matching

As the most important step of a stereo vision system, stereo matching, which finds the correspondences in stereo image pairs, requires high-quality real-time depth computation. In this paper, a high throughput and high accuracy full-pipeline hardware architecture with disparity and row parallelism is proposed. In the semi-global aggregation stage, to improve the accuracy in discontinuous regions, adaptive weighted path costs are adopted, and, five aggregation paths are used without consuming external memory resources. Finally, L-R check, spike removal and disparity confidence check handle the outlier regions to enhance the disparity map. Disparity map quality evaluations on Middlebury benchmark and real-world outdoor scenery show that the proposed implementation is able to generate high quality disparity maps. The proposed hardware architecture is implemented on a Stratix V FPGA, which results in a throughput of 1280 × 960/197fps with 64 disparity levels at 156MHz. Compared with the latest work, we can get 1.48 times speedup and use 19% more memory with much less computation resources (34.5% less ALUTs and 46.1% less Regis-ters).

Quaternion representation based visual saliency for stereoscopic image quality assessment

In this paper, a novel visual saliency detection method for stereoscopic images is proposed for the stereoscopic image quality assessment (SIQA) by considering the disparity map and difference image between the stereo image pairs. Firstly, a new quaternion representation (QR) of each stereo image (left/right view image) is constructed, which comprises the image content, the inter-view disparity, and the difference map. The quaternion Fourier transform (QFT) is performed on the constructed QR to generate the visual saliency maps for left and right views of stereoscopic image pairs, respectively. The generated visual saliency maps are further incorporated into the quality metrics for SIQA. Experimental results demonstrate that the visual saliency maps generated by the proposed method can help significantly boost the performance of SIQA, comparing with other visual saliency models proposed for stereoscopic images. It further confirms that the proposed visual saliency model can accurate depict the acuity property of human visual system (HVS) in judging the perceptual quality of stereoscopic images.

Planar self-calibration for stereo cameras with radial distortion.

In this paper, we present a robust technique of stereo calibration using homography constraints. Our method is novel as stereo calibration is performed by solving a polynomial equation system including two radial distortion parameters, using a minimal number of five image point correspondences. This enables us to calibrate from only a pair of stereo images of a planar scene, and to provide the exact algebraic solution to the stereo calibration problem. The minimal case solution is useful to reduce the computation time and increase the calibration robustness when using random sample consensus (RANSAC) from the correspondences of the stereo image pair. Further, a non-linear parameter optimization for the intrinsic and extrinsic parameters of stereo cameras is performed using the inliers, which are determined after RANSAC. In addition, our method can achieve more robust calibration results with multiple stereo image pairs by performing joint optimization. In contrast to the previous stereo calibration methods, our method works without requiring any special hardware and has no problems with one stereo image pair, even corrupted by severe radial distortions. Finally, by evaluating our method on both synthetic and real scene data, we demonstrate that our method is both efficient and accurate for stereo calibration.

DESIGN AND IMPLEMENTATION OF A NOVEL PORTABLE 360° STEREO CAMERA SYSTEM WITH LOW-COST ACTION CAMERAS

Abstract. The demand for capturing indoor spaces is rising with the digitalization trend in the construction industry. An efficient solution for measuring challenging indoor environments is mobile mapping. Image-based systems with 360° panoramic coverage allow a rapid data acquisition and can be processed to georeferenced 3D images hosted in cloud-based 3D geoinformation services. For the multiview stereo camera system presented in this paper, a 360° coverage is achieved with a layout consisting of five horizontal stereo image pairs in a circular arrangement. The design is implemented as a low-cost solution based on a 3D printed camera rig and action cameras with fisheye lenses. The fisheye stereo system is successfully calibrated with accuracies sufficient for the applied measurement task. A comparison of 3D distances with reference data delivers maximal deviations of 3 cm on typical distances in indoor space of 2-8 m. Also the automatic computation of coloured point clouds from the stereo pairs is demonstrated.

Determining Surface Roughness in Erosion Testing Using Digital Photogrammetry

Abstract Surface roughness is an imporant characteristic in numerical and physical geotechnical soil models. The methods for measuring surface roughness are currently operator dependent, time consuming, or require highly specialized equipment. The objective of this study was to develop a method to measure the roughness on soil samples in erosion tests and reduce the operator dependency. A stereophotogrammetry computational program was developed in which three-dimensional (3D) soil surface coordinates were derived from digital stereo pair images of soil samples following erosion tests. The 3D coordinates were then used to calculate a measure of surface roughness for the soil samples. A validation study showed that the stereophotogrammetry system was able to measure the roughness of the sample within 0.09 mm of a high-density scan taken using a structured light scanner. This stereophotogrammetry system is automated and yields more precise, repeatable roughness measurements, and the results are less variable than the traditional caliper method.

Stereo matching using cost volume fusion for high dynamic range scenes

A method is proposed to obtain accurate disparity maps for high dynamic range (HDR) scenes using stereo image pairs acquired under different exposure times and viewpoints. In HDR scenes, saturated pixels such as too-dark or too-bright pixels are inevitable because of the limited bit-depth representations of images. These saturated regions include little information for matching images and harm the performance of conventional stereo matching methods. To address this problem, a method is proposed that generates a new cost volume by fusing cost volumes obtained using different stereo image pairs taken under different exposure times as well as different points of view. In addition, the directional semi-global matching method is proposed to efficiently optimise the cost volume. Experimental results show that the proposed method generates more accurate results for saturated regions than existing methods on various datasets.

Moving-Object Detection From Consecutive Stereo Pairs Using Slanted Plane Smoothing

Detecting moving objects is of great importance for autonomous unmanned vehicle systems, and a challenging task especially in complex dynamic environments. This paper proposes a novel approach for the detection of moving objects and the estimation of their motion states using consecutive stereo image pairs on mobile platforms. First, we use a variant of the semi-global matching algorithm to compute initial disparity maps. Second, assisted by the initial disparities, boundaries in the image segmentation produced by simple linear iterative clustering are classified into coplanar, hinge, and occlusion. Moving points are obtained during ego-motion estimation by a modified random sample consensus) algorithm without resorting to time-consuming dense optical flow. Finally, the moving objects are extracted by merging superpixels according to the boundary types and their movements. The proposed method is accelerated on the GPU at 20 frames per second. The data which we use for testing and benchmarking is released, thus completing similar data sets. It includes 812 image pairs and 924 moving objects with ground truth for better algorithms evaluation. Experimental results demonstrate that the proposed method achieves competitive results in terms of moving-object detection and their motion state estimation in challenging urban scenarios.

A comparative evaluation of combined feature detectors and descriptors in different color spaces for stereo image matching of tree

Abstract Tree canopy geometric characteristics are directly related to tree growth and productivity, and this information has been used to predict yield, fertilizer application in citrus crops, water consumption or biomass. So a 3D model and a depth map of the tree can be useful. One method of creating 3D model is stereo vision technique. The comparison and performance assessment of different combinations of feature detectors and descriptors is very important in this technique. In this study, the performance of 12 combinations of well-known detectors and descriptors including BRISK with SURF, BRISK with BRISK, BRISK with FREAK, Harris with SURF, Harris with BRISK, Harris with FREAK, SURF with SURF, SURF with BRISK, SURF with FREAK, MSER with SURF, MSER with BRISK, and MSER with FREAK was evaluated. The included color spaces were HSV, H, YCbCr, Y, NTSC and RGB. The performance comparison for each combination was carried out in terms of precision and recall values using stereo image pairs of tree. It was observed that the largest number of keypoints were detected by MSER and SURF detectors almost in all possible spaces. Best combinations used from SURF detector and descriptor when the precision and recall results were considered. RGB and Y spaces were the best spaces to implement combinations. The combination of SURF with SURF, SURF with FREAK, and HARRIS-SURF were found to be preferable.

Optimal Weight Design Approach for the Geometrically-Constrained Matching of Satellite Stereo Images

This study presents an optimal weighting approach for combined image matching of high-resolution satellite stereo images (HRSI). When the rational polynomial coefficients (RPCs) for a pair of stereo images are available, some geometric constraints can be combined in image matching equations. Combining least squares image matching (LSM) equations with geometric constraints equations necessitates determining the appropriate weights for different types of observations. The common terms between the two sets of equations are the image coordinates of the corresponding points in the search image. Considering the fact that the RPCs of a stereo pair are produced in compliance with the coplanarity condition, geometric constraints are expected to play an important role in the image matching process. In this study, in order to control the impacts of the imposed constraint, optimal weights for observations were assigned through equalizing their average redundancy numbers. For a detailed assessment of the proposed approach, a pair of CARTOSAT-1 sub-images, along with their precise RPCs, were used. On top of obtaining different matching results, the dimension of the error ellipses of the intersection points in the object space were compared. It was shown through analysis that the geometric mean of the semi-minor and semi-major axis by our method was reduced 0.17 times relative to the unit weighting approach.

2D EVOLUTION PATTERN AND 3D RECONSTRUCTION OF TWO LARGE RIFTS IN FILCHNER ICE SHELF FROM SATELLITE IMAGES

Abstract. The floating ice shelves surrounding the Antarctic Ice Sheet is important for the stability of the grounded ice-sheet. Fracturing and calving event of an ice shelf can damage the structural integrity and contributes to an increase in ice discharge to the ocean. In this study, we focus on two large newly-developed traverse rifts in the frontal region of Filchner Ice Shelf which may lead to next major calving in the region. Two-dimensional evolution of the two rifts were investigated in detail using multisource remote sensing data, Two abrupt extensions in 2006 and 2011 were found with a rate of more than 1 km d-1. Moreover, a stereo pair of ZY-3 images was utilized for the first time to reconstruct the 3D rift model where an average rift depth of 48 m and ice mélange rise of ~10 m were measured.

Epipolar Rectification by Singular Value Decomposition of Essential Matrix

Image rectification is an important stage of applying a pair of projective transformations, or homographies, to a pair of stereo images, so that epipolar lines in the original images map to horizontally aligned lines in the rectified images. Considering that for some stereo rigs the intrinsic parameters of the cameras are known but their external parameters are unknown, in this paper, we present a novel method for stereo rectification based on the essential matrix which is derived from the fundamental matrix. Without any optimization process, closed-form analytical solutions of the projective transformations for epipolar rectification can be directly obtained by conducting SVD on the essential matrix. Experimental results show the proposed rectification method not only has higher efficiency and rectification precision, but also its scale invariance and robustness are superior to existing methods.

Stereo-vision three-dimensional reconstruction of curvilinear structures imaged with a TEM

Deriving accurate three-dimensional (3-D) structural information of materials at the nanometre level is often crucial for understanding their properties. Tomography in transmission electron microscopy (TEM) is a powerful technique that provides such information. It is however demanding and sometimes inapplicable, as it requires the acquisition of multiple images within a large tilt arc and hence prolonged exposure to electrons. In some cases, prior knowledge about the structure can tremendously simplify the 3-D reconstruction if incorporated adequately. Here, a novel algorithm is presented that is able to produce a full 3-D reconstruction of curvilinear structures from stereo pair of TEM images acquired within a small tilt range that spans from only a few to tens of degrees. Reliability of the algorithm is demonstrated through reconstruction of a model 3-D object from its simulated projections, and is compared with that of conventional tomography. This method is experimentally demonstrated for the 3-D visualization of dislocation arrangements in a deformed metallic micro-pillar.

Monocular Stereo Measurement Using High-Speed Catadioptric Tracking.

This paper presents a novel concept of real-time catadioptric stereo tracking using a single ultrafast mirror-drive pan-tilt active vision system that can simultaneously switch between hundreds of different views in a second. By accelerating video-shooting, computation, and actuation at the millisecond-granularity level for time-division multithreaded processing in ultrafast gaze control, the active vision system can function virtually as two or more tracking cameras with different views. It enables a single active vision system to act as virtual left and right pan-tilt cameras that can simultaneously shoot a pair of stereo images for the same object to be observed at arbitrary viewpoints by switching the direction of the mirrors of the active vision system frame by frame. We developed a monocular galvano-mirror-based stereo tracking system that can switch between 500 different views in a second, and it functions as a catadioptric active stereo with left and right pan-tilt tracking cameras that can virtually capture 8-bit color images each operating at 250 fps to mechanically track a fast-moving object with a sufficient parallax for accurate 3D measurement. Several tracking experiments for moving objects in 3D space are described to demonstrate the performance of our monocular stereo tracking system.

A robust method for the reconstruction of disparity maps based on multilevel processing of stereo color image pairs

A novel method for the reconstruction of disparity maps (DMs) with robust properties to nonideal registration conditions, reflections, and noise in stereo color image pairs has been substantiated for the first time. The novel approach proposes a scheme for image DM reconstruction where Jaccard distance metric is used as a proximity criterion in stereo image pair matching. A physical interpretation of the method that allows the quality of the formed DMs to be improved significantly is given. A processing block diagram has been developed in accordance with the novel approach. Simulations of the novel DM reconstruction method have shown an advantage of the proposed DM reconstruction scheme in terms of generally recognized criteria, such as the structural similarity index measure and the bad matching pixels, and when visually comparing the formed DMs.

Vertical Accuracy Effect Verification for Satellite Imagery With Different GCPs

This letter expands the block adjustment method by, respectively, adopting high-precision global positioning system (GPS) points, geoscience laser altimeter system (GLAS) data, and Shuttle Radar Topography Mission (SRTM)-digital elevation model (DEM) as vertical control data. Block adjustment experiments were conducted with these control data, with 388 ZY-3 satellite stereo image pairs across over 18600 km2 of land in Hubei province of China utilized as experimental images and 180 GPS points as checkpoints. The experimental results obtained show that, with the adoption of 23 GPS points as control points, the horizontal root-mean-square error (RMSE) and vertical RMSE of the ZY-3 image improved from 10.97 to 5.72 m and from 7.12 to 1.65 m, respectively. Furthermore, with the adoption of 326 GLAS data points as vertical control points, the vertical RMSE of the ZY-3 images of the flat terrain areas and mountain terrain areas improved to 1.69 and 3.62 m, respectively. Through block adjustment constrained by SRTM-DEM, the vertical RMSE of the ZY-3 images was 1.44 m for flat terrain areas and 3.05 m for mountain terrain areas, respectively. The vertical accuracy of the satellite images was significantly enhanced, especially on the flat terrain area, when GLAS and STRM-DEM were used for vertical control block adjustment, which is similar to the vertical accuracy of block adjustments adopting GPS points as control.

PRECISION 3D SURFACE RECONSTRUCTION FROM LRO NAC IMAGES USING SEMI-GLOBAL MATCHING WITH COUPLED EPIPOLAR RECTIFICATION

Abstract. The Narrow-Angle Camera (NAC) on board the Lunar Reconnaissance Orbiter (LRO) comprises of a pair of closely attached high-resolution push-broom sensors, in order to improve the swath coverage. However, the two image sensors do not share the same lenses and cannot be modelled geometrically using a single physical model. Thus, previous works on dense matching of stereo pairs of NAC images would generally create two to four stereo models, each with an irregular and overlapping region of varying size. Semi-Global Matching (SGM) is a well-known dense matching method and has been widely used for image-based 3D surface reconstruction. SGM is a global matching algorithm relying on global inference in a larger context rather than individual pixels to establish stable correspondences. The stereo configuration of LRO NAC images causes severe problem for image matching methods such as SGM, which emphasizes global matching strategy. Aiming at using SGM for image matching of LRO NAC stereo pairs for precision 3D surface reconstruction, this paper presents a coupled epipolar rectification methods for LRO NAC stereo images, which merges the image pair in the disparity space and in this way, only one stereo model will be estimated. For a stereo pair (four) of NAC images, the method starts with the boresight calibration by finding correspondence in the small overlapping stripe between each pair of NAC images and bundle adjustment of the stereo pair, in order to clean the vertical disparities. Then, the dominate direction of the images are estimated by project the center of the coverage area to the reference image and back-projected to the bounding box plane determined by the image orientation parameters iteratively. The dominate direction will determine an affine model, by which the pair of NAC images are warped onto the object space with a given ground resolution and in the meantime, a mask is produced indicating the owner of each pixel. SGM is then used to generate a disparity map for the stereo pair and each correspondence is transformed back to the owner and 3D points are derived through photogrammetric space intersection. Experimental results reveal that the proposed method is able to reduce gaps and inconsistencies caused by the inaccurate boresight offsets between the two NAC cameras and the irregular overlapping regions, and finally generate precise and consistent 3D surface models from the NAC stereo images automatically.

ELEMENTAL AND TOPOGRAPHIC MAPPING OF LAVA FLOWSTRUCTURES IN MARE SERENITATIS ON THE MOON

Abstract. The detection of lunar lava flows based on local morphology highly depends on the available images. The thickness of lava flows, however, has been studied by many researchers and lunar lava flows are shown to be as thick as 200 m. Lunar lava flows are supposed to be concentrated on the northwestern lunar nearside. In this study we present elemental abundance maps, a petrological map and a digital terrain model (DTM) of a lava flow structure in northern Mare Serenitatis at (18.0° E, 32.4° N) and two possible volcanic vents at (11.2° E, 24.6° N) and (13.5° E, 37.5° N), respectively. Our abundance maps of the refractory elements Ca, Mg and our petrological map were obtained based on hyperspectral image data of the Moon Mineralogy Mapper (M3) instrument. Our DTM was constructed using GLD100 data in combination with a shape from shading based method to M3 and Lunar Reconnaissance Orbiter (LRO) Narrow Angle Camera (NAC) image data. The obtained NAC-based DEM has a very high effective resolution of about 1–2 m which comes close to the resolution of the utilized NAC images without requiring intricate processing of NAC stereo image pairs. As revealed by our elemental maps and DEM, the examined lava flow structure occurs on a boundary between basalts consisting of low-Ca/high-Mg pyroxene and high-Ca/low-Mg pyroxene, respectively. The total thickness of the lava flow is about 100 m, which is a relatively large value, but according to our DEM the lava flow may also be composed of two or more layers.

Hiding of phase-based stereo disparity for ghost-free viewing without glasses

When a conventional stereoscopic display is viewed without stereo glasses, image blurs, or 'ghosts', are visible due to the fusion of stereo image pairs. This artifact severely degrades 2D image quality, making it difficult to simultaneously present clear 2D and 3D contents. To overcome this limitation (backward incompatibility), here we propose a novel method to synthesize ghost-free stereoscopic images. Our method gives binocular disparity to a 2D image, and drives human binocular disparity detectors, by the addition of a quadrature-phase pattern that induces spatial subband phase shifts. The disparity-inducer patterns added to the left and right images are identical except for the contrast polarity. Physical fusion of the two images cancels out the disparity-inducer components and makes only the original 2D pattern visible to viewers without glasses. Unlike previous solutions, our method perfectly excludes stereo ghosts without using special hardware. A simple algorithm can transform 3D contents from the conventional stereo format into ours. Furthermore, our method can alter the depth impression of a real object without its being noticed by naked-eye viewers by means of light projection of the disparity-inducer components onto the object's surface. Psychophysical evaluations have confirmed the practical utility of our method.

Mixing body‐parts model for 2D human pose estimation in stereo videos

This study targets 2D articulated human pose estimation (i.e. localisation of body limbs) in stereo videos. Although in recent years depth-based devices (e.g. Microsoft Kinect) have gained popularity, as they perform very well in controlled indoor environments (e.g. living rooms, operating theatres or gyms), they suffer clear problems in outdoor scenarios and, therefore, human pose estimation is still an interesting unsolved problem. The authors propose here a novel approach that is able to localise upper-body keypoints (i.e. shoulders, elbows, and wrists) in temporal sequences of stereo image pairs. The authors’ method starts by locating and segmenting people in the image pairs by using disparity and appearance information. Then, a set of candidate body poses is computed for each view independently. Finally, temporal and stereo consistency is applied to estimate a final 2D pose. The authors’ validate their model on three challenging datasets: ‘stereo human pose estimation dataset’, ‘poses in the wild’ and ‘INRIA 3DMovie’. The experimental results show that the authors’ model not only establishes new state-of-the-art results on stereo sequences, but also brings improvements in monocular sequences.