Stereo Image Pairs Research Articles

A framework for high-precision DEM reconstruction based on the radargrammetry technique

ABSTRACTIn recent years, with the continuous development of synthetic aperture radar (SAR) techniques, radargrammetry technique has become an important method to obtain high-precision digital elevation model (DEM). Based on radargrammetry techniques, a new processing framework is proposed in this letter. In the new framework, in respect of radiometry, a one-to-many stereo image pairs structure and a multiple image matching algorithm based on this structure are introduced, which improves the robustness and quality of the results in the matching process. In terms of geometry, a fusion algorithm of stereo pairs with different intersection angle based on the terrain relief slope threshold is proposed, which improves the accuracy of the results. Subsequently, the experiment is conducted in the Hengduan Mountains in Southwest China and demonstrate the effectiveness of this framework compared with the classical radargrammetry technique.

HPM-TDP: An efficient hierarchical PatchMatch depth estimation approach using tree dynamic programming

Accurate and efficient estimation of the dense depth information from a pair of stereo images is a key step for many applications such as digital surface model production, 3D reconstruction and visualization, autonomous driving, and robotic navigation. Although great progress has been achieved in stereo matching over the past decade, the matching difficulties in poor and repetitive texture regions remain an issue. Aiming at solving the shortcomings of the current methods, this paper proposes HPM-TDP, which is an efficient hierarchical PatchMatch depth estimation approach that integrates a coarse-to-fine image pyramid strategy with a continuous Markov random field (MRF)-based global energy optimization framework, and minimizes the energy function by combining a hierarchical PatchMatch (HPM) framework and local α-expansion based tree dynamic programming (TDP). Firstly, the coarse-to-fine image pyramid strategy is integrated with the PatchMatch filter algorithm to quickly generate the hierarchical disparity plane prior for initializing each pixel’s disparity plane of the energy function optimization. Secondly, a multi-resolution cost aggregation strategy is adopted to boost the robustness of the matching cost function in the poor and repetitive texture areas. Finally, the HPM framework and local α-expansion based TDP are adopted to solve the non-submodular energy optimization problem, resulting in a globally optimized disparity plane map. Three benchmark datasets—the Middlebury 3.0, KITTI 2015, and Vaihingen datasets—were used to test the performance of HPM-TDP. The comprehensive experimental results demonstrate that HPM-TDP obtains a good performance on all datasets in terms of the (“Out-Noc”, “Avg-Noc”, “Out-All”, “Avg-All”) of (15.45%, 4.16px, 24.26%, 12.14px) and (5.46%, 1.20px, 6.55%, 1.54px) for Middlebury 3.0 and KITTI 2015 training datasets, and the (“Out-All”, “Avg-All”) of (26.32%, 4.04px) for Vaihingen dataset, respectively.

Object occlusion guided stereo image retargeting

The resizing of stereo images while preserving salient content and geometric consistency between the image pair is an essential problem to be addressed for 3D visualization. Existing stereo image retargeting techniques are observed to be incurring salient object deformation. In this paper, we formulate a seam carving method to perform stereo image retargeting using graph-cuts having node size as the number of pixels in one of the stereo images. We define the object map with depth ordering of each object from the camera. The seams are allowed to pass along the object-object occlusion boundary at depth discontinuity in order to prevent salient object deformation. We propose adaptive occlusion boundary weights as a function of an object area to be occluded to preserve small objects. The seam passing through object-object occlusion boundary in one image may not follow the exact boundary in the other image. We propose a refinement of the seam to follow object-object occlusion boundary in both the stereo images. By comparing the proposed method with other state-of-the-art stereo image retargeting methods, we show that the proposed method is able to preserve the salient object(s) along with geometric consistency between the stereo image pair.

Depth Map Reconstruction Based on Features Formed by Descriptor of Stereo Color Pairs

An approach to depth map reconstruction from stereo pairs of color images in visualization of three-dimensional objects is proposed and substantiated for the first time. The approach makes use of a novel local image descriptor based on visual primitives and relations between them, namely, cocolority, coplanarity, distance, and angle. The new approach is compared with other well-known descriptors, such as DAISY and SID. Numerical experiments and an analysis and physical interpretation of their results obtained in the case of actual radiometric differences in the exposition or illumination of stereo image pairs have shown that the new approach is superior to other existing descriptors.

Bergman glial cell morphology under the high voltage Electron microscope

The detailed morphology of Bergam glial cell was observed in single field of view during observation of Golgi stained mouse cerebellar cortex under the high voltage electron microscopy. The 3-dimensional organization of Bergman glial cell fully demonstrated with 8-degree stereo-paired images. The morphology of Bergman glial fiber and its appendages forming microdomains connected to other glial fiber are clearly presented in this image. This image provides a valuable insight for understanding morphology of Bergman glial cell.

Dense Stereo Matching Based on Multiobjective Fitness Function—A Genetic Algorithm Optimization Approach for Stereo Correspondence

Dense stereo image matching in remotely sensed images is a challenging problem, though it has been studied for more than two decades, due to occlusions, discontinuities, geometric, and radiometric distortions. A novel multiobjective fitness function-based dense stereo matching approach using genetic algorithms (GAs) is proposed in this paper. The proposed method is useful for estimating dense disparity map with an improved number of inliers for a stereo image pair, despite the constraint of finding correct disparity at depth discontinuities. In this paper, the steps of GA, such as initialization of the population, fitness function, and crossover and mutation operation, are designed and implemented to effectively deal with the problem of dense stereo image matching. To initialize the population, a Scale Invariant Feature Transform (SIFT) descriptor is computed for each pixel and multiple-size window-based matching is performed, using the similarity measures: 1) Euclidean distance and 2) spectral angle mapper. The generated disparity maps are pruned to choose a suitable subset using the designed fitness functions, considering the constraints related to stereo image pair, such as epipolar constraint, which encodes the epipolar geometry and the similarity measure that is useful to decide accuracy of the correspondences. The two objective functions are the number of inliers computed using the fundamental matrix and an energy minimization function, considering discontinuities and occlusions. The usefulness of this approach for remotely sensed stereo image pairs is demonstrated by improving the number of inliers and favorably comparing with state-of-the-art dense stereo image matching methods.

Improvement of stereo matching algorithm based on guided filtering and Kernel Regression

Stereo matching is one of the most dynamic fields in computer vision. Though its relevant research has already stepped into a mature stage, there are still certain challenges to obtain real-time and high-precision disparity maps from stereo image pairs. This paper presents a novel local stereo matching algorithm with better performance in edge preserving. In the first stage, this paper measures matching cost through combining truncated absolute differences (TAD) of the color and gradient. In the cost aggregation stage, this paper is creatively to combined the weighted guided filtering and adaptive steering kernel regression algorithm, which effectively preserves image edge and depth information. In the final stage, an adaptive steering kernel regression algorithm is employed in interpolation to refine the final disparity map. According to the Middlebury benchmark experiments, the algorithm proposed in this paper could have better performance than other local stereo matching algorithms.

A Computational Model for the Neural Representation and Estimation of the Binocular Vector Disparity from Convergent Stereo Image Pairs.

The depth cue is a fundamental piece of information for artificial and living beings who interact with the surrounding environment in order to handle objects and to avoid obstacles: in such situations, the disparity patterns, which arise when agents fixate objects, are vector fields. We propose a biologically-inspired computational model to estimate dense horizontal and vertical disparity maps by exploiting the cortical paradigms of the primate visual system: in particular, we aim to model the disparity sensitivity of the V1-MT visual pathway. The proposed model is based on a first processing stage composed of a bank of spatial band-pass filters and a static nonlinearity, mimicking complex binocular cells. Then, subsequent pooling stages and decoding strategies allow the model to estimate the vector disparity, after having represented it as a population of MT-like units. We assess the proposed model by using standard benchmarking stereo images, the Middlebury dataset, and specific stereo images that have horizontal and vertical disparities, which characterize the stimuli produced by active vision systems. Moreover, we systemically analyze how the different processing stages affect the model performance, and we discuss their implications for the neural modeling.

Intrinsic color correction for stereo matching

The improvement of color similarity between stereo images can bring better performance to stereo matching algorithms. For this purpose, we present a color correction method to alleviate the color discrepancy between a pair of stereo images, so that the color appearance of one image, i.e., the target image, is consistent with the other image, i.e., the source image. Our method starts with decomposing both the target image and the source image into two intrinsic layers, i.e., the reflectance layer and the shading layer, using intrinsic decomposition. The purpose of intrinsic decomposition is to distinguish and then process different color discrepancies caused by different factors (shading and reflectance) separately and appropriately. Then, a practical and effective consistent segmentation algorithm, which applied to the original stereo images, is proposed to establish the region correspondences. Next, luminance correction method and color correction method, based on the local region correspondences, are adopted to correct the shading layer and the reflectance layer in the target image, respectively, making them as similar as possible to those of the source image. Eventually, the two corrected layers of the target image are combined to yield the final corrected image. The experimental results demonstrate that the proposed method, which using intrinsic decomposition to handle color discrepancy caused by different factors, not only enhances the visual color similarity between the stereo images but also improves the accuracy of their stereo matching.

Measuring shape and motion of a high-speed object with designed features from motion blurred images

Vision-based geometry measurement plays a crucial role in many science and industrial areas. Plenty of researches devoted to measuring static objects, while few focused on motion blurred situations, which inevitably arise when the object being measured moves fast relative to the camera(s). Motion blur usually invalids the vision-based measurement algorithms designated for static objects. In this paper, we devote to accurate three dimensional (3D) reconstruction of moving objects from motion blurred stereo image pairs. A convolutional neural network (CNN) based method is first proposed to recognize the motion blurred visual targets. A motion blur model based on inner-frame path superposition imaging is then established. Finally, an optimization framework is set up to reconstruct the 3D target motion path during the camera exposure. Experiments are involved to demonstrate the validity and accuracy of the method.

Toward the Ghosting Phenomenon in a Stereo-Based Map With a Collaborative RGB-D Repair

Although 3-D reconstruction of dynamic road environment by moving cameras has been broadly applied in recognition and navigation systems, this task is still considered challenging, especially under circumstances with moving objects, where the reconstruction precision is strongly harassed by the ghosting problem. To address this issue, in this paper, we propose a novel approach for reconstructing 3-D maps of complete static scenes, based on a combination of an elaborately designed moving-object filtering mechanism and a map repairing and blank refilling procedure, where both plausible color and depth information from stereo image pairs are utilized. In this approach, first, we employ the planarity knowledge into the initial depth map based on the simple linear iterative cluster (SLIC) superpixel segmentation. The dynamic area in the image is determined under the supervision of odometry calculation. After wiping off moving objects, by collaboratively repairing color and depth information, the final 3-D map containing only static scene is obtained. The experimental results on extensive challenging real-world scenarios demonstrate the effectiveness and robustness of our approach.

Reliable image matching via photometric and geometric constraints structured by Delaunay triangulation

Image matching is a basic task in the field of photogrammetry and remote sensing. By using the advantages of the Delaunay triangulation, this paper proposes a novel image matching method. First, neighboring structures of randomly distributed feature points are formed with the assistance of the Delaunay triangulation and its corresponding graph, and the image planes are simultaneously divided into patches of near-regular triangles. Second, two constraints, a photometric constraint and a geometric constraint, are implemented based on the constructed neighboring structures, which incorporate the hierarchical elimination and left-right checking strategies to deliver the influences of outliers on the decision of inliers and ensure the high precision of the final matches. The former utilizes a line descriptor as a second-order photometric constraint, and the latter adopts the spatial angular order (SAO) to achieve a geometric constraint for the calculation of dissimilarity scores between correspondences. In addition, with the constraints between triangles of the refined Delaunay triangulation and its corresponding graph, a match expansion is designed to exploit as many inliers as possible. Finally, a reliable image matching algorithm is proposed by sequentially executing the three constraints for outlier elimination and match expansion. Under comprehensive analysis and comparison with five state-of-the-art algorithms, the performance of the proposed method is verified by using both rigid and non-rigid datasets. The experimental results demonstrate that the Delaunay triangulation is sufficient to construct neighboring structures for the implementation of local photometric and geometric constraints, and the proposed method can achieve good performance in terms of the precision, recall and number of inliers, and provide reliable matches for stereo image pairs with both rigid and non-rigid transformations.

Disparity estimation using multilevel and global information

Deep convolutional neural networks have shown prominent performance in stereo matching. However, current network architectures lack performance in exploiting context and global information to finding corresponding points in ill-posed regions. A stereo matching network without postprocessing is proposed to solve this problem. This network combines the improved multilevel feature pyramid pooling module with the light two-dimensional (2-D) convolution subnetwork to efficiently utilize multilevel information and global information. In the multilevel feature pyramid pooling module, the base image feature is extracted by cascading three small convolution filters. Features of a stereo image pair are calculated by hierarchically fusing and pooling features information of the same scale after using the residual network. Multilevel semantic information is fully utilized to improve the robustness of image feature representation in multilevel feature pyramid pooling module. In the light 2-D convolution subnetwork, low-level structural information is obtained from the target image by three concatenated convolution layers with small convolution filters. Low-level information is used to rectify matching cost with global view to improve matching accuracy. The experimental results on the Scene Flow dataset, the MPI Sintel dataset, and the Middlebury dataset show that the performance obtained by the proposed network can be improved in the ill-posed regions. Matching accuracy is competitive compared to other results obtained by end-to-end networks without postprocessing.

2D/3D information fusion for building extraction from high-resolution satellite stereo images using kernel graph cuts

ABSTRACTThis study presents a building extraction strategy from High-resolution satellite stereo images (HRSSI) using 2D and 3D information fusion. In the 2D processing strategy, a visible vegetation index (VVI) is generated. In the 3D processing, a disparity map is generated using semi-global matching (SGM). To remove defects from the disparity map, an object-based approach is proposed by using mean-shift image segmentation and extracting rectangles. By removing terrain effects, a normalized disparity map (nDM) is produced. In the next step, vegetation pixels are removed from nDM and an initial building mask is generated. As nDM does not have precise building boundaries, hybrid segmentation by the kernel graph cut (KGC) is applied to the feature space including the RGB, nDM, and VVI and the results are used in a decision level fusion step. By this methodology, segments that are highly intersected with initial building mask are classified as buildings. Finally, a building boundary refinement (BBR) algorithm is applied to buildings for removing the remaining defects. The proposed method is applied to two pairs of GeoEye-1 stereo images including residential and industrial test areas. Evaluation results show the completeness and correctness level of higher than 90% for the two test areas. Further evaluations show that the quality metric has significantly changed after decision level fusion using the KGC.

A Computational Model that Recovers Depth from Stereo-Input without Using Any Oculomotor Information

It is commonly believed that the visual system requires oculomotor information to perceive depth from binocular disparity. However, any effect of the oculomotor information on depth perception is too restricted to explain depth perception under natural viewing conditions. In this study, I describe a computational model that can recover depth from a stereo-pair of retinal images without using any oculomotor information. The model shows that, at least from a computational perspective, any oculomotor information is not necessary for perceiving depth from the stereo retinal images.

Segment-Based Disparity Refinement With Occlusion Handling for Stereo Matching.

In this paper, we propose a disparity refinement method that directly refines the winner-take-all (WTA) disparity map by exploring its statistical significance. According to the primary steps of the segment-based stereo matching, the reference image is over-segmented into superpixels and a disparity plane is fitted for each superpixel by an improved random sample consensus (RANSAC). We design a two-layer optimization to refine the disparity plane. In the global optimization, mean disparities of superpixels are estimated by Markov random field (MRF) inference, and then, a 3D neighborhood system is derived from the mean disparities for occlusion handling. In the local optimization, a probability model exploiting Bayesian inference and Bayesian prediction is adopted and achieves second-order smoothness implicitly among 3D neighbors. The two-layer optimization is a pure disparity refinement method because no correlation information between stereo image pairs is demanded during the refinement. Experimental results on the Middlebury and KITTI datasets demonstrate that the proposed method can perform accurate stereo matching with a faster speed and handle the occlusion effectively. It can be indicated that the "matching cost computation + disparity refinement" framework is a possible solution to produce accurate disparity map at low computational cost.

A spatio temporal spectral framework for plant stress phenotyping

BackgroundRecent advances in high throughput phenotyping have made it possible to collect large datasets following plant growth and development over time, and those in machine learning have made inferring phenotypic plant traits from such datasets possible. However, there remains a dirth of datasets following plant growth under stress conditions along with methods for inferring them using only remotely sensed data, especially under a combination of multiple stress factors such as drought, weeds and nutrient deficiency. Such stress factors and their combinations are commonly encountered during crop production and being able to accurately detect and treat such stress conditions in an automated and timely manner can provide a major boost to farm yields with minimal resource input.ResultsWe present a generic framework for remote plant stress phenotyping that consists of a dataset with spatio-temporal-spectral data following sugarbeet crop growth under optimal, drought, low and surplus nitrogen fertilization, and weed stress conditions, along with a machine learning based methodology for systematically inferring these stress conditions from the remotely measured data. The dataset contains biweekly color images, infra-red stereo image pairs and hyperspectral camera images along with applied treatment parameters and environmental factors like temperature and humidity, collected over two months. We present a plant agnostic methodology for deriving plant trait indicators such as canopy cover, height, hyperspectral reflectance and vegetation indices along with a spectral 3D reconstruction of the plants from the raw data to serve as a benchmark. Additionally, we provide fresh and dry weight measurements for both the above (canopy) and below (beet) ground biomass at the end of the growing period to serve as indicators of expected yield. We further describe a data driven, machine learning based method to infer water, Nitrogen and weed stress using the derived plant trait indicators. We use the plant trait indicators to evaluate 8 different classification approaches from which the best classifier achieved a mean cross validation accuracy of approx 93, 76 and 83% for drought, nitrogen and weed stress severity classification respectively. We also show that our multi-modal approach significantly improves classifier performance over using any single modality.ConclusionThe presented framework and dataset can serve as a valuable reference for creating and comparing processing pipelines which extract plant trait indicators and infer prevalent stress factors from remote sensing data under a variety of environments and cropping conditions. These techniques can then be deployed on farm machinery or robots enabling automated, precise and timely corrective interventions for maximising yield.

Improvement of viewing experience on stereoscopic image guided by human stereo vision

Recent 3D visual quality assessment methods still have difficulties in providing the best viewing experience from the viewer’s perspective due to the ambiguous understanding of human stereo vision. One of the key reasons is that the disparity gradient, which affects human depth perception, is hard to control for the input stereo image pair. In this paper, we mathematically formulated the human disparity gradient and optimized the disparity gradients for each stereo image pair. Considering that the disparity gradient needs to be limited to a specific range to satisfy the human visual preference and comfortableness, we proposed a new quantitative definition of disparity gradient and trained the optimal disparity gradients were learned from the pilot study to enhance the viewing experience. Extensive subjective evaluations have demonstrated the competitiveness of this proposed method for the improvement of the viewing experience.

Modified Depth-Map Inpainting Method Using the Neural Network

This paper proposes a method for reconstructing a depth map obtained using a stereo pair image. The proposed approach is based on a geometric model for the synthesis of patches. The entire image is preliminarily divided into blocks of different size, where large blocks are used to restore homogeneous areas, and small blocks are used to restore details of the image structure. Lost pixels are recovered by copying the pixel values from the source based on the similarity criterion. We used a trained neural network to select the “best like” patch. Experimental results show that the proposed method gives better results than other modern methods, both in subjective and objective measurements for reconstructing a depth map.

Автоматизована побудова цифрової моделі мікроповерхні об’єкта за стереопарою цифрових РЕМ-зображень

Автоматизована побудова цифрової моделі мікроповерхні об’єкта за стереопарою цифрових РЕМ-зображень