Stereo Image Pairs Research Articles

360 ° Stereo Image Composition With Depth Adaption.

360 ° images and videos have become an economic and popular way to provide VR experiences using real-world content. However, the manipulation of the stereo panoramic content remains less explored. In this article, we focus on the 360 ° image composition problem, and develop a solution that can take an object from a stereo image pair and insert it at a given 3D position in a target stereo panorama, with well-preserved geometry information. Our method uses recovered 3D point clouds to guide the composited image generation. More specifically, we observe that using only a one-off operation to insert objects into equirectangular images will never produce satisfactory depth perception and generate ghost artifacts when users are watching the result from different view directions. Therefore, we propose a novel per-view projection method that segments the object in 3D spherical space with the stereo camera pair facing in that direction. A deep depth densification network is proposed to generate depth guidance for the stereo image generation of each view segment according to the desired position and pose of the inserted object. We finally combine the synthesized view segments and blend the objects into the target stereo 360 ° scene. A user study demonstrates that our method can provide good depth perception and removes ghost artifacts. The per-view solution is a potential paradigm for other content manipulation methods for 360 ° images and videos.

Self‐supervised binocular depth estimation algorithm with self‐rectification for autonomous driving

AbstractAiming to address the challenge where existing methods struggle to predict accurate disparities for imperfectly rectified stereo images, and that supervised training requires a considerable amount of ground truth, a self‐supervised binocular depth estimation algorithm with self‐rectification for autonomous driving is proposed. Firstly, a subnetwork dedicated to stereo rectification, aiming to estimate the homography between stereo images is developed. This homography facilitates the transformation of stereo image pairs, aligning their corresponding pixels horizontally. Secondly, a foundational self‐supervised framework primarily centred on minimizing errors in stereo image reconstruction, combined with the generative‐adversarial strategy is introduced. Finally, a vertical offset prediction module (VOPM) is incorporated into the basic framework to further enhance the resistance of the stereo matching network to pixel‐level vertical offset errors. Experimental results on the public KITTI dataset for autonomous driving demonstrate the effectiveness of this approach in improving the disparity prediction performance for imperfectly rectified stereo images. Moreover, the self‐supervised training framework exhibits superiority over state‐of‐the‐art methods.

Refining disparity maps using deep learning and edge-aware smoothing filter

Stereo matching algorithm is crucial for applications that rely on three-dimensional (3D) surface reconstruction, producing a disparity map that contains depth information by computing the disparity values between corresponding points from a stereo image pair. In order to yield desirable results, the proposed stereo matching algorithm must possess a high degree of resilience against radiometric variation and edge inconsistencies. In this article convolutional neural network (CNN) is employed in the first stage to generate the raw matching cost, which is subsequently filtered with a bilateral filter (BF) and applied with cross-based cost aggregation (CBCA) during the cost aggregation stage to enhance precision. Winner-take-all (WTA) strategy is implemented to normalise the disparity map values. Finally, the resulting output is subjected to an edge-aware smoothing filter (EASF) to reduce the noise. Due to its resistance to high contrast and brightness, the filter is found to be effective in refining and eliminating noise from the output image. Despite discontinuities like adiron's lost cup handle or artl's shattered rods, this approach, based on experimental research utilizing a Middlebury standard validation benchmark, yields a high level of accuracy, with an average non-occluded error of 6.79%, comparable to other published methods.

CVGSR: Stereo image Super-Resolution with Cross-View guidance

The ability to capture the complementary information of stereo images is critical to the development of stereo image super-resolution. Most existing studies have attempted to integrate reliable stereo correspondence along the polar directions through parallax attention and various fusion strategies. However, most of these approaches ignore the large parallax differences in stereo images, resulting in poor performance of convolutional-based parallax attention in capturing the long-range dependencies between images. In this paper, we propose a novel cross-view guided stereo image super-resolution network (CVGSR) for reconstructing high-resolution stereo image pairs with rich texture details by fully exploiting the complementary nature of stereo image pairs. Specifically, we first deploy a cross-view interaction module (CVIM) to explore intra/cross-view dependencies from local to global to compensate for the incomplete compatibility of information between the left and right views. This module uses a progressive cross-guiding strategy to better merge features from occluded and non-occluded regions. Based on this, an efficient attention Transformer (EAT) is improved to activate more input information and further mine the cross-view complementarity. Furthermore, we design a texture loss to optimize the visual perceptual quality of reconstructed images with sharp boundaries and rich texture details. Extensive experiments on four stereo image datasets demonstrate that the proposed CVGSR achieves a competitive and excellent performance.

A road surface reconstruction dataset for autonomous driving

Recent developments in intelligent robot systems, especially autonomous vehicles, put forward higher requirements for safety and comfort. Road conditions are crucial factors affecting the comprehensive performance of ground vehicles. Nonetheless, existing environment perception datasets for autonomous driving lack attention to road surface areas. In this paper, we introduce the road surface reconstruction dataset, providing multi-modal, high-resolution, and high-precision data collected by real-vehicle platform in diverse driving conditions. It covers common road types containing approximately 16,000 pairs of stereo images, point clouds, and ground-truth depth/disparity maps, with accurate data processing pipelines to ensure its quality. Preliminary evaluations reveal the effectiveness of our dataset and the challenge of the task, underscoring substantial opportunities of it as a valuable resource for advancing computer vision techniques. The reconstructed road structure and texture contribute to the analysis and prediction of vehicle responses for motion planning and control systems.

Stereo matching from monocular images using feature consistency

AbstractSynthetic images facilitate stereo matching. However, synthetic images may suffer from image distortion, domain bias, and stereo mismatch, which would significantly restrict the widespread use of stereo matching models in the real world. The first goal in this paper is to synthesize real‐looking images for minimizing the domain bias between the synthesized and real images. For this purpose, sharpened disparity maps are produced from a mono real image. Then, stereo image pairs are synthesized using these imperfect disparity maps and the single real image in the proposed pipeline. Although the synthesized images are as realistic as possible, the domain styles of the synthesized images are always very different from the real images. Thus, the second goal is to enhance the domain generalization ability of the stereo matching network. For that, the feature extraction layer is replaced with a teacher–student model. Then, a constraint of binocular contrast features is imposed on the output of the model. When tested on the KITTI, ETH3D, and Middlebury datasets, the accuracy of the method outperforms traditional methods by at least 30%. Experiments demonstrate that the approaches are general and can be conveniently embedded into existing stereo networks.

A dense matching method for remote sensing images fused with CPS denoising

Dense matching of remote sensing images is crucial for 3D reconstruction. This study proposes an enhanced dense matching method employing the CPS image denoising algorithm, aiming to boost the SGM algorithm's accuracy and efficiency in remote sensing image matching. The stereo image pair's quality is evaluated using the PSNR index, and a decision-making criterion based on the CPS algorithm is incorporated to determine the need for denoising. Preprocessing steps, including image cropping and pixel coordinate transformation, significantly reduce computational requirements. An epipolar line model, minimizing the disparity between two pixels, is used for calculations. This model is employed to construct an epipolar image, enhancing the accuracy and efficiency of the process. The study conducted experimental validation and analysis of the mismatch rate, running time, and denoising effect of the algorithm using the Middlebury 2021 stereo datasets. Additionally, the matching results of the World-View3 satellite stereo image pairs were visualized and analyzed. The experimental results indicate that the proposed algorithm reduces the average mismatch rate by 13.1% and increases the running speed by about 3 to 4 times compared to the SGBM algorithm. Specifically, the denoising effect reduces the mismatch rate of the reconstructed image by an average of 8.97%. The results indicate that the CPS method effectively addresses dense matching challenges in the presence of image blur and noise, thereby improving the operational efficiency and accuracy of the dense matching algorithm.

Observation of Chiral Channels in Helical Covalent Organic Frameworks.

Unraveling the mechanism of chirality transfer across length scales is crucial to the rational development of functional materials with hierarchical chirality. The key obstacle is the lack of structural information, especially at the mesoscopic level. We report herein the structural identification of helical covalent organic frameworks (heliCOFs) with hierarchical chirality, which integrate molecular chirality, channel chirality, and morphology chirality into one crystalline entity. Specifically, benefiting from the highly ordered structure of heliCOFs, the existence of chiral channels at the mesoscopic level has been confirmed by electron crystallography, and the handedness of these chiral channels has been directly determined through the stereopair imaging technique. Accordingly, the chirality transfer in heliCOFs from microscopic to macroscopic levels could be rationalized with a layer-rotating model that has been supported by both crystal structure analysis and theoretical calculations. Observation of chiral channels in heliCOFs not only provides unprecedented data for the understanding of the chirality transfer process but also sheds new light on the rational construction of highly ordered polymeric materials with hierarchical chirality.

Modeling Stereo-Confidence out of the End-to-End Stereo-Matching Network via Disparity Plane Sweep

We propose a novel stereo-confidence that can be measured externally to various stereo-matching networks, offering an alternative input modality choice of the cost volume for learning-based approaches, especially in safety-critical systems. Grounded in the foundational concepts of disparity definition and the disparity plane sweep, the proposed stereo-confidence method is built upon the idea that any shift in a stereo-image pair should be updated in a corresponding amount shift in the disparity map. Based on this idea, the proposed stereo-confidence method can be summarized in three folds. 1) Using the disparity plane sweep, multiple disparity maps can be obtained and treated as a 3-D volume (predicted disparity volume), like the cost volume is constructed. 2) One of these disparity maps serves as an anchor, allowing us to define a desirable (or ideal) disparity profile at every spatial point. 3) By comparing the desirable and predicted disparity profiles, we can quantify the level of matching ambiguity between left and right images for confidence measurement. Extensive experimental results using various stereo-matching networks and datasets demonstrate that the proposed stereo-confidence method not only shows competitive performance on its own but also consistent performance improvements when it is used as an input modality for learning-based stereo-confidence methods.

Irregular boundaries stereo images dataset creating using depth estimation model

AbstractThis paper introduces a stereoscopic image and depth dataset created using a deep learning model. It addresses the challenge of obtaining accurate and annotated stereo image pairs with irregular boundaries for deep learning model training. Stereoscopic image and depth dataset provides a unique resource for training deep learning models to handle irregular boundary stereoscopic images, which are valuable for real-world scenarios with complex shapes or occlusions. The dataset is created using monocular depth estimation, a state-of-the-art depth estimation model, and it can be used in applications like rectifying images, estimating depth, detecting objects, and autonomous driving. Overall, this paper presents a novel dataset that demonstrates its effectiveness and potential for advancing stereo vision and developing deep learning models for computer vision applications.

Bridging local and global representations for self-supervised monocular depth estimation

Monocular depth estimation is a challenging problem, especially in a self-supervised manner without relying on the depth ground truth. The self-supervised approach places higher demands on the global and local feature extraction capabilities of the network. Based on this view, we propose to perform efficient hybrid feature extraction by exploiting the capacity of the Transformer and convolutional neural network to model long-range dependencies and local correlations at the same time. A bowknot-type fuser is designed to align features extracted from different sources as well as bridge global and local semantic representations. To obtain higher-quality pseudo-labels from the extracted features, we suggest the pseudo-label smoothing technique to fully utilize the multi-scale features, consequently boosting the effect of self-distillation loss as an auxiliary supervision to train the neural network. In addition, we propose pixel adaptive smoothness loss to refine the predicted depth map by introducing the image’s textural and spatial information. The suggested method is trained on the KITTI benchmark using stereo image pairs and achieves competitive depth estimation performance in contrast with previous approaches. The code and models are available at https://github.com/MaylingLin/BLGR-Depth.

ERROR ANALYSIS OF VISUAL ODOMETRY FOR A SMALL SIZE UNMANNED AERIAL VEHICLE

Abstract. Usage of small size Unmanned Aerial Vehicle (UAV) with autonomous flight capability became widespread at both civil and military applications because of its low costs and ease of use. However, inertial or GPS based positioning and orientation sensors can be disabled at battlefields and the UAV losses its path in a short time. Visual odometry becomes a solution for the aforementioned hassle. Real-time detection of ground control points, matching the same control points at the stereo pair images and correcting the position and orientation data with least square adjustment are the challenges of visual odometry. In this study, error analysis of position and orientation of a UAV flying through mountainous region which cannot obtain any augmenting data for position and orientation is conducted by error propagation law. The flight of UAV is simulated with exact initial position and orientation data. Optic images of ground obtained by nadir directed camera are georeferenced using the digital elevation model of the region, interior and exterior camera parameters. The digital elevation model is used to correct the relief displacement. The stereo-pair image is obtained when the UAV flies 200 m and image coordinates of the common control points are detected by colinearity equations. Position and orientation of UAV is corrected by least square adjustment and the process is repeated for each obtained image acquired at 200 m interval. The analysis revealed that the precisions of horizontal and vertical positioning becomes 75 and 200 meter, respectively. The precision of roll angle becomes worse than 0.5 degree, while the precisions of pitch and yaw angle are estimate to be better that 0.1 degree.

Deformation mechanisms and their role in the lack of ductility in the refractory-based high entropy alloy AlMo0.5NbTa0.5TiZr

The deformation mechanisms in the refractory high entropy alloy AlMo0.5NbTa0.5TiZr have been investigated, with their potential impact on ductility. Specifically, the modes of plasticity active in the B2 matrix and in an alloyed version of the phase Al4Zr5 found at grain and subgrain boundaries were explored by use of nano-indentation. For the B2 phase, a combination of weak-beam dark-field and superlattice imaging revealed dissociation of 〈111〉 superdislocations into two superpartial dislocations, bounding an anti-phase boundary (APB), following the reaction: a〈111〉= a/2〈111〉+APB+a/2〈111〉. The dislocations in the B2 phase exhibited long, relatively straight screw segments and short edge segments, such that the mobility of the edge dislocations appears to be higher than that of the screw segments. Evidence of a B2 to bcc slip transmission process has been observed through use of stereo-pair imaging and anaglyph construction. This process involves the transmission of slip from the B2 matrix, followed by the decorrelated motion of the two a/2〈111〉 dislocations in the bcc precipitate, demonstrated by the increased separation of the dislocations within the bcc phase. Regarding the deformation of the Al4Zr5 intermetallic phase, it was found that indentation of this phase resulted in no evidence of dislocation generation and motion, and cracking at the edges of the given indent. It has been concluded that the lack of plasticity of this phase is the cause of the room temperature brittle nature of this alloy.

End-to-End Edge-Guided Multi-Scale Matching Network for Optical Satellite Stereo Image Pairs

Acquiring disparity maps by dense stereo matching is one of the most important methods for producing digital surface models. However, the characteristics of optical satellite imagery, including significant occlusions and long baselines, increase the challenges of dense matching. In this study, we propose an end-to-end edge-guided multi-scale matching network (EGMS-Net) tailored for optical satellite stereo image pairs. Using small convolutional filters and residual blocks, the EGMS-Net captures rich high-frequency signals during the initial feature extraction phase. Subsequently, pyramid features are derived through efficient down-sampling and consolidated into cost volumes. To regularize these cost volumes, we design a top–down multi-scale fusion network that integrates an attention mechanism. Finally, we innovate the use of trainable guided filter layers in disparity refinement to improve edge detail recovery. The network is trained and evaluated using the Urban Semantic 3D and WHU-Stereo datasets, with subsequent analysis of the disparity maps. The results show that the EGMS-Net provides superior results, achieving endpoint errors of 1.515 and 2.459 pixels, respectively. In challenging scenarios, particularly in regions with textureless surfaces and dense buildings, our network consistently delivers satisfactory matching performance. In addition, EGMS-Net reduces training time and increases network efficiency, improving overall results.

Multi-Scale Interaction Network for Low-Light Stereo Image Enhancement

Low-light stereo image enhancement (LLSIE) is a task that improves the visual quality of stereo images captured in low-light environment. The core idea of LLSIE is to exploit the cross-view cues for enhancing the stereo image pairs. However, LLSIE is still a challenging task due to the perspective difference and insufficient information interaction between left and right views. In this paper, we propose a novel Multi-Scale Interaction Network (MSINet) for LLSIE. To be specific, we present a new multi-scale stereo Interaction module (MIM), which can promote the multi-scale cross-view information flow. Firstly, MIM extracts multi-scale features from both the left and right views. Then, the features from two views are fused in different scales, so that the information from stereo image pairs can be fully exchanged. Compared to the existing parallax attention module (PAM), our MIM explores the cross-view interaction in multiple scales with less computational cost more sufficiently. Besides, we also develop a new Spatial Gating Block (SGB) for refining intra-view features. By incorporating the long residual and spatial gating mechanism, SGB is capable of completing dynamic spatial modeling and obtaining strong feature representations. Extensive experiments on the Flickr1024, KITTI 2015 and Middlebury datasets show that the proposed MSINet reaches SOTA performance and obtains better visual results than other compared methods.

Disparity Refinement for Stereo Matching of High-Resolution Remote Sensing Images Based on GIS Data

With the emergence of the Smart City concept, the rapid advancement of urban three-dimensional (3D) reconstruction becomes imperative. While current developments in the field of 3D reconstruction have enabled the generation of 3D products such as Digital Surface Models (DSM), challenges persist in accurately reconstructing shadows, handling occlusions, and addressing low-texture areas in very-high-resolution remote sensing images. These challenges often lead to difficulties in calculating satisfactory disparity maps using existing stereo matching methods, thereby reducing the accuracy of 3D reconstruction. This issue is particularly pronounced in urban scenes, which contain numerous super high-rise and densely distributed buildings, resulting in large disparity values and occluded regions in stereo image pairs, and further leading to a large number of mismatched points in the obtained disparity map. In response to these challenges, this paper proposes a method to refine the disparity in urban scenes based on open-source GIS data. First, we register the GIS data with the epipolar-rectified images since there always exists unignorable geolocation errors between them. Specifically, buildings with different heights present different offsets in GIS data registering; thus, we perform multi-modal matching for each building and merge them into the final building mask. Subsequently, a two-layer optimization process is applied to the initial disparity map based on the building mask, encompassing both global and local optimization. Finally, we perform a post-correction on the building facades to obtain the final refined disparity map that can be employed for high-precision 3D reconstruction. Experimental results on SuperView-1, GaoFen-7, and GeoEye satellite images show that the proposed method has the ability to correct the occluded and mismatched areas in the initial disparity map generated by both hand-crafted and deep-learning stereo matching methods. The DSM generated by the refined disparity reduces the average height error from 2.2 m to 1.6 m, which demonstrates superior performance compared with other disparity refinement methods. Furthermore, the proposed method is able to improve the integrity of the target structure and present steeper building facades and complete roofs, which are conducive to subsequent 3D model generation.

Stereo Matching Method for Remote Sensing Images Based on Attention and Scale Fusion

With the development of remote sensing satellite technology for Earth observation, remote sensing stereo images have been used for three-dimensional reconstruction in various fields, such as urban planning and construction. However, remote sensing images often contain noise, occluded regions, untextured areas, and repeated textures, which can lead to reduced accuracy in stereo matching and affect the quality of 3D reconstruction results. To reduce the impact of complex scenes in remote sensing images on stereo matching and to ensure both speed and accuracy, we propose a new end-to-end stereo matching network based on convolutional neural networks (CNNs). The proposed stereo matching network can learn features at different scales from the original images and construct cost volumes with varying scales to obtain richer scale information. Additionally, when constructing the cost volume, we introduce negative disparity to adapt to the common occurrence of both negative and non-negative disparities in remote sensing stereo image pairs. For cost aggregation, we employ a 3D convolution-based encoder–decoder structure that allows the network to adaptively aggregate information. Before feature aggregation, we also introduce an attention module to retain more valuable feature information, enhance feature representation, and obtain a higher-quality disparity map. By training on the publicly available US3D dataset, we obtain an accuracy of 1.115 pixels in end-point error (EPE) and 5.32% in the error pixel ratio (D1) on the test dataset, and the inference speed is 92 ms. Comparing our model with existing state-of-the-art models, we achieve higher accuracy, and the network is beneficial for the three-dimensional reconstruction of remote sensing images.

Disparity Computation With Low Intensity Quantization on Stereo Image Pairs

Disparity Computation With Low Intensity Quantization on Stereo Image Pairs

Dense Matching With Optimized Penalty and Interpolation for High-Resolution Optical Stereo Image Pairs

Dense Matching With Optimized Penalty and Interpolation for High-Resolution Optical Stereo Image Pairs

Stereo Superpixel Segmentation via Decoupled Dynamic Spatial-Embedding Fusion Network

Stereo superpixel segmentation aims at grouping the discretizing pixels into perceptual regions through left and right views more collaboratively and efficiently. Existing superpixel segmentation algorithms mostly utilize color and spatial features as input, which may impose strong constraints on spatial information while utilizing the disparity information in terms of stereo image pairs. To alleviate this issue, we propose a stereo superpixel segmentation method with a decoupling mechanism of spatial information in this work. To decouple stereo disparity information and spatial information, the spatial information is temporarily removed before fusing the features of stereo image pairs, and a decoupled stereo fusion module (DSFM) is designed to handle the stereo features alignment as well as occlusion problems. Moreover, since the spatial information is vital to superpixel segmentation, we further design a dynamic spatiality embedding module (DSEM) to re-add spatial information, and the weights of spatial information will be adaptively adjusted through the dynamic fusion (DF) mechanism in DSEM for achieving a finer segmentation. Comprehensive experimental results demonstrate that our method can achieve the state-of-the-art performance on the KITTI2015 and Cityscapes datasets, and also verify the efficiency when applied in salient object detection on NJU2K dataset. The source code will be available publicly after paper is accepted.