Stereo Disparity Research Articles

Stereo disparity estimation based on Siamese network

Abstract This paper proposes an approach for estimating stereo disparity based on a Siamese network. The existing Top-k disparity regression strategy may overlook the true disparity at object edges. To address this issue, an additional maximum disparity estimation value is introduced to optimize the Top-k strategy, creating the Top-k+ strategy. Furthermore, this research improves the MobileNetv2 (MV2) block by introducing an attention mechanism in the frequency domain, resulting in a more efficient FMV2 block that extracts high-frequency information such as textures and edges. Considering the excellent performance of the gated recurrent unit (GRU) in disparity optimization, the algorithm adopts an iterative optimization method based on GRU. Comparative studies pertaining to the Scene Flow, KITTI 2015, and ETH3D benchmark datasets show outstanding results in disparity estimation, with an end-point error of 0.58 pixels on the Scene Flow dataset, a D1-all error of 1.67% on the KITTI 2015 dataset, and an average disparity error of 0.22 pixels on the ETH3D dataset. Compared to other GRU-based iterative optimization algorithms, the proposed method not only exhibits significant performance advantages but also demonstrates a lightweight design.

DOT-SLAM: A Stereo Visual Simultaneous Localization and Mapping (SLAM) System with Dynamic Object Tracking Based on Graph Optimization.

Most visual simultaneous localization and mapping (SLAM) systems are based on the assumption of a static environment in autonomous vehicles. However, when dynamic objects, particularly vehicles, occupy a large portion of the image, the localization accuracy of the system decreases significantly. To mitigate this challenge, this paper unveils DOT-SLAM, a novel stereo visual SLAM system that integrates dynamic object tracking through graph optimization. By integrating dynamic object pose estimation into the SLAM system, the system can effectively utilize both foreground and background points for ego vehicle localization and obtain a static feature points map. To rectify the inaccuracies in depth estimation from stereo disparity directly on the foreground points of dynamic objects due to their self-similarity characteristics, a coarse-to-fine depth estimation method based on camera-road plane geometry is presented. This method uses rough depth to guide fine stereo matching, thereby obtaining the 3 dimensions (3D)spatial positions of feature points on dynamic objects. Subsequently, by establishing constraints on the dynamic object's pose using the road plane and non-holonomic constraints (NHCs) of the vehicle, reducing the initial pose uncertainty of dynamic objects leads to more accurate dynamic object initialization. Finally, by considering foreground points, background points, the local road plane, the ego vehicle pose, and dynamic object poses as optimization nodes, through the establishment and joint optimization of a nonlinear model based on graph optimization, accurate six degrees of freedom (DoFs) pose estimations are obtained for both the ego vehicle and dynamic objects. Experimental validation on the KITTI-360 dataset demonstrates that DOT-SLAM effectively utilizes features from the background and dynamic objects in the environment, resulting in more accurate vehicle trajectory estimation and a static environment map. Results obtained from a real-world dataset test reinforce the effectiveness.

Survey on Embedding Economical Autonomous Navigation System for Mobile Robots and UAV

Navigation has traditionally served the purpose of determining one's position, locating destinations, and charting a course towards them. It furnishes accurate details about the whereabouts of specific places or objects. Despite numerous advancements and enhancements in navigation technology, there have been ongoing discussions about its potential for autonomy. This suggests a scenario where navigation operates independently, without human intervention. Devices equipped with this capability comprehend their destination and chart the most efficient route to reach it. A crucial concept in this context is Visual Odometry (VO), which calculates the relative position between successive image frames. Likewise, the positioning of mobile robots relies on similar principles. However, a significant challenge arises over time as VO is susceptible to accumulating errors, known as drift. The Inertial Measurement Unit (IMU), which consists of accelerometers, gyroscopes, and magnetometers, is added to counteract this. These elements provide data that is more accurate and helps reduce noise. The integration of IMU with VO results in the creation of Visual Inertial Odometry (VIO). Furthermore, combining VIO with Global Positioning System (GPS) data through an Extended Kalman Filter (EKF) enhances localization accuracy both locally and globally. Additionally, stereo disparity estimation is employed to generate a depth perception map for obstacle detection, converted into a 2D grid map of occupancy after. While a waypoint follower directs the robot or devices toward its intended objective, local route planning algorithms create interim waypoints to prevent obstructions.

Stereo Superpixel Segmentation via Decoupled Dynamic Spatial-Embedding Fusion Network

Stereo superpixel segmentation aims at grouping the discretizing pixels into perceptual regions through left and right views more collaboratively and efficiently. Existing superpixel segmentation algorithms mostly utilize color and spatial features as input, which may impose strong constraints on spatial information while utilizing the disparity information in terms of stereo image pairs. To alleviate this issue, we propose a stereo superpixel segmentation method with a decoupling mechanism of spatial information in this work. To decouple stereo disparity information and spatial information, the spatial information is temporarily removed before fusing the features of stereo image pairs, and a decoupled stereo fusion module (DSFM) is designed to handle the stereo features alignment as well as occlusion problems. Moreover, since the spatial information is vital to superpixel segmentation, we further design a dynamic spatiality embedding module (DSEM) to re-add spatial information, and the weights of spatial information will be adaptively adjusted through the dynamic fusion (DF) mechanism in DSEM for achieving a finer segmentation. Comprehensive experimental results demonstrate that our method can achieve the state-of-the-art performance on the KITTI2015 and Cityscapes datasets, and also verify the efficiency when applied in salient object detection on NJU2K dataset. The source code will be available publicly after paper is accepted.

A Temporal Learning Approach to Inpainting Endoscopic Specularities and Its Effect on Image Correspondence

Video streams are utilised to guide minimally-invasive surgery and diagnosis in a wide range of procedures, and many computer-assisted techniques have been developed to automatically analyse them. These approaches can provide additional information to the surgeon such as lesion detection, instrument navigation, or anatomy 3D shape modelling. However, the necessary image features to recognise these patterns are not always reliably detected due to the presence of irregular light patterns such as specular highlight reflections. In this paper, we aim at removing specular highlights from endoscopic videos using machine learning. We propose using a temporal generative adversarial network (GAN) to inpaint the hidden anatomy under specularities, inferring its appearance spatially and from neighbouring frames, where they are not present in the same location. This is achieved using in-vivo data from gastric endoscopy (Hyper Kvasir) in a fully unsupervised manner that relies on the automatic detection of specular highlights. System evaluations show significant improvements to other methods through direct comparison and ablation studies that depict the importance of the network’s temporal and transfer learning components. The generalisability of our system to different surgical setups and procedures was also evaluated qualitatively on in-vivo data of gastric endoscopy and ex-vivo porcine data (SERV-CT, SCARED). We also assess the effect of our method in comparison to other methods on computer vision tasks that underpin 3D reconstruction and camera motion estimation, namely stereo disparity, optical flow, and sparse point feature matching. These are evaluated quantitatively and qualitatively and results show a positive effect of our specular inpainting method on these tasks in a novel comprehensive analysis. Our code and dataset are made available at https://github.com/endomapper/Endo-STTN.

Multilevel Inverse Patchmatch Network with Local and Global Refinement for Underwater Stereo Matching

Vision-based underwater autonomous systems play a significant role in marine exploration. Stereo matching is one of the most popular applications for vision-based underwater autonomous systems, which recovers the geometric information of underwater scenes via stereo disparity estimation. While stereo matching in the air has achieved great progress with the development of neural networks, it generalizes poorly to the underwater scenario due to the challenging underwater degradation. In this paper, we propose a novel Multilevel Inverse Patchmatch Network (MIPNet) to iteratively model pair-wise correlations under underwater degradation and estimate stereo disparity with both local and global refinements. Specifically, we first utilized the inverse Patchmatch module in a novel multilevel pyramid structure to recover the detailed stereo disparity from the input stereo images. Secondly, we introduced a powerful Attentional Feature Fusion module to model pair-wise correlations with global context, ensuring high-quality stereo disparity estimation for both in-air and underwater scenarios. We evaluate the proposed method on the popular real-world ETH3D benchmark, and the highly competitive performance against the popular baselines demonstrates the effectiveness of the proposed method. Moreover, with its superior performance on our real-world underwater dataset, e.g., our method outperforms the popular baseline RAFT-Stereo by 27.1%, we show the good generalization ability of our method to underwater scenarios. We finally discuss the potential challenges for underwater stereo matching via our experiments on the impact of water.

Real-Time Stereo Disparity Prediction Based on Patch-Embedded Extraction and Depthwise Hierarchical Refinement for 3-D Sensing of Autonomous Vehicles on Energy-Efficient Edge Computing Devices

Sensing and communications are dispensable for autonomous vehicles and IoT. One key task in autonomous driving is the sensing of 3D information surrounding a vehicle. Most existing stereo disparity prediction networks pursue accurate disparity maps on high-performance GPUs with high energy consumption. While very few stereo networks achieving real-time on resource-constrained edge devices are hardly satisfactory in terms of accuracy. To tackle this, we propose a lightweight and efficient three-stage stereo disparity prediction network named HRSNet for real-time stereo matching on energy-efficient edge devices with limited resource, reducing energy consumption and requirement on high-performance computing resources. The network consists of a serial patch-embedded feature extractor with short shortcut connections and a depthwise hierarchical refinement. In the stage of refining disparity maps, we employ a novel 2D aggregation network containing expanded depthwise separable convolutions with residual connections to regularize 3D group-wise cost volumes, resulting in accuracy improvement and inference acceleration. By evaluating on KITTI 2015, the proposed network achieves approaching results to state-of-the-art metrics on high-end GPU, and frame-rate@D1-all results of 64.6 FPS@6.16%, 41.7 FPS@3.49%, 32.2 FPS@2.29% in Stages 1-3 on Jetson Nano with ultra-low average power consumption of 6.3 W after tensorRT optimization.

DecoupledPoseNet: Cascade Decoupled Pose Learning for Unsupervised Camera Ego-Motion Estimation

Although many impressive works on learning-based camera ego-motion estimation methods have been proposed recently, most of them promote the accuracy of camera pose estimation by various sequential learning with loop closure optimization, while neglecting the improvement of PoseNet itself. In this paper, we focus on the coupling of rotation and translation in ego-motion estimation, and design a cascade decoupling structure to separately learn the rotation and translation of camera relative motion between adjacent frames. Meanwhile, a rigid-aware unsupervised learning framework with iterative pose refinement scheme is proposed for camera ego-motion estimation. It can disambiguate rigid motion and deformations in dynamic scenarios by jointly learning of optical flow, stereo disparity and camera pose. Validated with evaluation experiments on the public available datasets, our method is superior to the state-of-the-art unsupervised methods, and can achieve comparable results with the supervised ones.

Stereopsis for rapidly moving targets.

Stereoscopic depth perception is possible with luminance-defined target velocities at least as high as 600° s-1, up to the limit of 30 Hz imposed by the high-temporal frequency cut-off of the eye. The limitation for perceiving depth from stereo disparity of moving targets is not their velocity but the temporal frequency bandwidth of the eye, which is affected by adaption state. Stereoacuity for a depth shift in a horizontally moving grating depends not on spatial disparity between corresponding luminance points in spatial units of arc min, but on the spatial shift as a fixed proportion of the period of the grating, in other words, on the phase angle difference between the two eyes, as is also the case for obliquely orientated, stationary gratings. Phase differences explain not only the classic Pulfrich stereophenomenon but its equivalent with dynamic visual noise, and a new effect in which depth results from interocular phase differences in luminance modulation. This article is part of a discussion meeting issue 'New approaches to 3D vision'.

A Multifeature-Assisted Road and Vehicle Detection Method Based on Monocular Depth Estimation and Refined U-V Disparity Mapping

Usually, the detection process of traffic objects, such as vehicles, finds the visual input lacks the necessary depth information, so it is difficult to directly and quickly obtain the results. Typically, to separate the objects from the complex background, it is necessary to utilize a complex model or prior knowledge, which can be computationally expensive or simply infeasible. To battle this issue, depth visual information is used in this paper to accurately segment roads and vehicles, so it doesn’t need to use complex models to detect objects in the visual input. First, an unsupervised deep learning-based monocular depth estimation method is used to obtain the stereo disparity map. Then a non-parametric, refined U-V disparity mapping method is used to obtain the road region of interest. Next, this paper uses the road parallel scanning to determine the source and vanishing points and uses the adjacent disparity similarity algorithm to complement and extract the target region to detect roads and vehicles. This algorithm uses multi-feature fusion such as height-width ratio, perspective ratio, and area ratio to accurately segment the target region, and the effectiveness of the proposed method is tested on a public dataset. The experimental results show that the proposed model can accurately and efficiently detect roads and vehicles in a variety of scenarios.

SBIN: A stereo disparity estimation network using binary convolutions

Although the current advances on convolutional networks are outstanding, they mainly depend on extensive computational power, limiting the areas of applications. The latter applies for stereo disparity estimation, where current solutions can barely run on embedded devices. This work shows that it is possible to binarize an end-to-end stereo disparity network, which can be considered a step towards lightweight and potentially faster disparity estimation networks. This work shows the validity of the proposed approach through experimentation in two well-known datasets, sceneflow and kitti2012. The results show that a binary disparity model is possible but at the cost of performance. An EPE of 5.14 and 2.09 is achieved in sceneflow and kitti2012 accordingly.

Fast and Compact Image Segmentation Using Instance Stixels

State-of-the-art stixel methods fuse dense stereo disparity and semantic class information, e.g., from a Convolutional Neural Network (CNN), into a compact representation of driveable space, obstacles and background. However, they do not explicitly differentiate instances within the same semantic class. We investigate several ways to augment single-frame stixels with instance information, which can be extracted by a CNN from the RGB image input. As a result, our novel Instance Stixels method efficiently computes stixels that account for boundaries of individual objects, and represents instances as grouped stixels that express connectivity. Experiments on the Cityscapes dataset demonstrate that including instance information into the stixel computation itself, rather than as a post-processing step, increases the segmentation performance (i.e., Intersection over Union and Average Precision). This holds especially for overlapping objects of the same class. Furthermore, we show the superiority of our approach in terms of segmentation performance and computational efficiency compared to combining the separate outputs of Semantic Stixels and a state-of-the-art pixel-level CNN. We achieve processing throughput of 28 frames per second on average for 8 pixel wide stixels on images from the Cityscapes dataset at 1792 × 784 pixels. Our Instance Stixels software is made freely available for non-commercial research purposes.

Monocular contextual constraint for stereo matching with adaptive weights assignment

Matching-based stereo disparity estimation has difficulty in dealing with occlusion, weak and repetitive textures in binocular vision. By contrast, monocular vision, estimating depth from a single image, is not subject to these challenges. Inspired by this, in this study, we propose an adaptive co-learning framework with monocular and stereo branches named CLStereo to improve stereo performance. This framework introduces a monocular branch as contextual constraints to transfer the prior knowledge learned from the monocular branch to the stereo branch. An adaptive weights assignment is further proposed to balance the co-learning of both branches without mutually tuning. CLStereo can be seamlessly embedded into many existing deep stereo models to boost their performance, especially in occluded, weak, and repetitive texture areas. Extensive experiments demonstrate that we achieve the state-of-the-art performance on the Scene Flow dataset and improve deep stereo models by at least 4% on KITTI 2012 and 2015 benchmarks.

Binocular vision training for professional athletes

Optimal visual abilities including stereo acuity seem to be an important issue in sports. There is increasing evidence that stereo acuity can be sustainably improved by digital vision training even for people with good stereo acuity. In this study 31male and female tennis players (professionals, young professionals, coaches and former professionals) completed at least 6training units each with 192 dynamic stereoscopic tasks (N = 1152) within 6 weeks including a4-option testwith different levels of difficulty on a3D screen at a distance of 5 m. The parameter reaction time and correctness at 15-300 arcseconds was determined. For amore precise representation of the reaction time improvement as afunction of the difficulty level, the parameter reaction time increase per stereo disparity reduction (ReST) was defined. Reaction time to 15 arcsecond stimuli significantly decreased from 3.9 s to 1.6 s (59%) as aresult of digital vision training. The correctness at 30 arcsecond stimuli significantly increased by 23%. The observed improvement in reaction time during vision training did not result in decreasing correctness when answering the visual questions. This represents an overall improvement in stereo vision. Dynamic visual training over 6 weeks improves stereoscopic performance including stereo acuity, response time and correctness.

Three-Stream 3D deep CNN for no-Reference stereoscopic video quality assessment

• An end-to-end learning structure for the Stereoscopic Video Quality Assessment with three pipelines to extract the quality degradation from spatial, temporal, and disparity information. • A non-homogeneous data augmentation system that splits the stereoscopic video into small cube patches and removes the outliers from the created dataset using entropy criteria. • A robust deep depth quality-aware feature extractor used to extract motion and depth maps jointly and feed them synchronously to the deep network to extract quality discriminatory features. • Two popular datasets are used to validate the proposed learning architecture for Stereoscopic Video Quality Assessment and received state-of-the-art results in both datasets. Convolutional Neural Networks (CNNs) have achieved great success in learning computer vision tasks, particularly 3D CNNs, for extracting spatio-temporal features from the given videos. However, 3D CNNs have not been well-examined for the Stereoscopic Video Quality Assessment (SVQA). To our best knowledge, most of the state-of-the-art methods used the traditional hand-crafted feature extraction methods for the SVQA. Very few methods used the power of deep learning for SVQA, and they just considered the spatial information, ignoring the disparity and motion information. In this paper, we propose a No-Reference (NR) deep 3D CNN architecture that jointly focuses on spatial, disparity, and temporal information between consecutive frames. A 3-Stream 3D CNN, shortly 3S-3DCNN , by performing 3D CNNs, extracts features from spatial, motion, and depth channels to estimate the stereo video’s quality. It captures the degradations in the quality of the stereoscopic video in multiple dimensions. Firstly, the scene flow, which is the joint prediction of the optical flow and stereo disparity, is calculated. Then, the spatial information, optical flow, and disparity map of a given video are used as input to the 3S-3DCNN model. The extracted features are concatenated and utilized as inputs to the fully connected layers for doing the regression. We split the input videos into cube patches for data augmentation and remove the cubes that confuse our model from the training and testing sets. Two standard stereoscopic video quality assessment benchmarks of LFOVIAS3DPh2 and NAMA3DS1-COSPAD1 were used to evaluate our method. Experimental results show that our 3S-3DCNN method’s objective score significantly correlates with the subjective SVQ scores in multiple video datasets. The RMSE for NAMA3DS1-COSPAD1 dataset is 0.2757, which outperforms other methods by a large margin. The SROCC value for the blur distortion of the LFOVIAS3DPh2 dataset is more than 98%, indicating that the 3S-3DCNN is consistent with human visual perception.

Learning stereo disparity with feature consistency and confidence

Learning stereo disparity with feature consistency and confidence

Learning stereo disparity with feature consistency and confidence

Learning stereo disparity with feature consistency and confidence

A unified model for binocular fusion and depth perception: Stereo spatial resolution and disparity variance limits

A unified model for binocular fusion and depth perception: Stereo spatial resolution and disparity variance limits

Stereo disparity optimization with depth change constraint based on a continuous video

Three-dimensional reconstruction based on stereo vision technology is an important research direction in the field of computer vision, and has a wide range of applications in industrial measurement, medical image reconstruction, cultural relic preservation, robot navigation, virtual reality and other fields. However, the three-dimensional reconstruction of moving objects usually has poor accuracy, low efficiency and poor visualization effect due to the image noise, motion blur, complex and time-consuming calculation etc. In this article, a disparity optimization method based on depth change constraint is proposed, which utilizes the correlation of the adjacent frames in the continuous video sequence to eliminate mismatches and correct the wrong disparity values by introducing a depth change constraint threshold. The experiments on the video images which are taken by a binocular stereo vision system demonstrate that our method of removing incorrect matches bears satisfactory results and it can greatly improve the effect of the three-dimensional reconstruction of the moving objects.

An Improved Sum of Squared Difference Algorithm for Automated Distance Measurement

In recent years, with improvement of photoelectric conversion efficiency and accuracy, photoelectric sensor was arranged to simulate binocular stereo vision for 3D measurement, and it has become an important distance measurement method. In this paper, an improved sum of squared difference (SSD) algorithm which can use binocular cameras to measure distance of vehicle ahead was proposed. Firstly, consistency matching calibration was performed when images were acquired. Then, Gaussian blur was used to smooth the image, and grayscale transformation was performed. Next, the Sobel operator was used to detect the edge of images. Finally, the improved SSD was used for stereo matching and disparity calculation, and the distance value could be obtained corresponding to each point. Experimental results showed that the improved SSD algorithm had an accuracy rate of 95.06% when stereo matching and disparity calculation were performed. This algorithm fully meets the requirements of distance measurement.