Binocular Stereo Matching Research Articles

Binocular Vision-Based Pole-Shaped Obstacle Detection and Ranging Study

(1) Background: In real road scenarios, various complex environmental conditions may occur, including bright lights, nighttime, rain, and snow. In such a complex environment for detecting pole-shaped obstacles, it is easy to lose the feature information. A high rate of leakage detection, false positives, and measurement errors are generated as a result. (2) Methods: The first part of this paper utilizes the improved YOLOv5 algorithm to detect and classify pole-shaped obstacles. Then, the identified target frame information is combined with binocular stereo matching to obtain more accurate distance information. (3) Results: The experimental results demonstrate that this method achieves a mean average precision (mAP) of 97.4% for detecting pole-shaped obstacles, which is 3.1% higher than the original model. The image inference time is only 1.6 ms, which is 1.8 ms faster than the original algorithm. Additionally, the model size is only 19.0 MB. Furthermore, the range error of this system is less than 7% within the range of 3–15 m. (4) Conclusions: Therefore, the algorithm not only achieves real-time and accurate identification and classification but also ensures precise measurement within a specific range. Meanwhile, the model is lightweight and better suited for deploying sensing systems.

UAMD-Net: A Unified Adaptive Multimodal Neural Network for Dense Depth Completion

Depth estimation is a critical problem in robotics applications especially autonomous driving. Currently, depth prediction based on binocular stereo matching and depth completion based on fusion of monocular image and laser point cloud are two mainstream methods. However, the former usually suffers from lack of constraint while building cost volume, and the latter could not be trained in self-supervised way and haven’t utilized the geometric constraint of stereo matching, which we think will further improve the performance. Therefore, we propose a novel multimodal neural network, namely UAMD-Net, for dense depth completion based on fusion of binocular stereo matching and the weak constraint from the sparse point clouds. Specifically, the sparse point clouds are converted to sparse depth map and filled to the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">multimodal feature encoder</i> (MFE) with binocular image, constructing a cross-modal cost volume. Then, it will be further processed by the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">multimodal feature aggregator</i> (MFA) and the depth regression layer. Furthermore, since previous multimodal depth estimation methods ignore the problem of modality dependence, we propose a new training strategy called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">random modality dropout</i> (RMD) which enables the network to be adaptively trained with multiple modality inputs and inference with specific modality inputs. Benefiting from the flexible network structure and adaptive training method, our proposed network can realize unified training under various modality input conditions. Comprehensive experiments conducted on KITTI and DrivingStereo depth completion datasets demonstrate that our method produces robust results and outperforms other state-of-the-art methods.

TOF-assisted binocular vision accuracy improvement method for underwater fish size inspection.

To address problems such as the lack of accuracy in acquiring depth maps for dynamic fish 3D measurements by usual binocular vision or a time-of-flight (TOF) depth camera, a TOF-assisted binocular vision depth acquisition algorithm is used to obtain high-quality depth maps. The TOF depth energy function is designed to guide the binocular stereo matching process, which improves the correct matching rate of binocular matching in low-texture regions; the TOF and binocular stereo matching confidence weighting functions are designed to achieve the fusion of the two at pixel level to improve the matching quality of fish in the occluded overlapping regions. The experimental results show that the TOF-assisted binocular vision system improves the accuracy of fish size measurement compared to single binocular vision while reducing the measurement error when the fish body has a significant inclination along the depth axis.

Depth Information Precise Completion-GAN: A Precisely Guided Method for Completing Ill Regions in Depth Maps

In the depth map obtained through binocular stereo matching, there are many ill regions due to reasons such as lighting or occlusion. These ill regions cannot be accurately obtained due to the lack of information required for matching. Since the completion model based on Gan generates random results, it cannot accurately complete the depth map. Therefore, it is necessary to accurately complete the depth map according to reality. To address this issue, this paper proposes a depth information precise completion GAN (DIPC-GAN) that effectively uses the Guid layer normalization (GuidLN) module to guide the model for precise completion by utilizing depth edges. GuidLN flexibly adjusts the weights of the guiding conditions based on intermediate results, allowing modules to accurately and effectively incorporate the guiding information. The model employs multiscale discriminators to discriminate results of different resolutions at different generator stages, enhancing the generator’s grasp of overall image and detail information. Additionally, this paper proposes Attention-ResBlock, which enables all ResBlocks in each task module of the GAN-based multitask model to focus on their own task by sharing a mask. Even when the ill regions are large, the model can effectively complement the missing details in these regions. Additionally, the multiscale discriminator in the model enhances the generator’s robustness. Finally, the proposed task-specific residual module can effectively focus different subnetworks of a multitask model on their respective tasks. The model has shown good repair results on datasets, including artificial, real, and remote sensing images. The final experimental results showed that the model’s REL and RMSE decreased by 9.3% and 9.7%, respectively, compared to RDFGan.

Multi-Scale Aggregation Stereo Matching Network Based on Dense Grouping Atrous Convolution

The key to image depth estimation is to accurately find corresponding points between the left and right images. A binocular camera can directly estimate the depth of the left and right image range, which completely avoids the requirement of target recognition accuracy in a monocular depth estimation. However, it is difficult for binocular stereo matching to accurately segment objects and find matching points in the ill-posed areas (weak texture, deformation, object edge, etc.) of the left and right images. In the semantic segmentation task, atrous convolution is used to solve the contradiction between the receptive field and the segmentation accuracy. This research focused on balancing the impact of holes on the segmentation task. In addition, in order to solve the issue where matching points in the ill-posed regions of left and right images are affected by noise, we used a 3D convolution to aggregate the cost volume to obtain better accuracy. However, the 3D convolution method is prone to mismatching in ill-posed areas of the image. To tackle the problems above, we proposed a dense grouping atrous convolution spatial pyramid pooling (DenseGASPP) method. The feature of the DenseGASPP method is that there is a dense connection between the group atrous convolutions to fully integrate feature information. This method can expand the receptive field and balance the effect of holes on the segmentation task. Moreover, we introduced multi-scale cost aggregation into our method, which uses the repeated exchange of information between cost volumes of different scales to obtain rich contextual information and reduce the mismatching of the network. To evaluate the performance of our method, we conducted several groups of typical algorithm experiments on the scene flow and KITTI 2015 standard datasets. From the results, our model achieves better performance, reducing the EPE from 1.09 to 0.67, which improves the mismatching ability of binocular depth estimation algorithm in ill-posed regions.

An improved binocular stereo matching algorithm based on AANet

An improved binocular stereo matching algorithm based on AANet

High-precision single-pixel 3D calibration method using pseudo-phase matching.

Compressive sensing makes it possible to explore two-dimensional spatial information using a single-point detector. However, the reconstruction of the three-dimensional (3D) morphology using a single-point sensor is largely limited by the calibration. Here we demonstrate a pseudo-single-pixel camera calibration (PSPC) method using pseudo phase matching in stereo, which can perform 3D calibration of low-resolution images with the help of a high-resolution digital micromirror device (DMD) in the system. In this paper, we use a high-resolution CMOS to pre-image the DMD surface and successfully calibrate the spatial position of a single-point detector and the projector with the support of binocular stereo matching. Our system achieved sub-millimeter reconstructions of spheres, steps, and plaster portraits at low compression ratios with a high-speed digital light projector (DLP) and a highly sensitive single-point detector.

An Algorithm for Single View Occlusion Area Detection in Binocular Stereo Matching

Occlusion area detection is a crucial step affecting the performance of the binocular stereo matching algorithm, but the traditional method of occlusion area detection has two major problems, including left–right consistency detection (LRC). First, these algorithms must obtain the left and right disparity maps with precision. Second, these algorithms cannot detect the occlusion region at the image’s borders. We propose the single view occlusion area detective (SVOAD) algorithm to detect these occlusion areas and better deal with them. The SVOAD can detect the area of occlusion from a single image, thereby reducing the computational cost. Additionally, the algorithm can detect the occlusion region in all image regions. This paper also improves the guided filter so that it works better with the end-to-end neural network and makes the SVOAD algorithm work better.

Design of laser scanning binocular stereo vision imaging system and target measurement

Binocular stereo vision is an important scheme of road environment perception for intelligent driving, because it can provide image information and spatial three-dimensional position information. However, binocular stereo matching is often disturbed by the environment, and noise, shadow, light transformation, and so on, will all cause mismatching, therefore it is impossible to carry out accurate 3D reconstruction. In this paper, an active imaging perception and target measurement method combining laser scanning and binocular vision is proposed. The binocular vision structure is used to simulate the human eye's observation of the environment in front of the vehicle. At the same time, through the laser lines scanned to the target, the strong characteristics of binocular matching are provided to enhance the anti-interference of binocular matching, and calculate the three-dimensional point cloud data of the front object according to the binocular stereo vision theory. Firstly, this paper studies the influence of the parameters changing of the system on the accuracy of object depth value in indoor environment, and then selects the most suitable system parameters to carry out three-dimensional reconstruction experiment on the actual target object in indoor environment. Finally, the scene experiment in outdoor environment is carried out. In the outdoor environment, the red spectrum laser equipped in the system can still effectively reconstruct the target surface radian, but there are some limitations due to the limitation of outdoor light intensity. This paper preliminarily summarizes the current structural design and experiment, and further discusses the follow-up research and improvement ideas. Research and experiments show that the laser scanning binocular stereo vision imaging system proposed in this paper has the function of forward target ranging, especially under some bad imaging environment, this method can also effectively match the binocular images, and present the three-dimensional information of the internal details of the target. Under the indoor experimental environment, the minimum relative error is 1.42 % for the target with a distance of about 1.5 m, and in the evening outdoor environment, the minimum the minimum relative error is 0.87 % for the target with a distance of about 3 m. Due to the advantages of laser line intensity feature matching, it can accelerate the speed of binocular matching, and has potential application prospects in autonomous system.

4D dual-mode staring hyperspectral-depth imager for simultaneous spectral sensing and surface shape measurement.

A 4D dual-mode staring hyperspectral-depth imager (DSHI), which acquire reflectance spectra, fluorescence spectra, and 3D structural information by combining a staring hyperspectral scanner and a binocular line laser stereo vision system, is introduced. A 405 nm laser line generated by a focal laser line generation module is used for both fluorescence excitation and binocular stereo matching of the irradiated line region. Under the configuration, the two kinds of hyperspectral data collected by the hyperspectral scanner can be merged into the corresponding points in the 3D model, forming a dual-mode 4D model. The DSHI shows excellent performance with spectral resolution of 3 nm, depth accuracy of 26.2 µm. Sample experiments on a fluorescent figurine, real and plastic sunflowers and a clam are presented to demonstrate system's with potential within a broad range of applications such as, e.g., digital documentation, plant phenotyping, and biological analysis.

UAV Landing Aid Hexapod Robot based on ArUco Marker and Sparse Optical Flow

A UAV (Unmanned Aerial Vehicle) landing robot system based on a six-degree-of-freedom hexapod and computer vision with ArUco marker and sparse optical flow is implemented in this paper. By using binocular camera measuring the real-time position and orientation of UAV, a dynamic target tracking method based on ArUco marker and sparse optical flow is proposed. In this method, the ArUco marker is combined with sparse optical flow to improve the accuracy of target detection and the matching error of binocular stereo matching. The binocular vision system with ArUco marker is used to measure and calculate the relative coordinates of markers in the camera coordinate system. Then the UAV contour fitting is conducted in order to obtain the real-time position and orientation of this mobile target. The effectiveness of the novel UAV landing hexapod robot system based on ArUco marker combined with sparse optical flow is verified through the UAV capture experiment.

The stereo matching algorithm based on an improved adaptive support window

In binocular stereo matching, there has been a problem of low matching accuracy and noise immunity in discontinuous regions and weak-textured regions. This paper proposes a stereo matching algorithm based on an improved adaptive support window. In the cost computation stage, first, according to the preset arm length and colour threshold, a cross-based arm is obtained, which centres on the pixel to be matched; then the adaptive regions of the vertical arm and the horizontal arm are constructed respectively, which have different shape and size. Finally, the union of the two adaptive regions is used as the final support window of Census transform. Performance evaluations on Middlebury stereo data sets demonstrate that the proposed algorithm outperforms other seven most challenging stereo matching algorithms. The mismatching rate of this algorithm is greatly reduced, and the anti-noise performance is also improved considerably. Because the construction of the adaptive region is based on strict criteria and comprehensive consideration, the algorithm proposed in the paper can improve the matching accuracy in the weak-textured regions and discontinuous disparity regions.

Three-dimensional face point cloud hole-filling algorithm based on binocular stereo matching and a B-spline

Three-dimensional face point cloud hole-filling algorithm based on binocular stereo matching and a B-spline

Research on recognition of painted faces

In the field of public safety surveillance, suspects may disguise themselves by painting their faces, challenging the face recognition system. The well-developed ordinary, two-dimensional image face recognition methods rely mainly on the texture features of the face images, which leaves the traditional face recognition system relatively vulnerable to the camouflage attacks. We set our sights on three-dimensional (3D) alternatives. A 3D facial recognition method is proposed which extracts 3D features of each individual that are not related to face texture. Considering actual requirements, we applied binocular stereo matching to obtain 3D face point clouds. Hereafter, we included a spatial representation for classification and recognition of 3D face, based on which we then adopted a multicascade classifier. By reconstructing, extracting features, and identifying multiple 3D faces, comparison experiment results demonstrate that our proposed method discerned 3D faces correctly, defeated the face camouflage attacks effectively, and showed promising application prospects for public safety surveillance.

HCVNet: Binocular Stereo Matching via Hybrid Cost Volume Computation Module With Attention

Binocular stereo matching, a computer vision task typically using cost volume constructed from the left and right feature maps to estimate disparity and depth, is widely applied in 3D reconstruction, autonomous driving and robotics navigation. Though recent study brings an awareness of the convolution neural networks and the attention algorithms used in this field can make great progress, it is still difficult to satisfy the demand of high-precision applications due to many reasons. Study finds that the exist methods usually incline to ignore the intermediate feature map of other scales, pay less attention to the relationship between left and right feature maps and even just tend to use one type of cost volume to train the model. In this article, we mainly focus on solving the three problems mentioned above. Firstly, we present the Multi-scale Feature Extraction and Fusion Module(MFEFM) to get the informational feature maps via fusing all scale feature maps. And then we design the Effective Channel Attention Module(ECAM) applied to better capture and utilize the channel-wise independencies. Finally, we adopt the Hybrid Cost Volume Computation Module(HCVCM) to construct and aggregate cost volume. With these solutions, we build an end-to-end stereo matching network named HCVNet. Comparison with other state-of-the-art models, it can achieve 0.714px EPE on SceneFlow dataset, descending PSMNet(1.09px EPE) by 37.6%.

Improvement of AnyNet-based end-to-end phased binocular stereo matching network

To improve the accuracy of the parallax maps obtained by the binocular stereo matching network for better 3D reconstruction, this thesis is based on the AnyNet network, with the corresponding optimization to get MyNet. The Hourglass structure allows for the acquisition of multiscale image information. The original 3D Conv is replaced by a 3D residual block in the parallax network to avoid the gradient disappearance or explosion and model degradation problems of traditional convolutional neural network models, thus further improving the cost volume obtained. The experiments were tested on the KITTI2012 and KITTI2015 datasets and showed some improvement compared to the original algorithm. The experimental results show that the accuracy of the parallax map has improved significantly with almost the same time spent.

Stereo matching algorithm based on improved Census transform and texture filtering

To solve the problem that existing binocular stereo matching algorithms have low matching accuracy in discontinuous disparity and low texture area, a stereo matching algorithm based on Census transform and texture filtering is proposed. The weighted Census transform circular template is used to carry out the matching cost, which reflects the influence of the distance between neighborhood pixels and target pixels on the calculation and expands the perception range of target pixels; the texture filtering method is used for cost aggregation, which highlights the image structure information and smooths the internal texture. The experimental results show that the stereo matching algorithm proposed in this paper can effectively reduce the mismatching rate of images, the disparity map obtained has less noise, and the matching effect is better when the pattern texture is relatively dense.

Depth Estimation from Light Field Geometry Using Convolutional Neural Networks.

Depth estimation based on light field imaging is a new methodology that has succeeded the traditional binocular stereo matching and depth from monocular images. Significant progress has been made in light-field depth estimation. Nevertheless, the balance between computational time and the accuracy of depth estimation is still worth exploring. The geometry in light field imaging is the basis of depth estimation, and the abundant light-field data provides convenience for applying deep learning algorithms. The Epipolar Plane Image (EPI) generated from the light-field data has a line texture containing geometric information. The slope of the line is proportional to the depth of the corresponding object. Considering the light field depth estimation as a spatial density prediction task, we design a convolutional neural network (ESTNet) to estimate the accurate depth quickly. Inspired by the strong image feature extraction ability of convolutional neural networks, especially for texture images, we propose to generate EPI synthetic images from light field data as the input of ESTNet to improve the effect of feature extraction and depth estimation. The architecture of ESTNet is characterized by three input streams, encoding-decoding structure, and skipconnections. The three input streams receive horizontal EPI synthetic image (EPIh), vertical EPI synthetic image (EPIv), and central view image (CV), respectively. EPIh and EPIv contain rich texture and depth cues, while CV provides pixel position association information. ESTNet consists of two stages: encoding and decoding. The encoding stage includes several convolution modules, and correspondingly, the decoding stage embodies some transposed convolution modules. In addition to the forward propagation of the network ESTNet, some skip-connections are added between the convolution module and the corresponding transposed convolution module to fuse the shallow local and deep semantic features. ESTNet is trained on one part of a synthetic light-field dataset and then tested on another part of the synthetic light-field dataset and real light-field dataset. Ablation experiments show that our ESTNet structure is reasonable. Experiments on the synthetic light-field dataset and real light-field dataset show that our ESTNet can balance the accuracy of depth estimation and computational time.

A Fast Stereo Matching Network with Multi-Cross Attention

Stereo matching networks based on deep learning are widely developed and can obtain excellent disparity estimation. We present a new end-to-end fast deep learning stereo matching network in this work that aims to determine the corresponding disparity from two stereo image pairs. We extract the characteristics of the low-resolution feature images using the stacked hourglass structure feature extractor and build a multi-level detailed cost volume. We also use the edge of the left image to guide disparity optimization and sub-sample with the low-resolution data, ensuring excellent accuracy and speed at the same time. Furthermore, we design a multi-cross attention model for binocular stereo matching to improve the matching accuracy and achieve end-to-end disparity regression effectively. We evaluate our network on Scene Flow, KITTI2012, and KITTI2015 datasets, and the experimental results show that the speed and accuracy of our method are excellent.

Binocular stereo matching of real scenes based on a convolutional neural network and computer graphics.

The binocular stereo matching method based on deep learning has limited cross-domain generalization ability, and it is a tricky problem to obtain a large amount of data from real scenes for training. The most advanced stereo matching network is difficult to apply to new real scenes. In this paper, we propose a real-scene stereo matching method based on a convolutional neural network and computer graphics. A virtual binocular imaging system is constructed by introducing graphics software, and a high-quality semi-synthetic dataset close to the texture characteristics of the real scene is constructed for training the network. A feature standardization layer is embedded in the feature extraction module of the proposed network to further reduce the feature space difference between semi-synthetic data and real scene data. Three small 4D cost volumes are constructed to replace one large 4D cost volume, which reduces GPU memory consumption and improves the matching performance of the network. The experimental results show that compared with the traditional stereo matching method, the matching accuracy of the proposed method is significantly improved by about 60%. Compared with other learning-based methods, the matching accuracy is increased by about 30%, the matching speed is increased by 38%, and it has good robustness to the interference of defocus blur and Gaussian noise.