Stereo Vision Research Articles

Vision-based environmental perception for autonomous driving

Visual perception plays an important role in autonomous driving. One of the primary tasks is object detection and identification. Since the vision sensor is rich in color and texture information, it can quickly and accurately identify various road information. The commonly used technique is based on extracting and calculating various features of the image. The recent development of deep learning-based method has better reliability and processing speed and has a greater advantage in recognizing complex elements. For depth estimation, vision sensor is also used for ranging due to their small size and low cost. Monocular camera uses image data from a single viewpoint as input to estimate object depth. In contrast, stereo vision is based on parallax and matching feature points of different views, and the application of Deep learning also further improves the accuracy. In addition, Simultaneous Location and Mapping (SLAM) can establish a model of the road environment, thus helping the vehicle perceive the surrounding environment and complete the tasks. In this paper, we introduce and compare various methods of object detection and identification, then explain the development of depth estimation and compare various methods based on monocular, stereo, and RGB-D sensors, next review and compare various methods of SLAM, and finally summarize the current problems and present the future development trends of vision technologies.

A system and method for person identification and positioning incorporating object edge detection and scale-invariant feature transformation

During the overhead crane, due to the complex background and large image size, the direct processing algorithm usually leads to time-consuming operation and difficult to achieve accurate personnel positioning. To address these issues, we propose the feature points recognition and matching algorithm incorporating object edge detection and scale-invariant feature transform (SIFT), which is applied to the designed object detection and positioning algorithm for binocular stereo vision. The proposed method applies the Canny operator to extract contours from the locally cropped regions obtained using YOLOv5. The contour points are used as the key points of SIFT, and corresponding feature points are created descriptors, and feature points matching are performed. Experimental results indicate that the proposed method can filter feature points successfully. Moreover, the binocular stereo vision system with the binocular positioning algorithm designed has good performance, with the overall depth direction positioning error within 5% in a range of 12 m.

Electroencephalography recognition based on encephalic region and temporal sequence transformer

Stereoscopic vision is the key to good motor control and accurate cognition, and its formation is closely related to brain control. The early methods of measuring stereoscopic vision rely on the subject’s judgment, which might be influenced by inadvertent misjudgments. To solve this problem, we collected the Electroencephalography (EEG) of subjects watching dynamic random dot stereogram for stereogram recognition. To analyze stereogram EEG signals, this paper proposed a transformer-based encephalic region temporal sequence analysis network. Inspired by the concept of brain regions, this network designs an encephalic region Transformer module to capture global spatial features in each brain region and among the whole brain regions. Based on the spatial features of electrodes in different brain regions, the global spatial dependence of all electrodes can be further obtained. Then, the temporal sequence Transformer module is adopted to learn the global temporal EEG features. Finally, we utilize the spatial-temporal multi-scale convolution module to extract advanced spatial and temporal fusion features for recognition. The simulation results on two public EEG datasets illustrate the excellent classification performance of the proposed model, which is better than 9 existing comparison models in EEG recognition.

GelStereo Palm: A Novel Curved Visuotactile Sensor for 3-D Geometry Sensing

Recently, visuotactile sensors have shown promising potential in robotics due to their high-resolution sensing ability. Unfortunately, the majority of available visuotactile sensors are limited to flat shapes, which severely limits their application possibilities. In this paper, we propose a novel curved visuotactile sensor, the GelStereo Palm, which senses the 3D contact geometry on a curved surface using a binocular vision system. Meanwhile, to solve the light refraction problem in the binocular stereo vision system under a curved medium, a Refractive Stereo Ray Tracing model for GelStereo Palm (GP-RSRT) is presented. Moreover, a 3D tactile point cloud sensing pipeline is introduced to reconstruct the 3D contact geometry in real-time. Finally, extensive experiments are conducted to verify the accuracy and robustness of the 3D contact geometry sensing of our GelStereo Palm sensor.

A target spatial location method for fuze detonation point based on deep learning and sensor fusion

The spatial location of fuze detonation point is crucial for evaluating the working condition and improving the performance of fuze. Considering the observation safety, the non-contact long-distance accurate measurement technology is essential. In this paper, we propose a method that takes the sensor data of optics, spatial attitude and GPS as input and outputs the spatial position of the fuze detonation point. The proposed method consists of two steps. First, an object detection algorithm with post-processing algorithm is proposed to obtain rich information of the target. The algorithm achieves high-accuracy detection by introducing powerful backbone, attention mechanism, group convolution, and improved multi-scale feature fusion. Second, a Variational Auto-Encoder (VAE) algorithm model improved by dense connection structure and multiple source heterogeneous sensor information fusion structure is proposed as the position regression algorithm. It receives the status information of the observer camera and the output of the object detection algorithm, and then outputs the three-dimensional coordinates of the explosion point. Finally, method validation and performance analysis are realized through virtual scene simulation. Experiment results show the superiority of the proposed object detection algorithm over other typical algorithms on explosion flare detection, with its Average Precision (AP) of 0.889. The positioning error of the spatial location method is 0.896 m, while that of the binocular stereo vision method is 2.863 m. Therefore, the proposed target spatial location method is proved to be effective and accurate.

Train Distance Estimation in Turnout Area Based on Monocular Vision.

Train distance estimation in a turnout area is an important task for the autonomous driving of urban railway transit, since this function can assist trains in sensing the positions of other trains within the turnout area and prevent potential collision accidents. However, because of large incident angles on object surfaces and far distances, Lidar or stereo vision cannot provide satisfactory precision for such scenarios. In this paper, we propose a method for train distance estimation in a turnout area based on monocular vision: firstly, the side windows of trains in turnout areas are detected by instance segmentation based on YOLOv8; secondly, the vertical directions, the upper edges and lower edges of side windows of the train are extracted by feature extraction; finally, the distance to the target train is calculated with an appropriated pinhole camera model. The proposed method is validated by practical data captured from Hong Kong Metro Tsuen Wan Line. A dataset of 2477 images is built to train the instance segmentation neural network, and the network is able to attain an MIoU of 92.43% and a MPA of 97.47% for segmentation. The accuracy of train distance estimation is then evaluated in four typical turnout area scenarios with ground truth data from on-board Lidar. The experiment results indicate that the proposed method achieves a mean RMSE of 0.9523 m for train distance estimation in four typical turnout area scenarios, which is sufficient for determining the occupancy of crossover in turnout areas.

Calibration method for monocular laser speckle projection system.

This paper proposes a novel calibration method for the monocular laser speckle projection system. By capturing images of a calibration board with speckles under different poses, projector's optical axis is fitted and utilized to calibrate the rotation between the camera and projector. The translation is solved in closed form subsequently and projector's virtual image is recovered via homography. After calibration, the system can be regarded and operated as a binocular stereo vision system with speckle pattern. The proposed method is efficient and convenient, without need of reference image or high-precision auxiliary equipment. Validated by experiments on Astra-s and Astra-pro, it presents significant improvement in depth-estimation compared to the traditional method.

Three-dimensional reconstruction of mobile binocular stereo vision based on push-broom line structured light for a workpiece surface

Stereo vision technology based on line structured light can effectively solve the problem of a three-dimensional (3D) reconstruction of a smooth surface. A method for 3D reconstruction of mobile binocular stereo vision based on push-broom line structured light for a workpiece surface is presented in this paper. The subpixel coordinates of the light strip centers of the line structured light are obtained by the Steger algorithm while the binocular module moves along the guide rail, and the polar constraint is used to achieve the matching of the extracted light strip centers. As a result, the 3D coordinates of the light strip centers in each location can be calculated because of the known interior and external parameters of the binocular module. To obtain the 3D point cloud data of the entire surface, a relative pose optimization method with respect to the initial frame is proposed, which accurately estimates the pose of the cameras in each location with respect to that in the initial location and unifies the 3D coordinates of the light strip centers in each location to the datum coordinates. The relative pose optimization method first estimates the rough values by using the direct linear transform method, and then iteratively calculates the refined solutions based on the principle of minimizing the re-projection errors. Simulation data and substantial experimental results validate the effectiveness of our method. Our method is compared to the direct linear transform method and the frame-by-frame transfer method, and the root mean square error (RMSE) of the distance from 3D point cloud to fitted plane is used to evaluate the 3D reconstruction accuracy. The repeatability experiment shows that the RMSE from our method is as low as 0.83 mm.

Tomato flower pollination features recognition based on binocular gray value-deformation coupled template matching

It is necessary to recognize the tomato pollination features for the designing demand of intelligent and precise tomato supplementary pollination equipment. Mentioned pollination features include flower opening state and the three-dimensional position and pose of flower anther. Tomato flower pollination features recognition method is designed in this paper based on the deep learning full opened flower recognition model and binocular template matching three-dimensional information recognition method. First of all, a binocular stereo vision system is built to acquire the tomato flower images in greenhouse with natural lighting. Which can help vision system avoid the impact of inconsistent light intensity on image recognition. The acquired images were equalized with three groups parameters to labeled images. And the improved MC-AlexNet deep learning model is established to recognize the full opened tomato flowers in the left image of image pair acquired with binocular vision system. Then, the template is created with the full opened flower recognition result of deep learning model in left image based on gray value and deformation template matching method. And the template matching is established to recognize the corresponding full opened flowers in the right image of image pair. Finally, with the template matching result, the anther segmentation is conducted to calculate three-dimensional position and pose of anther. The experimental results show that the accuracy of full opened tomato flowers recognition model is 96.23%. The average position recognition deviation of anther is 5.94 mm. And the average anther pose recognition deviation in three plane is 6.24° for the images that anthers can be observed. And the average time consuming is about 143.18 ms per image pair. It can be concluded that the method of binocular template matching established in this paper can fulfill the demand of supplementary pollination equipment design, and the research result lays a foundation for designing and improvement of tomato supplementary pollination equipment in greenhouse.

Fire Detection and Geo-Localization Using UAV’s Aerial Images and Yolo-Based Models

The past decade has witnessed a growing demand for drone-based fire detection systems, driven by escalating concerns about wildfires exacerbated by climate change, as corroborated by environmental studies. However, deploying existing drone-based fire detection systems in real-world operational conditions poses practical challenges, notably the intricate and unstructured environments and the dynamic nature of UAV-mounted cameras, often leading to false alarms and inaccurate detections. In this paper, we describe a two-stage framework for fire detection and geo-localization. The key features of the proposed work included the compilation of a large dataset from several sources to capture various visual contexts related to fire scenes. The bounding boxes of the regions of interest were labeled using three target levels, namely fire, non-fire, and smoke. The second feature was the investigation of YOLO models to undertake the detection and localization tasks. YOLO-NAS was retained as the best performing model using the compiled dataset with an average mAP50 of 0.71 and an F1_score of 0.68. Additionally, a fire localization scheme based on stereo vision was introduced, and the hardware implementation was executed on a drone equipped with a Pixhawk microcontroller. The test results were very promising and showed the ability of the proposed approach to contribute to a comprehensive and effective fire detection system.

Single-stripe-enhanced spacetime stereo reconstruction for concrete defect identification

Few studies explored the 3D reconstruction of concrete defects, and existing methods often suffer from noise interference or impracticality in real-world applications. This study proposes a novel single-stripe-enhanced spacetime stereo reconstruction method that combines a laser liner for structured light generation and a stereo vision camera. This method analyses the morphology and intensity variation of laser stripes from images captured from single scans, achieving accurate defect reconstructions. It is designed for handheld scenarios with image re-alignment eliminating the vibration caused by handheld operation. The morphology of laser stripes at crack sections is analysed for robust fine crack reconstruction. Active frame searching is introduced to improve the calculation efficiency, enabling spacetime stereo to operate over hundreds of frames. A comparative study of the proposed method and two commonly-used stereo matching methods demonstrates its ability to mitigate the impact of handheld operation and generate 3D reconstructions that highlight potential crack regions.

How Do Background and Remote User Representations Affect Social Telepresence in Remote Collaboration?: A Study with Portal Display, a Head Pose-Responsive Video Teleconferencing System

This study presents Portal Display, a screen-based telepresence system that mediates the interaction between two distinct spaces, each using a single display system. The system synchronizes the users’ viewpoint with their head position and orientation to provide stereoscopic vision through this single monitor. This research evaluates the impact of graphically rendered and video-streamed backgrounds and remote user representations on social telepresence, usability, and concentration during conversations and collaborative tasks. Our results indicate that the type of background has a negligible impact on these metrics. However, point cloud streaming of remote users significantly improves social telepresence, usability, and concentration compared with graphical avatars. This study implies that Portal Display can operate more efficiently by substituting the background with graphical rendering and focusing on higher-resolution 3D point cloud streaming for narrower regions for remote user representations. This configuration may be especially advantageous for applications where the remote user’s background is not essential to the task, potentially enhancing social telepresence.

Effect of visual perception training on binocular visual function reconstruction in patients after strabismus surgery.

To explore the effect of visual perception learning software training (VPT) on binocular visual function reconstruction in children with intermittent exotropia after strabismus surgery. Ninety children with intermittent exotropia admitted to our hospital from June 2018 to December 2018 were included, and randomly divided into VPT and control groups. Children in the control group received basic binocular vision training, while those in the VPT group received VPT after strabismus surgery. Tertiary visual function, visual perception function, Newcastle Control Score (NCS), and ocular position retraction rate were compared at 3 and 12mo after the surgery. At 3 and 12mo after the surgery, the proportion of simultaneous perception, binocular fusion version and binocular stereo vision in the VPT group was conspicuously higher than that in the control group (P<0.05). After the vision training, the binocular visual perception functions of children in both groups were significantly improved compared with that before training (P<0.05). Interestingly, the grating sharpness, texture perception and texture motion perception in the VPT group were dramatically better than control group (P<0.01). The NCS in the VPT group was significantly lower than that in the control group (P<0.05). The ocular position retraction rate in the VPT group was significantly lower than that in the control group at 12mo (8.89% vs 26.67%, P=0.03). VPT effectively promotes binocular visual function reconstruction in intermittent exotropia children after strabismus surgery and reduces the strabismus severity and ocular position retraction rate.

Retracted: Application of Rotationally Symmetrical Triangulation Stereo Vision Sensor in National Dance Movement Detection and Recognition

Retracted: Application of Rotationally Symmetrical Triangulation Stereo Vision Sensor in National Dance Movement Detection and Recognition

A Stable, Efficient, and High-Precision Non-Coplanar Calibration Method: Applied for Multi-Camera-Based Stereo Vision Measurements.

Traditional non-coplanar calibration methods, represented by Tsai's method, are difficult to apply in multi-camera-based stereo vision measurements because of insufficient calibration accuracy, inconvenient operation, etc. Based on projective theory and matrix transformation theory, a novel mathematical model is established to characterize the transformation from targets' 3D affine coordinates to cameras' image coordinates. Then, novel non-coplanar calibration methods for both monocular and binocular camera systems are proposed in this paper. To further improve the stability and accuracy of calibration methods, a novel circular feature points extraction method based on region Otsu algorithm and radial section scanning method is proposed to precisely extract the circular feature points. Experiments verify that our novel calibration methods are easy to operate, and have better accuracy than several classical methods, including Tsai's and Zhang's methods. Intrinsic and extrinsic parameters of multi-camera-systems can be calibrated simultaneously by our methods. Our novel circular feature points extraction algorithm is stable, and with high precision can effectively improve calibration accuracy for coplanar and non-coplanar methods. Real stereo measurement experiments demonstrate that the proposed calibration method and feature extraction method have high accuracy and stability, and can further serve for complicated shape and deformation measurements, for instance, stereo-DIC measurements, etc.

The effect of training on sensitivity and stability of double fusion in Panum's limiting case.

Panum's limiting case is a phenomenon of monocular occlusion in binocular vision. This occurs when one object is occluded by the other object for one eye, but the two objects are both visiblefor the other eye. Although previous studies have found that vertical gradient of horizontal disparity and cue conflict are two important factors for double fusion, the effect of training on the sensitivity and stability of Panum's limiting case remains unknown. The current study trained 26 participants for 5 days with several of Panum's configurations (Gilliam, Frisby, and Wang series). The latency and duration of double fusion were recorded to examine the effects of training on sensitivity and stability of double fusion in Panum's limiting case. For each level of vertical gradient of horizontal disparity and cue conflict, the latency of double fusion decreased and the duration of double fusion increased with each additional training session. The results showed that vertical gradient of horizontal disparity and cue conflict interacted, and the duration of high cue conflict was significantly shorter than that of medium and low cue conflict for each level of vertical gradient of horizontal disparity. The findings suggest that there is an effect of training for vertical gradient of horizontal disparity and cue conflict in Panum's limiting case, and that the three factors jointly affect the sensitivity and stability of double fusion.

DIC measurement method based on binocular stereo vision for image 3D displacement detection

The deformation detection of large machinery is usually achieved using three-dimensional displacement measurement. Binocular stereo vision measurement technology, as a commonly used digital image correlation method, has received widespread attention in the academic community. Binocular stereo vision achieves the goal of three-dimensional displacement measurement by simulating the working mode of the human eyes, but the measurement is easily affected by light refraction. Based on this, the study introduces particle swarm optimization algorithm for target displacement measurement on Canon imaging dataset, and introduces backpropagation neural network for mutation processing of particles in particle swarm algorithm to generate fusion algorithm. It combines the four coordinate systems of world, pixel, physics, and camera to establish connections. Taking into account environmental factors and lens errors, the camera parameters and deformation coefficients were revised by shooting a black and white checkerboard. Finally, the study first conducted error analysis on binocular stereo vision technology in three dimensions, and the relative error remained stable at 1 % within about 60 seconds. At the same time, three algorithms, including the spotted hyena algorithm, were introduced to conduct performance comparison experiments using particle swarm optimization and backpropagation network algorithms. The experiment shows that the three-dimensional error of the fusion algorithm gradually stabilizes within the range of [–0.5 %, 0.5 %] over time, while the two-dimensional error generally hovers around 0 value. Its performance is significantly superior to other algorithms, so the binocular stereo vision of this fusion algorithm can achieve good measurement results.

Refraction error analysis in stereo vision for system parameters optimization

The traditional stereo vision system with an air-based imaging model is unsuitable for multi-layer imaging conditions, where refraction effects can introduce significant errors in the measurement outcomes. Although numerous theoretical studies have developed refraction correction algorithms, it may be more beneficial and desirable to reduce the error by optimizing the system parameters in some cases. To address this dilemma, in this study, we first build a three-layer stereo vision model by forward projection and backward reconstruction. Then, the impact of various factors on the errors of reconstructed coordinates in the conditions of air-glass-air and air-glass-water is demonstrated through simulations. Real experiments are further carried out to verify the validity of the simulation. Finally, a practical guide for conducting experiments and tests is presented to assist in the design of more effective system parameters.

Research on the immersive multimedia virtual interactive experience technology of the Great Wall World Heritage based on VR vision system

Abstract This paper constructs an immersive multimedia virtual interactive experience technology with the Great Wall World Heritage Site as the experimental object. By analyzing the VR vision system and algorithm, the process of stereo vision perception and the general algorithm of somatic image production are explained. The distance equation and depth calculation in stereoscopic projection are studied, and the virtual interactive experience technology is constructed and applied to the World Heritage Site of the Great Wall. The result data show that the immersive multimedia virtual interactive experience technology highlights the real feeling of users more, in which the model has 50% of the influence on the real feeling, the mapping has 15%, the lighting has 15%, and the rendering has 10%. The youth population has a higher percentage of interest in the virtual field, accounting for 50%, 65%, and 60%, respectively, while the middle-aged stage has a more average degree of interest in the virtual field, with 50% of the average interest, 30% of happy to experience, 25% of not interested and 50% of interested. This thesis is conducive to enriching the project of the Great Wall World Heritage Site and providing an effective reference for technological innovation of immersive multimedia virtual interactive experience.

A Filter-Based Integration of GNSS, INS, and Stereo Vision in Tight Mode With Optimal Smoothing

A Filter-Based Integration of GNSS, INS, and Stereo Vision in Tight Mode With Optimal Smoothing