Binocular Stereo Vision Research Articles

Three-dimensional reconstruction optimization of tunnel face and intelligent extraction of discontinuity orientation based on binocular stereo vision

In the process of grading and dynamically optimizing the design and construction parameters of the surrounding rock mass of a rock tunnel face, efficiently and accurately acquiring the geometrical parameters of the rock discontinuities is an important basic task. To address the problems of time consuming, low accuracy, and high danger associated with traditional methods of obtaining the structural information of rock mass, this paper proposes a method for three-dimensional reconstruction and intelligent information extraction of tunnel face based on binocular stereo vision (BSV). First, the parallel binocular device with a single camera was improved, calibrated using the checkerboard calibration method. By integrating with the semi-global matching algorithm, the BSV based method for the three-dimensional reconstruction of the rock mass of the tunnel face was optimized. Furthermore, based on the results from on-site engineering applications, this study leveraged two parameters, point cloud density and algorithm runtime, to determine the optimal values for the disparity range and window size parameters within the semi-global stereo matching algorithm. This enhancement improved the performance of the 3D reconstruction method based on binocular stereo vision. Finally, efficient and refined intelligent methods for extracting structural parameters of the rock mass were proposed based on k-nearest neighbor search and kernel density estimation. The research results can provide reliable technical support for the intelligent and efficient acquisition of rock mass structural information in rock tunnel engineering faces.

Displacement Measurement and 3D Reconstruction of Segmental Retaining Wall Using Deep Convolutional Neural Networks and Binocular Stereovision

Computer vision techniques were employed to monitor the displacement of retaining walls using artificial markers, traditional feature detection algorithms, and photogrammetry‐based point cloud reconstruction. However, the use of artificial markers often increases both installation time and costs, whereas the performance of traditional feature matching is affected by uneven illumination, and photogrammetry techniques require multiple images for point cloud reconstruction. To overcome these limitations, a nontarget‐based displacement monitoring method for segmental retaining walls (SRWs) using a combination of deep learning and stereovision was proposed. Binocular stereovision was employed to reconstruct the geometry and surface properties of the SRW in a digital three‐dimensional (3D) model. Deep learning models were then used to extract natural features from SRW blocks, enabling displacement calculation without using artificial targets. The performance was evaluated by monitoring the behaviors of SRW experiments at both laboratory and field scales. The deep learning–based image segmentation models identified SRW block features in the experiment and real case datasets with an average F1 score from 0.910 to 0.965 under various environmental conditions. The reconstructed results of point cloud coordinates demonstrated high accuracy, ranging from 95.2% to 98.6%. Furthermore, the calculated displacement exhibited a high degree of agreement with the measured displacement. The accuracy of the calculated displacements for the laboratory and field experiments ranged from 89.5% to 99.1%. The proposed method can be used for automatic SRW displacement monitoring.

A Binocular Vision-Based Crack Detection and Measurement Method Incorporating Semantic Segmentation.

The morphological characteristics of a crack serve as crucial indicators for rating the condition of the concrete bridge components. Previous studies have predominantly employed deep learning techniques for pixel-level crack detection, while occasionally incorporating monocular devices to quantify the crack dimensions. However, the practical implementation of such methods with the assistance of robots or unmanned aerial vehicles (UAVs) is severely hindered due to their restrictions in frontal image acquisition at known distances. To explore a non-contact inspection approach with enhanced flexibility, efficiency and accuracy, a binocular stereo vision-based method incorporating full convolutional network (FCN) is proposed for detecting and measuring cracks. Firstly, our FCN leverages the benefits of the encoder-decoder architecture to enable precise crack segmentation while simultaneously emphasizing edge details at a rate of approximately four pictures per second in a database that is dominated by complex background cracks. The training results demonstrate a precision of 83.85%, a recall of 85.74% and an F1 score of 84.14%. Secondly, the utilization of binocular stereo vision improves the shooting flexibility and streamlines the image acquisition process. Furthermore, the introduction of a central projection scheme achieves reliable three-dimensional (3D) reconstruction of the crack morphology, effectively avoiding mismatches between the two views and providing more comprehensive dimensional depiction for cracks. An experimental test is also conducted on cracked concrete specimens, where the relative measurement error in crack width ranges from -3.9% to 36.0%, indicating the practical feasibility of our proposed method.

Applications of visual perception techniques using neural networks in autonomous driving

The perception system and the decision system are important components of a complete autonomous driving vehicle. The perception system can help the decision system to obtain the necessary information of external environment and vehicle status. The traditional perception system mainly relies on the on-board radar. But in recent years, vision-based perception techniques have become a hot research topic. Meanwhile, thanks to the excellent performance of neural networks in processing image data, the processing algorithms for visual perception images have also made great progress. Visual perception techniques can not only acquire more information, but also is more cost effective and easier to install. This paper provides an overview of the more mature and promising visual perception techniques, including their principles and data processing algorithms, in terms of acquiring 2D image data and 3D depth information. For acquiring 2D image data, this paper introduces the principle of event camera and reviews the current progress on the event camera. Regarding the acquisition of 3D depth information, three techniques are introduced, namely binocular stereo-vision, time of flight (TOF), and structured light. Their performance when combined with neural networks for autonomous driving applications is also reviewed. Finally, this paper lists the current dilemmas faced by the above 2D and 3D imaging techniques and the possible solutions.

Fully automatic system for fish biomass estimation based on deep neural network

The approach for estimating biomass in non-contact, free-swimming fish has encountered difficulties such as fish body occlusion, bending, non-orthogonal angles, and low efficiency. To address these issues, this study had combined fish posture recognition (using deep learning technology) with biomass estimation (utilizing stereo vision technology) for the first time, and developed a fast, precise, and fully automatic fish biomass estimation system. The improved single-stage target detection algorithm significantly improved the direct detection and extraction of high-quality images of moving fish in real-time, eliminating the need for manual processing of images that may have imperfect posture. Fish body length and height were measured in the real world using binocular stereo vision technology. Finally, the fish body weight can be estimated by considering the relationship among their body length, height, and weight. The experiment confirmed that the system had successfully avoided influencing factors that could affect fully automatic estimation. The results demonstrated a strong linear relationship between the estimated and measured fish body weights, with a mean relative error (MRE) of 2.87%. There were no significant differences between the estimated and measured weights (p = 0.94). The MRE of the multi-factor model was much lower than that of the single-factor model (length-weight of 8.86% and height-weight of 7.41%). The results indicate that the system developed is a highly effective approach to fully automated biomass estimation. This can be used to guide actual production and further study of the mechanism of fish growth.

A markless 3D human motion data acquisition method based on the binocular stereo vision and lightweight open pose algorithm

This study aimed to achieve precise 3D human motion data acquisition without the need for markers. Specifically, the proposed system utilized two affordable cameras to capture human motion from diverse angles. It developed a lightweight Open Pose algorithm to extract 2D human body joints from recorded videos. Subsequently, a stereo-matching method was applied to derive accurate 3D human body joint data. Comprehensive experiments with 10 healthy participants showcased the system's exceptional performance, demonstrating superior measurement accuracy compared to established standards like the NOKOV system (RMSE values consistently below 9°) and commonly used Microsoft Kinect devices (V2 and Azure). This innovative binocular vision system presents a new approach to motion capture, potentially offering a cost-effective and precise tool for practical applications across various scenarios (e.g., motor functional assessment, virtual avatar streaming), contributing significantly to both scientific and academic communities.

A system and method for person identification and positioning incorporating object edge detection and scale-invariant feature transformation

During the overhead crane, due to the complex background and large image size, the direct processing algorithm usually leads to time-consuming operation and difficult to achieve accurate personnel positioning. To address these issues, we propose the feature points recognition and matching algorithm incorporating object edge detection and scale-invariant feature transform (SIFT), which is applied to the designed object detection and positioning algorithm for binocular stereo vision. The proposed method applies the Canny operator to extract contours from the locally cropped regions obtained using YOLOv5. The contour points are used as the key points of SIFT, and corresponding feature points are created descriptors, and feature points matching are performed. Experimental results indicate that the proposed method can filter feature points successfully. Moreover, the binocular stereo vision system with the binocular positioning algorithm designed has good performance, with the overall depth direction positioning error within 5% in a range of 12 m.

GelStereo Palm: A Novel Curved Visuotactile Sensor for 3-D Geometry Sensing

Recently, visuotactile sensors have shown promising potential in robotics due to their high-resolution sensing ability. Unfortunately, the majority of available visuotactile sensors are limited to flat shapes, which severely limits their application possibilities. In this paper, we propose a novel curved visuotactile sensor, the GelStereo Palm, which senses the 3D contact geometry on a curved surface using a binocular vision system. Meanwhile, to solve the light refraction problem in the binocular stereo vision system under a curved medium, a Refractive Stereo Ray Tracing model for GelStereo Palm (GP-RSRT) is presented. Moreover, a 3D tactile point cloud sensing pipeline is introduced to reconstruct the 3D contact geometry in real-time. Finally, extensive experiments are conducted to verify the accuracy and robustness of the 3D contact geometry sensing of our GelStereo Palm sensor.

A target spatial location method for fuze detonation point based on deep learning and sensor fusion

The spatial location of fuze detonation point is crucial for evaluating the working condition and improving the performance of fuze. Considering the observation safety, the non-contact long-distance accurate measurement technology is essential. In this paper, we propose a method that takes the sensor data of optics, spatial attitude and GPS as input and outputs the spatial position of the fuze detonation point. The proposed method consists of two steps. First, an object detection algorithm with post-processing algorithm is proposed to obtain rich information of the target. The algorithm achieves high-accuracy detection by introducing powerful backbone, attention mechanism, group convolution, and improved multi-scale feature fusion. Second, a Variational Auto-Encoder (VAE) algorithm model improved by dense connection structure and multiple source heterogeneous sensor information fusion structure is proposed as the position regression algorithm. It receives the status information of the observer camera and the output of the object detection algorithm, and then outputs the three-dimensional coordinates of the explosion point. Finally, method validation and performance analysis are realized through virtual scene simulation. Experiment results show the superiority of the proposed object detection algorithm over other typical algorithms on explosion flare detection, with its Average Precision (AP) of 0.889. The positioning error of the spatial location method is 0.896 m, while that of the binocular stereo vision method is 2.863 m. Therefore, the proposed target spatial location method is proved to be effective and accurate.

Calibration method for monocular laser speckle projection system.

This paper proposes a novel calibration method for the monocular laser speckle projection system. By capturing images of a calibration board with speckles under different poses, projector's optical axis is fitted and utilized to calibrate the rotation between the camera and projector. The translation is solved in closed form subsequently and projector's virtual image is recovered via homography. After calibration, the system can be regarded and operated as a binocular stereo vision system with speckle pattern. The proposed method is efficient and convenient, without need of reference image or high-precision auxiliary equipment. Validated by experiments on Astra-s and Astra-pro, it presents significant improvement in depth-estimation compared to the traditional method.

Three-dimensional reconstruction of mobile binocular stereo vision based on push-broom line structured light for a workpiece surface

Stereo vision technology based on line structured light can effectively solve the problem of a three-dimensional (3D) reconstruction of a smooth surface. A method for 3D reconstruction of mobile binocular stereo vision based on push-broom line structured light for a workpiece surface is presented in this paper. The subpixel coordinates of the light strip centers of the line structured light are obtained by the Steger algorithm while the binocular module moves along the guide rail, and the polar constraint is used to achieve the matching of the extracted light strip centers. As a result, the 3D coordinates of the light strip centers in each location can be calculated because of the known interior and external parameters of the binocular module. To obtain the 3D point cloud data of the entire surface, a relative pose optimization method with respect to the initial frame is proposed, which accurately estimates the pose of the cameras in each location with respect to that in the initial location and unifies the 3D coordinates of the light strip centers in each location to the datum coordinates. The relative pose optimization method first estimates the rough values by using the direct linear transform method, and then iteratively calculates the refined solutions based on the principle of minimizing the re-projection errors. Simulation data and substantial experimental results validate the effectiveness of our method. Our method is compared to the direct linear transform method and the frame-by-frame transfer method, and the root mean square error (RMSE) of the distance from 3D point cloud to fitted plane is used to evaluate the 3D reconstruction accuracy. The repeatability experiment shows that the RMSE from our method is as low as 0.83 mm.

Tomato flower pollination features recognition based on binocular gray value-deformation coupled template matching

It is necessary to recognize the tomato pollination features for the designing demand of intelligent and precise tomato supplementary pollination equipment. Mentioned pollination features include flower opening state and the three-dimensional position and pose of flower anther. Tomato flower pollination features recognition method is designed in this paper based on the deep learning full opened flower recognition model and binocular template matching three-dimensional information recognition method. First of all, a binocular stereo vision system is built to acquire the tomato flower images in greenhouse with natural lighting. Which can help vision system avoid the impact of inconsistent light intensity on image recognition. The acquired images were equalized with three groups parameters to labeled images. And the improved MC-AlexNet deep learning model is established to recognize the full opened tomato flowers in the left image of image pair acquired with binocular vision system. Then, the template is created with the full opened flower recognition result of deep learning model in left image based on gray value and deformation template matching method. And the template matching is established to recognize the corresponding full opened flowers in the right image of image pair. Finally, with the template matching result, the anther segmentation is conducted to calculate three-dimensional position and pose of anther. The experimental results show that the accuracy of full opened tomato flowers recognition model is 96.23%. The average position recognition deviation of anther is 5.94 mm. And the average anther pose recognition deviation in three plane is 6.24° for the images that anthers can be observed. And the average time consuming is about 143.18 ms per image pair. It can be concluded that the method of binocular template matching established in this paper can fulfill the demand of supplementary pollination equipment design, and the research result lays a foundation for designing and improvement of tomato supplementary pollination equipment in greenhouse.

Effect of visual perception training on binocular visual function reconstruction in patients after strabismus surgery.

To explore the effect of visual perception learning software training (VPT) on binocular visual function reconstruction in children with intermittent exotropia after strabismus surgery. Ninety children with intermittent exotropia admitted to our hospital from June 2018 to December 2018 were included, and randomly divided into VPT and control groups. Children in the control group received basic binocular vision training, while those in the VPT group received VPT after strabismus surgery. Tertiary visual function, visual perception function, Newcastle Control Score (NCS), and ocular position retraction rate were compared at 3 and 12mo after the surgery. At 3 and 12mo after the surgery, the proportion of simultaneous perception, binocular fusion version and binocular stereo vision in the VPT group was conspicuously higher than that in the control group (P<0.05). After the vision training, the binocular visual perception functions of children in both groups were significantly improved compared with that before training (P<0.05). Interestingly, the grating sharpness, texture perception and texture motion perception in the VPT group were dramatically better than control group (P<0.01). The NCS in the VPT group was significantly lower than that in the control group (P<0.05). The ocular position retraction rate in the VPT group was significantly lower than that in the control group at 12mo (8.89% vs 26.67%, P=0.03). VPT effectively promotes binocular visual function reconstruction in intermittent exotropia children after strabismus surgery and reduces the strabismus severity and ocular position retraction rate.

DIC measurement method based on binocular stereo vision for image 3D displacement detection

The deformation detection of large machinery is usually achieved using three-dimensional displacement measurement. Binocular stereo vision measurement technology, as a commonly used digital image correlation method, has received widespread attention in the academic community. Binocular stereo vision achieves the goal of three-dimensional displacement measurement by simulating the working mode of the human eyes, but the measurement is easily affected by light refraction. Based on this, the study introduces particle swarm optimization algorithm for target displacement measurement on Canon imaging dataset, and introduces backpropagation neural network for mutation processing of particles in particle swarm algorithm to generate fusion algorithm. It combines the four coordinate systems of world, pixel, physics, and camera to establish connections. Taking into account environmental factors and lens errors, the camera parameters and deformation coefficients were revised by shooting a black and white checkerboard. Finally, the study first conducted error analysis on binocular stereo vision technology in three dimensions, and the relative error remained stable at 1 % within about 60 seconds. At the same time, three algorithms, including the spotted hyena algorithm, were introduced to conduct performance comparison experiments using particle swarm optimization and backpropagation network algorithms. The experiment shows that the three-dimensional error of the fusion algorithm gradually stabilizes within the range of [–0.5 %, 0.5 %] over time, while the two-dimensional error generally hovers around 0 value. Its performance is significantly superior to other algorithms, so the binocular stereo vision of this fusion algorithm can achieve good measurement results.

Digital twin technology for wind turbine towers based on joint load–response estimation: A laboratory experimental study

An accurate estimation of dynamic loads and structural dynamic responses is deemed an indispensable prerequisite for developing trustworthy digital twin (DT) models of dynamically excited structures. Traditional joint load–response estimation (JLRE) algorithms necessitate a full-rank feedthrough matrix, implying that accelerometers are required at all degrees of freedom with input loads. If an unknown input is a bending moment or torque, the rotational acceleration must be measured, which is usually difficult, if not impractical, in real applications. Therefore, the applications of the traditional JLRE in wind turbines (WTs) are rather limited due to the complex excitation mechanism. However, the unified linear input and state estimator (ULISE) algorithm presented in this paper eliminates this constraint. This paper experimentally tested this algorithm on an operating 1:50 scaled WT model. The influences from blade rotation were considered. A comprehensive structural health monitoring (SHM) sensing system was deployed on the WT tower to measure the tower's dynamic responses. A camera system, which integrated digital image correlation (DIC) technology and binocular stereo vision technology, was also adopted. The unknown excitations and unmonitored dynamic responses of the WT tower were estimated simultaneously using the ULISE algorithm. The reconstructed strain and acceleration responses achieved high accuracy, with a normalized root mean square error less than 9%. The blade rotation had a notable impact on the WT tower, primarily manifested as a bending moment whose dominant frequency was corresponding to the blade's rotational speed. Such joint estimations could function as the virtual sensing of unknown inputs and responses in the DT model of the WT tower, which will enable DT-based online remote structural condition assessment and preventive maintenance in the future.

Three-Dimensional Point Cloud Reconstruction Method of Cardiac Soft Tissue Based on Binocular Endoscopic Images

Three-dimensional reconstruction technology based on binocular stereo vision is a key research area with potential clinical applications. Mainstream research has focused on sparse point reconstruction within the soft tissue domain, limiting the comprehensive 3D data acquisition required for effective surgical robot navigation. This study introduces a new paradigm to address existing challenges. An innovative stereoscopic endoscopic image correction algorithm is proposed, exploiting intrinsic insights into stereoscopic calibration parameters. The synergy between the stereoscopic endoscope parameters and the disparity map derived from the cardiac soft tissue images ultimately leads to the acquisition of precise 3D points. Guided by deliberate filtering and optimization methods, the triangulation process subsequently facilitates the reconstruction of the complex surface of the cardiac soft tissue. The experimental results strongly emphasize the accuracy of the calibration algorithm, confirming its utility in stereoscopic endoscopy. Furthermore, the image rectification algorithm exhibits a significant reduction in vertical parallax, which effectively enhances the stereo matching process. The resulting 3D reconstruction technique enables the targeted surface reconstruction of different regions of interest in the cardiac soft tissue landscape. This study demonstrates the potential of binocular stereo vision-based 3D reconstruction techniques for integration into clinical settings. The combination of joint calibration algorithms, image correction innovations, and precise tissue reconstruction enhances the promise of improved surgical precision and outcomes in the field of cardiac interventions.

Coral Detection, Ranging, and Assessment (CDRA) algorithm-based automatic estimation of coral reef coverage

Climate change poses significant threats to the health of coral reefs. The detection of coral coverage is a quantitative method for assessing the health of coral reefs. Valuable insights into the effects of climate change on coral reef ecosystems and the trends in coral reef development can be gained through the monitoring of coral coverage. However, traditional manual methods for estimating coral coverage are time-consuming and labor-intensive, limiting the scalability and efficiency of coral surveys. The field of automated processing for coral video imagery is currently limited, impeding the development of effective techniques for comprehensive coral monitoring and further hindering progress in coral ecosystem monitoring. To cope with these challenges, there is a crucial need for automated systems to efficiently handle large volumes of video data, thereby transforming imaging devices, such as underwater robots, into autonomous sensors capable of conducting reliable and comprehensive surveys. In response to this pressing issue, this study presents a novel video analysis approach, termed the Coral Detection, Ranging, and Assessment (CDRA) algorithm, which combines cutting-edge computer vision techniques including You Only Look Once version 5 (YOLOv5) object detection, binocular stereo vision, and tracking algorithms. The CDRA algorithm aims to analyze continuous video segments of coral reefs, enabling the identification, counting, and estimation of size and location of individual coral colonies. Experimental evaluations conducted in controlled pool environments demonstrated the high accuracy of the proposed method, with an average accuracy of 90.7% in estimating coral coverage at different locations. Moreover, field trials conducted in Xidao Island of China verified the effectiveness and robustness of the CDRA algorithm under varying water quality and lighting conditions. The findings of this study represent a significant step towards the development of reliable and automated techniques for coral reef detection, contributing to the advancement of coral reef conservation efforts.

Detection and Positioning of Camellia oleifera Fruit Based on LBP Image Texture Matching and Binocular Stereo Vision

To achieve the rapid recognition and accurate picking of Camellia oleifera fruits, a binocular vision system composed of two industrial cameras was used to collect images of Camellia oleifera fruits in natural environments. The YOLOv7 convolutional neural network model was used for iterative training, and the optimal weight model was selected to recognize the images and obtain the anchor frame region of the Camellia oleifera fruits. The local binary pattern (LBP) maps of the anchor frame region were extracted and matched by using the normalized correlation coefficient template matching algorithm to obtain the positions of the center point in the left and right images. The recognition experimental results showed that the accuracy rate, recall rate, mAP and F1 of the model were 97.3%, 97.6%, 97.7% and 97.4%. The recognition rate of the Camellia oleifera fruit with slight shading was 93.13%, and the recognition rate with severe shading was 75.21%. The recognition rate of the Camellia oleifera fruit was 90.64% under sunlight condition, and the recognition rate was 91.34% under shading condition. The orchard experiment results showed that, in the depth range of 400–600 mm, the maximum error value of the binocular stereo vision system in the depth direction was 4.279 mm, and the standard deviation was 1.142 mm. The detection and three-dimensional positioning accuracy of the binocular stereo vision system for Camellia oleifera fruits could basically meet the working requirements of the Camellia oleifera fruit-picking robot.

Analysis of the current situation and teaching strategies of university table tennis teaching in the context of big data

Abstract As an important part of university physical education, table tennis teaching occupies a crucial position in physical education reform and requires continuous innovation and reform. This paper first starts from the current situation of table tennis teaching and proposes the application of a binocular stereo vision system based on a binocular stereo vision system for university table tennis teaching, which lays the foundation for three-dimensional reconstruction and skill learning. Secondly, the mathematical model of the 3D reconstruction algorithm is constructed by the tangential distortion of least squares, which avoids the complex algorithm of iteratively solving the 3D world coordinates of table tennis and improves the speed of the 3D reconstruction algorithm while the accuracy meets the subsequent prediction requirements. Finally, students in the experimental and control groups were compared in the technical evaluation of table tennis backhand pushing, attacking techniques, and left pushing and right attacking to verify the effectiveness of the table tennis binocular stereo vision teaching system. The study results showed that the experimental group students trained by the binocular stereo vision system for table tennis had 41.77% higher accuracy, 23.13% higher coordination, 12.31% higher return speed, and 14.84% higher hitting power compared to the control group. Applying a binocular stereo-vision system to table tennis teaching is novel and is conducive to stimulating students’ curiosity and challenge, enhancing their interest in learning, and effectively improving their understanding and knowledge of table tennis techniques.

Environment Modeling Method for Intelligent Robots Based on Binocular Vision

With the development of technology, the demand for intelligent mobile robots is growing steadily. Environment perception and modeling, are the key elements to achieve robot intelligence. Therefore, binocular stereo vision sensor is used as the core device to perform operations such as binocular camera calibration, obtaining binocular camera internal/external parameters, and distortion correction. On this basis, a multi-dimensional and all-round environment perception system is constructed, and the dynamic target detection results and 3D scene map reconstruction results are displayed dynamically in real time. The experimental results of binocular camera calibration show that the re-projection error lies in the sub-pixel interval, and the image distortion caused by the camera is basically eliminated to meet the experimental expectation of distortion correction; the experimental results of environment modeling show that the mean value of deviation between the object representation value of 3D scene map and the actual object real value is -0.041m, among which the minimum deviation value is 0.001m and the maximum deviation value is 0.532m, in The experimental results are excellent in the tasks of 3D scene map reconstruction and feature point matching. In addition, the storage space of 3D scene map is significantly reduced, and the real-time performance of the system is significantly better than the traditional method.