Stereo Vision System Research Articles

Passive vision based three-dimensional seam tracking using a stereo vision system with a single HDR camera

Seam tracking is a crucial feature that an automatic welding system must have. The laser tracking algorithm offers high accuracy but faces challenges in identifying seams with extremely narrow gaps, as the deformation of the laser stripe becomes difficult to detect in such cases. In contrast, the passive vision-based algorithm can effectively capture narrow gap information from the welding scene, making it more suitable for recognizing narrow seams. However, most of proposed passive vision-based seam tracking is based on “teaching and playback” mode and can only realize planar tracking. In medium and thick steel plate welding, the workpieces are so large that the “teaching and playback” based control algorithm is unsuitable. Therefore, in this paper, a stereo vision system with a single camera and the corresponding calibration algorithm are developed to achieve three dimensional seam tracking. Compared to stereo system with dual camera, the proposed system is cheaper and easier to synchronize acquisition. Moreover, to get out of the “teaching and playback” control strategy, a vector control algorithm without cumulative error is proposed. Results of nine experiments show that the average error of the proposed seam tracking system is less than 1.0662mm and the maximum error is 1.8030mm.

High-Precision Disparity Estimation for Lunar Scene Using Optimized Census Transform and Superpixel Refinement

High-precision lunar scene 3D data are essential for lunar exploration and the construction of scientific research stations. Currently, most existing data from orbital imagery offers resolutions up to 0.5–2 m, which is inadequate for tasks requiring centimeter-level precision. To overcome this, our research focuses on using in situ stereo vision systems for finer 3D reconstructions directly from the lunar surface. However, the scarcity and homogeneity of available lunar surface stereo datasets, combined with the Moon’s unique conditions—such as variable lighting from low albedo, sparse surface textures, and extensive shadow occlusions—pose significant challenges to the effectiveness of traditional stereo matching techniques. To address the dataset gap, we propose a method using Unreal Engine 4 (UE4) for high-fidelity physical simulation of lunar surface scenes, generating high-resolution images under realistic and challenging conditions. Additionally, we propose an optimized cost calculation method based on Census transform and color intensity fusion, along with a multi-level super-pixel disparity optimization, to improve matching accuracy under harsh lunar conditions. Experimental results demonstrate that the proposed method exhibits exceptional robustness and accuracy in our soon-to-be-released multi-scene lunar dataset, effectively addressing issues related to special lighting conditions, weak textures, and shadow occlusion, ultimately enhancing disparity estimation accuracy.

Responses of Vehicular Occupants During Emergency Braking and Aggressive Lane-Change Maneuvers.

To validate active human body models for investigating occupant safety in autonomous cars, it is crucial to comprehend the responses of vehicle occupants during evasive maneuvers. This study sought to quantify the behavior of midsize male and small female passenger seat occupants in both upright and reclined postures during three types of vehicle maneuvers. Volunteer tests were conducted using a minivan, where vehicle kinematics were measured with a DGPS sensor and occupant kinematics were captured with a stereo-vision motion capture system. Seatbelt loads, belt pull-out, and footrest reaction forces were also documented. The interior of the vehicle was 3D-scanned for modeling purposes. Results indicated that seatback angles significantly affected occupant kinematics, with small female volunteers displaying reduced head and torso movements, except during emergency braking with a upright posture seatback. Lane-change maneuvers revealed that maximum lateral head excursions varied depending on the maneuver's direction. The study concluded that seatback angles were crucial in determining the extent of occupant movement, with notable variations in head and torso excursions observed. The collected data assist in understanding occupant behavior during evasive maneuvers and contribute to the validation of human body models, offering essential insights for enhancing safety systems in autonomous vehicles.

Characterization of woven composite material under multiaxial loading regimes using FE-based stereocorrelation

In this paper, woven glass fiber composite samples were subjected to three different cyclic loading histories (i.e., tensile, shear and combined loading at 45° via the Modified Arcan Fixture. During the experiments, the samples were monitored by a stereovision system. Finite Element based stereocorrelation was used to measure the displacement and strain fields. The analysis of the experiments revealed different damage mechanisms. Furthermore, for the 45° experiment, the highest strain levels were reached, whereas for the tensile test, the highest stress levels were achieved.

Aberration-robust monocular passive depth sensing using a meta-imaging camera

Depth sensing plays a crucial role in various applications, including robotics, augmented reality, and autonomous driving. Monocular passive depth sensing techniques have come into their own for the cost-effectiveness and compact design, offering an alternative to the expensive and bulky active depth sensors and stereo vision systems. While the light-field camera can address the defocus ambiguity inherent in 2D cameras and achieve unambiguous depth perception, it compromises the spatial resolution and usually struggles with the effect of optical aberration. In contrast, our previously proposed meta-imaging sensor1 has overcome such hurdles by reconciling the spatial-angular resolution trade-off and achieving the multi-site aberration correction for high-resolution imaging. Here, we present a compact meta-imaging camera and an analytical framework for the quantification of monocular depth sensing precision by calculating the Cramér–Rao lower bound of depth estimation. Quantitative evaluations reveal that the meta-imaging camera exhibits not only higher precision over a broader depth range than the light-field camera but also superior robustness against changes in signal-background ratio. Moreover, both the simulation and experimental results demonstrate that the meta-imaging camera maintains the capability of providing precise depth information even in the presence of aberrations. Showing the promising compatibility with other point-spread-function engineering methods, we anticipate that the meta-imaging camera may facilitate the advancement of monocular passive depth sensing in various applications.

DOT-SLAM: A Stereo Visual Simultaneous Localization and Mapping (SLAM) System with Dynamic Object Tracking Based on Graph Optimization.

Most visual simultaneous localization and mapping (SLAM) systems are based on the assumption of a static environment in autonomous vehicles. However, when dynamic objects, particularly vehicles, occupy a large portion of the image, the localization accuracy of the system decreases significantly. To mitigate this challenge, this paper unveils DOT-SLAM, a novel stereo visual SLAM system that integrates dynamic object tracking through graph optimization. By integrating dynamic object pose estimation into the SLAM system, the system can effectively utilize both foreground and background points for ego vehicle localization and obtain a static feature points map. To rectify the inaccuracies in depth estimation from stereo disparity directly on the foreground points of dynamic objects due to their self-similarity characteristics, a coarse-to-fine depth estimation method based on camera-road plane geometry is presented. This method uses rough depth to guide fine stereo matching, thereby obtaining the 3 dimensions (3D)spatial positions of feature points on dynamic objects. Subsequently, by establishing constraints on the dynamic object's pose using the road plane and non-holonomic constraints (NHCs) of the vehicle, reducing the initial pose uncertainty of dynamic objects leads to more accurate dynamic object initialization. Finally, by considering foreground points, background points, the local road plane, the ego vehicle pose, and dynamic object poses as optimization nodes, through the establishment and joint optimization of a nonlinear model based on graph optimization, accurate six degrees of freedom (DoFs) pose estimations are obtained for both the ego vehicle and dynamic objects. Experimental validation on the KITTI-360 dataset demonstrates that DOT-SLAM effectively utilizes features from the background and dynamic objects in the environment, resulting in more accurate vehicle trajectory estimation and a static environment map. Results obtained from a real-world dataset test reinforce the effectiveness.

Motion-error-free calibration of event camera systems using a flashing target.

Event cameras, inspired by biological vision, offer high dynamic range, excellent temporal resolution, and minimal data redundancy. Precise calibration of event camera systems is essential for applications such as 3D vision. The cessation of extra gray frame production in popular models like the dynamic vision sensor (DVS) poses significant challenges to achieving high-accuracy calibration. Traditional calibration methods, which rely on motion to trigger events, are prone to movement-related errors. This paper introduces a motion-error-free calibration method for event cameras using a flashing target produced by a standard electronic display that elicits high-fidelity events. We propose an improved events-accumulator to reconstruct gray images with distinct calibration features and develop an optimization method that adjusts camera parameters and control point positions simultaneously, enhancing the calibration accuracy of event camera systems. Experimental results demonstrated higher accuracy compared to the traditional motion-based calibration method (reprojection error: 0.03 vs. 0.96 pixels). The 3D reconstruction error remained around 0.15 mm, significantly improving over the motion-based method's 8.00 mm. Additionally, the method's adaptability for hybrid calibration in event-based stereovision systems was verified (e.g., with frame cameras or projectors).

Configuration reconstruction and all-joint synchronous measurement based on vision for segmented linkage manipulator of rigid-flexible dual-arm space robot

A rigid-flexible dual-arm space robot offers promising potential for on-orbit operation due to complementary strengths of its rigid and flexible manipulators. The rigid manipulator has the advantage of large payload capacity and high motion accuracy. The segmented linkage flexible manipulator is ideal for maneuvering in narrow, unstructured environments like internal satellite inspections and maintenance, due to its flexibility and slender body. However, there are inevitable tracking errors due to the multiple cable-driven mechanisms in the flexible manipulator. It’s difficult to get the configuration and end pose of flexible manipulator without adding external sensors. To address these issues, this paper presents a method for configuration reconstruction and all-joint synchronous measurement of the flexible manipulator. The flexible manipulator is captured by the stereovision system mounted on the rigid manipulator. The links’ equivalent center points are recognized by central moment method and edge line extraction method based on the natural characteristics. Joint-to-link kinematic model of flexible manipulator is established to measure joint angles and reconstruct configuration. Several simulations and experiments are carried out to validate the proposed method. The experimental results showed links’ average measurement position errors were less than 13 mm and joint angles reconstructed using the proposed method had an error of approximately 0.35°. These results demonstrate the accuracy and robustness of the proposed method.

Application of Stereo Vision to Control the Movement of the Robot Arm Towards the Position of Red Chilies

The trend of decreasing young workers in the agricultural sector needs to be anticipated by developing intelligent machines known as agricultural robots. This research aims to apply a stereo vision system to control the movement of the robot's grip towards the 3D position of the red chili fruit. The stereo vision system installed on the robot waist (joint-2) is used to capture plant images and process them using HSV masking filters and triangulation principal to obtain the 3D center point position of the fruit. The robot joint movement is calculated using geometric based inverse kinematics. The research results show that the average accuracy of the stereo vision system is 93.9 %. The average grip positioning accuracy is 95.6 % to the actual chili fruit position and 98.5 % to the stereo vision calculation value. The average stability of the stereo vision values is 99.5 %, while the average positioning stability of the robot's grip is 99.6 %. Time consumption for image processing is 0.053 s while time consumption for robot grip movement is 9 s. Therefore, the stereo vision system can be used to control robot's grip movement with a good accuracy. Keywords: Red chili fruit, Robot arm, Stereo vision, Three-dimensional position.

A Novel Simulation Method for 3D Digital-Image Correlation: Combining Virtual Stereo Vision and Image Super-Resolution Reconstruction.

3D digital-image correlation (3D-DIC) is a non-contact optical technique for full-field shape, displacement, and deformation measurement. Given the high experimental hardware costs associated with 3D-DIC, the development of high-fidelity 3D-DIC simulations holds significant value. However, existing research on 3D-DIC simulation was mainly carried out through the generation of random speckle images. This study innovatively proposes a complete 3D-DIC simulation method involving optical simulation and mechanical simulation and integrating 3D-DIC, virtual stereo vision, and image super-resolution reconstruction technology. Virtual stereo vision can reduce hardware costs and eliminate camera-synchronization errors. Image super-resolution reconstruction can compensate for the decrease in precision caused by image-resolution loss. An array of software tools such as ANSYS SPEOS 2024R1, ZEMAX 2024R1, MECHANICAL 2024R1, and MULTIDIC v1.1.0 are used to implement this simulation. Measurement systems based on stereo vision and virtual stereo vision were built and tested for use in 3D-DIC. The results of the simulation experiment show that when the synchronization error of the basic stereo-vision system (BSS) is within 10-3 time steps, the reconstruction error is within 0.005 mm and the accuracy of the virtual stereo-vision system is between the BSS's synchronization error of 10-7 and 10-6 time steps. In addition, after image super-resolution reconstruction technology is applied, the reconstruction error will be reduced to within 0.002 mm. The simulation method proposed in this study can provide a novel research path for existing researchers in the field while also offering the opportunity for researchers without access to costly hardware to participate in related research.

Methodological Selection of Optimal Features for Object Classification Based on Stereovision System.

With the expansion of green energy, more and more data show that wind turbines can pose a significant threat to some endangered bird species. The birds of prey are more frequently exposed to collision risk with the wind turbine blades due to their unique flight path patterns. This paper shows how data from a stereovision system can be used for an efficient classification of detected objects. A method for distinguishing endangered birds from common birds and other flying objects has been developed and tested. The research focused on the selection of a suitable feature extraction methodology. Both motion and visual features are extracted from the Bioseco BPS system and retested using a correlation-based and a wrapper-type approach with genetic algorithms (GAs). With optimal features and fine-tuned classifiers, birds can be distinguished from aeroplanes with a 98.6% recall and 97% accuracy, whereas endangered birds are delimited from common ones with 93.5% recall and 77.2% accuracy.

A Binocular Color Line-Scanning Stereo Vision System for Heavy Rail Surface Detection and Correction Method of Motion Distortion.

Thanks to the line-scanning camera, the measurement method based on line-scanning stereo vision has high optical accuracy, data transmission efficiency, and a wide field of vision. It is more suitable for continuous operation and high-speed transmission of industrial product detection sites. However, the one-dimensional imaging characteristics of the line-scanning camera cause motion distortion during image data acquisition, which directly affects the accuracy of detection. Effectively reducing the influence of motion distortion is the primary problem to ensure detection accuracy. To obtain the two-dimensional color image and three-dimensional contour data of the heavy rail surface at the same time, a binocular color line-scanning stereo vision system is designed to collect the heavy rail surface data combined with the bright field illumination of the symmetrical linear light source. Aiming at the image motion distortion caused by system installation error and collaborative acquisition frame rate mismatch, this paper uses the checkerboard target and two-step cubature Kalman filter algorithm to solve the nonlinear parameters in the motion distortion model, estimate the real motion, and correct the image information. The experiments show that the accuracy of the data contained in the image is improved by 57.3% after correction.

Robust distance measurement using illumination map estimation and MAHNet in underground coal mines

An effective binocular stereo distance measurement method is proposed to address challenges posed by low brightness and weak texture of images captured in underground coal mines for the machine vision method. This approach is based on illumination map estimation and the MobileNetV3 attention hourglass stereo matching network (MAHNet) model. First, a binocular stereo vision system is established in which infrared LEDs are uniformly distributed on both sides of the belt conveyor bracket as visual feature points. Second, images are preprocessed using illumination map estimation, and the optimization of inhomogeneous brightness image enhancement is achieved by adopting adaptive Gamma correction. Third, the YOLOv5 target detection network and Gaussian fitting fusion algorithm are utilized to detect infrared LED feature points. Fourth, the MAHNet model is employed to generate the cost volume and perform disparity regression, resulting in the acquisition of accurate disparity images. Finally, triangulation is applied to determine the depth of feature points. The experimental results of distance measurement demonstrate that an average relative ranging accuracy of 1.52% within the range of 50.0 cm to 250.0 cm can be achieved by the optimized method, thereby validating the effectiveness of this binocular distance measurement method in underground coal mines.

A novel data-driven algorithm for object detection, tracking, distance estimation, and size measurement in stereo vision systems

A novel data-driven algorithm for object detection, tracking, distance estimation, and size measurement in stereo vision systems

The Development of a Stereo Vision System to Study the Nutation Movement of Climbing Plants.

Climbing plants, such as common beans (Phaseolus vulgaris L.), exhibit complex motion patterns that have long captivated researchers. In this study, we introduce a stereo vision machine system for the in-depth analysis of the movement of climbing plants, using image processing and computer vision. Our approach involves two synchronized cameras, one lateral to the plant and the other overhead, enabling the simultaneous 2D position tracking of the plant tip. These data are then leveraged to reconstruct the 3D position of the tip. Furthermore, we investigate the impact of external factors, particularly the presence of support structures, on plant movement dynamics. The proposed method is able to extract the position of the tip in 86-98% of cases, achieving an average reprojection error below 4 px, which means an approximate error in the 3D localization of about 0.5 cm. Our method makes it possible to analyze how the plant nutation responds to its environment, offering insights into the interplay between climbing plants and their surroundings.

RC-SLAM: Road Constrained Stereo Visual SLAM System Based on Graph Optimization.

Intelligent vehicles are constrained by road, resulting in a disparity between the assumed six degrees of freedom (DoF) motion within the Visual Simultaneous Localization and Mapping (SLAM) system and the approximate planar motion of vehicles in local areas, inevitably causing additional pose estimation errors. To address this problem, a stereo Visual SLAM system with road constraints based on graph optimization is proposed, called RC-SLAM. Addressing the challenge of representing roads parametrically, a novel method is proposed to approximate local roads as discrete planes and extract parameters of local road planes (LRPs) using homography. Unlike conventional methods, constraints between the vehicle and LRPs are established, effectively mitigating errors arising from assumed six DoF motion in the system. Furthermore, to avoid the impact of depth uncertainty in road features, epipolar constraints are employed to estimate rotation by minimizing the distance between road feature points and epipolar lines, robust rotation estimation is achieved despite depth uncertainties. Notably, a distinctive nonlinear optimization model based on graph optimization is presented, jointly optimizing the poses of vehicle trajectories, LPRs, and map points. The experiments on two datasets demonstrate that the proposed system achieved more accurate estimations of vehicle trajectories by introducing constraints between the vehicle and LRPs. The experiments on a real-world dataset further validate the effectiveness of the proposed system.

A Novel Approach for Simultaneous Localization and Dense Mapping Based on Binocular Vision in Forest Ecological Environment

The three-dimensional reconstruction of forest ecological environment by low-altitude remote sensing photography from Unmanned Aerial Vehicles (UAVs) provides a powerful basis for the fine surveying of forest resources and forest management. A stereo vision system, D-SLAM, is proposed to realize simultaneous localization and dense mapping for UAVs in complex forest ecological environments. The system takes binocular images as input and 3D dense maps as target outputs, while the 3D sparse maps and the camera poses can be obtained. The tracking thread utilizes temporal clue to match sparse map points for zero-drift localization. The relative motion amount and data association between frames are used as constraints for new keyframes selection, and the binocular image spatial clue compensation strategy is proposed to increase the robustness of the algorithm tracking. The dense mapping thread uses Linear Attention Network (LANet) to predict reliable disparity maps in ill-posed regions, which are transformed to depth maps for constructing dense point cloud maps. Evaluations of three datasets, EuRoC, KITTI and Forest, show that the proposed system can run at 30 ordinary frames and 3 keyframes per second with Forest, with a high localization accuracy of several centimeters for Root Mean Squared Absolute Trajectory Error (RMS ATE) on EuRoC and a Relative Root Mean Squared Error (RMSE) with two average values of 0.64 and 0.2 for trel and Rrel with KITTI, outperforming most mainstream models in terms of tracking accuracy and robustness. Moreover, the advantage of dense mapping compensates for the shortcomings of sparse mapping in most Smultaneous Localization and Mapping (SLAM) systems and the proposed system meets the requirements of real-time localization and dense mapping in the complex ecological environment of forests.

Long-Range 3D Reconstruction Based on Flexible Configuration Stereo Vision Using Multiple Aerial Robots

Aerial robots, or unmanned aerial vehicles (UAVs), are widely used in 3D reconstruction tasks employing a wide range of sensors. In this work, we explore the use of wide baseline and non-parallel stereo vision for fast and movement-efficient long-range 3D reconstruction with multiple aerial robots. Each viewpoint of the stereo vision system is distributed on separate aerial robots, facilitating the adjustment of various parameters, including baseline length, configuration axis, and inward yaw tilt angle. Additionally, multiple aerial robots with different sets of parameters can be used simultaneously, including the use of multiple baselines, which allows for 3D monitoring at various depth ranges simultaneously, and the combined use of horizontal and vertical stereo, which improves the quality and completeness of depth estimation. Depth estimation at a distance of up to 400 m with less than 10% error using only 10 m of active flight distance is demonstrated in the simulation. Additionally, estimation of a distance of up to 100 m with flight distance of up to 10 m on the vertical axis and horizontal axis is demonstrated in an outdoor mapping experiment using the developed prototype UAVs.

Stereo vision based systems for sea-state measurement and floating structures monitoring

Using computer vision techniques such as stereo vision systems for sea state measurement or for offshore structures monitoring can improve the measurement fidelity and accuracy with no significant additional cost. In this paper, two experiments (in-lab/open-sea) are conducted to study the performance of stereo vision system to measure the water wave surface elevation and rigid body heaving motion. For the in-lab experiment, regular water waves are generated in a wave tank for different frequencies and wave heights, where the water surface is scanned by the stereo vision camera installed on the top of the tank. Surface elevation inferred by the stereo vision is verified by an installed stationary side camera that records the water surface through the tank transparent side window, water surface elevation measured by the side camera recordings is extracted using edge detection algorithm. During the in-lab experiment a heaving buoy is installed to test the performance of Visual Simultaneous Localization and Mapping (VSLAM) algorithm to monitor the buoy heave motion. The VSLAM algorithm fuses a buoy onboard stereo vision recordings with an embedded Inertial Measurement Unit (IMU) to estimate the 6-DOF of a rigid body. The Buoy motion VSLAM measurements are verified by a KLT tracking algorithm implemented on the video recordings of the stationary side camera. The open-sea experiment is implemented in Lake Somerville, Texas. The stereo vision system is installed to measure the water surface elevation and directional spectrum of the wind generated irregular waves. The open-sea wave measurements by the stereo vision are verified by a Sofar commercial wave buoys deployed in the testing location.

Stereo vision based pose estimation for hybrid Additive Manufacturing with Laser Powder Bed Fusion

Laser Powder Bed Fusion (PBF-LB/M) is a resource-efficient Additive Manufacturing (AM) technology adopted for creation of intricate metallic components. To increase the production rate, combination of PBF-LB/M and conventional manufacturing is crucial. Currently, the placing of AM structures onto conventional pre-formed parts is lacking in accuracy, resulting in failed prints. To enhance the accuracy of PBF-LB/M for Hybrid Additive Manufacturing, it is essential to precisely determine the 6D pose of the base component on the substrate plate to minimize any misalignment between the component and the AM build. To tackle this challenge, a stereo vision system has been developed. This system can be easily integrated within the closed process chamber of many PBF-LB/M machine, enabling precise measurement of the base component’s position relative to the machine coordinate frame leading to a precise alignment of the part to the additive structures.