Binocular Stereo Research Articles

Measurement and optimization method for aero-engine rotors based on binocular multi-line laser sensing and virtual assembly

During the assembly of aero-engine rotors, eccentricity and tilt errors along the radial and axial direction result in uneven loading along the mating surface, which significantly impacts the operational lifespan of the rotor. This paper proposes a three-dimensional contour measurement system based on binocular multi-line laser sensing and conducts an aircraft engine rotor stacking virtual assembly technology system, to response the practical requirements of aircraft engine rotor unit production assembly engineering. We combine multi-line structured light active projection device with the binocular stereo vision model to construct the contour measurement system, which enhances the surface features of the projected object by adhering to the common constraint of structured line and stereo vision. Furthermore, the error propagation model for multi-stage rotor assembly to investigate the mechanism of error impact is established aiming to facilitate quantitative research on virtual assembly mechanisms. By manipulating the rotor phases and evaluating the coaxiality under various conditions, effective prediction of phase variation can be achieved, the multi-objective benchmark virtual assembly method based on the error propagation model of multi-stage rotor coaxiality is presented. The experimental results indicate that the proposed measurement system achieves the maximum disparity between the spatial data points procured by the proposed measurement system and the predetermined value of 2.1 μm, and the maximum disparity between the target plane’s average movement distance and the preset value of 3.7 μm. The precision provided by this system effectively meets the objective of assembly phase optimization for multi-stage rotors. In the empirical experiment conducted on the three-stage rotors, both systems yielded identical optimized phases: 0° for the first-stage rotor, 270° for the second-stage rotor, and 120° for the third-stage rotor, thereby validating the efficacy of the proposed measurement and optimization method for aero-engine multi-stage rotors with virtual assembly.

System Structural Error Analysis in Binocular Vision Measurement Systems

A binocular stereo vision measurement system is widely used in fields such as industrial inspection and marine engineering due to its high accuracy, low cost, and ease of deployment. An unreasonable structural design can lead to difficulties in image matching and inaccuracies in depth computation during subsequent processing, thereby limiting the system’s performance and applicability. This paper establishes a systemic error analysis model to enable the validation of changes in structural parameters on the performance of the binocular vision measurement. Specifically, the impact of structural parameters such as baseline distance and object distance on measurement error is analyzed. Extensive experiments reveal that when the ratio of baseline length to object distance is between 1 and 1.5, and the angle between the baseline and the optical axis is between 30 and 40 degrees, the system measurement error is minimized. The experimental conclusions provide guidance for subsequent measurement system research and parameter design.

Hemispherical Retina Emulated by Plasmonic Optoelectronic Memristors with All-Optical Modulation for Neuromorphic Stereo Vision.

Binocular stereo vision relies on imaging disparity between two hemispherical retinas, which is essential to acquire image information in three dimensional environment. Therefore, retinomorphic electronics with structural and functional similarities to biological eyes are always highly desired to develop stereo vision perception system. In this work, a hemispherical optoelectronic memristor array based on Ag-TiO2 nanoclusters/sodium alginate film is developed to realize binocular stereo vision. All-optical modulation induced by plasmonic thermal effect and optical excitation in Ag-TiO2 nanoclusters is exploited to realize in-pixel image sensing and storage. Wide field of view (FOV) and spatial angle detection are experimentally demonstrated owing to the device arrangement and incident-angle-dependent characteristics in hemispherical geometry. Furthermore, depth perception and motion detection based on binocular disparity have been realized by constructing two retinomorphic memristive arrays. The results demonstrated in this work provide a promising strategy to develop all-optically controlled memristor and promote the future development of binocular vision system with in-sensor architecture.

Enhanced precision in greenhouse tomato recognition and localization

The core challenge in realizing automatic tomato harvesting in greenhouse environments lies in the precise identification and localization of the fruits. This paper introduces a comprehensive approach based on an improved YOLOv5 detection algorithm and optimized binocular stereo vision technology. Firstly, by introducing the C3-Transformer Encoder (CTM) structure and Bidirectional Feature Pyramid Network (Bi-FPN), this study enhanced the model’s ability to recognize tomatoes, especially under complex backgrounds and occlusion conditions. After field testing, the mAP50 accuracy reached 97.1%, an increase of 1.2 percentage points, enhancing detection precision. In addition, the ZED binocular camera was used, and the census stereo matching algorithm was optimized, significantly reducing disparity errors, thereby improving the accuracy of depth information. This allows the model to accurately calculate the three-dimensional spatial position of tomatoes obscured by branches and leaves, greatly improving the efficiency of the harvesting robot. Through field debugging verification with the harvesting robot, the method proposed in this study has shown high accuracy and reliability in the recognition and localization of tomatoes in complex greenhouse environments.

Multi-line structured light stripes clustering based on a custom iterative window

ObjectiveThis paper proposes a multi-line structured light stripe clustering method based on a custom iterative window to address the issue of stripes confusion caused by stripe breakage and noise points during the multi-line laser stripe matching process in binocular stereo vision. MethodsFirstly, the paper segments the measured object from the carrier plane. Secondly, the extracted centerlines are clustered using the custom iterative window. Finally, the stripes are numbered one by one after clustering the left and right images using barycentric coordinates of the stripes, based on the property of sequential consistency constraint, to complete the stereo matching of the stripes with the same number. ResultsThe experimental results show that the method can effectively cluster the centerline stripes on the measured object and solve the problem of stripes mis-corresponding when stereo matching of multi-line structured light stripes.

Trinocular 360-degree stereo for accurate all-round 3D reconstruction considering uncertainty

This research proposes a method for accurate all-round 3D reconstruction of an indoor environment in one-shot using a system of trinocular 360-degree cameras. Binocular 360-degree stereo is unable to reconstruct in all directions due to lack of disparity along epipolar directions. Thus, a third camera along a perpendicular epipolar direction is introduced to cover for this, making the system trinocular. However, previous works with trinocular stereo did not adequately take into account the uncertainty of disparity estimation and geometric constraints around 3D reconstruction. Therefore, we propose a geometric optimization scheme considering disparity estimation uncertainty and show that this results in both higher accuracy and lesser outliers along epipolar directions, in both simulated and real environments.

On the Parameter Calibration Method for Dual-Spectral Triocular Camera

The dual-spectrum triocular camera system composed of a binocular visible light camera and mid-infrared camera is used for high-precision thermal fault detection and thermal field reconstruction of industrial equipment. The realization of its function depends on the high-precision camera parameter calibration. The difficulty lies in how to realize the infrared camera calibration quickly and improve the parameter accuracy of multi-lens camera. In this paper, according to the characteristics of the dual-spectral triocular camera system, the theoretical model is constructed, and the circular asymmetric calibration plate under infrared supplementary light is selected as the calibration object through experiments. A method for combining infrared image adaptive histogram enhancement and binarization processing based on the Sauvola algorithm is proposed to effectively calibrate the infrared camera. A global parameter optimization method based on traditional passive vision binocular stereo calibration is proposed. The optimization of experimental parameters finds the reprojection error value is 0.16, which meets the demand of high-precision calibration and solves the problem of difficult-to-calibrate weak image texture of infrared camera and the calibration accuracy of the camera under different imaging capabilities.

Multi-Scale Binocular Stereo Matching Based on Semantic Association

Multi-Scale Binocular Stereo Matching Based on Semantic Association

Multiple object characterization of recyclable domestic waste using binocular stereo vision

Recycling and recovery of domestic waste pose significant challenges in terms of accurate object positioning and ranging, areas where conventional methods, heavily reliant on monocular image processing, often fail due to limited distance data and reduced precision. This study introduces an innovative strategy for the localization and ranging of recyclable waste objects, leveraging the NanoDet-Plus model within a binocular stereo vision context. A dataset, the Binocular Recyclable Domestic Waste Dataset (BRDWD), was constructed, with transfer learning executed via the pretrained MULTI-TRASH monocular dataset. Procedures encompassed stereo camera calibration and rectification to address geometric distortions, while the integration of the Semi-Global Block Matching (SGBM) algorithm supplied disparity details for triangulation-based distance computations. Enhancements to the NanoDet-Plus model led to improved target accuracy, evidenced by a mean average precision at a threshold of 0.5 (mAP@0.5) reaching 91.88% and a detection speed of 15 frames per second (FPS). Assessments verified a ranging error below 5% within the 0.5 to 1.2-meter span, aligning well with the requirements of typical recyclable waste sorting and recycling situations. Additionally, a human-machine interface was developed utilizing Pyside6, enabling image uploads, real-time result visualization, and interactive process control, thereby furnishing pivotal advancements for the automation and intellectualization of waste categorization and recycling.

A Binocular Color Line-Scanning Stereo Vision System for Heavy Rail Surface Detection and Correction Method of Motion Distortion.

Thanks to the line-scanning camera, the measurement method based on line-scanning stereo vision has high optical accuracy, data transmission efficiency, and a wide field of vision. It is more suitable for continuous operation and high-speed transmission of industrial product detection sites. However, the one-dimensional imaging characteristics of the line-scanning camera cause motion distortion during image data acquisition, which directly affects the accuracy of detection. Effectively reducing the influence of motion distortion is the primary problem to ensure detection accuracy. To obtain the two-dimensional color image and three-dimensional contour data of the heavy rail surface at the same time, a binocular color line-scanning stereo vision system is designed to collect the heavy rail surface data combined with the bright field illumination of the symmetrical linear light source. Aiming at the image motion distortion caused by system installation error and collaborative acquisition frame rate mismatch, this paper uses the checkerboard target and two-step cubature Kalman filter algorithm to solve the nonlinear parameters in the motion distortion model, estimate the real motion, and correct the image information. The experiments show that the accuracy of the data contained in the image is improved by 57.3% after correction.

A lightweight detection method of pavement potholes based on binocular stereo vision and deep learning

The identification of pavement defects is crucial and must be done efficiently, accurately, and cost-effectively. A commercial-grade sports camera and mobile vehicle work together to detect pavement potholes in a lightweight manner. A dataset of 6,186 images with a resolution of 1500 by 1500 pixels has been constructed based on vehicle-mounted tilted images. Object detection and image segmentation are performed using a single-model MASK R-CNN, and the principle of binocular stereo vision is employed to reconstruct the three-dimensional (3D) structures of potholes. A novel methodology is introduced to calculate the 3D feature parameters of potholes, enabling the precise determination of the pavement pothole damage ratio. An average accuracy of 98% for detection and 94% for segmentation is achieved with the Mask R-CNN model. The proposed algorithms achieve an average DR calculation accuracy of 82%, facilitating precise identification of surface irregularities. The paper presents an intelligent approach for detecting pavement potholes.

Visual training for the treatment of retinal photoreceptor inner and outer segment connection (IS/OS) fracture: A case report.

The inner segments and outer segments (IS/OS) of the retinal photoreceptors are the areas that receive light signals and are the most initial sites for generating visual impulses, and the integrity of the IS/OS has a direct impact on visual sensitivity. We performed OCT on a 6-year-old child with vision loss and found that the cause of his vision loss was a retinal IS/OS fracture, and the child underwent some treatments to improve microcirculation and nourish the retina at a higher-level hospital, but his vision never improved. Our examination of this child revealed that this child not only had decreased visual acuity, but also hypermetropia, but his near stereopsis was normal. The symptoms were similar to those of amblyopia, so we tried to use visual training as a treatment. First, 6 sessions of fine visual stimulation were given, followed by 3 sessions of accommodation training, and we followed the 4 stages of accommodation training: perception of accommodation, amplitude of accommodation, sensitivity of accommodation, and autonomic accommodation. After 9 consecutive visual training sessions, the child's visual acuity was stabilized at 0.6, and then we added eye movement training, and after the child's visual acuity was improved to 0.7, we suppressed the visual acuity of the left eye to 0.6, so as to make the visual acuity of both eyes similar, which would promote the establishment of binocular stereo vision, and then we carried out 9 more visual training sessions, and the patient's visual acuity was stabilized at 0.8 gradually. OCT review showed that the child's retinal IS/OS fracture was basically closed. In conclusion, our study found that visual training can restore visual acuity in children with monocular IS/OS fracture and also promote repair of IS/OS fracture, which increases our understanding and knowledge of the treatment of retinal IS/OS fracture, and this case may provide some lessons for the treatment of retinal IS/OS fracture in children. We hope to have more samples of retinal IS/OS fracture in the future to evaluate the efficacy of visual training for retinal IS/OS fracture.

Robust distance measurement using illumination map estimation and MAHNet in underground coal mines

An effective binocular stereo distance measurement method is proposed to address challenges posed by low brightness and weak texture of images captured in underground coal mines for the machine vision method. This approach is based on illumination map estimation and the MobileNetV3 attention hourglass stereo matching network (MAHNet) model. First, a binocular stereo vision system is established in which infrared LEDs are uniformly distributed on both sides of the belt conveyor bracket as visual feature points. Second, images are preprocessed using illumination map estimation, and the optimization of inhomogeneous brightness image enhancement is achieved by adopting adaptive Gamma correction. Third, the YOLOv5 target detection network and Gaussian fitting fusion algorithm are utilized to detect infrared LED feature points. Fourth, the MAHNet model is employed to generate the cost volume and perform disparity regression, resulting in the acquisition of accurate disparity images. Finally, triangulation is applied to determine the depth of feature points. The experimental results of distance measurement demonstrate that an average relative ranging accuracy of 1.52% within the range of 50.0 cm to 250.0 cm can be achieved by the optimized method, thereby validating the effectiveness of this binocular distance measurement method in underground coal mines.

Estimation of rock joint roughness using binocular stereo vision

To quickly and cost-effectively evaluate the joint roughness coefficient (JRC) of the rock joints, a technique based on binocular stereo vision was developed to reconstruct the three-dimensional (3D) model. After camera calibration, only two images of rock joints were needed to be captured for 3D reconstruction. Then, the generated point clouds were processed to extract the geometric information, mainly focusing on the JRC calculation. A comparison was made with the point clouds obtained from laser scanning to verify the proposed method, and the possibility of the method in large-scale modeling was also explored. The results show that the proposed technique for 3D reconstruction of rock joints can meet the basic accuracy requirements in laboratory work, and it has a good consistency in roughness estimation with laser scanning. Compared to traditional photogrammetry, this method significantly simplifies the working procedure to improve measurement efficiency and reduce the project cost.

Non-Contact Tilapia Mass Estimation Method Based on Underwater Binocular Vision

The non-destructive measurement of fish is an important link in intelligent aquaculture, and realizing the accurate estimation of fish mass is the key to the stable operation of this link. Taking tilapia as the object, this study proposes an underwater tilapia mass estimation method, which can accurately estimate the mass of free-swimming tilapia under non-contact conditions. First, image enhancement is performed on the original image, and the depth image is obtained by correcting and stereo matching the enhanced image using binocular stereo vision technology. And the fish body is segmented by an SAM model. Then, the segmented fish body is labeled with key points, thus realizing the 3D reconstruction of tilapia. Five mass estimation models are established based on the relationship between the body length and the mass of tilapia, so as to realize the mass estimation of tilapia. The results showed that the average relative errors of the method models were 5.34%~7.25%. The coefficient of determination of the final tilapia mass estimation with manual measurement was 0.99, and the average relative error was 5.90%. The improvement over existing deep learning methods is about 1.54%. This study will provide key technical support for the non-destructive measurement of tilapia, which is of great significance to the information management of aquaculture, the assessment of fish growth condition, and baiting control.

An empirical study of virtual museum based on dual-mode mixed visualization: the Sanxingdui bronzes

The design and implementation of virtual museums have evolved over decades, broadening the scope of traditional museum exhibitions and serving as an effective complement to physical museums. However, virtual visual effects have consistently fallen short of replicating the authentic exhibition experience and cannot substitute for physical museums. In this paper, we report a dual-mode mixed visualization system combining computational photography and binocular stereo vision techniques, and introduce an innovative virtual museum for Sanxingdui bronzes. This approach enables visitors to engage with the artifacts in a multifaceted manner-allowing for navigation, interaction, auditory guidance, and the examination of textural details as if handling the artifacts directly. Further analysis of participant responses in terms of visual cognition and aesthetic appreciation across different settings demonstrates that this dual-mode mixed visualization delivers not only a spatiotemporal experience consistent with the real world but also surreal visual effects. This immersive experience allows audiences to experience Sanxingdui art intimately, offering both high-resolution visuals and cost-effectiveness.

Measurement Of Spatial-Temporal Waves In The Laboratory Using Computer Vision Technolog

Given the complexity of ocean wave phenomena, physical modeling continues to be essential in various engineering and research applications. Wave height, a characteristic measured in nearly all projects, is commonly assessed using resistance/capacitance wave gauges. However, a significant limitation of these traditional wave gauges is their point-based nature, which restricts their ability to describe detailed spatial wave characteristics, especially during wave nonlinearity or when waveforms change. Furthermore, to reconstruct the 3D wave field, an array of wave gauges is often employed alongside an appropriate interpolation algorithm. Additionally, wave gauges may alter flow/wave conditions in specific scenarios. To address these limitations, various non-intrusive measurement techniques have been proposed. Gomit et al. (2022) classified these into stereoscopic, projection, and light-based methods, including LiDAR (Blenkinsopp et al., 2012), RGB-D cameras (Martínez-Aranda et al., 2018), and binocular stereo vision (Li et al., 2022). However, their applicability is constrained by factors such as optical setup complexity and device cost. With the rapid advancement of computer vision technology, it is possible to improve these issues and better describe temporal and spatial wave field variations using such techniques. This conference contribution details the reconstruction of the temporal and spatial 3D free water surface using photogrammetry techniques applied to images from consumer webcams. It also compares these results with measurements obtained from a wave gauge.

Wearable apparatus for correction of visual alignment under torsional strabismus.

A wearable optical apparatus that compensates for eye misalignment (strabismus) to correct for double vision (diplopia) is proposed. In contrast to prism lenses, commonly used to compensate for horizontal and/or vertical misalignment, the proposed approach is able to compensate for any combination of horizontal, vertical, and torsional misalignment. If the action of the extraocular muscles is compromised (e.g., by nerve damage), a patient may lose their ability to maintain visual alignment, negatively affecting their binocular fusion and stereo depth perception capability. Torsional misalignment cannot be mitigated by standard Fresnel prism lenses. Surgical procedures intended to correct torsional misalignment may be unpredictable. A wearable device able to rectify visual alignment and restore stereo depth perception without surgical intervention could potentially be of great value to people with strabismus. We propose a novel lightweight wearable optical device for visual alignment correction. The device comprises two mirrors and a Fresnel prism, arranged in such a way that together they rotationally shift the view seen by the affected eye horizontally, vertically, and torsionally. The extent of the alignment correction on each axis can be arbitrarily adjusted according to the patient's particular misalignment characteristics. The proposed approach was tested by computer simulation, and a prototype device was manufactured. The prototype device was tested by a strabismus patient exhibiting horizontal and torsional misalignment. In these tests, the device was found to function as intended, allowing the patient to enjoy binocular fusion and stereo depth perception while wearing the device for daily activities over a period of several months. The proposed device is effective in correcting arbitrary horizontal, vertical, and torsional misalignment of the eyes. The results of the initial testing performed are highly encouraging. Future study is warranted to formally assess the effectiveness of the device on multiple test patients.

Headland Identification and Ranging Method for Autonomous Agricultural Machines

Headland boundary identification and ranging are the key supporting technologies for the automatic driving of intelligent agricultural machinery, and they are also the basis for controlling operational behaviors such as autonomous turning and machine lifting. The complex, unstructured environments of farmland headlands render traditional image feature extraction methods less accurate and adaptable. This study utilizes deep learning and binocular vision technologies to develop a headland boundary identification and ranging system built upon the existing automatic guided tractor test platform. A headland image annotation dataset was constructed, and the MobileNetV3 network, notable for its compact model structure, was employed to achieve binary classification recognition of farmland and headland images. An improved MV3-DeeplabV3+ image segmentation network model, leveraging an attention mechanism, was constructed, achieving a high mean intersection over union (MIoU) value of 92.08% and enabling fast and accurate detection of headland boundaries. Following the detection of headland boundaries, binocular stereo vision technology was employed to measure the boundary distances. Field experiment results indicate that the system’s average relative errors of distance in ranging at distances of 25 m, 20 m, and 15 m are 6.72%, 4.80%, and 4.35%, respectively. This system is capable of meeting the real-time detection requirements for headland boundaries.

Design of a teleoperated and mixed reality-based electric power live line working robot

This paper presents a novel teleoperation force feedback approach combining graphic and haptic feedback for enhanced telepresence. Visual force feedback is modeled through a fitting process, with the force direction established through coordinate transformations. To construct operational environments, a new scene recognition and reconstruction technique is proposed, based on the principles of binocular stereovision. This method is primarily designed for live-line work scenarios, focusing on power line wires. It involves image preprocessing, target image recognition and extraction, binocular stereo measurements, generating depth point clouds, and ultimately reconstructing three-dimensional models of the target objects. Ultimately, a novel electric power live line working robot system with a strong sense of telepresence is achieved, based on teleoperation and mixed reality.