Stereo Vision System Research Articles

Falling styles of perforated disks

We experimentally investigate the dynamics of thin perforated disks freely falling under gravity in still water. With a stereo vision system, three-dimensional paths are reconstructed. Three motions corresponding to different paths are identified, namely spiral motion, Hula-Hoop motion and helical precession motion. Compared with previous studies, it is found that the lateral displacements of perforated disks are much smaller than those of the other regular particles under the same number of Re and I*. Flow visualizations for freely falling perforated disks are performed using sodium fluorescein dye, and significant differences can be found in the wake. Hairpin vortices can be clearly distinguished in Hula-Hoop motion, which are hardly found in the other motions. Due to the presence of vortices shed from holes, there are complex vortex interactions in the wake, such as adjacent vortices fusion and the secondary induced vortex. Mechanisms behind different wake patterns are analyzed. And different three-dimensional paths under the same I* and Re are observed due to different wake patterns.

Extraction of 3D distribution of potato plant CWSI based on thermal infrared image and binocular stereovision system.

As the largest component of crops, water has an important impact on the growth and development of crops. Timely, rapid, continuous, and non-destructive detection of crop water stress status is crucial for crop water-saving irrigation, production, and breeding. Indices based on leaf or canopy temperature acquired by thermal imaging are widely used for crop water stress diagnosis. However, most studies fail to achieve high-throughput, continuous water stress detection and mostly focus on two-dimension measurements. This study developed a low-cost three-dimension (3D) motion robotic system, which is equipped with a designed 3D imaging system to automatically collect potato plant data, including thermal and binocular RGB data. A method is developed to obtain 3D plant fusion point cloud with depth, temperature, and RGB color information using the acquired thermal and binocular RGB data. Firstly, the developed system is used to automatically collect the data of the potato plants in the scene. Secondly, the collected data was processed, and the green canopy was extracted from the color image, which is convenient for the speeded-up robust features algorithm to detect more effective matching features. Photogrammetry combined with structural similarity index was applied to calculate the optimal homography transform matrix between thermal and color images and used for image registration. Thirdly, based on the registration of the two images, 3D reconstruction was carried out using binocular stereo vision technology to generate the original 3D point cloud with temperature information. The original 3D point cloud data were further processed through canopy extraction, denoising, and k-means based temperature clustering steps to optimize the data. Finally, the crop water stress index (CWSI) of each point and average CWSI in the canopy were calculated, and its daily variation and influencing factors were analyzed in combination with environmental parameters. The developed system and the proposed method can effectively detect the water stress status of potato plants in 3D, which can provide support for analyzing the differences in the three-dimensional distribution and spatial and temporal variation patterns of CWSI in potato.

Underwater Binocular Meta-lens

Underwater optics in all-aquatic environments is vital for environmental management, biogeochemistry, phytoplankton ecology, benthic processes, global change, etc. Many optical techniques of observational systems for underwater sensing, imaging, and applications have been developed. For the demands of compact, miniaturized, portable, lightweight, and low-energy consumption, a novel underwater binocular depth-sensing and imaging meta-optic device is developed and reported here. A GaN binocular meta-lens is specifically designed and fabricated to demonstrate underwater stereo vision and depth sensing. The diameter of each meta-lens is 2.6 mm, and the measured distance between the two meta-lens centers is 4.04 mm. The advantage of our binocular meta-lens is no need of distortion correction or camera calibration, which is necessary for traditional two-camera stereo vision systems. Based on the experimental results, we developed the generalized depth calculation formula for all-size binocular vision systems. With deep-learning support, this stereo vision system can realize the fast underwater object’s depth and image computation for real-time processing capability. Our artificial intelligent imaging results show that depth measurement accuracy is down to 50 μm. Besides the aberration-free advantage of flat meta-optic components, the intrinsic superhydrophobicity properties of our nanostructured GaN meta-lens enable an antiadhesion, stain-resistant, and self-cleaning novel underwater imaging device. This stereo vision binocular meta-lens will significantly benefit underwater micro/nanorobots, autonomous submarines, machine vision in the ocean, marine ecological surveys, etc.

A comparative evaluation of 3D geometries of scenes estimated using factor graph based disparity estimation algorithms

Passive stereo vision systems are useful for estimating 3D geometries from digital images similar to the human biological system. In general, two cameras are situated at a known distance from the object and simultaneously capture images of the same scene from different views. This paper presents a comparative evaluation of 3D geometries of scenes estimated by three disparity estimation algorithms, namely the hybrid stereo matching algorithm (HCS), factor graph-based stereo matching algorithm (FGS), and a multi-resolution FGS algorithm (MR-FGS). Comparative studies were conducted using our stereo imaging system as well as hand-held, consumer-market digital cameras and camera phones of a variety of makes/models. Based on our experimental results, the factor graph algorithm (FGS) and multi-resolution factor graph algorithm (MR-FGS) result in a higher level of 3D reconstruction accuracy than the HCS algorithm. When compared with the FGS algorithm, MR-FGS provides a significant improvement in the disparity contrast along the depth boundaries and minimal depth discontinuities.

Stereo Vision Controlled Variable Rate Sprayer for Specialty Crops: Part II. Sprayer Development and Performance Evaluation

Highlights A real time stereo vision controlled variable rate sprayer for specialty crops was developed. The stereo vision system of the sprayer detected outdoor trees with similar canopy profiles under travel speeds ranging from 3.2 to 8 km h-1. Canopy volume measurements of the sprayer were impacted by lateral distances between the sprayer and the tree center and travel speeds. The sprayer required less than 200 ms from tree canopy detection to spray decisions. The sprayer achieved spray volume reductions from 72.6% to 80.5% compared to constant rate spray application. Abstract. A real time variable rate sprayer controlled by a stereo vision system was developed to increase the accuracy of spray applications and reduce the use of crop protection products. The sprayer was designed to detect tree canopies and calculate its volume using depth images from the stereo vision system, and discharge corresponding spray volumes every 200 ms through the embedded software in the graphical user interface. The sprayer was evaluated in an apple orchard at different travel speeds (3.2 to 8.0 km h-1) for its performance in detecting canopy and measuring its volume. In addition, spray volume, deposition, and coverage of the variable rate application of the sprayer were evaluated against a constant rate application. Test results showed that the sprayer detected visually similar tree canopies during the evaluations, although its canopy volume measurements deviated from manually measured canopy volume from 0.11 to 0.83 m3 due to lateral position changes of the sprayer. The sprayer adjusted duty cycles of pulse width modulated valves to accurately spray the intended volume for detected canopies (0.073 to 0.083 L m-3) and only used spray volumes of 19.5% to 26.7% compared to a constant rate spray application (338 L ha-1). The constant rate spray application generally had more spray deposition and coverage in tree canopies than the variable rate sprayer, as expected since its spray volume was approximately 3.7 times higher. However, the mean spray depositions from the constant rate spray application were significantly varied (p=0.05) by tree sizes, while the variable rate spray application achieved statistically equivalent mean spray depositions regardless of tree sizes. The stereo vision controlled sprayer offers a cost-effective real-time variable rate spray option for growers with the potential to perform other tasks by using image processing algorithms while applying crop protection products. Keywords: Automation, Canopy volume, Crop protection, Depth image, Orchard, Precision agriculture, Real-time application.

Road to Repair (R2R): An Afrocentric Sensor-Based Solution to Enhanced Road Maintenance

Potholes are one of the most important issues in African road-networks. They pose a major threat to mobility and, with time, cause accelerated degradation of the underlying road infrastructure as well as extensive vehicle damage. To address the need for improved infrastructure management, an advanced data gathering solution is required. This paper presents one such solution. The pothole detection, classification and logging (PDCL) system is under active development by Sensorit (Pty) Ltd in collaboration with the University of Cape Town (UCT) Radar Remote Sensing Group (RRSG). This system represents the initial step in Sensorit’s modular approach to producing fully autonomous vehicles for African markets. In this paper, an overview of the PDCL system is presented and early results are shown. The efficacy of the system’s unique combination of active infrared stereo vision and mmWave frequency-modulated continuous-wave (FMCW) radar sensors is explored. Under various experimental conditions, range-Doppler maps (RDMs) produced by the radar were unable to provide meaningful pothole detections. In contrast, processed depth maps produced by the stereo vision system are demonstrated to successfully detect even shallow potholes.

Особенности вычисления взаимно корреляционной функции при определении диспарантности в стереоскопической системе технического зрения

This article considers the methods of calculating the cross-correlation function in determining the disparity from the images of a stereo pair formed by a stereo vision system. When determining the disparity, it is necessary to find a point on the second image of the stereo pair at a given point of the first image, which can be done by finding the maximum of the cross-correlation function. The possibility of reducing the amount of calculations in the time domain when using a non-positional calculus system, also called a residue number system, is considered. As a result of modeling based on a system of positional calculus and a residue number system using similar-shaped functions shifted along the X axis relative to each other, as well as using real stereo images, the coincidence of the main maxima of the cross-correlation function is shown with a significant reduction in the amount of calculations.

Visual Odometry With Point and Line Features Based on Underground Tunnel Environment

Compared with the interior environment, there exists a lot of noise and dust in an underground tunnel, and the light is unstable, which is difficult to track by the direct method. Moreover, errors in line segment projection and line feature drift under the influence of light can lead to significant deviations in the odometry. Consequently, to improve the accuracy and robustness of visual odometry, a point-line feature stereo visual odometry system is proposed in this paper. The system combines ORB features and LSD line features; using the angle relationship of line projection, we propose a new method for calculating the reprojection error of line features, reconstruct the reprojection model based on line features, and construct a new reprojection error model based on point-line features, which adds an angle constraint to the reprojection of line features and solves the instability caused by line projection error. It is shown in our experiments on the KITTI, New College dataset, that the translation error of our system is reduced by about 40% on average compared to PLVO, with a reduction in relative positioning error. Experiments in the hallway and underground tunnel environments have shown that the maximum positioning error of our system has been reduced by 75% in hallways and by 56.7% in an underground tunnel. Therefore, our algorithm effectively improves the localization accuracy and is more advantageous in low-texture environments.

Cell-Based Refinement Processor Utilizing Disparity Characteristics of Road Environment for SGM-Based Stereo Vision Systems

Cell-Based Refinement Processor Utilizing Disparity Characteristics of Road Environment for SGM-Based Stereo Vision Systems

Impact factors on rockfall movement using high-speed camera binocular stereo vision technology

Rockfall is a frequent and serious geological hazard in mountainous areas worldwide. It is of great significance to fully understand rockfall movement for disaster assessment and prevention. In this paper, based on the principle of binocular stereo vision, a filed experimental monitoring technology was developed to conduct the rockfall movement experiment research. Through the orthogonal experimental method, taking the block velocities obtained by binocular stereo vision system as a reference index, the impact factors such as the slope angles and the falling heights, shapes and masses of blocks on rockfall movement are studied. The lateral deviations and jumping heights of different blocks when they are released from the same falling height and moving along the same slope are presented. The results show that slope angle is the most key factor affecting block movement, while the falling heights, shapes and masses of blocks are the secondary factors. Moreover, the experimental methods and results of this paper can lay a foundation for the further study of rockfall movement characteristics.

Automatic mileage positioning for road inspection using binocular stereo vision system and global navigation satellite system

Road inspections ensure an adequate infrastructure operation and traffic safety, but the widely used positioning modes of road inspection have large deviations with road mileage. This paper describes an automatic method of road mileage positioning for road inspection. It integrates binocular stereo vision system, Global Navigation Satellite System (GNSS), objection detection algorithm, and Optical Character Recognition into road mileage positioning. A validation test result shows the average deviation of the mileage positioning is 4.06 m, which meets the requirement of road inspection. The significance of this paper consists of improving the automatic method of mileage positioning with the rapid disposal of abnormal targets on roads. In future research, the targets of road inspection, such as pavement potholes, patch, wearing lane markings, will be detected by objection detection algorithms and be positioned by the mileage positioning method.

VDI/VDE 2634–1 performance evaluation tests and systematic errors in passive stereo vision systems

In this paper, we explore the sensitivity of the length tests described in the VDI/VDE 2634-1 guideline to systematic errors in a passive stereo vision system. The results reported here are based on a stereo vision system set up on the table of a Cartesian coordinate measuring machine (CMM) and simulations that match the experimental conditions. The stereo vision system is calibrated using a checkerboard pattern resulting in estimates for 26 camera model parameters. Errors in these model parameters cause errors in the triangulated 3D points, thus resulting in length errors evaluated from those points. In the first part of this study, we consider a grid of points in the measurement volume of the stereo vision system and simulate the effect of induced systematic errors in each model parameter to determine the resulting error in the length for all lines (i.e., between all pairs of points) in that grid. We identify the lines that provide the maximum error in the length for each model parameter. Thus, we identify a collection of lines, which we refer to as model-based lines, that provide maximum sensitivity to the model parameters. Also, through simulations, we assess whether the lines defined in the VDI/VDE guideline are sufficient to detect the systematic errors in the stereo vision system. In the second part of this study, we evaluate the performance of the stereo vision system, i.e., assess the effect of errors in the model parameters, by using the CMM as the reference. For this purpose, the CMM is commanded to move to the set of points in the rectangular grid previously considered in the simulations with the stereo vision system tracking a small checkerboard target installed on the ram of the CMM. We compare the VDI/VDE and model-based lines in terms of the experimentally obtained length errors. The studies show that selection of lines based on the model (determined through simulations and verified experimentally) is advantageous because of the superior ability in detecting systematic errors in stereo vision systems.

An Automatic Recognition Method for Fish Species and Length Using an Underwater Stereo Vision System

Developing new methods to detect biomass information on freshwater fish in farm conditions enables the creation of decision bases for precision feeding. In this study, an approach based on Keypoints R-CNN is presented to identify species and measure length automatically using an underwater stereo vision system. To enhance the model’s robustness, stochastic enhancement is performed on image datasets. For further promotion of the features extraction capability of the backbone network, an attention module is integrated into the ResNeXt50 network. Concurrently, the feature pyramid network (FPN) is replaced by an improved path aggregation network (I-PANet) to achieve a greater fusion of effective feature maps. Compared to the original model, the mAP of the improved one in object and key point detection tasks increases by 4.55% and 2.38%, respectively, with a small increase in the number of model parameters. In addition, a new algorithm is introduced for matching the detection results of neural networks. On the foundation of the above contents, coordinates of head and tail points in stereo images as well as fish species can be obtained rapidly and accurately. A 3D reconstruction of the fish head and tail points is performed utilizing the calibration parameters and projection matrix of the stereo camera. The estimated length of the fish is acquired by calculating the Euclidean distance between two points. Finally, the precision of the proposed approach proved to be acceptable for five kinds of common freshwater fish. The accuracy of species identification exceeds 94%, and the relative errors of length measurement are less than 10%. In summary, this method can be utilized to help aquaculture farmers efficiently collect real-time information about fish length.

A non-iterative calibration method for the extrinsic parameters of binocular stereo vision considering the line constraints

The calibration accuracy of the binocular stereo vision system is important for precision measurement. The extrinsic parameters (structure parameters), which describe the relative position of the two cameras, need to be optimized before measurement. The optimization of extrinsic parameters is accomplished through iterations of minimizing an objective function. It is hard to decide the optimum objective function between various objective functions. The calibration accuracy is decreased when the objective function is incorrect, which can result in excessive optimization or fall into a local minimum (ambiguous solution), notably for planar targets. In this paper, we proposed a non-iterative calibration method for the extrinsic parameters of binocular stereo vision considering the line constraints of a planar target. Each line formed by every two calibration points can provide two independent constraints. Additional equations for solving the extrinsic parameters can be obtained using these constraints. We added these equations to the equations system established by the homography of the target plane and image plane. The extrinsic parameters of the binocular stereo vision system can be solved non-iteratively. Using the parameters calibrated by our method, greater improvements in the accuracy and stability of measurements occurred, compared to other existing methods. In the −45° to +45° measuring range, the maximum measurement error is no more than 0.037° and the average measurement error is no more than 0.022°. In the 0 to 28 mm measuring range, the maximum measurement error is no more than 0.039 mm and the average measurement error is no more than 0.012 mm. The maximum length measurement error is no more than 0.038 mm and the average length measurement error is no more than 0.011 mm while measuring a 240 × 180 mm2 target.

Integration of stereo vision system calibration and kinematic calibration for an autonomous kiwifruit harvesting system

Stereo vision system and manipulator are two major components of an autonomous fruit harvesting system. In order to raise the fruit-harvesting rate, stereo vision system calibration and kinematic calibration are two significant processes to improve the positional accuracy of the system. This article reviews the mathematics of these two calibration processes and presents an integrated approach for acquiring calibration data and calibrating both components of an autonomous kiwifruit harvesting system. The calibrated harvesting system yields good positional accuracy in the laboratory tests, especially in harvesting individual kiwifruit. However, the performance is not in line with the outcomes in the orchard field tests due to the cluster growing style of kiwifruit. In the orchard test, the calibrations reduce the fruit drop rate but it does not impressively raise the fruit harvesting rate. Most of the fruit in the clusters remain in the canopy due to the invisibility of the stereo vision system. After analyzing the existing stereo vision system, a future visual sensing system research direction for an autonomous fruit harvesting system is justified.

A Lightweight Stereo Visual Odometry System for Navigation of Autonomous Vehicles in Low-Light Conditions

Localization of vehicles in a 3D environment is a basic task in autonomous driving. In the low-light environments, it is difficult to navigate independently using a visual odometry for autonomous driving. The main reason for this challenge is the blurred images in the scenes with insufficient illumination. Although numerous works focused on this issue, it still has a number of inherent drawbacks. In this paper, we propose a lightweight stereo visual odometry system for navigation of autonomous vehicles in low-light situations. Contrary to the existing recovery methods, we aim to divide the captured image into the illumination image as well as the reflectance image and only estimate the illumination one, where the enhanced map of the low-light image is acquired by using the retinex theory. In addition, we further utilize a simplified and rapid feature detection scheme, which reduces the computation time by about 85% but maintaining the matching accuracy similar to that of ORB features. Finally, the experiments show that our average memory consumption of our proposed method is much less than the conventional algorithm.

Development of Multi-Robotic Arm System for Sorting System Using Computer Vision

This paper develops a multi-robotic arm system and a stereo vision system to sort objects in the right position according to size and shape attributes. The robotic arm system consists of one master and three slave robots associated with three conveyor belts. Each robotic arm is controlled by a robot controller based on a microcontroller. A master controller is used for the vision system and communicating with slave robotic arms using the Modbus RTU protocol through an RS485 serial interface. The stereo vision system is built to determine the 3D coordinates of the object. Instead of rebuilding the entire disparity map, which is computationally expensive, the centroids of the objects in the two images are calculated to determine the depth value. After that, we can calculate the 3D coordinates of the object by using the formula of the pinhole camera model. Objects are picked up and placed on a conveyor branch according to their shape. The conveyor transports the object to the location of the slave robot. Based on the size attribute that the slave robot receives from the master, the object is picked and placed in the right position. Experiment results reveal the effectiveness of the system. The system can be used in industrial processes to reduce the required time and improve the performance of the production line.

Identification of a Mathematical Model for the Transformation of Images for Stereo Correspondence Measurements of Mining Equipment

The stereometry of the working units of mining machines is optimized at the design stage, in terms of selected criteria based on computer simulations of the mining process. The recovered bodies of cutting heads or drums used in manufacturing are regenerated in the overhaul process. Ensuring that their dimensions comply with the nominal ones is labor-intensive and raises production costs. However, deviations of these components from the nominal shape can make it difficult to position the pick holders (which can cause collisions) or make welding them impossible (which results from too large a distance between the pick holders’ base and the side surface of the working unit). This applies especially to robotic technologies. By utilizing automatic (online) measurements of the distribution of the actual distances of the pick holders’ bases from the side surface of the working unit (taken during their positioning using a robot), it is possible to correct their positions without changing the setting of the pick itself. This measurement can be carried out using the non-contact stereovision method. This paper presents a method of matching raster images obtained from a stereovision system installed on an experimental robotic station. The results are presented of the numerical research carried out to select a mathematical model of the geometric transformation of images, for ensuring the effectiveness of the developed method.

A Meta-Device for Intelligent Depth Perception.

The optical illusion affects depth-sensing due to the limited and specific light-field information acquired by single-lens imaging. The incomplete depth information or visual deception would cause cognitive errors. To resolve this problem, an intelligent and compact depth-sensing meta-device that is miniaturized, integrated, and applicable for diverse scenes in all light levels is demonstrated. The compact and multifunction stereo vision system adopts an array with 3600 achromatic meta-lenses and a size of 1.2 × 1.2 mm2 to measure the depth over a 30cm range with deep-learning support. The meta-lens array can act as multiple imaging lenses to collect light field information. It can also work with a light source as an active optical device to project a structured light. The meta-lens array can serve as the core functional component of a light-field imaging system under bright conditions or a structured-light projection system in the dark. The depth information in both ways can be analyzed and extracted by the convolutional neural network. This work provides a new avenue for the applications such as autonomous driving, machine vision, human-computer interaction, augmented reality, biometric identification, etc.

Automatic Cutting System Design of Robot Hand Based on Stereo Vision

At present, the pouring riser of valve castings is mainly removed by manual cutting, polluting the environment and causing harm to the human body with a low efficiency. Therefore, an automatic cutting pouring riser method using the stereo vision system and manipulator is proposed. The relative position of the valve casting and the end of the manipulator is obtained through the position transformation of the valve casting coordinate system, the manipulator end of the coordinate system, and the camera coordinate system. The spatial motion trajectory of the manipulator is planned, implementing automatic pouring riser cutting of the same valve casting with the same pose. The experimental results show that the position deviation and the angle deviation of the repetitive positioning accuracy and the random positioning accuracy of the visual system are within ±1 mm and ±1°; in the pouring riser cutting test, the maximum deviation between the actual cutting trajectory and the theoretical cutting trajectory is 3 mm. In summary, the method shows a good reliability and could meet the requirements of cutting accuracy.