Stereo Vision Applications Research Articles

Application of Stereo Vision to Control the Movement of the Robot Arm Towards the Position of Red Chilies

The trend of decreasing young workers in the agricultural sector needs to be anticipated by developing intelligent machines known as agricultural robots. This research aims to apply a stereo vision system to control the movement of the robot's grip towards the 3D position of the red chili fruit. The stereo vision system installed on the robot waist (joint-2) is used to capture plant images and process them using HSV masking filters and triangulation principal to obtain the 3D center point position of the fruit. The robot joint movement is calculated using geometric based inverse kinematics. The research results show that the average accuracy of the stereo vision system is 93.9 %. The average grip positioning accuracy is 95.6 % to the actual chili fruit position and 98.5 % to the stereo vision calculation value. The average stability of the stereo vision values is 99.5 %, while the average positioning stability of the robot's grip is 99.6 %. Time consumption for image processing is 0.053 s while time consumption for robot grip movement is 9 s. Therefore, the stereo vision system can be used to control robot's grip movement with a good accuracy. Keywords: Red chili fruit, Robot arm, Stereo vision, Three-dimensional position.

Advances in the application of stereo vision in aquaculture with emphasis on fish: A review

AbstractThe effective implementation of machine vision has played a crucial role in advancing intelligent aquaculture across various domains. Stereo vision, as a branch of machine vision, has become a mainstream technology in aquaculture. Its distinctive capability to conduct comprehensive underwater monitoring from multiple angles, unaffected by object occlusion has propelled it to the forefront of aquaculture applications. This article offers a comprehensive review of the diverse applications of stereo vision in aquaculture spanning from its inception to present. The exploration encompasses its role in crucial areas such as biomass estimation and behavioural analysis, which include fish counting, weight estimation, swimming behaviour, feeding behaviour and abnormal behaviour. Furthermore, the paper delves into the advantages of stereo vision over traditional 2D machine vision approaches, while also acknowledging limitations, and identifying future challenges that must be addressed to fully leverage its potential in aquaculture. The review emphasizes the prospect of advancement in deep learning stereo‐matching algorithms specifically designed for underwater environments to catalyse a breakthrough in stereo vision technology. In summary, this review aims to provide researchers and practitioners with a better understanding of the current development of stereo vision in aquaculture, optimizing stereo vision technology and better serving the aquaculture field.

Object pose estimation based on stereo vision with improvedK‐DtreeICPalgorithm

SummaryWith the wide application of stereovision in SLAM, object pose estimation has gradually become one of the research hotspots. This article proposes an object pose estimation for robotic grasping based on stereo vision with improved K‐D tree ICP algorithm. The feature points and feature descriptors of the point cloud of the object to be captured are extracted, and the feature template set is established. The SAC‐IA algorithm is used to carry out initial registration of the point cloud of the target, and the ICP algorithm based on K‐D tree is used for fine registration. The experimental results show that the average coincidence degree of the final registration of the proposed object pose estimation method reaches 94.1%, and the accurate 6D pose of the object to be grasped is obtained.

Omni-Directional Semi-Global Stereo Matching with Reliable Information Propagation

High efficiency and accuracy of semi-global matching (SGM) make it widely used in many stereo vision applications. However, SGM not only struggles in dealing with pixels in homogeneous area, but also suffers from streak artifacts. In this paper, we propose a novel omni-directional SGM (OmniSGM) with a cost volume update scheme to aggregate costs from paths along all directions and to encourage reliable information to propagate across entire image. Specifically, we perform SGM along four tree structures, namely trees in the left, right, top and bottom of root node, and then fuse the outputs to obtain final result. The contributions of pixels on each tree can be recursively computed from leaf nodes to root node, ensuring our method has linear time computational complexity. Moreover, An iterative cost volume update scheme is proposed using aggregated cost in the last pass to enhance the robustness of initial matching cost. Thus, useful information is more likely to propagate in a long distance to handle the ambiguities in low textural area. Finally, we present an efficient strategy to propagate disparities of stable pixels along the minimum spanning tree (MST) for disparity refinement. Extensive experiments in stereo matching on Middlebury and KITTI datasets demonstrate that our method outperforms typical traditional SGM-based cost aggregation methods.

Overcurrent-driven LEDs for consistent image colour and brightness in agricultural machine vision applications

Machine vision systems are being utilized extensively in agriculture applications. Daytime imaging in outdoor field conditions presents challenges such as variable lighting and colour inconsistencies due to sunlight. Motion blur can occur due to vehicle movement and vibrations from ground terrain. A camera system with active lighting can be a solution to overcome these challenges. In this study, the usage of over-current driven LEDs to produce a powerful flash was investigated as a viable light source for daytime imaging. The current drawn by an LED was increased by a factor of six times its normal rating resulting in increased illuminance. A circuit was designed for storing and releasing energy to the LEDs for a strobe-like effect and a controller was used for synchronizing the strobe with a camera to acquire images. The system was deployed in an apple orchard on three days in summer of 2020. Images were taken throughout the day in both sunny and cloudy conditions of different canopy structures. There was substantial improvement in image brightness and colour consistency by using the LED flashes. Images captured by the prototype system during an 11-hour period showed an average decrease of 85% in standard deviation for the Hue-Saturation-Value (HSV) channels compared to that of the auto-exposure setting. Additionally, the prototype system was able to fix motion blur in images averaging 7 mm in error for a stereo vision application with the camera moving at 7 km/hr. These results show that the designed LED flash system can reduce the undesirable effects of lighting variability and motion blur in images stemming from outdoor field conditions.

Application of Binocular Stereo Vision in Radioactive Source Image Reconstruction and Multimodal Imaging Fusion

Current methods for the imaging of radioactive sources ignore the surrounding scene information. Reconstruction results do not have real scale meaning, creating difficulties for further disposal and management of nuclear waste. In addition, the traditional reconstruction method requires fixed hardware to directly measure the reconstruction parameters. In recent years, the combination of computer vision and traditional radiation imaging has been greatly performed. The binocular camera is a small, light-insusceptible visual instrument, suitable for integration with other instruments. In this article, we present a method based on binocular vision technology to radioactive source image reconstruction. The binocular camera provides the assisting information about the surrounding scene and replaces the fixed hardware to indirectly achieve the measurement target of reconstruction parameters, which includes an azimuth angle and a distance. In addition, we designed a simple algorithm based on the Hough transform and the regular shape of the waste drum to improve the spatial resolution for good visualization. The experimental rendering results with different transparency degrees provide a good sense of spatial scale.

Application of stereovision on a saddle-shaped membrane structure in aero-elastic wind tunnel test.

Some conventional measurements (e.g., accelerometer) are usually used to measure the aero-elastic responses of membrane structures in the field of civil engineering, which are usually measured by using contact sensors, and just one-dimensional data of a single point can be obtained. To deal with this disadvantage, a stereovision measurement is equipped after putting forward a series of algorithms. This newly formed measurement is adopted to measure the aero-elastic responses of a saddle-shaped membrane structure. Before measuring, a series of observations are carried out and discussed by considering the characteristic of the test model, wind tunnel, and interference from the inflow, which is merely mentioned in the previous research. Finally, the full field distribution of average displacement of membrane, under different wind directions and wind speeds, is given. The time history of full field displacement distribution is also presented as a gif file. This successful application implies that when adopting such a stereovision measurement system, it is necessary to optimize the installation of cameras. After an appropriate installation design, the proposed stereovision measurement will make up the disadvantages of conventional measurements and can obtain the full field aero-elastic responses of membrane structures.

A performance analysis of stereo matching algorithms for stereo vision applications in smart environments

A performance analysis of stereo matching algorithms for stereo vision applications in smart environments

Radiometric Invariant Dense Disparity Estimation for Real Time Stereo Correspondence

Computer stereo vision tries to mimic human vision by grabbing multiple views of the same scene and cognizing it. The stereo correspondence will find out the matching pixels between the two views based on the Lambertian criteria, which results in disparity. The distance of the objects from the camera can be calculated using this disparity. But in the real world scenario, this Lambertian assumption may not work always due to the radiometric variation between the image pairs and the conventional approaches results in erroneous disparity. In this work, for doing the radiometric invariant stereo matching, the simple local binary pattern is used. The correspondence is done by using semi global block matching method, which can handle the depth changes of curved surfaces and slanting surfaces by adding suitable penalty terms. The performance evaluation of the proposed shows lesser error rate in the range of 0.14% - 0.3883% and run time requirement of 0.20 milliseconds only. This radiometric invariant stereo correspondence attains accuracy as that of global method with run time speed as that of local method and is suitable for most of the real time stereo vision applications.

Runtime Performance and Power Optimization of Parallel Disparity Estimation on Many-Core Platforms

This article investigates the use of many-core systems to execute the disparity estimation algorithm, used in stereo vision applications, as these systems can provide flexibility between performance scaling and power consumption. We present a learning-based runtime management approach that achieves a required performance threshold while minimizing power consumption through dynamic control of frequency and core allocation. Experimental results are obtained from a 61-core Intel Xeon Phi platform for the aforementioned investigation. The same performance can be achieved with an average reduction in power consumption of 27.8% and increased energy efficiency by 30.04% when compared to Dynamic Voltage and Frequency Scaling control alone without runtime management.

FPGA based real-time on-road stereo vision system

Abstract Accurate and Real-Time stereo vision is an essential need for many computer vision applications, such as On-Road stereo vision system. In this paper, a fast and accurate system for On-Road stereo vision application is presented. In order to achieve this purpose, first, an algorithm is presented that is optimized for hardware implementation and On-Road application, then an appropriate hardware architecture for this algorithm is proposed. The approach uses gradient and Census transform as initial cost function, then cost is aggregate in cross-based supported reign. LR-check and disparity refinement is also utilized to improve final disparity. The proposed design is a standalone stereo vision system where all steps of algorithm is implemented on a hardware platform. Overall system is implemented on FPGA platform and it is tested on KITTI Database. The proposed system is also tested on Middlebury database without any changes in parameters. Experimental results show that the proposed system has high accuracy in KITTI and Middlebury database. In term of hardware resource, the system occupy %%66 of XC6VLX240T FPGA device for 1920 × 1080 image with 96 disparity levels that operated at 53.5 Fps.

Binary descriptor-based dense line-scan stereo matching

We present a line-scan stereo system and descriptor-based dense stereo matching for high-performance vision applications. The stochastic binary local descriptor (STABLE) descriptor is a local binary descriptor that builds upon the principles of compressed sensing theory. The most important properties of STABLE are the independence of the descriptor length from the matching window size and the possibility that more than one pair of pixels contributes to a single-descriptor bit. Individual descriptor bits are computed by comparing image intensities over pairs of balanced random subsets of pixels chosen from the whole described area. On a synthetic as well as real-world examples, we demonstrate that STABLE provides competitive or superior performance than other state-of-the-art local binary descriptors in the task of dense stereo matching. The real-world example is derived from line-scan binocular stereo imaging, i.e., two line-scan cameras are observing the same object line and 2-D images are generated due to relative motion. We show that STABLE performs significantly better than the census transform and local binary patterns (LBP) in all considered geometric and radiometric distortion categories to be expected in practical applications of stereo vision. Moreover, we show as well that STABLE provides comparable or better matching quality than the binary robust-independent elementary features descriptor. The low computational complexity and flexible memory footprint make STABLE well suited for most hardware architectures. We present quantitative results based on the Middlebury stereo dataset as well as illustrative results for road surface reconstruction.

A Point Cloud Alignment Algorithm Based on Stereo Vision Using Random Pattern Projection

This paper proposes a point cloud alignment algorithm based on stereo vision using Random Pattern Projection (RPP). In the application of stereo vision, it is rather difficult to find correspondences between stereo images of texture-less objects. To overcome this issue, RPP is used to enhance the object’s features, thus increasing the accuracy of the identified correspondences of the stereo images. In the 3D alignment algorithm, the down sample technique is used to filter out the outliers of the point cloud data to improve system efficiency. Furthermore, the extracted features of the down sample point cloud data were applied in the matching process. Finally, the object’s pose was estimated by the alignment algorithm based on object features. In experiments, the maximum error and standard deviation of rotation are respectively about 0.031°and 0.199°, while the maximum error and standard deviation of translation are respectively about 0.565 mm and 0.902 mm . The execution time for pose estimation is about 230ms.

Edge-Directed Hardware Architecture for Real-Time Disparity Map Computation

Stereo Vision, a technique aimed at inferring depth information from stereo images, has been used in a wide range of computer vision applications, with real-time requirements in emerging embedded vision systems. Computation of the disparity map, a vital step in extracting depth information from stereo images, requires a significant amount of computational resources. As such, existing software implementations require high-end hardware platforms to achieve real-time frame rates, suggesting that dedicated hardware mechanisms might be more suitable for embedded applications. In this paper, we present a disparity map computation architecture targeting embedded stereo vision applications with hard real-time requirements. The architecture integrates a hardware edge detection mechanism that reduces the search space, improving the overall performance, and is configurable in terms of various application parameters, making it suitable for a number of application environments. The paper also presents a study on the impact of the various parameters in terms of the performance and hardware/power overheads. An experimental prototype of the architecture was implemented on the Xilinx ML505 FPGA Evaluation Platform, achieving 50 Frames Per Second (fps) for 1,280 × 1,024 image sizes. Moreover, the quality of the disparity maps generated by the proposed system is comparable to other existing hardware implementations featuring local stereo correspondence methods.

SAD-Based Stereo Vision Machine on a System-on-Programmable-Chip (SoPC)

This paper, proposes a novel solution for a stereo vision machine based on the System-on-Programmable-Chip (SoPC) architecture. The SOPC technology provides great convenience for accessing many hardware devices such as DDRII, SSRAM, Flash, etc., by IP reuse. The system hardware is implemented in a single FPGA chip involving a 32-bit Nios II microprocessor, which is a configurable soft IP core in charge of managing the image buffer and users' configuration data. The Sum of Absolute Differences (SAD) algorithm is used for dense disparity map computation. The circuits of the algorithmic module are modeled by the Matlab-based DSP Builder. With a set of configuration interfaces, the machine can process many different sizes of stereo pair images. The maximum image size is up to 512 K pixels. This machine is designed to focus on real time stereo vision applications. The stereo vision machine offers good performance and high efficiency in real time. Considering a hardware FPGA clock of 90 MHz, 23 frames of 640 × 480 disparity maps can be obtained in one second with 5 × 5 matching window and maximum 64 disparity pixels.

System Optimization for Prism Based Stereovision

In this paper, we proposed a method for analyzing and optimizing the prism based single-lens stereovision system so that the pictures captured by this system are adequate for the application of stereovision. We first analyzed this system from the standpoint of system setup and parameters selection. Then we introduced a method to calculate the FOV of the prism based single-lens stereovision system. By adding some restricted condition, we redefined the area of FOV and gave a detailed connection between FOV and captured picture to obtain the appropriate image pair for stereovision.

Extraction of Visual Landmarks Using Improved Feature Matching Technique for Stereo Vision Applications

Stereo vision is becoming more and more common in three-dimensional visualization, autonomous vehicle navigation, path finding, object recognition, and other computer vision applications. In this paper, we present an approach for extracting visual landmarks from images acquired by a stereoscopic camera. The scale invariant feature transform (SIFT) technique with self-organizing map is used to detect and recognize visual landmarks. Our methodology is based on winner calculation technique, and the main idea is to keep in the database only distinctive features or landmarks in order to minimize detection time. We will demonstrate that this methodology is more efficient than ordinary SIFT algorithm or speeded up robust features (SURF) matching technique. Improved feature group matching with computation time better than SIFT and the SURF has been observed in the experiments with a variety of stereo image pairs.

Application of balanced neural tree for classifying tentative matches in stereo vision

Here, we present a new application of the supervised learning based classifier in stereo matching. In particular, the chosen classifier is a balanced neural tree. The tentative matches are obtained using the speeded-up robust feature (SURF) matching. The feature vector corresponding to each tentative match is formed based on a similarity measure between SURF descriptors and their neighborhoods in the two stereo images, and these feature vectors are classified into inlier or outlier classes. Further, accuracy of the obtained results have been evaluated in terms of stereovision applications such as in the estimation of the homography and rectification matrices between two stereo images. The experiments based on these stereo estimates show the applicability of the proposed application in the stereo vision.

An improved feature matching technique for stereo vision applications with the use of self-organizing map

Stereo vision cameras are widely used for finding a path for obstacle avoidance in autonomous mobile robots. The Scale Invariant Feature Transform (SIFT) algorithm proposed by Lowe is used to extract distinctive invariant features from images. While it has been successfully applied to a variety of computer vision problems based on feature matching including machine vision, object recognition, image retrieval, and many others, this algorithm has high complexity and long computational time. In order to reduce the computation time, this paper proposes a SIFT improvement technique based on a Self-Organizing Map (SOM) to perform the matching procedure more efficiently for feature matching problems. Matching for multi-dimension SIFT features is implemented with a self-organizing map that introduces competitive learning for matching features. Experimental results on real stereo images show that the proposed algorithm performs feature group matching with lower computation time than the SIFT algorithm proposed by Lowe. We performed experiments on various set of stereo images under dynamic environment with different camera viewpoints that is based on rotation and illumination conditions. The numbers of matched features were increased to double as compared to the algorithm developed by Lowe. The results showing improvement over the SIFT proposed by Lowe are validated through matching examples between different pairs of stereo images. The proposed algorithm can be applied to stereo vision based robot navigation for obstacle avoidance, as well as many other feature matching and computer vision applications.

Stereo-vision applications to reconstruct the 3D texture of pavement surface

Characterisation of pavement surface texture has significant effects on ride comfort and road safety. Pavement texture is typically reported as a single attribute, such as mean profile depth, root mean square roughness or hydraulic radius, which limits the usefulness of information extracted from texture measurements. Therefore, advanced methods that characterise pavement texture in three dimensions are needed. This paper reviews recent advances in the development of two imaging-based texture evaluation methods. The main objective of these methods is to recover the 3D heights of the pavement surface. Also, the validation of the proposed image-based texture indicators is examined. Results show that image-based techniques can be successfully applied to recover the 3D heights of pavement surface textures and provide substantial information on the friction and noise characteristics of the surface.