Real-time Stereo Vision Research Articles

Real-Time Stereo Vision Hardware Accelerator: Fusion of SAD and Adaptive Census Algorithm

Real-Time Stereo Vision Hardware Accelerator: Fusion of SAD and Adaptive Census Algorithm

Stereo Vision Controlled Variable Rate Sprayer for Specialty Crops: Part II. Sprayer Development and Performance Evaluation

Highlights A real time stereo vision controlled variable rate sprayer for specialty crops was developed. The stereo vision system of the sprayer detected outdoor trees with similar canopy profiles under travel speeds ranging from 3.2 to 8 km h-1. Canopy volume measurements of the sprayer were impacted by lateral distances between the sprayer and the tree center and travel speeds. The sprayer required less than 200 ms from tree canopy detection to spray decisions. The sprayer achieved spray volume reductions from 72.6% to 80.5% compared to constant rate spray application. Abstract. A real time variable rate sprayer controlled by a stereo vision system was developed to increase the accuracy of spray applications and reduce the use of crop protection products. The sprayer was designed to detect tree canopies and calculate its volume using depth images from the stereo vision system, and discharge corresponding spray volumes every 200 ms through the embedded software in the graphical user interface. The sprayer was evaluated in an apple orchard at different travel speeds (3.2 to 8.0 km h-1) for its performance in detecting canopy and measuring its volume. In addition, spray volume, deposition, and coverage of the variable rate application of the sprayer were evaluated against a constant rate application. Test results showed that the sprayer detected visually similar tree canopies during the evaluations, although its canopy volume measurements deviated from manually measured canopy volume from 0.11 to 0.83 m3 due to lateral position changes of the sprayer. The sprayer adjusted duty cycles of pulse width modulated valves to accurately spray the intended volume for detected canopies (0.073 to 0.083 L m-3) and only used spray volumes of 19.5% to 26.7% compared to a constant rate spray application (338 L ha-1). The constant rate spray application generally had more spray deposition and coverage in tree canopies than the variable rate sprayer, as expected since its spray volume was approximately 3.7 times higher. However, the mean spray depositions from the constant rate spray application were significantly varied (p=0.05) by tree sizes, while the variable rate spray application achieved statistically equivalent mean spray depositions regardless of tree sizes. The stereo vision controlled sprayer offers a cost-effective real-time variable rate spray option for growers with the potential to perform other tasks by using image processing algorithms while applying crop protection products. Keywords: Automation, Canopy volume, Crop protection, Depth image, Orchard, Precision agriculture, Real-time application.

Real-time low-power binocular stereo vision based on FPGA

Real-time low-power binocular stereo vision based on FPGA

Variant center-symmetric census transform for real-time stereo vision architecture on chip

Stereo vision is one of the most crucial operations in many computer vision applications, and stereo matching is its most important step. In recent years, stereo matching has developed in the direction of increasing accuracy and higher computational speed, and the real-time stereo matching algorithm based on hardware architecture has been increasingly emphasized due to its applicability in embedded systems. The most frequently used method on FPGA is the census transform (CT) because of its simple structure and easy parallelization and the high quality of the generated disparity maps, but CT has the drawbacks of mismatches in some regions and dependence on a central pixel. In this paper, an improved CT-based semi-global stereo matching algorithm with pipeline and parallel operation based on FPGA is proposed in order to increase matching accuracy in specific regions while satisfying real-time constraints. In the matching cost step, pixels are divided into two parts, with the two methods calculated to generate the bit-vector feature. A four-path semi-global matching algorithm with twice aggregate is proposed in the cost aggregation stage. The left–right check, mismatching point filling, and the median filter to enhance the final disparity map are also required. The novel algorithm is evaluated on the Middleburry benchmark and implemented on Xilinx ZYNQ-7000 SoCs, which results in a throughput of 640 × 480/60 fps, with 64 disparity levels at 100 MHz. Compared with the related work, we improve the average accuracy by 1.61%, making our approach suitable for real-time embedded systems.

Automatic sound synthesis using the fly algorithm

Our study is demonstrated a new type of evolutionary sound synthesis method. This work based on the fly algorithm, a cooperative co-evolution algorithm; it is derived from the Parisian evolution approach. The algorithm has relatively amended the position of individuals (the Flies) represented by 3-D points. The fly algorithm has successfully investigated in different applications, starting with a real-time stereo vision for robotics. Also, the algorithm shows promising results in tomography to reconstruct 3-D images. The final application of the fly algorithm was generating artistic images, such as digital mosaics. In all these applications, the flies’representation started for simple, 3-D points, to complex one, the structure of 9-elements. Our method follows evolutionary digital art with the fly algorithm in representing the pattern of the flies. They represented in a way of having their structure. This structure includes position, colour, rotation angle, and size. Our algorithm has the benefit of graphics processing units (GPUs) to generate the sound waveform using the modern OpenGL shading language.

Radiometric Invariant Dense Disparity Estimation for Real Time Stereo Correspondence

Computer stereo vision tries to mimic human vision by grabbing multiple views of the same scene and cognizing it. The stereo correspondence will find out the matching pixels between the two views based on the Lambertian criteria, which results in disparity. The distance of the objects from the camera can be calculated using this disparity. But in the real world scenario, this Lambertian assumption may not work always due to the radiometric variation between the image pairs and the conventional approaches results in erroneous disparity. In this work, for doing the radiometric invariant stereo matching, the simple local binary pattern is used. The correspondence is done by using semi global block matching method, which can handle the depth changes of curved surfaces and slanting surfaces by adding suitable penalty terms. The performance evaluation of the proposed shows lesser error rate in the range of 0.14% - 0.3883% and run time requirement of 0.20 milliseconds only. This radiometric invariant stereo correspondence attains accuracy as that of global method with run time speed as that of local method and is suitable for most of the real time stereo vision applications.

Real-Time Stereo Vision System: A Multi-Block Matching on GPU

Real-time stereo vision is attractive in many areas such as outdoor mapping and navigation. As a popular accelerator in the image processing field, GPU is widely used for the studies of the stereo vision algorithms. Recently, many stereo vision systems on GPU have achieved low error rate, as a result of the development of deep learning. However, their processing speed is normally far from the real-time requirement. In this paper, we propose a real-time stereo vision system on GPU for the high-resolution images. This system also maintains a low error rate compared with other fast systems. In our approach, the image is resized to reduce the computational complexity and to realize the real-time processing. The low error rate is kept by using the cost aggregation with multiple blocks, secondary matching and sub-pixel estimation. Its processing speed is 41 fps for $2888\times 1920$ pixels images when the maximum disparity is 760.

FPGA based real-time on-road stereo vision system

Abstract Accurate and Real-Time stereo vision is an essential need for many computer vision applications, such as On-Road stereo vision system. In this paper, a fast and accurate system for On-Road stereo vision application is presented. In order to achieve this purpose, first, an algorithm is presented that is optimized for hardware implementation and On-Road application, then an appropriate hardware architecture for this algorithm is proposed. The approach uses gradient and Census transform as initial cost function, then cost is aggregate in cross-based supported reign. LR-check and disparity refinement is also utilized to improve final disparity. The proposed design is a standalone stereo vision system where all steps of algorithm is implemented on a hardware platform. Overall system is implemented on FPGA platform and it is tested on KITTI Database. The proposed system is also tested on Middlebury database without any changes in parameters. Experimental results show that the proposed system has high accuracy in KITTI and Middlebury database. In term of hardware resource, the system occupy %%66 of XC6VLX240T FPGA device for 1920 × 1080 image with 96 disparity levels that operated at 53.5 Fps.

Fly4Arts : Art numérique évolutionnaire avec l’Algorithme des mouches

The aim of this study is to generate artistic images, such as digital mosaics, as an optimisation problem without the introduction of any a priori knowledge or constraint other than an input image. The usual practice to produce digital mosaic images heavily relies on Centroidal Voronoi diagrams. We demonstrate here that it can be modelled as an optimisation problem solved using a cooperative co-evolution strategy based on the Parisian evolution approach, the Fly algorithm. An individual is called a fly. Its aim of the algorithm is to optimise the position of infinitely small 3-D points (the flies). The Fly algorithm has been initially used in real-time stereo vision for robotics. It has also demonstrated promising results in image reconstruction for tomography. In this new application, a much more complex representation has been study. A y is a tile. It has its own position, size, colour, and rotation angle. Our method takes advantage of graphics processing units (GPUs) to generate the images using the modern OpenGL Shading Language (GLSL) and Open Computing Language (OpenCL) to compute the difference between the input image and simulated image. Different types of tiles are implemented, some with transparency, to generate different visual effects, such as digital mosaic and spray paint. An online study with 41 participants has been conducted to compare some of our results with those generated using an open source software for image manipulation. It demonstrates that our method leads to more visually appealing images.

Stereo vision during adverse weather — Using priors to increase robustness in real-time stereo vision

Stereo vision can deliver a dense 3D reconstruction of the environment in real-time for driver assistance as well as autonomous driving. Semi-Global Matching (SGM) is a popular method of choice for solving this task which is already in use for production vehicles. Despite the enormous progress in the field and the high level of performance of modern stereo methods, one key challenge remains: robust stereo vision in automotive scenarios during rain, snow and darkness. Under these circumstances, current methods generate strong temporal noise, many disparity outliers and false positives on object level. These problems are addressed in this work by regularizing stereo vision via prior information. We formulate a temporal prior and a scene prior, which we apply to SGM in order to overcome the deficiencies. The temporal prior integrates knowledge from the previous disparity map to exploit the high temporal correlation, the scene prior exploits knowledge of a representative traffic scene. Using these priors, the object detection rate improves significantly on a driver assistance dataset of 3000 frames including bad weather while reducing the rate of erroneous object detections. We also outperform the ECCV Robust Vision Challenge 2012 winner, iSGM, on this dataset. In addition, results are presented for the KITTI dataset, even showing improvements under good weather conditions when exploiting the temporal prior.We also show that the temporal and scene priors are easy and efficient to implement on a hybrid CPU/reconfigurable hardware platform. The use of these priors can be extended to other application areas such as mobile robotics.

Improve a 3D distance measurement accuracy in stereo vision systems using optimization methods’ approach

Abstract This paper presents a 3D distance measurement accuracy improvement for stereo vision systems using optimization methods A Stereo Vision system is developed and tested to identify common uncertainty sources. As the optimization methods are used to train a neural network, the resulting equation can be implemented in real time stereo vision systems. Computational experiments and a comparative analysis are conducted to identify a training function with a minimal error performance for such method. The offered method provides a general purpose modelling technique, attending diverse problems that affect stereo vision systems. Finally, the proposed method is applied in the developed stereo vision system and a statistical analysis is performed to validate the obtained improvements.

A real-time stereo vision system for distance measurement and underwater image restoration

This paper presents the development of an embedded real-time system that performs distance measurement and restoration of underwater images, using stereo vision techniques. To achieve a high performance low-cost implementation, the overall system has been developed using a hardware/software codesign approach. Several hardware modules have been designed to implement the several pixel intensive tasks, such as background image storage, background subtraction, center of mass calculation, and image restoration. On the other hand, less intensive tasks, such as the estimation of the disparity and the distance tasks (performed just once for each image), are executed using an embedded soft processor (Altera Nios II). The developed platform employs a pair of identical CMOS cameras for the stereo vision system, a low-cost FPGA, and an small screen for visualization of the images. In this paper, we describe both the overall design of the system and the calibration procedure used to determine the stereo vision system parameters. The Altera Quartus II was used as a synthesis tool, which estimates that the system consumes 115.25 mW and achieves an output of 26.56 frames per second for images of $$800\times 480$$ pixels. The synthesis results and the measurement precision show that the developed system is suitable for real-time tasks.

GPU-Based Optimization of Self-Organizing Map Feature Matching for Real-Time Stereo Vision

In this paper, we present a graphics processing unit (GPU)-based matching technique for the purpose of fast feature matching between different images. The scale invariant feature transform algorithm developed by Lowe for various feature matching applications, such as stereo vision and object recognition, is computationally intensive. To address this problem, we propose a matching technique optimized for GPUs to perform computations in less time. We optimize GPUs for fast computation of keypoints to make our system quick and efficient. The proposed method uses a self-organizing map feature matching technique to perform efficient matching between the different images. The experiments are performed on various image sets to examine the performance of the system under varying conditions, such as image rotation, scaling, and blurring. The experimental results show that the proposed algorithm outperforms the existing feature matching methods, resulting in fast feature matching due to the optimization of the GPU.

Design of a Redundant Manipulator for Playing Table Tennis towards Human-Like Stroke Patterns

This study investigates the design of a 7-DOF humanoid manipulator capable of playing table tennis with human-like stroke patterns. The manipulator system includes a redundant arm, real-time stereo vision system, and a distributed motion control system. First, the size, weight, workspace, and motion capability of the designed arm are similar to those of a human's arm. The forward and inverse kinematics, and the Jacobian matrix of the redundant manipulator are formulated. Next, a distributed motion control system is designed. The ball trajectory prediction method is proposed. Then, a human-inspired optimization method based on Jacobian pseudoinverse and the comfort of the arm posture for stroke pattern trajectory is proposed to achieve human-like stroke patterns and decrease the counterforce exerted on the manipulator. Finally, the validity of the proposed system and methods is demonstrated via human-like stroke pattern experiments.

Hardware-Efficient Design of Real-Time Profile Shape Matching Stereo Vision Algorithm on FPGA

A variety of platforms, such as micro-unmanned vehicles, are limited in the amount of computational hardware they can support due to weight and power constraints. An efficient stereo vision algorithm implemented on an FPGA would be able to minimize payload and power consumption in microunmanned vehicles, while providing 3D information and still leaving computational resources available for other processing tasks. This work presents a hardware design of the efficient profile shape matching stereo vision algorithm. Hardware resource usage is presented for the targeted micro-UV platform, Helio-copter, that uses the Xilinx Virtex 4 FX60 FPGA. Less than a fifth of the resources on this FGPA were used to produce dense disparity maps for image sizes up to 450 × 375, with the ability to scale up easily by increasing BRAM usage. A comparison is given of accuracy, speed performance, and resource usage of a census transform-based stereo vision FPGA implementation by Jin et al. Results show that the profile shape matching algorithm is an efficient real-time stereo vision algorithm for hardware implementation for resource limited systems such as microunmanned vehicles.

SAD-Based Stereo Vision Machine on a System-on-Programmable-Chip (SoPC)

This paper, proposes a novel solution for a stereo vision machine based on the System-on-Programmable-Chip (SoPC) architecture. The SOPC technology provides great convenience for accessing many hardware devices such as DDRII, SSRAM, Flash, etc., by IP reuse. The system hardware is implemented in a single FPGA chip involving a 32-bit Nios II microprocessor, which is a configurable soft IP core in charge of managing the image buffer and users' configuration data. The Sum of Absolute Differences (SAD) algorithm is used for dense disparity map computation. The circuits of the algorithmic module are modeled by the Matlab-based DSP Builder. With a set of configuration interfaces, the machine can process many different sizes of stereo pair images. The maximum image size is up to 512 K pixels. This machine is designed to focus on real time stereo vision applications. The stereo vision machine offers good performance and high efficiency in real time. Considering a hardware FPGA clock of 90 MHz, 23 frames of 640 × 480 disparity maps can be obtained in one second with 5 × 5 matching window and maximum 64 disparity pixels.

Object Tracking and Positioning Based on Stereo Vision

The goal of this paper is to present a method for object tracking and positioning based on stereo vision in real time. The method effectively combined stereo matching algorithm with object tracking algorithm, and calculated the spatial location information of the object by using binocular stereo vision while the object is being tracked. The stereo matching used dynamic programming, image pyramids and control points modification algorithm, and the object tracking mainly utilized CamShift algorithm in this paper. The experimental results have confirmed that the proposed method realized real-time tracking for moving object, accurate calculating for the object three-dimensional coordinates, which meet the applied needs of servo follow-up system.

Improved Census Transforms for Resource-Optimized Stereo Vision

Real-time stereo vision has proven to be a useful technology with many applications. However, the computationally intensive nature of stereo vision algorithms makes real-time implementation difficult in resource-limited systems. The field-programmable gate array (FPGA) has proven to be very useful in the implementation of local stereo methods, yet the resource requirements can still be a significant challenge. This paper proposes a variety of sparse census transforms that dramatically reduce the resource requirements of census-based stereo systems while maintaining stereo correlation accuracy. This paper also proposes and analyzes a new class of census-like transforms, called the generalized census transforms. This new transform allows a variety of very sparse census-like stereo correlation algorithms to be implemented while demonstrating increased robustness and flexibility. The resource savings and performance of these transforms is demonstrated by the design and implementation of a parameterizable stereo system that can implement stereo correlation using any census transform. Several optimizations for typical FPGA-based correlation systems are also proposed. The resulting system is capable of running at over 500 MHz on a modern FPGA, resulting in a throughput of over 500 million input pixel pairs per second.

Efficient stereo vision algorithms for resource-limited systems

In most circumstances, determining an acceptable trade-off between speed and accuracy when selecting a stereo vision algorithm for implementation is dependent on the target application. This work attempts to provide a perspective on the efficiency of existing real-time stereo vision algorithms in terms of this trade-off. This work also provides an example of modifying an existing highly accurate stereo vision algorithm to increase its runtime performance while trying to limit the loss in accuracy. The modifications can be used to increase efficiency of several other local stereo vision algorithms due to sharing some common components. Such an example demonstrates the challenge of making efficient trade-offs in accuracy for runtime performance. It is shown that the modifications resulted in an 8X speedup over the original algorithm, with accuracy results comparable to existing real-time algorithms.

A Platform-Based SoC Design for Real-Time Stereo Vision

A stereo vision is able to build threedimensional maps of its environment. It can provide much more complete information than a 2D image based vision but has to process, at least, that much more data. In the past decade, real-time stereo has become a reality. Some solutions are based on reconfigurable hardware and others rely on specialized hardware. However, they are designed for their own specific applications and are difficult to extend their functionalities. This paper describes a vision system based on a System on a Chip (SoC) platform. A real-time stereo image correlator is implemented using Sum of Absolute Difference (SAD) algorithm and is integrated into the vision system using AMBA bus protocol. Since the system is designed on a pre-verified platform it can be easily extended in its functionality increasing design productivity. Simulation results show that the vision system is suitable for various real-time applications