Binocular Sensor Research Articles

Improved Calibration of Eye-in-Hand Robotic Vision System Based on Binocular Sensor.

Eye-in-hand robotic binocular sensor systems are indispensable equipment in the modern manufacturing industry. However, because of the intrinsic deficiencies of the binocular sensor, such as the circle of confusion and observed error, the accuracy of the calibration matrix between the binocular sensor and the robot end is likely to decline. These deficiencies cause low accuracy of the matrix calibrated by the traditional method. In order to address this, an improved calibration method for the eye-in-hand robotic vision system based on the binocular sensor is proposed. First, to improve the accuracy of data used for solving the calibration matrix, a circle of confusion rectification method is proposed, which rectifies the position of the pixel in images in order to make the detected geometric feature close to the real situation. Subsequently, a transformation error correction method with the strong geometric constraint of a standard multi-target reference calibrator is developed, which introduces the observed error to the calibration matrix updating model. Finally, the effectiveness of the proposed method is validated by a series of experiments. The results show that the distance error is reduced to 0.080 mm from 0.192 mm compared with the traditional calibration method. Moreover, the measurement accuracy of local reference points with updated calibration results from the field is superior to 0.056 mm.

RGB-D-Based Stair Detection and Estimation Using Deep Learning.

Stairs are common vertical traffic structures in buildings, and stair detection tasks are important in environmental perception for autonomous mobile robots. Most existing algorithms have difficulty combining the visual information from binocular sensors effectively and ensuring reliable detection at night and in the case of extremely fuzzy visual clues. To solve these problems, we propose a stair detection network with red-green-blue (RGB) and depth inputs. Specifically, we design a selective module, which can make the network learn the complementary relationship between the RGB feature maps and the depth feature maps and fuse the features effectively in different scenes. In addition, we propose several postprocessing algorithms, including a stair line clustering algorithm and a coordinate transformation algorithm, to obtain the stair geometric parameters. Experiments show that our method has better performance than existing the state-of-the-art deep learning method, and the accuracy, recall, and runtime are improved by 5.64%, 7.97%, and 3.81 ms, respectively. The improved indexes show the effectiveness of the multimodal inputs and the selective module. The estimation values of stair geometric parameters have root mean square errors within 15 mm when ascending stairs and 25 mm when descending stairs. Our method also has extremely fast detection speed, which can meet the requirements of most real-time applications.

A Literature Survey on Vision Assistance System Based on Binocular Sensors for Visually Impaired Users

According to research, in 2015, there were an estimated 253 million people with visual impairment worldwide. Of these, 36 million were completely blind and a further 217 million had moderate to severe visual impairment (MSVI). This forms a considerable part of the population if you look at it. But the visually impaired still face a lot of difficulties in just going about their daily life. Most have to rely on the people around, on a basic walking cane or on a guide dog. This takes away a good part of their independence as they have to rely on someone for their basic needs. Many wearable technologies have been proposed to assist the visually impaired in their daily routines in order to solve this problem. This paper is an attempt to comprehend all of the past research done on the subject.

A Literature Survey on Vision Assistance System Based on Binocular Sensors for Visually Impaired Users

A Literature Survey on Vision Assistance System Based on Binocular Sensors for Visually Impaired Users

Region Growing Based on 2-D–3-D Mutual Projections for Visible Point Cloud Segmentation

In recent years, with the rapid development of multisensor fusion technology, point clouds used are no longer limited to those including 3-D coordinates acquired by visual sensors, such as binocular sensors or structured light sensors. The 4-D or more multidimensional data are needed to analyze information in a more intuitive way. The 3-D point cloud and 2-D image have complementary information, and the point cloud can be colored by the fusion of coordinate data and intensity data. However, due to the limitation of sight occlusion, only some points in the point cloud are visible in a single image, i.e., they have intensity information. Most existing methods rely on surface reconstruction, which has always been a complex problem in theory and implementation. In this article, a new algorithm named region growing based on 2-D–3-D mutual projections is proposed. Based on the idea of regional growing, we select the initial seed points by the geometric information of the point cloud in the 3-D space and projection plane, and then estimate the visibility of each point according to the growth criteria defined by us. The results show that the proposed method successfully divides the visible points and the occlusion points and achieves satisfactory results in the subsequent intensity fusion. Our method is more robust than the large curvature change and large density change regions. For the car model, the false negative rate of our algorithm decreases by 4.4% compared with Katz’s method, and the score of our algorithm is 15.4% higher than that of Biasutti et al .’s method.

A spraying path planning algorithm based on colour-depth fusion segmentation in peach orchards

Automatic and precise spraying in orchards is currently an active area in precision agriculture research. Research on this topic mainly focuses on the recognition of fruit trees and the planning of driving paths. The dense canopy and complex background of peach tree orchards present great challenges in fruit tree recognition. In this study, an algorithm based on colour-depth vision was developed to solve the path planning problem for precise spraying in peach orchards. First, a colour-depth binocular sensor was used to acquire video images in peach orchards. Then, a colour-depth fusion segmentation method (CDFS) based on the leaf wall area (LWA) of the colour-depth images was proposed. This method combines the result of Otsu segmentation for the green layer of the colour image and the K-means layering result of the depth image to accurately detect the LWA of peach trees. Then, image erosion was used to delimit the two largest LWAs as the region of interest (ROI). Finally, the spraying path was planned by detecting the midpoint of ROI spacing as the end of the spraying path. A comparative analysis of five image segmentation algorithms showed that the proposed CDFS algorithm produced the closest LWA to that obtained by manual segmentation and thus achieved the best segmentation effect. Comparing this automatic path plan method with the drivable areas showed that the overall accuracy rate of the path planning was 97.5%. Therefore, the proposed algorithm planned paths within the drivable area and can thus be used to accurately and automatically plan spraying paths in peach orchards, except when there were gaps between rows in the fruit orchard.

Wearable Vision Assistance System Based on Binocular Sensors for Visually Impaired Users

Blind or visually impaired people face special difficulties in daily life. With the advances in vision sensors and computer vision, the design of wearable vision assistance system is promising. In order to improve the life quality of the visually impaired group, a wearable system is proposed in this paper. Typically the performance of visual sensors is affected by a variety of complex factors in practice, resulting in a large number of noise and distortion. In this paper, we will creatively leverage image quality evaluation to select the captured images through vision sensors, which can ensure the input quality of scenes for the final identification system. First, we use binocular vision sensors to capture images in a fixed frequency and choose the informative ones based on stereo image quality assessment. Then the captured images will be sent to cloud for further computing. Specially, the detection and automatic result will be done for all the received images. Convolutional neural network based on big data will be used in this step. According to image analysis, the cloud computing can return the requested information for users, which can help them make a more reasonable decision in further action. Simulations and experiments show that the proposed method can solve the problem effectively. In addition, statistical results also demonstrate that wearable vision system can make visually impaired group more satisfied in visual needed situations.

Semantic Edge Based Disparity Estimation Using Adaptive Dynamic Programming for Binocular Sensors.

Disparity calculation is crucial for binocular sensor ranging. The disparity estimation based on edges is an important branch in the research of sparse stereo matching and plays an important role in visual navigation. In this paper, we propose a robust sparse stereo matching method based on the semantic edges. Some simple matching costs are used first, and then a novel adaptive dynamic programming algorithm is proposed to obtain optimal solutions. This algorithm makes use of the disparity or semantic consistency constraint between the stereo images to adaptively search parameters, which can improve the robustness of our method. The proposed method is compared quantitatively and qualitatively with the traditional dynamic programming method, some dense stereo matching methods, and the advanced edge-based method respectively. Experiments show that our method can provide superior performance on the above comparison.

Epipolar constraint of single-camera mirror binocular stereo vision systems

Virtual binocular sensors, composed of a camera and catoptric mirrors, have become popular among machine vision researchers, owing to their high flexibility and compactness. Usually, the tested target is projected onto a camera at different reflection times, and feature matching is performed using one image. To establish the geometric principles of the feature-matching process of a mirror binocular stereo vision system, we proposed a single-camera model with the epipolar constraint for matching the mirrored features. The constraint between the image coordinates of the real target and its mirror reflection is determined, which can be used to eliminate nonmatching points in the feature-matching process of a mirror binocular system. To validate the epipolar constraint model and to evaluate its performance in practical applications, we performed realistic matching experiments and analysis using a mirror binocular stereo vision system. Our results demonstrate the feasibility of the proposed model, suggesting a method for considerable improvement of efficacy of the process for matching mirrored features.