Robot Vision System Research Articles

Viewpoint Planning for Range Sensors Using Feature Cluster Constrained Spaces for Robot Vision Systems.

The efficient computation of viewpoints for solving vision tasks comprising multi-features (regions of interest) represents a common challenge that any robot vision system (RVS) using range sensors faces. The characterization of valid and robust viewpoints is even more complex within real applications that require the consideration of various system constraints and model uncertainties. Hence, to address some of the challenges, our previous work outlined the computation of valid viewpoints as a geometrical problem and proposed feature-based constrained spaces (C-spaces) to tackle this problem efficiently for acquiring one feature. The present paper extends the concept of C-spaces to consider multi-feature problems using feature cluster constrained spaces (GC-spaces). A GC-space represents a closed-form, geometrical solution that provides an infinite set of valid viewpoints for acquiring a cluster of features satisfying diverse viewpoint constraints. Furthermore, the current study outlines a generic viewpoint planning strategy based on GC-spaces for solving vision tasks comprising multi-feature scenarios effectively and efficiently. The applicability of the proposed framework is validated on two different industrial vision systems used for dimensional metrology tasks.

YOLACTFusion: An instance segmentation method for RGB-NIR multimodal image fusion based on an attention mechanism

The tomato plant’s main-stem is a feasible lead for robotic searching the grows discretely-growing targets of harvesting, pruning or pollinating. Owing to the highlighted reflection characteristics of the main-stem in the near-infrared (NIR) waveband, this study proposes a multimodal hierarchical fusion method (YOLACTFusion) based on the attention mechanism, to achieve an instance segmentation of the main-stem from similar-colored differentiation (i.e., green leaf and green fruit) in robotic vision systems. The model inputs RGB images and 900–1100 nm NIR images into two ResNet50 backbone networks and uses a parallel attention mechanism to fuse feature maps of various scales together into the head network, to improve the segmentation performance of the main-stem of RGB images. The loss function for the multimodal image weights the original loss on the RGB image and the position offset loss and classification loss on the NIR image. Furthermore, the local depthwise separable convolution is used for the backbone network, and Conv-BN layers are merged to reduce the computational complexity. The results show that the precision and recall of YOLACTFusion of the main-stem detection, respectively reached 93.90 % and 62.60 %; and the precision and recall of instance segmentation reached 95.12 % and 63.41 %, respectively. Compared to YOLACT, the mean average precision (mAP) of YOLACTFusion is increased from 39.20 % to 46.29 %, the model size is reduced from 199.03 MB to 165.52 MB, while the image processing efficiency remains similar. The overall results show that the multimodal instance segmentation method proposed in this study significantly improves the detection and segmentation of tomato main-stems under a similar-colored background, which would be a potential method for improving agricultural robot’s visual perception.

Evaluation of Computer Vision Systems and Applications to Estimate Trunk Cross-Sectional Area, Flower Cluster Number, Thinning Efficacy and Yield of Apple

Precision crop load management of apple requires counting fruiting structures at various times during the year to guide management decisions. The objective of the current study was to evaluate the accuracy of and compare different commercial computer vision systems and computer applications to estimate trunk cross-sectional area (TCSA), flower cluster number, thinning efficacy, and yield estimation. These studies evaluated two companies that offer different vision systems in a series of trials across 23 orchards in four states. Orchard Robotics uses a proprietary camera system, and Pometa (previously Farm Vision) uses a cell phone camera system. The cultivars used in the trials were ‘NY1’, ‘NY2’, ‘Empire’, ‘Granny Smith’, ‘Gala’, ‘Fuji’, and ‘Honeycrisp’. TCSA and flowering were evaluated with the Orchard Robotics camera in full rows. Flowering, fruit set, and yield estimation were evaluated with Pometa. Both systems were compared with manual measurements. Our results showed a positive linear correlation between the TCSA with the Orchard Robotics vision system and manual measurements, but the vision system underestimated the TCSA in comparison with the manual measurements (R2s between 0.5 and 0.79). Both vision systems showed a positive linear correlation between nubers of flowers and manual counts (R2s between 0.5 and 0.95). Thinning efficacy predictions (in June) were evaluated using the fruit growth rate model, by comparing manual measurements and the MaluSim computer app with the computer vision system of Pometa. Both systems showed accurate predictions when the numbers of fruits at harvest were lower than 200 fruit/tree, but our results suggest that, when the numbers of fruits at harvest were higher than 200 fruit/tree, both methods overestimated final fruit numbers per tree when compared with final fruit numbers at harvest (R2s 0.67 with both systems). Yield estimation was evaluated just before harvest (August) with the Pometa system. Yield estimation was accurate when fruit numbers were fewer than 75 fruit per tree, but, when the numbers of fruit at harvest were higher than 75 fruit per tree, the Pometa vision system underestimated the final yield (R2 = 0.67). Our results concluded that the Pometa system using a smartphone offered advantages such as low cost, quick access, simple operation, and accurate precision. The Orchard Robotics vision system with an advanced camera system provided more detailed and accurate information in terms of geo-referenced information for individual trees. Both vision systems evaluated are still in early development and have the potential to provide important information for orchard managers to improve crop load management decisions.

Algorithms for the visual analysis of an environment by an autonomous mobile robot for area cleanup

Objectives. At present, increasing rates of pollution of vast areas by various types of household waste are becoming an increasingly serious problem. In this connection, the creation of a robotic complex capable of performing autonomous litter collection functions becomes an urgent need. One of the key components of such a complex comprises a vision system for detecting and interacting with target objects. The purpose of this work is to develop the underlying algorithmics for the vision system of robots executing area cleaning functions.Methods. Within the framework ofthe proposed structure ofthe system for visual analysis ofthe external environment, algorithms for detecting and classifying objects of various appearance have been developed using convolutional neural networks. The neural network detector was set up by gradient descent on the open dataset of TACO training samples. To determine the geometric parameters of a surface in the field of view of the robot and estimate the coordinates of objects on the ground, a homography matrix was formed to take into account information about the characteristics and location of the video camera.Results. The developed software and algorithms for a mobile robot equipped with a monocular video camera are capable of implementing the functions of neural network detection and classification of litter objects in the frame, as well as projection of found objects on a terrain map for their subsequent collection.Conclusions. Experimental studies have shown that the developed system of visual analysis of the external environment of an autonomous mobile robot has sufficient efficiency to solve the tasks of detecting litter in the field of view of an autonomous mobile robot.

Investigation of IBVS control method utilizing vanishing vector subject to spatial constraint

When a visual servoing system has to move a workpiece to an unmarked spatial point, conventional control methods have difficulty accomplishing the task. This is referred to as the spatial constraint problem. This research proposes an improved solution by introducing a new image feature based on previous research. Initially, a vanishing vector, considered a new type of image feature, is designed in the image plane, and the viability of employing the vanishing vector to represent the unmarked spatial point is demonstrated. Afterward, using a robotic vision system with an eye-to-hand configuration, simulations and experiments are conducted in real-time to validate the effectiveness of the proposed method. The results indicate that the spatial constraint problem can be solved effectively, allowing the robot to move the workpiece to the desired position.

Viewpoint Generation Using Feature-Based Constrained Spaces for Robot Vision Systems

The efficient computation of viewpoints while considering various system and process constraints is a common challenge that any robot vision system is confronted with when trying to execute a vision task. Although fundamental research has provided solid and sound solutions for tackling this problem, a holistic framework that poses its formal description, considers the heterogeneity of robot vision systems, and offers an integrated solution remains unaddressed. Hence, this publication outlines the generation of viewpoints as a geometrical problem and introduces a generalized theoretical framework based on Feature-Based Constrained Spaces (C-spaces) as the backbone for solving it. A C-space can be understood as the topological space that a viewpoint constraint spans, where the sensor can be positioned for acquiring a feature while fulfilling the constraint. The present study demonstrates that many viewpoint constraints can be efficiently formulated as C-spaces, providing geometric, deterministic, and closed solutions. The introduced C-spaces are characterized based on generic domain and viewpoint constraints models to ease the transferability of the present framework to different applications and robot vision systems. The effectiveness and efficiency of the concepts introduced are verified on a simulation-based scenario and validated on a real robot vision system comprising two different sensors.

Sliding mode observer based fault identification in automatic vision system of robot

It is analyzed the possibility to apply sliding mode observers for fault identification and consecutively the improvement of Technical Vision System (TVS) functioning in the tasks of mobile robotics. It is important that in swarm robotics tasks, where the Machine Vision is based on collaborative scanning of frame by several TVSs, the value of each single fault is multiplied. The original approach to system modeling permits us to build the sliding mode observer of reduced order, and apply its estimation for control improvement of scanning aperture DC motor in TVS. It is shown by simulations that estimation of scanner’s motor instability is robust and not sensitive to many practical perturbations.

A novel multidimensional uncalibration method applied to six-axis manipulators.

This study proposes a multidimensional uncalibrated technique for tracking and grasping dynamic targets by a robotic arm in the eye-in-hand mode. This method avoids complex and cumbersome calibration processes, enabling machine vision tasks to be adaptively applied in a variety of complex environments, which solved the problem of traditional calibration methods being unstable in complex environments. The specific method used in this study is first, in the eye-in-hand mode, the robotic arm moves along the x, y, and z axes in sequence, and images are taken before and after each movement. Thereafter, the image Jacobian matrix is calculated from the three (or more) sets of images collected. Finally, the robotic arm converts the target coordinates in the real-time captured images by the camera into coordinates in the robotic arm coordinate system through the image Jacobian matrix and performs real-time tracking. This study tests the dynamic quasi-Newton method for estimating the Jacobian matrix and optimizes the initialization coupling problem using the orthogonal moving method. This optimization scheme significantly shortens the iteration process, making the uncalibrated technology more fully applied in the field of dynamic object tracking. In addition, this study proposes a servo control algorithm with predictive compensation to mitigate or even eliminate the systematic error caused by time delay in dynamic target tracking in robot visual servo systems.

ADAPTIVE IMAGE ENHANCEMENT MODEL FOR THE ROBOT VISION SYSTEM

Robotics is one of the important trends in the current development of science and technology. Most modern robots and drones have their own vision system, including a video camera, which they use to take digital photos and video streams. These data are used to analyze the situation in the robot's camera field of view, as well as to determine a real-time robot's behavior algorithm. In this regard, the novelty of the paper is special polynomial mathematical model and method for adaptive gradational correction of a digital image. The proposed model and method make it possible to independently adjust to brightness scales and image formats and optimally perform gradational image correction in various lighting conditions. Thus, ensuring the efficiency of the entire subsequent cycle of image analysis in the robot's vision system. In addition, the paper presents the results of numerous experiments of such gradational correction for images of various classes, as well as conditions of reduced and increased levels of illumination of the field of view objects. Conclusions and recommendations are given regarding the practical application of the proposed model and method.

ROAD: Robotics-Assisted Onsite Data Collection and Deep Learning Enabled Robotic Vision System for Identification of Cracks on Diverse Surfaces

Crack detection on roads is essential nowadays because it has a significant impact on ensuring the safety and reliability of road infrastructure. Thus, it is necessary to create more effective and precise crack detection techniques. A safer road network and a better driving experience for all road users can result from the implementation of the ROAD (Robotics-Assisted Onsite Data Collecting) system for spotting road cracks using deep learning and robots. The suggested solution makes use of a robot vision system’s capabilities to gather high-quality data about the road and incorporates deep learning methods for automatically identifying cracks. Among the tested algorithms, Xception stands out as the most accurate and predictive model, with an accuracy of over 90% during the validation process and a mean square error of only 0.03. In contrast, other deep neural networks, such as DenseNet201, InceptionResNetV2, MobileNetV2, VGG16, and VGG19, result in inferior accuracy and higher losses. Xception also achieves high accuracy and recall scores, indicating its capability to accurately identify and classify different data points. The high accuracy and superior performance of Xception make it a valuable tool for various machine learning tasks, including image classification and object recognition.

A Deep Learning-Based Visual Map Generation for Mobile Robot Navigation

Visual map-based robot navigation is a strategy that only uses the robot vision system, involving four fundamental stages: learning or mapping, localization, planning, and navigation. Therefore, it is paramount to model the environment optimally to perform the aforementioned stages. In this paper, we propose a novel framework to generate a visual map for environments both indoors and outdoors. The visual map comprises key images sharing visual information between consecutive key images. This learning stage employs a pre-trained local feature transformer (LoFTR) constrained with a 3D projective transformation (a fundamental matrix) between two consecutive key images. Outliers are efficiently detected using marginalizing sample consensus (MAGSAC) while estimating the fundamental matrix. We conducted extensive experiments to validate our approach in six different datasets and compare its performance against hand-crafted methods.

Contingency Planning of Visual Contamination for Wheeled Mobile Robots with Chameleon-Inspired Visual System

To enable mobile robots to effectively deal with the emergency of visual contamination, contingency planning based on case-based reasoning (CBR) was performed in this paper. First, for a wheeled mobile robot (WMR) equipped with a chameleon-inspired visual system, a target search model in chameleon-inspired binocular negative correlation movement (CIBNCM) mode was established. Second, a CBR-based contingency planning model of visual contamination for WMRs was established, where the reasoning process using CBR for visual contamination was analyzed in detail. Third, through the analysis of environment perception when visual contamination occurs, a perception model in chameleon-inspired visual contamination for WMRs was built. Finally, to validate the proposed approach, a contingency planning experiment scheme for visual contamination was designed based on the robot’s general planning of target tracking, and the experimental result is discussed. The proposed CBR-based contingency planning approach for visual contamination can reason out effective solutions corresponding to the contamination situations. The rationality of the approach was verified by experiments with satisfactory results. Moreover, compared with the contingency planning method based on rule-based reasoning, the accuracy of target retracking after the robot visual system is contaminated is significantly higher for the CBR-based contingent planning method used in this paper.

Environment Perception with Chameleon-Inspired Active Vision Based on Shifty Behavior for WMRs

To improve the environment perception ability of wheeled mobile robots (WMRs), the visual behavior mechanism of the negative-correlation motion of chameleons is introduced into the binocular vision system of WMRs, and a shifty-behavior-based environment perception model with chameleon-inspired active vision for WMRs is established, where vision–motor coordination is achieved. First, a target search sub-model with chameleon-inspired binocular negative-correlation motion is built. The relationship between the rotation angles of two cameras and the neck and the camera’s field of view (FOV), overlapping angle, region of interest, etc., is analyzed to highlight the binocular negative-correlation motion compared with binocular synchronous motion. The search efficiency of the negative-correlation motion is doubled compared with binocular synchronous motion, and the search range is also greatly improved. Second, the FOV model of chameleon-inspired vision perception based on a shifty-behavior mode is set up. According to the different functional requirements of target searching and tracking stages, the shift of the robot visual behavior is analyzed from two aspects, measuring range and accuracy. Finally, a chameleon-inspired active-vision-based environment perception strategy for mobile robots is constructed based on the shifty-behavior mode, and experimental verification is deployed, which achieves the reproduction of the visual behavior of chameleons in the vision system of mobile robots with satisfactory results.

Ball tracking and trajectory prediction system for tennis robots

Abstract Recently, as the service robot market has grown, robots have emerged in various fields such as industry, service, and sports. In the field of sports, robots that can play with humans have been developed. We proposed a novel vision system for measuring the trajectory of a tennis ball and predicting its bound position, which can be utilized in the development of tennis robots. In this paper, we introduce a ball detection algorithm using an artificial neural network and a ball trajectory prediction algorithm using stereo vision. Our approach involved the use of a net vision system and a robot vision system to accurately detect and track the ball as it moves across the court. By combining these two systems, we were able to predict the trajectory and bound position of the tennis ball with high accuracy. As a result, the accuracy of the neural network for ball detection in actual tennis images reaches 81.4%. The ball trajectory prediction error in Gazebo simulation is 29.6 cm in the x-axis, 7.2 cm in the y-axis, and 11.7 cm in the z-axis on average.

DOPESLAM: High-Precision ROS-Based Semantic 3D SLAM in a Dynamic Environment.

Recent advancements in deep learning techniques have accelerated the growth of robotic vision systems. One way this technology can be applied is to use a mobile robot to automatically generate a 3D map and identify objects within it. This paper addresses the important challenge of labeling objects and generating 3D maps in a dynamic environment. It explores a solution to this problem by combining Deep Object Pose Estimation (DOPE) with Real-Time Appearance-Based Mapping (RTAB-Map) through means of loose-coupled parallel fusion. DOPE's abilities are enhanced by leveraging its belief map system to filter uncertain key points, which increases precision to ensure that only the best object labels end up on the map. Additionally, DOPE's pipeline is modified to enable shape-based object recognition using depth maps, allowing it to identify objects in complete darkness. Three experiments are performed to find the ideal training dataset, quantify the increased precision, and evaluate the overall performance of the system. The results show that the proposed solution outperforms existing methods in most intended scenarios, such as in unilluminated scenes. The proposed key point filtering technique has demonstrated an improvement in the average inference speed, achieving a speedup of 2.6× and improving the average distance to the ground truth compared to the original DOPE algorithm.

A novel hand-eye calibration method using double-layer optimization and outlier sample screening for monocular vision robots

In monocular vision robot systems, the hand-eye calibration approach is crucial for ensuring operational accuracy. A double-layer (Levenberg–Marquardt ([LM] method) optimization (DLMO) method with outlier points screening is proposed to reduce the influence of random errors in robot vision systems and improve the calibration accuracy of the robot hand-eye matrix. First, the equation of the hand-eye matrix is established, and the initial value of the hand-eye matrix is solved by the linear least square method. Second, the Euler angle transformation is applied to the rotation matrix part to ensure its orthogonality. Next, an optimization model of the hand-eye matrix is constructed, and the traditional LM optimization method is used to optimize the initial hand-eye matrix for the first time. Finally, the optimization model of the hand-eye matrix is modified, and the LM optimization method with the outlier sample points screening is applied to optimize the hand-eye matrix for the second time. Hand-eye calibration tests are conducted on an industrial robot equipped with a monocular vision system using the proposed method. Experimental results demonstrate that the average position error of the calibration results obtained by the proposed DLMO method is 0.22 mm, which is superior to the traditional hand-eye calibration method and meets the working requirements of the vision robot in the industrial field.

Using lightweight deep learning algorithm for real-time detection of apple flowers in natural environments

Flower thinning at the most appropriate stage could achieve high and stable yield of apple. Achieving the accurate and real-time detection of apple flowers can provide necessary technical support for the vision system of thinning robots. An apple flower detection method based on lightweight YOLOv5s algorithm was proposed. The original Backbone of YOLOv5s was replaced by ShuffleNetv2, and the Conv module of the Neck part of YOLOv5s network was replaced by Ghost module. ShuffleNetv2 reduced the memory access cost through Channel Split operation. Ghost module reduced the computing cost of the general volume layer while maintaining the similar detection performance. The combination of these two methods in the improvement of YOLOv5s network can greatly reduce the size of the model and improve the detection speed, which was convenient for the migration and application of the model. To verify the effectiveness of the model, 3005 apple flower images in different environments were used for training and testing. The Precision, Recall, and mean Average Precision (mAP) of YOLOv5s-ShuffleNetv2-Ghost model were 88.40 %, 86.10 %, and 91.80 %, respectively, the model size was only 0.61 MB, and the detection speed was 86.21 fps. The detection speed of YOLOv5s-ShuffleNetv2-Ghost model on the Jetson nano B01 development board was 2.48 fps. The results showed that the method was feasible for real-time and accurate detection of apple flowers. The research can provide technical reference for the development of orchard flower thinning robots.

An adaptive image enhancement approach for safety monitoring robot under insufficient illumination condition

The safety monitoring robots have been a potential solution to the task of patrol and inspection under uncontrolled complex environments such as tunnels. However, unclear semantic information and noises in low-light images will cause disturbance to the robot vision system. The deep learning-based enhancement is proven to advance the vision under poor illuminating conditions. However, the difficulties of excessive manual cost in collecting image pairs and image noise amplification are still challenging. In this work, we introduce a new adaptive image enhancement pipeline, which enables robots to self-adaptive to complex illumination conditions. With the pipeline, the IE-Module is proposed to learn image enhancement information from multi-scale feature blocks, and the SEnhance Module is used to eliminate the damage of denoising to normal pixels of the image. To make the algorithm better coupled with the safety monitoring robot, the algorithm is further enhanced with a filter threshold for adaptive to varied illumination conditions. The effectiveness of our method on both standard datasets and real-world scenarios was verified with experiments and real applications. The experiments on a low-light image benchmark as well as SLAM and 6D pose estimation experiments demonstrated that our method is superior to state-of-the-art methods qualitatively and quantitatively. Moreover, the proposed algorithm has been deployed in a real safety monitoring robot, which successfully demonstrates its capacity to perform patrol and inspection tasks in a low-light environment.

Method of gaze extraction in bionic vision

Aiming at the low efficiency of information processing in a robot vision system, our work studies how to extract fixed points during tasks so that significant and interesting target areas in the scene can be located. First, the architecture of a gaze extraction model of bionic vision is proposed, which consists of a spatial visual saliency model and a task-driven fixed point extraction model. The task-driven fixed point extraction process is represented as a closed-loop control problem. To couple exploration with the sampling process, a Q learning algorithm based on a random strategy is adopted, which is a top-down task-driven gaze extraction in the time dimension. It is fused with the spatial-based visual saliency model to form a spatiotemporal hybrid gaze extraction model to determine the fixed point in the final image. Finally, our qualitative image visualization experiment indicates that the fixed point extraction results of the model in two consecutive frames are close to the truth value obtained by an eye tracker. The quantitative area under the curve and average angular error data confirms the effectiveness of the model in predicting and extracting fixed points.

A Time-Delay Feedback Neural Network for Discriminating Small, Fast-Moving Targets in Complex Dynamic Environments.

Discriminating small moving objects within complex visual environments is a significant challenge for autonomous micro-robots that are generally limited in computational power. By exploiting their highly evolved visual systems, flying insects can effectively detect mates and track prey during rapid pursuits, even though the small targets equate to only a few pixels in their visual field. The high degree of sensitivity to small target movement is supported by a class of specialized neurons called small target motion detectors (STMDs). Existing STMD-based computational models normally comprise four sequentially arranged neural layers interconnected via feedforward loops to extract information on small target motion from raw visual inputs. However, feedback, another important regulatory circuit for motion perception, has not been investigated in the STMD pathway and its functional roles for small target motion detection are not clear. In this article, we propose an STMD-based neural network with feedback connection (feedback STMD), where the network output is temporally delayed, then fed back to the lower layers to mediate neural responses. We compare the properties of the model with and without the time-delay feedback loop and find that it shows a preference for high-velocity objects. Extensive experiments suggest that the feedback STMD achieves superior detection performance for fast-moving small targets, while significantly suppressing background false positive movements which display lower velocities. The proposed feedback model provides an effective solution in robotic visual systems for detecting fast-moving small targets that are always salient and potentially threatening.