Robot Vision System Research Articles

Visual Perception for Multiple Human–Robot Interaction From Motion Behavior

Visual perception is an important component for human–robot interaction processes in robotic systems. Interaction between humans and robots depends on the reliability of the robotic vision systems. The variation of camera sensors and the capability of these sensors to detect many types of sensory inputs improve the visual perception. The analysis of activities, motions, skills, and behaviors of humans and robots have been addressed by utilizing the heat signatures of the human body. The human motion behavior is analyzed by body movement kinematics, and the trajectory of the target is used to identify the objects and the human target in the omnidirectional (O-D) thermal images. The process of human target identification and gesture recognition by traditional sensors have problem for multitarget scenarios since these sensors may not keep all targets in their narrow field of view (FOV) at the same time. O-D thermal view increases the robots’ line-of-sights and ability to obtain better perception in the absence of light. The human target is informed of its position, surrounding objects and any other human targets in its proximity so that humans with limited vision or vision disability can be assisted to improve their ability in their environment. The proposed method helps to identify the human targets in a wide FOV and light independent conditions to assist the human target and improve the human–robot and robot–robot interactions. The experimental results show that the identification of the human targets is achieved with a high accuracy.

Target Recognition Method of Rehabilitation Robot Based on Image Local Features

Target recognition technology is an important topic in the field of artificial intelligence, which is widely used in fields such as medicine, robot vision, highway traffic, and VR technology. In the robot vision system, the target recognition technology requires high precision, real-time, and practicality, and it must be able to cope with the harsh recognition environment on site. This paper mainly studies the target recognition method of rehabilitation robot based on image local features. In this paper, a rehabilitation robot recognition system is established based on the local features of the image and the target recognition technology. Firstly, after collecting or processing the initially prepared images, feature extraction is performed, and the feature vectors extracted from the features are classified and recognized and then transferred to the recognition model of the rehabilitation robot. Then, according to the functional requirements of the rehabilitation robot recognition system, the “one-to-many” rehabilitation mode and real-time system monitoring are realized to accurately identify the target state. Finally, the Harris algorithm is used to convert the predetermined image set into a grayscale image, and the Gaussian distribution is used uniformly to determine the location and proportion of the feature points, and extract the corresponding image. The Harris matching method used in this article is very effective and has the highest accuracy. The number of correct matches accounted for 58.3%. Compared with the SIFT algorithm and the Harris+SIFT algorithm, the accuracy rate increases by 12.3% and 14.4%, respectively. The recall rate increased by 13.2% and 8.9% respectively. The experimental results show that the target recognition technology of rehabilitation robot based on image local features is more accurate than general technology.

Rapid Fine-Grained Classification of Butterflies Based on FCM-KM and Mask R-CNN Fusion

Butterfly recognition is a key link in the field of animal and plant observation. In order to realize the location and recognition of butterflies by robot vision system in complex environment, rapid fine-grained classification of butterflies based on the FCM-KM and Mask R-CNN fusion was proposed. First, an adaptive image enhancement algorithm based on fuzzy sets optimized by FOA was used to realize the adaptive fuzzy enhancement of butterfly images in image pre-processing. Then, K-Means clustering algorithm optimized by dynamic population firefly algorithm based on chaos theory and max-min distance algorithm, FCM-KM, was used to determine the optimal clustering number K instead of manual parameter tuning. Finally, while effectively segmenting the butterfly images, the Softmax in Mask R-CNN was used to classify the butterfly images. The recognition accuracy of the trained model in the verification set was 83.62%. To verify the feasibility and effectiveness of the model in complex environment, the rapid fine-grained classification method of butterflies based on the FCM-KM and Mask R-CNN fusion was compared with CNN, Resnet and original Mask R-CNN. The experiment results show that the method proposed in this paper has a good classification effect on butterflies in complex environment.

Integrated computer vision system of autonomous robots

Integrated computer vision system of autonomous robots

Research on Structural Design and Vision System of Narrow Shallow Water Multipurpose Underwater Robot

Research on Structural Design and Vision System of Narrow Shallow Water Multipurpose Underwater Robot

РЕАЛІЗАЦІЙНА МОДЕЛЬ АДАПТИВНОГО СУМАТОРА ДЛЯ НЕЙРОПОДІБНИХ ЕЛЕМЕНТІВ

One of the promising areas for the use of neurotechnologies is robotics, namely, systems of technical vision and control systems for mobile robots of various applications. In particular, one of the basic tasks for these systems as part of autonomous robots is the task of object recognizing and determining of the obstacles contours in the movement of mobile robots in a non-deterministic environment. For a compact and reliable implementation of the basic units of these systems, there is no alternative) the use of neural network technologies with to focusing on perspective modern tools (FPGA). It is necessary to take into account the simultaneous perception of visual information, which requires, in turn, parallel spatially distributed processing of large amounts of information. The work proposes the structure of the adaptive adder in composition of artificial neurons, which are basic neural-like elements of different types of neural networks. The proposed pipeline summing device has advanced functionality, as it simulates the operation of the adaptive adder in the formal neuron with the formation of the result of processing taking into account the external bias with the sign, and also performs parallel summation of vector array numbers with the formation of their sum. The proposed adaptive adder has a regular structure consisting of (n+1) cells with almost the same set of units and connections between them, and also implements a spatially distributed process of parallel processing over n input elements of the vector array. All this simplifies the process of placing the adaptive adder in the FPGA chip. The orientation on functionally and technologically powerful FPGA chips allows to get compact and full-featured neurostructures for various purposes, the need for which is extremely important in the control systems of mobile robots.

Automatic Image and Video Caption Generation With Deep Learning: A Concise Review and Algorithmic Overlap

Methodologies that utilize Deep Learning offer great potential for applications that automatically attempt to generate captions or descriptions about images and video frames. Image and video captioning are considered to be intellectually challenging problems in imaging science. The application domains include automatic caption (or description) generation for images and videos for people who suffer from various degrees of visual impairment; the automatic creation of metadata for images and videos (indexing) for use by search engines; general-purpose robot vision systems; and many others. Each of these application domains can positively and significantly impact many other task-specific applications. This article is not meant to be a comprehensive review of image captioning; rather, it is a concise review of both image captioning and video captioning methodologies based on deep learning. This study treats both image and video captioning by emphasizing the algorithmic overlap between the two.

Non-destructive evaluation of photostability of crop signaling compounds and dose effects on celery vigor for precision plant identification using computer vision

Vegetable crops grown in the field are particularly vulnerable to competition from weeds at the seedling stage. To avoid yield loss, weeds must be removed early in the crop cycle. Shortages of farm laborers for hand weeding and lack of effective vegetable herbicides is stimulating the development of intelligent weeders. One challenge for development of effective intelligent weeders is the precision differentiation of vegetable crops and weeds. Crop signaling is a technique that enables rapid and accurate identification of target crops. In this study, an appropriate dose of fluorescent compounds allowed the vegetable crop locations in the field to be reliably detectable by smart machines. The study examined protocols for crop root treatment, computer vision system development, crop signal detection and biomass evaluation. Rhodamine B (Rh-B) is a fluorescent compound with unique optical properties. Different doses of Rh-B were applied to the celery roots for various durations prior to transplantation to assess Rh-B transport in the plant, its photostability and potential impact on seedling growth in the natural outdoor environment of a farm. Systemic Rh-B absorbed via the roots moved throughout the celery plant in 24 h. Compared with 60 ppm solution of Rh-B, higher doses of Rh-B including 90, 180, and 270 ppm resulted in greater absorption by the plant, but such three concentrations injured the plant. The biomass test results showed that the treatment of celery roots for two days with a 60 ppm solution of Rh-B was safe for celery plants. This Rh-B dosage had good photostability in celery seedlings for about 5 weeks after transplanting. An effective dose of systemic Rh-B allowed for the rapid identification of celery plants at early growth stages, thereby facilitating the automatic differentiation of weeds and crops by a robotic machine vision system.

Faster R-CNN for multi-class fruit detection using a robotic vision system

An accurate and real-time image based multi-class fruit detection system is important for facilitating higher level smart farm tasks such as yield mapping and robotic harvesting. Robotic harvesting can reduce the costs of labour and increase fruit quality. This paper proposes a deep learning framework for multi-class fruits detection based on improved Faster R-CNN. The proposed framework includes fruits image library creation, data argumentation, improved Faster RCNN model generation, and performance evaluation. This work is a pioneer to create a multi-labeled and knowledge-based outdoor orchard image library using 4000 images in the real world. Also, improvement of the convolutional and pooling layers is achieved to have a more accurate and faster detection. The test results show the proposed algorithm has achieved higher detecting accuracy and lower processing time than the traditional detectors, which has excellent potential to build an autonomous and real-time harvesting or yield mapping/estimation system.

Target extraction of fruit picking robot vision system

Real-time and accurate target recognition and extraction of crop fruit images in the field is the key technology for picking robot vision system, and the essence of target extraction is image segmentation. When most of the water (vegetable) fruit is in the picking period, the surface color and the background color are quite different, while the surface colors of the same variety are similar, which are reflected in the different distribution characteristics of the fruit surface color and the background color in the color space. According to this characteristic, an image segmentation algorithm based on color space reference table for fruit target extraction of fruit picking robot vision system is proposed. The algorithm first establishes a color space reference table from the fruit sample image, and then uses a method similar to “convolution” to segment the image according to the color space reference table. Compared with other existing methods, the method is based on color information processing, which can remove the background more cleanly. Without segmentation processing and no complicated operation on the background, the method is beneficial to real-time image processing of the robot.

Structural design of anthropomorphic robot vision system

Anthropomorphic robot is one of the latest research directions in the field of intelligent robot research. It is an intelligent robot with similar shape and humanity. It has mobile function, operation function, sensing function, memory and autonomy, and can realize human-computer interaction. Robot technology has always been a hotspot of technology that people pay attention to. Because of the diverse development of robots, machine vision systems are gradually being established. The machine vision system realizes the functions of collecting images, analyzing images, acquiring information, processing images and others by simulating the human eye. It is a comprehensive technology integrating automation and computer. The article deeply studies the hardware composition and image processing technology of machine vision system and gives an example study on the application of machine vision in various fields. It also analyzes the current development status of machine vision system in China and looks forward to the bright future of machine vision.

Mobile robot visual navigation based on fuzzy logic and optical flow approaches

This paper presents the design of mobile robot visual navigation system in indoor environment based on fuzzy logic controllers (FLC) and optical flow (OF) approach. The proposed control system contains two Takagi–Sugeno fuzzy logic controllers for obstacle avoidance and goal seeking based on video acquisition and image processing algorithm. The first steering controller uses OF values calculated by Horn–Schunck algorithm to detect and estimate the positions of the obstacles. To extract information about the environment, the image is divided into two parts. The second FLC is used to guide the robot to the direction of the final destination. The efficiency of the proposed approach is verified in simulation using Visual Reality Toolbox. Simulation results demonstrate that the visual based control system allows autonomous navigation without any collision with obstacles.

Visual Perception System Guides Robots

AbstractThe perception abilities of industry robots and service robots alike need to be improved in order to enable man‐machine collaboration in the smart factory and reach higher autonomy of unmanned service vehicles. However, integration of intelligent computer vision solutions into robots still poses challenges, despite major developments such as the miniaturization of vision components. Therefore, the task is to develop an easy‐to‐integrate robot vision system that not only provides “eyes” to see anywhere, but also has a “brain” to create visual perception.

A real-time automated sorting of robotic vision system based on the interactive design approach

This research paper presents the proposes a robotic vision system to distinguish the color for the object and his position coordinate, and then sort the object (product) on the right branch conveyor belt according to color in real-time. The system was built based on the HVS mode algorithm for sorting product based on color. Furthermore, the system can be distinguished the object shape and then find his position to picking the object shape and putting on the right branch conveyor belt. The assumptions for the object shape were based on the shape properties, centroid algorithm, and border extraction. Both the object detection and the contour coordinate extraction methods are implemented using a series of image processing techniques. The main goal is met by sorting the object depends on the color feature from a gathering of objects. The robot movement (open and close griper, move up and down the arm, and move to the left and right) controlled by a microcontroller that controls the movement to the right branch conveyor belt. When the color or the object is detected, the microcontroller will initiate the actions of the robot. It was found that the accuracy of results based on the approach that developed in this paper which is 92% for shape sorting and 97% for colors sorting objects.

Visual Servo Tracking of Wheeled Mobile Robots With Unknown Extrinsic Parameters

In most visual servoing methods for wheeled mobile robots, the camera is supposed at the center of the robot, while it will bring much convenience for practical use when putting the camera a little away from the robot center. In this paper, to handle the challenge that is incurred by the uncalibrated translational camera-to-robot parameters, a novel visual servoing scheme is designed for wheeled mobile robots to track a given trajectory, which is defined by a sequence of images recorded previously. The trajectory tracking errors are defined with respect to both current and desired poses of the robot, and the open-loop error system is obtained after analyzing kinematic models of the visual robot system. Subsequently, to compensate both unknown extrinsic parameters and the visual depth, an adaptive visual servo tracking controller is elaborately designed, with disposing the nonholonomic motion constraint of the mobile robot. The tracking errors are strictly proven converging to zero asymptotically by employing Lyapunov techniques and Barbalat's lemma. Both simulation and comparative experiments are conducted to prove that the proposed strategy can drive the robot track the desired trajectory efficiently, even translational camera-to-robot parameters are unknown.

An image vision and automatic calibration system for universal robots

The rapid development of technology is causing the replacement of many traditional manufacture industries by automation. Robotic arms are now commonly used in many sectors. The requirements for robotic arms in different sectors are quite different; but, in general, robotic arms not only save cost and man power, they also improve safety. The aim of this study was an investigation of the integration of image identification with robotic arms. To do this, the Denavit–Hartenberg transformation matrix was used to analyze the mechanical kinematics of the joints of each robotic arm axis. This allowed the spatial relationships between the Cartesian 3 D coordinate system and the joint of each axis to be determined and communication between the robotic arm and image identification to be established. A robotic arm prototype platform with automatic image identification and calibration was constructed using a quick and robust method. Several of the variables that exist in real robotic arm applications have been solved in this study: primarily the accuracy error that occurs when repeated gripping of workpieces is done and a movement and placement track is set up. Deviations occur frequently and if image identification can be applied to offset repeated inaccuracy, the quality of finished products will be more consistent. The Universal Robots 5 six-axis robot and cameras, which use a filter, Sobel computation, and Hough transformation computer vision image processing technology, together comprise a system that can automatically identify the workpiece and carry out loading and unloading accurately.

Active Eye-in-Hand Data Management to Improve the Robotic Object Detection Performance

Adding to the number of sources of sensory information can be efficacious in enhancing the object detection capability of robots. In the realm of vision-based object detection, in addition to improving the general detection performance, observing objects of interest from different points of view can be central to handling occlusions. In this paper, a robotic vision system is proposed that constantly uses a 3D camera, while actively switching to make use of a second RGB camera in cases where it is necessary. The proposed system detects objects in the view seen by the 3D camera, which is mounted on a humanoid robot’s head, and computes a confidence measure for its recognitions. In the event of low confidence regarding the correctness of the detection, the secondary camera, which is installed on the robot’s arm, is moved toward the object to obtain another perspective of the object. With the objects detected in the scene viewed by the hand camera, they are matched to the detections of the head camera, and subsequently, their recognition decisions are fused together. The decision fusion method is a novel approach based on the Dempster–Shafer evidence theory. Significant improvements in object detection performance are observed after employing the proposed active vision system.

Flexible FTIR Spectral Imaging Enhancement for Industrial Robot Infrared Vision Sensing

Infrared (IR) spectral imaging sensing is a powerful visual technique for industrial material recognition in robot vision systems. However, the imaging sensing data have issues of random noise and band overlap. Resolution enhancement is usually the first step in the preprocessing procedure of industrial robot vision sensing. In this article, we develop a resolution-enhancement algorithm with total variation (TV) constraints for the degraded Fourier transform IR (FTIR) spectrum due to overlap and noise degradation in the robot vision sensing. The kernel function is calculated using the spectrometer imaging systems and Fourier optical theory. The proposed model not only can remove noises effectively but also can estimate the kernel function because of the adaptive TV as constraint regularization. This model is examined by a set of simulated FTIR spectra with the Poisson noises and a series of real FTIR spectra. The proposed model is compared with the other state-of-the-art methods in terms of performance. Experimental results demonstrate that the proposed approach can split the overlap band effectively while the spectral structure details are retained satisfactorily. The enhanced high-resolution imaging spectrum data can raise the robot vision sensing accuracy in industrial intelligent systems.

Optimal Trajectory Planning of Eye-head Gaze Shifts for Bionic Robot Vision System on Orchard Environment Perception

To make orchard robots obtain environmental information quickly and smoothly, this paper studies the neuro-mechanism of primates’ gaze shifts. Meanwhile, according to the optimal control theory, this paper considers time and energy as factors of formulating cost function and then analyses kinematic relationship between eye and head during gaze shifts to find constraint conditions. Solving the two-point boundary value differential equations of optimal control, this paper can extract the best control signal of coordinated eye and head movements. By building a multi-degree-of-freedom robot vision system corresponding to the biological model, this paper simulates primates’ gaze shifts and verifies the feasibility and validity of the method proposed.

Optical Flow for Real-Time Human Detection and Action Recognition Based on CNN Classifiers

Mobile robots equipped with camera sensors are required to perceive surrounding humans and their actions for safe and autonomous navigation. In this work, moving humans are the target objects. For robot vision, real-time performance is an important requirement. Therefore, we propose a robot vision system in which the original images captured by a camera sensor are described by optical flow. These images are then used as inputs to a classifier. For classifying images into human and not-human classifications, and the actions, we use a convolutional neural network (CNN), rather than coding invariant features. Moreover, we present a local search window as a novel detector for clipping partial images around target objects in an original image. Through the experiments, we ultimately show that the robot vision system is able to detect moving humans and recognize action in real time.