Robot Vision System Research Articles

Robust visual line-following navigation system for humanoid robots

This paper implements a novel line-following system for humanoid robots. Camera embedded on the robot’s head captures the image and then extracts the line using a high-speed and high-accuracy rectangular search method. This method divides the search location into three sides of rectangle and performs image convolution by edge detection matrix. The extracted line is used to calculate relative parameters, including forward velocity, lateral velocity and angular velocity that drive line-following walking. A proposed path curvature estimation method generates the forward velocity and guidance reference point of the robot. A classical PID controller and a PID controller with angle compensation are then used to set the lateral velocity and angular velocity of the robot, improving the performance in tracking a curved line. Line-following experiments for various shapes were conducted using humanoid robot NAO. Experimental results demonstrate the robot can follow different line shapes with the tracking error remaining at a low level. This is a significant improvement from existing biped robot visual navigation systems.

Stereo Camera Head-Eye Calibration Based on Minimum Variance Approach Using Surface Normal Vectors.

This paper presents a stereo camera-based head-eye calibration method that aims to find the globally optimal transformation between a robot’s head and its eye. This method is highly intuitive and simple, so it can be used in a vision system for humanoid robots without any complex procedures. To achieve this, we introduce an extended minimum variance approach for head-eye calibration using surface normal vectors instead of 3D point sets. The presented method considers both positional and orientational error variances between visual measurements and kinematic data in head-eye calibration. Experiments using both synthetic and real data show the accuracy and efficiency of the proposed method.

Intelligent vision-based navigation system for mobile robot: A technological review

Vision system is gradually becoming more important. As computing technology advances, it has been widely utilized in many industrial and service sectors. One of the critical applications for vision system is to navigate mobile robot safely. In order to do so, several technological elements are required. This article focuses on reviewing recent researches conducted on the intelligent vision-based navigation system for the mobile robot. These include the utilization of mobile robot in various sectors such as manufacturing, warehouse, agriculture, outdoor navigation and other service sectors. Multiple intelligent algorithms used in developing robot vision system were also reviewed.

Fruit Quantity and Ripeness Estimation Using a Robotic Vision System

Accurate localization of crop remains highly challenging in unstructured environments, such as farms. Many developed systems still rely on the use of hand selected features for crop identification and often neglect the estimation of crop quantity and ripeness, which is a key to assigning labor during farming processes. To alleviate these limitations, we present a robotic vision system that can accurately estimate the quantity and ripeness of sweet pepper ( Capsicum annuum L ), a key horticultural crop. This system consists of three parts: detection, ripeness estimation, and tracking. Efficient detection is achieved using the FasterRCNN (FRCNN) framework. Ripeness is then estimated in the same framework by learning a parallel layer which we experimentally show results in superior performance than treating ripeness as extra classes in the traditional FRCNN framework. Evaluation of these two techniques outlines the improved performance of the parallel layer, where we achieve an $F_1$ score of 77.3 for the parallel technique yet only 72.5 for the best scoring (red) of the multiclass implementation. To track the crop, we present a vision only tracking via detection approach, which uses the FRCNN with parallel layers as input. Being a vision only solution, this approach is cheap to implement as it only requires a camera and in experiments, using field data, we show that our proposed system can accurately estimate the number of sweet pepper present, to within 4.1% of the visual ground truth.

A Vision-Based Navigation Robot System Using Hybrid Color Models

Vision-based navigation robot system is a hot issue in the fields of robot vision and control system. As the development of image processing technique, the application of vision-based navigation robot system is increasing in household and industrial robots. This paper designs a navigation system based on robot vision named Patio2, using hybrid color models including white balance, color detection, and connect domain building. With necessary experimental verification, the developed system can find the color square efficiently, walk towards the right direction, and stop in a certain area with limited distance of 10cm.

Design and Implementation of a Computer Vision System for an Autonomous Chess-Playing Robot

This work describes a mechatronic system composed by a robot arm that can play chess autonomously. The system is based on an industrial-grade robot manipulator, a computer vision system, and an open source chess engine. Classification algorithms were implemented in order to detect whether a given chessboard square is occupied, and in that case, if the piece is black or white. Such algorithms were compared in terms of their complexity of implementation, execution time and accuracy of predictions. To achieve an uniform illumination of the chessboard, a theoretical model of an LED illuminance curve was used to find the best orientation for each diode using a genetic algorithm. Both the support base for the LEDs and the chess pieces were made using a 3D printer. This implementation demonstrates the capabilities of the proposed vision-based system, whose complexity can be increased in the future for a number of applications.

Vision System for an Autonomous Underwater Vehicle with a Benthos Sampling Function

We developed a vision system for an autonomous underwater robot with a benthos sampling function, specifically sampling-autonomous underwater vehicle (AUV). The sampling-AUV includes the following five modes: preparation mode (PM), observation mode (OM), return mode (RM), tracking mode (TM), and sampling mode (SM). To accomplish the mission objective, the proposed vision system comprises software modules for image acquisition, image enhancement, object detection, image selection, and object tracking. The camera in the proposed system acquires images in intervals of five seconds during OM and RM, and in intervals of one second during TM. The system completes all processing stages in the time required for image acquisition by employing high-speed algorithms. We verified the effective operation of the proposed system in a pool.

Special Issue on Advanced Robotics in Agriculture, Forestry and Fisheries

The importance of primary industries, agriculture, forestry and fisheries, is obvious and needless to mention, however, the reduction of the working population and the aging problem make the situation of primary industry more sever. To compensate for the issues, the advanced technology in robotics has attracted attentions and expected the contributions in terms of productivity, cost effectiveness, pesticide-less, monitoring of the growth and harvesting, etc. Recently, robotic technologies are gradually being used in primary industry and their application area will expand more in the near future. This special issue’s objectives include collecting recent advances, automation, mechanization, research trends and their applications in agriculture, forestry and fisheries to promote a deeper understanding of major conceptual and technical challenges and facilitate spreading of recent breakthroughs in primary industries, and contribute to the enhancement of the quality of agricultural, forestry and fisheries robots by introducing the state-of-the-art in sensing, mobility, manipulation and related technologies. In this special issue, twelve papers are included. The first paper by Noguchi is the survey paper of the state-of-the-art in the agricultural vehicle type robots and discusses the future scope of agriculture with robotics. The next three papers are on tomato-monitoring system, and Fukui et al. propose a tomato fruit volume estimation method using saliency-based image processing and point cloud and clustering technology, Yoshida et al. do the cutting point identification for tomato-harvesting using a RGBD sensor and evaluate in the real farm experiments, and Fujinaga et al. present an image mosaicking method of tomato yard based on the infrared images and color images of tomato-clusters in the large green house. The fifth paper by Sori et al. reports a paddy weeding robot in wet-rice field to realize the pesticide-free produce of rice, and the sixth paper by Shigeta et al. is about an image processing system to measure cow’s BCS (Body Condition Score) automatically before milking cows and analyzes the two months data by CNN (Convolutional Neural Network). The seventh paper by Inoue et al. reports on an upper-limb power assist robot with a single actuator to reduce the weight and cost. The assist machine supports the shoulder and elbow movements for viticulture operations and upper-limb holding for load transport tasks. In the next paper, Tominaga et al. show an autonomous robotic system to move between the trees without damaging them and to cut the weeds in the forest for the forest industry. The last four papers are for the fishery industry, and Komeyama et al. propose a methods for monitoring the size of fish, red sea bream (RSB) aquaculture by developing a stereo vision system to avoid the risks of physical injury and mental stress to the fish. Nishida et al. report on a hovering type underwater robot to measure seafloor for monitoring marine resources whose sensor can be replaced depending on missions as the open hardware system. Yasukawa et al. propose a vision system for an autonomous underwater robot with a benthos sampling function, especially, sampling-autonomous underwater vehicles (SAUVs) to achieve a new sampling mission. The last paper by Han et al. is for gait planning and simulation analysis of an amphibious quadruped robot in the field of fisheries and aquaculture. We hope that this special issue can contributes to find solutions in primary industries, agriculture, forestry and fisheries.

An Uncalibrated Visual Servo Method Based on Projective Homography

An uncalibrated visual servo method based on projective homography, denoted as Projective Homography based Uncalibrated Visual Servoing (PHUVS), is proposed in this paper, in which a novel task function based on the element of projective homography is devised to realize visual servo without a prior knowledge of the camera intrinsic parameters and hand-eye relationships. The main advantage of this method is that it is not only suitable for totally uncalibrated scenarios but also cheap in computation costs when compared with classical image-based uncalibrated visual servoing methods. Numerical experiments are performed and the results confirm that the new approach is capable of both static positioning and dynamic tracking tasks, and presents competitive computational efficiency and accuracy performance. Note to Practitioners —Vision guided robot has been widely used in industries. However the industrial state-of-art robot visual servo systems usually require system calibration, including camera calibration, hand-eye calibration, and robot calibration. These calibration process are time consuming and requires some expertise, which is rare for ordinary workers. Moreover, when hand-eye configuration changed due to certain reasons such as adjustment of production line layout or displacement due to machining force or switch of robot end effector, the calibrated parameters are no longer accurate. This paper offers a plug and play approach for eye-in-hand robot visual servo tasks. When a robot-camera system is set up, there is no need for the operator to calibrate the system. With reference (goal) image and the current images obtained by the moving camera, the robot can autonomously achieve the goal pose which is described simply by the reference camera. With our novel task function, the computation cost is reduced, which will benefit the real-time application. The accuracy and some dynamic performance in both static positioning and dynamic target tracking are evaluated by robotic toolbox. This approach has the potential to improve the flexibility and robustness of robot system in robotic manipulation tasks such as assemble and painting, or robotic machining tasks such as matching the origin point of operating trajectory in large part manufacturing. Furthermore, this approach is generic, and can be extended to mobile robot visual servo control tasks.

Determination of optimal measurement configurations for self-calibrating a robotic visual inspection system with multiple point constraints

In this paper, we propose an algorithm to determine optimal measurement configurations for self-calibrating a robotic visual inspection system with multiple point constraints. The algorithm aims to improve the robotic visual inspection system’s calibration accuracy. To do so, a pre-calibration of the robotic visual inspection system is needed to obtain the hand-eye and robot exterior relationship to implement the inverse kinematic algorithm. The candidate measurement configurations with one point constraint can be obtained using the inverse kinematic algorithm for the robotic visual inspection system, so DETMAX is implemented to determine a given number of optimal measurement configurations from the candidate measurement configurations. Particle swarm optimization is used to optimize the positions of the multiple points one by one. To verify the efficiency of the proposed approach, experiment evaluation is conducted on a robotic visual inspection system.

Design of an Embedded Multi-Camera Vision System—A Case Study in Mobile Robotics

The purpose of this work is to explore the design principles for a Real-Time Robotic Multi Camera Vision System, in a case study involving a real world competition of autonomous driving. Design practices from vision and real-time research areas are applied into a Real-Time Robotic Vision application, thus exemplifying good algorithm design practices, the advantages of employing the “zero copy one pass” methodology and associated trade-offs leading to the selection of a controller platform. The vision tasks under study are: (i) recognition of a “flat” signal; and (ii) track following, requiring 3D reconstruction. This research firstly improves the used algorithms for the mentioned tasks and finally selects the controller hardware. Optimization for the shown algorithms yielded from 1.5 times to 190 times improvements, always with acceptable quality for the target application, with algorithm optimization being more important on lower computing power platforms. Results also include a 3-cm and five-degree accuracy for lane tracking and 100% accuracy for signalling panel recognition, which are better than most results found in the literature for this application. Clear results comparing different PC platforms for the mentioned Robotic Vision tasks are also shown, demonstrating trade-offs between accuracy and computing power, leading to the proper choice of control platform. The presented design principles are portable to other applications, where Real-Time constraints exist.

Target detection and recognition algorithm for moving UAV based on machine vision

To solve the shortcomings that current robot vision system accuracy is instable, unit operation time is overlong and automatic compensation function is lacked, this paper proposes robot vision system based on trinocular vision from the perspective of system hardware structure improvement and software algorithm optimization. Firstly, logic communication channel among PLC sensor, robot and vision PC shall be established, to realize real-time acquisition and processing of operation condition of material. Then design nine-point calibration method according to scaling, rotating and translation parameters between robot and vision coordinate system, and calibrate three industrial cameras to realize accurate binding of image pixel coordinate and robot world coordinate. Finally, choose region-of-interest, optimize the minimum mean square error matching algorithm and realize balance of system accuracy and speed according to project standard of product. Experimental test result shows that compared with current robot vision system, system in this paper has higher vision accuracy and operation efficiency.

Dynamics-Based Motion Deblurring Improves the Performance of Optical Character Recognition During Fast Scanning of a Robotic Eye

This paper presents a quantitative evaluation of the dynamics-based deblurring method using an optical character recognition (OCR) technology. Although various image deblurring algorithms have been studied, there has been no standard performance metric; deblurred images have often been evaluated in a qualitative manner. In this study, blurry images containing alphanumeric characters were obtained in the course of rapid motion using a robotic vision system. The obtained blurry images were recovered by the dynamics-based deblurring method. For a quantitative evaluation, OCR rates from the deblurred images by the dynamics-based method were calculated and compared with those by other well-known methods. Experiment results show that the dynamics-based method has the best quantitative results.

Simultaneous 3D Object Recognition and Pose Estimation Based on RGB-D Images

Object recognition and pose estimation are essential functions in applications of computer vision, and they also are fundamental modules in robotic vision systems. In recent years, RGB-D cameras become more and more popular, and the 3D object recognition technology has got more and more attention. In this paper, a novel design of simultaneous 3D object recognition and pose estimation algorithm is proposed based on RGB-D images. The proposed system converts the input RGB-D image to colored point cloud data and extracts features of the scene from the colored point cloud. Then, the existing color signature of histograms of orientations (CSHOT) description algorithm is employed to build descriptors of the detected features based on local texture and shape information. Given the extracted feature descriptors, a two-stage matching process is performed to find correspondences between the scene and a colored point cloud model of an object. Next, a Hough voting algorithm is used to filter out matching errors in the correspondence set and estimate the initial 3D pose of the object. Finally, the pose estimation stage employs RANdom SAmple Consensus (RANSAC) and hypothesis verification algorithms to refine the initial pose and filter out poor estimation results with error hypotheses. Experimental results show that the proposed system not only successfully recognizes the object in a complex scene but also accurately estimates the 3D pose information of the object with respect to the camera.

Active stabilization in robotic vision systems

In the article considered prospective approaches to the design of active systems for stabilizing systems based on the parallel kinematics mechanism and possible applications of such systems. Attention is drawn to the fact that not only object fluctuations are an important object for stabilization, but it is also important to compensate for the body vibrations, along with its vibrations. Based on the analysis, it was concluded that it is perspective to use mechanisms with parallel kinematics for the design of active stabilization systems. Was obtained a mathematical model of the hexapod, according to which a computer model in the Simulink package was designed. Its analysis confirmed the possibility of using a mechanism with parallel kinematics in designing an active stabilization system and presented requirements to the actuators of the system.

Algorithm for recognition of details in the machine vision systems at automation of assembly processes

The aim is an experimental research on the flat objects recognition using dimensionless marks of the contours of their binary images and determining the possibility of applying this method in computer vision systems of assembly robots. The main problem with the automation of assembly operations is the recognition of parts for the subsequent picking up of the robot arm. The basis for the formation of attribute vectors is the characteristics of the image contour. Recognition of a class of an unknown object consists in receipt of its contour, calculation of primary parameters and forming of a vector of dimensionless marks. Further mean square deviations of its vector of dimensionless marks from all reference are calculated. The minimum value of a deviation will specify probable belonging to the corresponding class.

Experiments on mobile robot stereo vision system calibration under hardware imperfection

Calibration is essential for any robot vision system for achieving high accuracy in deriving objects metric information. One of typical requirements for a stereo vison system in order to obtain better calibration results is to guarantee that both cameras keep the same vertical level. However, cameras may be displaced due to severe conditions of a robot operating or some other circumstances. This paper presents our experimental approach to the problem of a mobile robot stereo vision system calibration under a hardware imperfection. In our experiments, we used crawler-type mobile robot «Servosila Engineer». Stereo system cameras of the robot were displaced relative to each other, causing loss of surrounding environment information. We implemented and verified checkerboard and circle grid based calibration methods. The two methods comparison demonstrated that a circle grid based calibration should be preferred over a classical checkerboard calibration approach.

Color Transfer Pulse-Coupled Neural Networks for Underwater Robotic Visual Systems

With rapid developments in cloud computing, artificial intelligence, and robotic systems, ever more complex tasks, such as space and ocean exploration, are being implemented by intelligent robots. Here, we propose an underwater image enhancement scheme for robotic visual systems. The proposed algorithm and its implementation enhances and outputs an image captured by an underwater robot in real time. In this scheme, pulse-coupled neural network (PCNN)-based image enhancement and color transfer algorithms are combined to enhance the underwater image. To avoid color imbalance in the underwater image and enhance details while suppressing noise, color correction is first carried out on the underwater image before converting it into the hue–saturation–intensity domain and enhancing it by PCNN. The enhanced result improves the color and contrast of the source image and enhances the details and edges of darker regions. Experiments are performed on real world data to demonstrate the effectiveness of the proposed scheme.

From biokinematics to a robotic active vision system

Barn owls move their heads in very particular motions, compensating for the quasi-immovability of their eyes. These efficient predators often perform peering side-to-side head motions when scanning their surroundings and seeking prey. In this work, we use the head movements of barn owls as a model to bridge between biological active vision and machine vision. The biomotions are measured and used to actuate a specially built robot equipped with a depth camera for scanning. We hypothesize that the biomotions improve scan accuracy of static objects. Our experiments show that barn owl biomotion-based trajectories consistently improve scan accuracy when compared to intuitive scanning motions. This constitutes proof-of-concept evidence that the vision of robotic systems can be enhanced by bio-inspired viewpoint manipulation. Such biomimetic scanning systems can have many applications, e.g. manufacturing inspection or in autonomous robots.

Object Detection and Recognition for Assistive Robots: Experimentation and Implementation

Technological advances are being made to assist humans in performing ordinary tasks in everyday settings. A key issue is the interaction with objects of varying size, shape, and degree of mobility. Autonomous assistive robots must be provided with the ability to process visual data in real time so that they can react adequately for quickly adapting to changes in the environment. Reliable object detection and recognition is usually a necessary early step to achieve this goal. In spite of significant research achievements, this issue still remains a challenge when real-life scenarios are considered. In this article, we present a vision system for assistive robots that is able to detect and recognize objects from a visual input in ordinary environments in real time. The system computes color, motion, and shape cues, combining them in a probabilistic manner to accurately achieve object detection and recognition, taking some inspiration from vision science. In addition, with the purpose of processing the input visual data in real time, a graphical processing unit (GPU) has been employed. The presented approach has been implemented and evaluated on a humanoid robot torso located at realistic scenarios. For further experimental validation, a public image repository for object recognition has been used, allowing a quantitative comparison with respect to other state-of-the-art techniques when realworld scenes are considered. Finally, a temporal analysis of the performance is provided with respect to image resolution and the number of target objects in the scene.