Robot Vision System Research Articles

Modelling of Eyeball with Pan/Tilt Mechanism and Intelligent Face Recognition Using Local Binary Pattern Operator

This paper describes the vision system for a humanoid robot, which includes the mechanism that controls eyeball orientation and blinking process. Along with the mechanism designed, the orientation of the camera, integrated with controlling servomotors. This vision system is a bio-mimic, which is designed to match the size of human eye. This prototype runs face recognition and identifies, matches with a face in the database. Recognition of face leads to capture the facial image and synchronize with the face. As the individual shows any motion, the system also moves according to it.

A wheel-type in-pipe robot for grinding weld beads

To improve the safety and efficiency of polishing operations in circular boiler headers, a new type of wheel-drive polishing robot was developed in this study. The robot was designed to grind weld beads on the inner walls of pipes in diameter between 550 mm and 714 mm. The robot consists of a moving structure, a positioning structure, and a polishing structure. Charge coupled device (CCD) cameras and line lasers are used in the robot's vision system, thus the robot can be manually controlled to move, locate, and grind quickly and accurately. The experimental results showed that the robot performed well in practical applications.

Visual Servo Technology Research of Industrial Palletizing Robot

With the wide application of palletizing robot in the field of logistics automation, The market is making higher demand for the adaptability and accuracy of the palletizing robot. In order to solve the problems of single function and poor adaptability for the ordinary palletizing robot, we combined the machine vision and palletizing robot control technology, built the palletizing robot visual servo control system platform, carried out the related graphic processing algorithms research. By test and verification, this palletizing robot visual servo control system platform can finish the detection, sorting and stacking for different goods fast and accurately. It can offer reference for the subsequent research of the palletizing robot visual servo technology.

HYBRID COMMUNICATION NETWORK OF MOBILE ROBOT AND QUAD-COPTER

This paper introduces the design and development of QMRS (Quadcopter Mobile Robotic System). QMRS is a real-time obstacle avoidance capability in Belarus-132N mobile robot with the cooperation of quadcopter Phantom-4. The function of QMRS consists of GPS used by Mobile Robot and image vision and image processing system from both robot and quad-copter and by using effective searching algorithm embedded inside the robot. Having the capacity to navigate accurately is one of the major abilities of a mobile robot to effectively execute a variety of jobs including manipulation, docking, and transportation. To achieve the desired navigation accuracy, mobile robots are typically equipped with on-board sensors to observe persistent features in the environment, to estimate their pose from these observations, and to adjust their motion accordingly. Quadcopter takes off from Mobile Robot, surveys the terrain and transmits the processed Image terrestrial robot. The main objective of research paper is to focus on the full coordination between robot and quadcopter by designing an efficient wireless communication using WIFI. In addition, it identify the method involving the use of vision and image processing system from both robot and quadcopter; analyzing path in real-time and avoiding obstacles based-on the computational algorithm embedded inside the robot. QMRS increases the efficiency and reliability of the whole system especially in robot navigation, image processing and obstacle avoidance due to the help and connection among the different parts of the system.

Visual and Robotic Guidance Systems for Transcatheter Implantation of Heart Value Prostheses

Transcatheter heart valve prosthesis implantation (TCHVPI) is a complex method whose results depend on each of the components of the biotechnological system, such that complications cannot always be avoided. New medical technologies, namely robot systems (RS) and visual guidance systems (VGS) have been developed to counter these problems and make the procedure less dependent on the human factor. This review discusses the main current and potential VGS and RS technologies; the role of their integration is discussed and the main requirements for next-generation TCHVPI RS are evaluated.

Branch localization method based on the skeleton feature extraction and stereo matching for apple harvesting robot

Aiming at the problem of apple branch obstacle localization in fruit picking process of harvesting robot manipulator, in order to obtain three-dimensional information of the apple branch obstacle, the binocular stereo vision localization method of apple branch obstacle is proposed. Firstly, branch skeleton is extracted by morphological thinning method and then the feature skeleton is obtained after removing the false branch and recovering the occluded branch. After that, the endpoints and bifurcation points regarded as match feature points are extracted from skeleton, and the stereo matching algorithm based on features is adopted. Then, the depth information of branch obstacle is obtained on the basis of triangulation theory. Finally, the experiment results for apple tree branches localization show that the error lies in ±6.2 mm. Moreover, the error is merely ±1.5 mm when the distance between the object and the binocular camera is 1000 mm, which meets with localization accuracy requirements of apple harvesting robot visual system.

Cherry recognition in natural environment based on the vision of picking robot

In order to realize the automatic recognition of cherry in the natural environment, this paper designed a robot vision system recognition method. The first step of this method is to pre-process the cherry image by median filtering. The second step is to identify the colour of the cherry through the 0.9R-G colour difference formula, and then use the Otsu algorithm for threshold segmentation. The third step is to remove noise by using the area threshold. The fourth step is to remove the holes in the cherry image by morphological closed and open operation. The fifth step is to obtain the centroid and contour of cherry by using the smallest external rectangular and the Hough transform. Through this recognition process, we can successfully identify 96% of the cherry without blocking and adhesion.

Methods of Study Requirements for the Complex Robotic Vision System

The article presents a methodology for substantiation of requirements for the technical vision system of a robotic complex. A technical vision system of the robotic complex is viewed as a combination of two subsystems: measurement and recognition. To implement the methodology we developed methods for calculating partial optimality criteria for substantiation of the technical requirements and evaluation of search area of the optimal values of measuring instruments characteristics of the technical vision system of a robotic complex; a recursive procedure for choosing the optimal values of measuring instruments characteristics of the technical vision system; a scheme of trade-off for evaluating the optimal technical requirements for advanced measurement instruments of the technical vision system in different technical and economical conceptions. The search for optimal solution is done according to partial optimality criteria: recognition efficiency, the cost and risks of creating measuring instruments. For creating the recursive procedure based on formulated assumptions and assertions, a criterion, which provides the search of Pareto-optimal solutions, was synthesized. The developed methodology takes into account the existing (more suitable) technical and economical conceptions of creating a robotic complex while choosing trade-offs.

Combined Vision System for Mobile Robots

The article is devoted to examination of the method for formation of a panoramic image in conjunction with formation of detailed fragments of space objects for the vision systems of the mobile robots. As is known, navigation of the mobile robots often should take into account three coordinates in conjunction with a detailed analysis of the individual objects. It is a common assumption that a vision system operates with a certain set of specific objects, which must be prepared or in some way marked and allocated. The dynamic objects also should be taken into account in the field review. It is proposed that the formation of three-dimensional images of the space objects should be done by combining the stereoscopy methods and highlighting the structural aspect. In preparation of the materials for publication a system was considered which combines vision with a panoramic optoelectronic system, which implements the method of the structural illumination and stereovision, at the same time.

Method for assessing the quality of data used in evaluating the performance of recognition algorithms for fruits and vegetables

In studies on agricultural robot vision systems, data used to evaluate algorithm performance, such as successful recognition rates, vary because of various factors. If the variation is too large, representation of the actual performance of algorithms by the data is bound to be poor. Here we present a method for analysing the quality of data used to evaluate the performance of a recognition algorithm for occluded tomatoes based on measurement system analysis. The measurement system included a soft measurement tool (a counting method for the number of successful recognitions), appraisers, measured objects (recognition results of 300 occluded tomato images), the usage method for the soft measurement tool and measurement environments. The measurement system was analysed on the basis of its repeatability and reproducibility. Repeatability and reproducibility were both evaluated based on Fleiss's Kappa values, free-marginal multirater Kappa values and Kendall coefficients. Test results showed that repeatability was excellent or fair to good based on Fleiss's Kappa values and excellent based on free-marginal multirater Kappa values and Kendall coefficients for the three appraisers. Further improvement in the soft type of measurement tool is necessary. Reproducibility was fair to good with Fleiss's Kappa values and free-marginal multirater Kappa values, and good with Kendall coefficients. Large values of measured feature resulted in inferior repeatability and reproducibility.

Development of an in vivo visual robot system with a magnetic anchoring mechanism and a lens cleaning mechanism for laparoendoscopic single-site surgery (LESS).

Surgical robot systems which can significantly improve surgical procedures have been widely used in laparoendoscopic single-site surgery (LESS). For a relative complex surgical procedure, the development of an in vivo visual robot system for LESS can effectively improve the visualization for surgical robot systems. In this work, an in vivo visual robot system with a new mechanism for LESS was investigated. A finite element method (FEM) analysis was carried out to ensure the safety of the in vivo visual robot during the movement, which was the most important concern for surgical purposes. A master-slave control strategy was adopted, in which the control model was established by off-line experiments. The in vivo visual robot system was verified by using a phantom box. The experiment results show that the robot system can successfully realize the expected functionalities and meet the demands of LESS. The experiment results indicate that the in vivo visual robot with high manipulability has great potential in clinical application.

Implementation of a following wheel robot featuring stereoscopic vision

This paper describes how two cameras were used to implement stereoscopic visual imaging through a robot vision system. Image processing was applied to construct a real-time stereoscopic image tracking system for use in robots. The target object was sought by image preprocessing. The deviation of the target object from the optical center of the two cameras was measured to calculate the stereoscopic image depth of the target object, and the target object was instantly locked for real-time tracking. This system consists of two parts. The first part uses depth calculation to determine the distance between the target object and robot platform and applies image processing to simplify the image. The deviation angles of the left and right images are subsequently measured to calculate the target object distance. The second part uses real-time image tracking so that the platform can lock the image of the target object into an overlapping image. Proportional–integral–derivative (PID) feedback controls the robot movement and instantly obtains the target object and offset direction. The signal is exported to dc servo motors to drive the rotation of the two cameras and the movement of the platform. The image of the target object is locked in the image center of the stereoscopic visual system for real-time tracking.

Multi-person detecting and tracking based on RGB-D sensor for a robot vision system

In this paper, we address the problem of automatically detecting and tracking a variable number of objects in complex scenes using a RGB-D sensor on the robot system. We propose a novel approach for multi-object detecting by fusing RGB information and depth information. Meanwhile, this paper presents a robust multi-cue approach for multi-object tracking. A spatiotemporal object representation is proposed, which combines a generative colour model and a discriminative texture classifier. We employ a Bayesian framework based on particle filtering to achieve integrated object detection and tracking from a robot vision system. The experimental results show that the proposed method yields good tracking performance in real world environment.

Effects of rotation and systematic occlusion on fiducial marker recognition

Fiducial marker systems consist of patterns that are placed in environment for miscellaneous applications and are further automatically detected with cameras. A variety of applications determines the criteria, which characterize qualitative properties of a marker and include such evaluation benchmarks as resilience to occlusion, distance to a marker, false positive and false negative rates, sensitivity to illumination, and others. The paper compares existing ARTag, AprilTag, and CALTag systems utilizing a high fidelity camera, which is a main vision sensor of a full-size Russian humanoid robot AR-601M. In experiments the comparison of the three marker systems reliability and detection rate in occlusions of various types and intensities was verified. Finally, a preferable for AR-601M robot visual applications marker system was selected.

Multi-person detecting and tracking based on RGB-D sensor for a robot vision system

In this paper, we address the problem of automatically detecting and tracking a variable number of objects in complex scenes using a RGB-D sensor on the robot system. We propose a novel approach for multi-object detecting by fusing RGB information and depth information. Meanwhile, this paper presents a robust multi-cue approach for multi-object tracking. A spatiotemporal object representation is proposed, which combines a generative colour model and a discriminative texture classifier. We employ a Bayesian framework based on particle filtering to achieve integrated object detection and tracking from a robot vision system. The experimental results show that the proposed method yields good tracking performance in real world environment.

Video compression and compensation of moving target image based on redundant wavelet theory

The redundant wavelet transform has the translation invariant property, and the motion estimation can achieve better results in the redundant wavelet domain. This paper proposes the redundant discrete wavelet transformation (RDWT) for motion estimation and compensation in video compression and compensation of moving target image, and a fast adaptive block matching motion estimation algorithm based on redundant wavelet transform. This paper also introduces the methods of image domain search, starting point prediction and motion estimation into the redundant wavelet transform, and the study method based on optical flow which can enhance the transmission robustness with low redundancy. The experimental results show that the algorithm can effectively reduce the time required for motion estimation and improve the efficiency of motion estimation, and the subjective quality of the motion estimation is very good. The research results can provide technical support for robot vision system.

Video compression and compensation of moving target image based on redundant wavelet theory

The redundant wavelet transform has the translation invariant property, and the motion estimation can achieve better results in the redundant wavelet domain. This paper proposes the redundant discrete wavelet transformation (RDWT) for motion estimation and compensation in video compression and compensation of moving target image, and a fast adaptive block matching motion estimation algorithm based on redundant wavelet transform. This paper also introduces the methods of image domain search, starting point prediction and motion estimation into the redundant wavelet transform, and the study method based on optical flow which can enhance the transmission robustness with low redundancy. The experimental results show that the algorithm can effectively reduce the time required for motion estimation and improve the efficiency of motion estimation, and the subjective quality of the motion estimation is very good. The research results can provide technical support for robot vision system.

结合图像内容匹配的机器人视觉导航定位与全局地图构建系统

结合图像内容匹配的机器人视觉导航定位与全局地图构建系统

CLFs-based optimization control for a class of constrained visual servoing systems

In this paper, we use the control Lyapunov function (CLF) technique to present an optimized visual servo control method for constrained eye-in-hand robot visual servoing systems. With the knowledge of camera intrinsic parameters and depth of target changes, visual servo control laws (i.e. translation speed) with adjustable parameters are derived by image point features and some known CLF of the visual servoing system. The Fibonacci method is employed to online compute the optimal value of those adjustable parameters, which yields an optimized control law to satisfy constraints of the visual servoing system. The Lyapunov's theorem and the properties of CLF are used to establish stability of the constrained visual servoing system in the closed-loop with the optimized control law. One merit of the presented method is that there is no requirement of online calculating the pseudo-inverse of the image Jacobian's matrix and the homography matrix. Simulation and experimental results illustrated the effectiveness of the method proposed here.

A robust fruit image segmentation algorithm against varying illumination for vision system of fruit harvesting robot

Vision system is the crucial component of fruit harvesting robot for recognising fruit, however, which is seriously affected by varying illumination when the robot works in real natural environment. A robust fruit segmentation algorithm against varying illumination for vision system was proposed with the aim of effectively extracting fruit object in the natural environment. The method involved the application of improved wavelet transform to fruit image to normalise illumination of object surface. Then Retinex-based image enhancement algorithm was used to highlight fruit object of illumination normalised image. Finally fruit image was segmented by implementing K-means clustering. Three kinds of fruit images of different colour under sunny and cloudy days were segment using the proposed method respectively and the experimental results showed that the proposed algorithm could be robust against the influence of varying illumination and precisely segment different colour fruits.