Hand-eye System Research Articles

Development of Annulus-Object Random Bin Picking System based on Rapid Establishment of RGB-D Images

With the development of the automation industry, robotic arms, and vision applications are no longer limited to fixed actions of the past. Production lines increasingly require the recognition and grasping of objects in complex environments, emphasizing quick setup and stability. In this paper, a rapidly constructed eye-hand system for robotic arm grasping, which enables fast and efficient object manipulation, particularly for stacking, is introduced. Initially, images were captured using a camera to generate extensive datasets from a limited number of images. Objects were subsequently segmented and categorized using deep learning networks for object detection and instance segmentation. Three-dimensional position information was obtained from an RGB-D camera. Finally, object poses were determined based on plane normal vectors, and gripping positions were manually marked. This reduced the time required for grab-point identification, model training, and pose localization. Based on experimental results, the grasping procedure proposed in this paper is suitable for various object-grasping scenarios. It achieved impressive picking success rates of 96% for unstacked annular objects and 90.86% for random bin annular objects, respectively. In the final experiment, following depth information filtering, a success rate of 95.1% was attained with random bin annular object picking.

Dynamic Parallel Mapping and Trajectory Planning of Robot Arm in Unknown Environment

A high real-time robot arm mapping and dynamic shortest path planning algorithm is proposed, which is used for low-cost depth cameras and robot arms. First of all, an inverse kinematics solution algorithm for a non-spherical wrist joint is proposed for the Kinova Mico2 manipulator. Based on the idea of position and attitude decoupling combined with the numerical solution, the solution of multiple sets of inverse kinematics is acquired. Secondly, a hand-eye system is established on the basis of the depth camera-RealSense. The point cloud is inversely solved through the depth map, which is pre-processed by the pass-through filtering and outlier removal algorithm. A scene parallel mapping method is designed to achieve the real-time reconstruction of the working scene for the robot arm, in which the maintenance of map and large matrix operations can be completed by GPU. Thirdly, a path planning algorithm with the shortest path for the manipulator end in Cartesian space is designed based on D* method and grid map. The attitude interpolation and smoothing are performed on the path points to obtain the executable joint trajectories on the time axis. Finally, the ROS system is used to build a simulation test verification environment, and the effectiveness of the related algorithms is verified through multiple simulation cases.

Object Segmentation by Spraying Robot Based on Multi-Layer Perceptron

The vision system provides an important way for construction robots to obtain the type and spatial location information of the object. The characteristics of the construction environment, construction object, and robot structure are jointly examined in this paper to propose an approach of object segmentation by spraying the robot based on multi-layer perceptron. Firstly, the hand-eye system experimental platform is built through establishing the mathematical model of the system and calibrating the parameters of the model. Secondly, effort is made to carry out research on image preprocessing algorithms and related experiments, and compare the effects of different binocular stereo-matching algorithms in the actual engineering environment. Finally, research and an experiment are conducted to identify the applicability and effect of the depth image object segmentation algorithm based on multi-layer perceptron. The experimental results prove that the application of multi-layer perceptron to object segmentation by spraying robots can meet the requirement on solution accuracy and is suitable for the object segmentation of complex projects in real life. This approach not only overcomes the shortcomings of the existing recognition methods that are poor in accuracy and difficult to be used widely, but also provides basic data for the subsequent three-dimensional reconstruction, thus making a significant contribution to the research of image processing by spraying robots.

Optimal robot-world and hand-eye calibration with rotation and translation coupling

AbstractA classic hand-eye system involves hand-eye calibration and robot-world and hand-eye calibration. Insofar as hand-eye calibration can solve only hand-eye transformation, this study aims to determine the robot-world and hand-eye transformations simultaneously based on the robot-world and hand-eye equation. According to whether the rotation part and the translation part of the equation are decoupled, the methods can be divided into separable solutions and simultaneous solutions. The separable solutions solve the rotation part before solving the translation part, so the estimated errors of the rotation will be transferred to the translation. In this study, a method was proposed for calculation with rotation and translation coupling; a closed-form solution based on Kronecker product and an iterative solution based on the Gauss–Newton algorithm were involved. The feasibility was further tested using simulated data and real data, and the superiority was verified by comparison with the results obtained by the available method. Finally, we improved a method that can solve the singularity problem caused by the parameterization of the rotation matrix, which can be widely used in the robot-world and hand-eye calibration. The results show that the prediction errors of rotation and translation based on the proposed method be reduced to $0.26^\circ$ and $1.67$ mm, respectively.

Research on the integrated manipulator of point cloud measurement and precise cutting for waste nuclear tank

PurposeNuclear waste tanks need to be cut into pieces before they can be safely disposed of, but the cutting process produces a large amount of aerosols with radiation, which is very harmful to the health of the operator. The purpose of this paper is to establish an intelligent strategy for an integrated robot designed for measurement and cutting, which can accurately identify and cut unknown nuclear waste tanks and realize autonomous precise processing.Design/methodology/approachA robot system integrating point cloud measurement and plasma cutting is designed in this paper. First, accurate calibration methods for the robot, tool and hand-eye system are established. Second, for eliminating the extremely scattered point cloud caused by metal surface refraction, an omnidirectional octree data structure with 26 vectors is proposed to extract the point cloud model more accurately. Then, a minimum bounding box is calculated for limiting the local area to be cut, the local three-dimensional shape of the nuclear tank is fitted within the bounding box, in which the cutting trajectories and normal vectors are planned accurately.FindingsThe cutting precision is verified by changing the tool into a dial indicator in the simulation and the experiment process. The octree data structure with omnidirectional pointing vectors can effectively improve the filtering accuracy of the scattered point cloud. The point cloud filter algorithm combined with the structure calibration methods for the integrated measurement and processing system can ensure the final machining accuracy of the robot.Originality/valueAiming at the problems of large measurement noise interference, complex transformations between coordinate systems and difficult accuracy guarantee, this paper proposes structure calibration, point cloud filtering and point cloud-based planning algorithm, which can greatly improve the reliability and accuracy of the system. Simulation and experiment verify the final cutting accuracy of the whole system.

A Comparative Review of Hand-Eye Calibration Techniques for Vision Guided Robots

Hand-eye calibration enables proper perception of the environment in which a vision guided robot operates. Additionally, it enables the mapping of the scene in the robots frame. Proper hand-eye calibration is crucial when sub-millimetre perceptual accuracy is needed. For example, in robot assisted surgery, a poorly calibrated robot would cause damage to surrounding vital tissues and organs, endangering the life of a patient. A lot of research has gone into ways of accurately calibrating the hand-eye system of a robot with different levels of success, challenges, resource requirements and complexities. As such, academics and industrial practitioners are faced with the challenge of choosing which algorithm meets the implementation requirements based on the identified constraints. This review aims to give a general overview of the strengths and weaknesses of different hand-eye calibration algorithms available to academics and industrial practitioners to make an informed design decision, as well as incite possible areas of research based on the identified challenges. We also discuss different calibration targets, which is an important part of the calibration process that is often overlooked in the design process.

A novel flexible two-step method for eye-to-hand calibration for robot assembly system

In the visual robot system, the calibration of the hand-eye system is very important, which has a great influence on the positioning accuracy of the robot. Traditional methods are either complicated or require advanced external equipment. This paper proposes a new flexible method for hand-eye calibration, which is simple and flexible. Firstly, the robot carries the target to perform two transformation motions to solve the rotation relationship, and then the robot tool coordinate system performs several rotation motions to solve the transformation relationship. The paper provides empirical insights about how the robot hand-eye system is calibrated by controlling the robot to perform the specified motion without expensive and complicated 3D measurement equipment. The experiment and analysis indicate that the developed hand-eye calibration has high precision in 6-DOF industrial robot assembly application.

A Simultaneous Optimization Method of Calibration and Measurement for a Typical Hand–Eye Positioning System

Hand-eye calibration is one of the key technologies of robot hand and eye coordination operation. The calibration accuracy directly affects the measurement accuracy of hand-eye system. Due to the influence of environmental noise and disturbance, the results of hand-eye calibration usually contain errors. The hand-eye parameters with errors are used for vision measurement, which will lead to the calibration error into the measurement results. This error transfer phenomenon will inevitably affect the accuracy of visual measurement. In this article, a simultaneous optimization method of calibration and measurement is proposed. Based on the analysis of the motion characteristics, the measurement and calibration model with nonlinear constraints is established. An iterative method is proposed along with the closed-form solution to improve the calibration accuracy. Furthermore, the advantages of the proposed method over traditional methods are discussed. Simulation and experiment are carried out to analyze the calibration and measurement accuracy, error propagation, and solution conditions. The results demonstrate that the proposed method realizes the simultaneous optimization of calibration and measurement and can effectively suppress the influence of calibration error on measurement accuracy.

Hybrid calibration and detection approach for mobile robotic manufacturing systems

PurposeThe purpose of this paper is to present a visual detection approach to predict the poses of target objects placed in arbitrary positions before completing the corresponding tasks in mobile robotic manufacturing systems.Design/methodology/approachA hybrid visual detection approach that combines monocular vision and laser ranging is proposed based on an eye-in-hand vision system. The laser displacement sensor is adopted to achieve normal alignment for an arbitrary plane and obtain depth information. The monocular camera measures the two-dimensional image information. In addition, a robot hand-eye relationship calibration method is presented in this paper.FindingsFirst, a hybrid visual detection approach for mobile robotic manufacturing systems is proposed. This detection approach is based on an eye-in-hand vision system consisting of one monocular camera and three laser displacement sensors and it can achieve normal alignment for an arbitrary plane and spatial positioning of the workpiece. Second, based on this vision system, a robot hand-eye relationship calibration method is presented and it was successfully applied to a mobile robotic manufacturing system designed by the authors’ team. As a result, the relationship between the workpiece coordinate system and the end-effector coordinate system could be established accurately.Practical implicationsThis approach can quickly and accurately establish the relationship between the coordinate system of the workpiece and that of the end-effector. The normal alignment accuracy of the hand-eye vision system was less than 0.5° and the spatial positioning accuracy could reach 0.5 mm.Originality/valueThis approach can achieve normal alignment for arbitrary planes and spatial positioning of the workpiece and it can quickly establish the pose relationship between the workpiece and end-effector coordinate systems. Moreover, the proposed approach can significantly improve the work efficiency, flexibility and intelligence of mobile robotic manufacturing systems.

Research on Calibration Technology of Five-Degrees-of-Freedom Tandem Robots

In the robot feedback control system of visual feedback, the calibration of the hand-eye system is very important, because it directly affects the accuracy of the robot. Aiming at the small operating space and compact structure of the micro-assembly robot system, a fixed-vision hand-eye calibration method is proposed. The method first obtains the coordinates of 15 spatial points in the robot base coordinate system, and then uses the camera pose measurement principle to obtain the coordinates of the corresponding spatial points in the camera coordinate system, and then establishes the calibration equations. Based on the least squares method, the coordinate transformation matrix between the hand-eye systems is calibrated. The experimental results show that the constructed hand-eye system coordinate transformation matrix is effective and reliable. The calibration accuracy and effectiveness of the method are verified by relevant grabbing experiments.

Research on Intelligent Grasping System of Monocular Vision Guided Manipulator

In order to solve the problem of how a cooperative robot can quickly and accurately locate and grasp the specified target work piece and avoid obstacles autonomously under the condition of man-machine cooperation and the variable position of the object to be grasped, a new method of robot arm grasping under the visual guidance of ROS system was proposed. Using ROS framework and tools, the color threshold algorithm in ROS-Opencv is used to realize the rapid recognition and positioning of grasping targets. Use the ROS-Moveit! The kinematics calculation, motion planning and obstacle avoidance of the manipulator are completed the relationship between the image coordinate system and the robot coordinate system is established by hand eye calibration technology. In addition, ROS tools can be used to obtain information such as the position and speed of each joint during the movement of the mechanical arm. Through experimental testing and data analysis, it is shown that the hand-eye system is accurate in calibration, object positioning, and the robot arm can accurately grasp the target object, which has certain practical application value.

Calibration method for hand-eye system with rotation and translation couplings.

This paper develops a novel hand-eye calibration method for hand-eye systems with rotation and translation coupling terms. First, a nonlinear camera model with distortion terms and a model of a hand-eye system with rotation and translation coupling terms are established. Based on a non-linear optimization method and a reverse projection method, a decoupling calibration method for a lower-degree-of-freedom hand-eye system is proposed. Then the path planning for the calibration process is carried out. Based on the analysis of coupling constraints and hand-eye system motion constraints, three types of hand-eye calibration paths with high efficiency and easy operation are developed. In addition, the influence of key parameters on hand-eye calibration accuracy is analyzed. Finally, calibration experiments and parametric influence experiments are carried out. The results demonstrate that the proposed method is effective and practical for calibrating the hand-eye system.

一种基于直线基元的手眼系统结构光标定方法

一种基于直线基元的手眼系统结构光标定方法

The use of the visuo-motor reaction training device for the improvement of the coordination in the eye-hand system of the children and adolescents following the completion of the antineoplastic treatment of brain tumours

The objective of the present study was to demonstrate the possibility and the effectiveness of the application of the new method for the stimulation of the information processing activity of the brain, the enhancement of its effectiveness, and the improvement of the quality of the visual-motor coordination in the children and adolescents following the completion of the antineoplastic treatment of brain tumours. This paper reports the first experience of the application of the Dynavision d2 visuo-motor reaction training device which has been used for the visual-motor integration and the increase of the information processing speed. The sample included in the study was comprised of a total of 46 children at the mean age of 10.6±3.34 years that had undergone the treatment of the brain neoplasm. Each patient participated in 5 to 8 sessions of exercises with the use of the the Dynavision d2 visuo-motor reaction training device during 21 days. All the treated patients exhibited the stable reduction of the mean time of the reaction from the first session to the last one. The comparison of the outcomes of the treatment between the children of different sexes has demonstrated better results (in absolute terms and in dynamics) in the girls in comparison with the boys. The exercises with the use of the Dynavision d2 visuo-motor reaction training device were not accompanied by any negative subjective sensations in the treated patients, nor did they induce the adverse somatic side effects. The results of the present study give evidence of both the applicability and the effectiveness of the application of the Dynavision d2 visuo-motor reaction training device for the diagnostics of the disturbances in the motor and visual coordination and the associated cognitive functions and for their correction in the children following the treatment of the neoplasm of the posterior cranial fossa.

Development of Sensory-Motor Fusion-Based Manipulation and Grasping Control for a Robotic Hand-Eye System

In this paper, a sensory-motor fusion-based manipulation and grasping control strategy has been developed for a robotic hand-eye system. The proposed hierarchical control architecture has three modules: 1) vision servoing; 2) surface electromyography (sEMG)-based movement recognition; and 3) hybrid force and motion optimization for manipulation and grasping. A stereo camera is used to obtain the 3-D point cloud of a target object and provides the desired operational position. The AdaBoost-based motion recognition is employed to discriminate different movements based on sEMG of human upper limbs. The operational space motion planning for bionic arm and force planning for multifingered robotic hand can be both transformed as a convex optimization problem with various constraints. A neural dynamics optimization solution is proposed and implemented online. The proposed formulation can achieve a substantial reduction of computational load. The actual implementation includes a bionic arm with dextrous hand, high-speed active vision, and an EMG sensors. A series of manipulation tasks consisting of tracking/recogniting/grasping of an object are implemented, and experiment results exhibit the responsiveness and flexibility of the proposed sensory motion fusion approach.

Eye-hand coordination during a double-step task: evidence for a common stochastic accumulator.

Many studies of reaching and pointing have shown significant spatial and temporal correlations between eye and hand movements. Nevertheless, it remains unclear whether these correlations are incidental, arising from common inputs (independent model); whether these correlations represent an interaction between otherwise independent eye and hand systems (interactive model); or whether these correlations arise from a single dedicated eye-hand system (common command model). Subjects were instructed to redirect gaze and pointing movements in a double-step task in an attempt to decouple eye-hand movements and causally distinguish between the three architectures. We used a drift-diffusion framework in the context of a race model, which has been previously used to explain redirect behavior for eye and hand movements separately, to predict the pattern of eye-hand decoupling. We found that the common command architecture could best explain the observed frequency of different eye and hand response patterns to the target step. A common stochastic accumulator for eye-hand coordination also predicts comparable variances, despite significant difference in the means of the eye and hand reaction time (RT) distributions, which we tested. Consistent with this prediction, we observed that the variances of the eye and hand RTs were similar, despite much larger hand RTs (∼90 ms). Moreover, changes in mean eye RTs, which also increased eye RT variance, produced a similar increase in mean and variance of the associated hand RT. Taken together, these data suggest that a dedicated circuit underlies coordinated eye-hand planning.

Extending mind, visuospatial integration, and the evolution of the parietal lobes in the human genus

Current theories in extended mind suggest that cognition is the result of an integrative process involving brain, body, and environment. The relationships between inner and outer components strictly depend on the functional interface, which is represented by the body. Posture and locomotion influence the sensorial and behavioral relationships between the body and the environment which, in Primates, are strongly dependent on the eye-hand system, and coordinated by processes of visuospatial integration. The upper and medial parietal areas (like the precuneus and the intraparietal sulcus) are crucial for such functions. These areas are associated with specific human cortical features, and have undergone relevant morphological changes in Homo sapiens. Therefore, it can be hypothesized that the visuospatial functions and the role of the body as an interface have experienced important evolutionary changes in our species. Neandertals did not display similar changes in terms of brain morphology, and at the same time they showed a different manipulative behavior: they needed their teeth and mouth to properly handle tools much more than any modern human group does. This may suggest a different (and probably less specialized) way to integrate inner and outer components through the body interface. Archaeology is essential to evaluate possible functional changes in extinct human species, by considering other kinds of visuospatial behaviors that are evident from human ecology and material culture. We suggest that changes in the visuospatial integration functions and in the parietal areas may have represented an essential component for enhancing embodiment capacity. What remains to be established is the role of genetic, epigenetic, and environmental factors, in generating anatomical and functional differences among human species and between human and non-human primates. Visuospatial integration, within the perspective of extended cognition, may have had a major influence in establishing current human intellectual abilities and social patterns.

ビンピッキング用ハンドアイシステムの開発

ビンピッキング用ハンドアイシステムの開発

Research on the Control Strategy of Eye-to-Hand Target Track Real-Time Tracking System

This paper presents an Eye-to-hand system, designed for hand-eye system presents a automatically real-time tracking of moving targets control strategy. The first strategy uses a color mark on the target, the use of binocular stereo vision system for color measurement and then in the visual system of the target three-dimensional reconstruction to give the three-dimensional coordinates in the visual coordinate system when the target position and orientation when changes according to the amount of change in the three-dimensional coordinates of the direct control of the robot arm motion of each joint, enabling the tracking of moving targets.

Line-feature-based calibration method of structured light plane parameters for robot hand-eye system

For monocular-structured light vision measurement, it is essential to calibrate the structured light plane parameters in addition to the camera intrinsic parameters. A line-feature-based calibration method of structured light plane parameters for a robot hand-eye system is proposed. Structured light stripes are selected as calibrating primitive elements, and the robot moves from one calibrating position to another with constraint in order that two misaligned stripe lines are generated. The images of stripe lines could then be captured by the camera fixed at the robot's end link. During calibration, the equations of two stripe lines in the camera coordinate system are calculated, and then the structured light plane could be determined. As the robot's motion may affect the effectiveness of calibration, so the robot's motion constraints are analyzed. A calibration experiment and two vision measurement experiments are implemented, and the results reveal that the calibration accuracy can meet the precision requirement of robot thick plate welding. Finally, analysis and discussion are provided to illustrate that the method has a high efficiency fit for industrial in-situ calibration. (C) 2013 Society of Photo-Optical Instrumentation Engineers (SPIE) [DOI: 10.1117/1.OE.52.3.037202]