Hand-eye Robot Research Articles

Enhanced Hand–Eye Coordination Control for Six-Axis Robots Using YOLOv5 with Attention Module

Utilizing machine vision technology based on YOLOv5, a six-axis robot can quickly identify and classify targets. However, when the YOLOv5 model is used for the recognition and grasping of small workpieces, issues such as low precision and missed detections frequently occur. This paper proposes an enhanced object recognition algorithm, integrating a CBAM attention module and an improved loss function into YOLOv5 to control the hand–eye coordination of the six-axis robot during grasping. The CBAM attention module is incorporated into the backbone network of YOLOv5 to enhance its feature extraction capabilities, while the original loss function is modified to accelerate convergence and improve regression accuracy. An experimental platform for six-axis robot hand–eye coordination grasping was built, and grasping experiments were conducted. The proposed method significantly improves the robot’s grasping accuracy, with a 99.59% mAP0.5 and a 90.83% successful grasping rate, effectively addressing the challenges of low accuracy and missed detections in traditional systems.

Automatic Robot Hand-Eye Calibration Enabled by Learning-Based 3D Vision

Hand-eye calibration, a fundamental task in vision-based robotic systems, is commonly equipped with collaborative robots, especially for robotic applications in small and medium-sized enterprises (SMEs). Most approaches to hand-eye calibration rely on external markers or human assistance. We proposed a novel methodology that addresses the hand-eye calibration problem using the robot base as a reference, eliminating the need for external calibration objects or human intervention. Using point clouds of the robot base, a transformation matrix from the coordinate frame of the camera to the robot base is established as “I=AXB.” To this end, we exploit learning-based 3D detection and registration algorithms to estimate the location and orientation of the robot base. The robustness and accuracy of the method are quantified by ground-truth-based evaluation, and the accuracy result is compared with other 3D vision-based calibration methods. To assess the feasibility of our methodology, we carried out experiments utilizing a low-cost structured light scanner across varying joint configurations and groups of experiments. The proposed hand-eye calibration method achieved a translation deviation of 0.930 mm and a rotation deviation of 0.265 degrees according to the experimental results. Additionally, the 3D reconstruction experiments demonstrated a rotation error of 0.994 degrees and a position error of 1.697 mm. Moreover, our method offers the potential to be completed in 1 second, which is the fastest compared to other 3D hand-eye calibration methods. We conduct indoor 3D reconstruction and robotic grasping experiments based on our hand-eye calibration method. Related code is released at https://github.com/leihui6/LRBO.

On flange-based 3D hand–eye calibration for soft robotic tactile welding

This paper investigates the direct application of standardized designs on the robot for conducting robot hand–eye calibration by employing 3D scanners with collaborative robots. The well-established geometric features of the robot flange are exploited by directly capturing its point cloud data. In particular, an iterative method is proposed to facilitate point cloud processing towards a refined calibration outcome. Several extensive experiments are conducted over a range of collaborative robots, including Universal Robots UR5 & UR10 e-series, Franka Emika, and AUBO i5 using an industrial-grade 3D scanner Photoneo Phoxi S & M and a commercial-grade 3D scanner Microsoft Azure Kinect DK. Experimental results show that translational and rotational errors converge efficiently to less than 0.28 mm and 0.25 degrees, respectively, achieving a hand–eye calibration accuracy as high as the camera’s resolution, probing the hardware limit. A welding seam tracking system is presented, combining the flange-based calibration method with soft tactile sensing. The experiment results show that the system enables the robot to adjust its motion in real-time, ensuring consistent weld quality and paving the way for more efficient and adaptable manufacturing processes.

Hand-eye calibration method and machine vision research based on sensor network

With the promotion of Industry 4.0 reform, the trend of intelligent and precise production in the production workshop is gradually highlighted. This directly leads to higher requirements for robot hand eye coordination accuracy in automated workshops. In order to achieve more precise robot hand eye coordination control, this study designed a new mean calculation method based on the probability density theory, and designed a new mean robot hand eye calibration algorithm based on this. After the test, it is found that the translation error and rotation error calculated by the new mean algorithm are 0.26 and 0.92 respectively, which are significantly lower than other comparison algorithms when using all test samples of normal distribution. And the calculation time of the algorithm when using all the test samples is 2115 ms, which is also significantly lower than the comparison algorithm. The simulation results show that the new mean hand eye calibration method designed in this study can achieve more accurate hand eye coordination control of robots, and has certain application potential in high-precision industrial production scenarios.

Application of combinatorial optimization algorithm in industrial robot hand eye calibration

In order to improve the measurement accuracy of laser profiler and industrial robot measurement system and reduce the influence of random error in calibration process, a hand eye calibration method based on combinatorial optimization is proposed. Firstly, the laser profiler is positioned by the robot to scan a standard sphere from different postures. Then, combinations from all the measurement data are selected to calculate several transformation matrices and the corresponding profile datasets in base coordinate system can be derived consequently. Since all the profile data come from the same stationary standard sphere, sphere fitting is applied to each dataset and the optimal hand eye transformation matrix can be derived which corresponds to the minimum fitting error. Experimental results show that the novel method can effectively improve the calibration accuracy and sphere loss which is weighted sum of sphere fitting error and radius error is reduced from 1.976 to 0.539 after combinatorial optimization.

Research on Robot Hand-Eye Calibration Method Based on Three-Dimensional Calibration Block

Research on Robot Hand-Eye Calibration Method Based on Three-Dimensional Calibration Block

An Efficient Calibration Method of Line Structured Light Vision Sensor in Robotic Eye-in-Hand System

For line structured light vision sensor in robotic eye-in-hand system, the calibration of camera parameters, robot hand eye parameters and structured light plane parameters is an indispensable procedure before conducting 3-D measurement. Traditionally, these three sets of parameters are calibrated separately, making the calibration process complicated and inefficient. To address this, an efficient calibration method is proposed in this paper. By changing the poses of robot and calibration target, a set of images of the calibration target is captured, upon which the camera calibration is conducted. The extrinsic parameters obtained from camera calibration are utilized in the calibration of the structured light plane parameters and hand eye parameters, using the same set of images. Therefore, the three sets of parameters can be simultaneously calibrated with simple on-site operations. Moreover, the calibration target used in this method is only a planar checkerboard, which can be easily acquired at low cost. It is verified in the experiment that the proposed method is efficient and accurate, with the calibration duration within 10 minutes and the measurement accuracy within 0.40mm. Additionally, object pose measurement experiment is conducted to validate the effectiveness of the proposed method in practical application.

Visual-Guided Robotic Object Grasping Using Dual Neural Network Controllers

It has been a challenging task for a robotic arm to accurately reach and grasp objects, which has drawn much research attention. This article proposes a robotic hand–eye coordination system by simulating the human behavior pattern to achieve a fast and robust reaching ability. This is achieved by two neural-network-based controllers, including a rough reaching movement controller implemented by a pretrained radial basis function for rough reaching movements, and a correction movement controller built from a specifically designed brain emotional nesting network (BENN) for smooth correction movements. In particular, the proposed BENN is designed with high nonlinear mapping ability, with its adaptive laws derived from the Lyapunov stability theorem; from this, the robust tracking performance and accordingly the stability of the proposed control system are guaranteed by the utilization of the $H^{\infty }$ control approach. The proposed BENN is validated and evaluated by a chaos synchronization simulation, and the overall control system by object grasping tasks through a physical robotic arm in a real-world environment. The experimental results demonstrate the superiority of the proposed control system in reference to those with single neural networks.

Simultaneous robot-world and hand-eye calibration by the alternative linear programming

A typical hand-eye robot system has two related kinematic loops, which proper functioning requires the accurate determination of robot-world (RW) and hand-eye (HY) transformations in them. Insofar as HY calibration can resolve only the issues related to HY transformation, this study is focused on the joint treatment of the RW&HY transformations, which process is referred to as simultaneous RW&HY calibration. An alternative linear programming is adopted for the construction of semi-convex objective functions via two algorithms. These involve the homogeneous matrix and dual quaternion parameterizations, respectively. Their feasibility was further tested using simulated and real experimental datasets and compared to the results obtained via two available methods (NL and LMI). The results obtained strongly suggest that the homogeneous matrix parametrization (ALP1) had better performance than the dual quaternion one (ALP2). Meanwhile, both of them yielded the good optimal calibration solutions instead of local minima ones. Therefore, both formulations provide new insights into the behavior and complexity of the simultaneous RW&HY calibration.

Enhanced Robotic Hand–Eye Coordination Inspired From Human-Like Behavioral Patterns

Robotic hand–eye coordination is recognized as an important skill to deal with complex real environments. Conventional robotic hand–eye coordination methods merely transfer stimulus signals from robotic visual space to hand actuator space. This paper introduces a reverse method. Build another channel that transfers stimulus signals from robotic hand space to visual space. Based on the reverse channel, a human-like behavior pattern: “Stop-to-Fixate,” is imparted to the robot, thereby giving the robot an enhanced reaching ability. A visual processing system inspired by the human retina structure is used to compress visual information so as to reduce the robot’s learning complexity. In addition, two constructive neural networks establish the two sensory delivery channels. The experimental results demonstrate that the robotic system gradually obtains a reaching ability. In particular, when the robotic hand touches an unseen object, the reverse channel successfully drives the visual system to notice the unseen object.

Learning to Predict Intent from Gaze During Robotic Hand-Eye Coordination

Effective human-aware robots should anticipate their user’s intentions. During hand-eye coordination tasks, gaze often precedes hand motion and can serve as a powerful predictor for intent. However, cooperative tasks where a semi-autonomous robot serves as an extension of the human hand have rarely been studied in the context of hand-eye coordination. We hypothesize that accounting for anticipatory eye movements in addition to the movements of the robot will improve intent estimation. This research compares the application of various machine learning methods to intent prediction from gaze tracking data during robotic hand-eye coordination tasks. We found that with proper feature selection, accuracies exceeding 94% and AUC greater than 91% are achievable with several classification algorithms but that anticipatory gaze data did not improve intent prediction.

Development of Sensory-Motor Fusion-Based Manipulation and Grasping Control for a Robotic Hand-Eye System

In this paper, a sensory-motor fusion-based manipulation and grasping control strategy has been developed for a robotic hand-eye system. The proposed hierarchical control architecture has three modules: 1) vision servoing; 2) surface electromyography (sEMG)-based movement recognition; and 3) hybrid force and motion optimization for manipulation and grasping. A stereo camera is used to obtain the 3-D point cloud of a target object and provides the desired operational position. The AdaBoost-based motion recognition is employed to discriminate different movements based on sEMG of human upper limbs. The operational space motion planning for bionic arm and force planning for multifingered robotic hand can be both transformed as a convex optimization problem with various constraints. A neural dynamics optimization solution is proposed and implemented online. The proposed formulation can achieve a substantial reduction of computational load. The actual implementation includes a bionic arm with dextrous hand, high-speed active vision, and an EMG sensors. A series of manipulation tasks consisting of tracking/recogniting/grasping of an object are implemented, and experiment results exhibit the responsiveness and flexibility of the proposed sensory motion fusion approach.

An Infant Development-inspired Approach to Robot Hand-eye Coordination

This paper presents a novel developmental learning approach for hand-eye coordination in an autonomous robotic system. Robotic hand-eye coordination plays an important role in dealing with real-time environments. Under the approach, infant developmental patterns are introduced to build our robot's learning system. The method works by first constructing a brain-like computational structure to control the robot, and then by using infant behavioural patterns to build a hand-eye coordination learning algorithm. This work is supported by an experimental evaluation, which shows that the control system is implemented simply, and that the learning approach provides fast and incremental learning of behavioural competence.

A flexible method combining camera calibration and hand–eye calibration

SUMMARYWe consider the conventional techniques of vision robot system calibration where camera parameters and robot hand–eye parameters are computed separately, i.e., first performing camera calibration and then carrying out hand–eye calibration based on the calibrated parameters of cameras. In this paper we propose a joint algorithm that combines the camera calibration and the hand–eye calibration together. The proposed algorithm gives the solutions of the cameras' parameters and the hand–eye parameters simultaneously by using nonlinear optimization. Both simulations and real experiments show the superiority of our algorithm. We also apply our algorithm in the real application of the robot-assisted surgical system, and very good results have been obtained.

A 3D Modeling Method for Robot's Hand-Eye Coordinated Grasping

A 3D Modeling Method for Robot's Hand-Eye Coordinated Grasping

Visual Tracking of a Hand–Eye Robot for a Moving Target Object with Multiple Feature Points: Translational Motion Compensation Approach

In this paper, we propose a visual tracking control method of a hand–eye robot for a moving target object with multiple feature points. The hand–eye robot is composed of a 3-d.o.f. planar manipulator and a single charge-coupled-device camera that is mounted on the manipulator's end-effector. The control objective is to keep all feature points of the target object around their desired coordinates on the image plane. In many conventional visual servo methods, it is assumed that the target object is static. Consequently, the visual tracking error arises in the case of a moving target object. We have already proposed a visual tracking control system that takes into consideration the target object motion. This method can reduce the visual tracking error, but can only deal with a single feature point. Therefore, this paper extends such a visual tracking control method to multiple feature points. The effectiveness of our control method is evaluated experimentally.

Design of Robot Welding Seam Tracking System with Structured Light Vision

Robot welding is an important developing direction of welding automation and intelligentization, and automatic seam tracking technology is one of principal research domains. Nowadays, seam tracking system with structured light vision becomes a hot research. Structured light vision seam tracking products abroad are generally very expensive and can only be applied on special occasions. In China, the research of structured light vision seam tracking system is still just on the stage of experiments. A robot real-time seam tracking system with line structured light vision is designed. The hardware system is set up, a filtering method for line structure seam image is improved, and compared with common filtering, it has better effect and characteristic of real time. Two methods, fast template matching and fast Hough transform, to recognize the image coordinates of seam center are improved. Two new image recognition methods, structure element matching and corner detecting, are proposed. The comparison of seam image recognition shows that fast template matching and corner detecting are more precise and stable than the other two methods, and corner detecting is the best in real time. A simultaneous calibration for camera parameters and robot hand-eye is also proposed, and calculation shows that the calibration is effective and feasible. The robot seam tracking tests for linear and folded lap-joint are performed, which are based on the above four image recognition methods, and the results indicate that four image recognition methods are all applicable to real-time seam tracking, and the whole system sufficed for the requirements of real-time seam tracking. Automatic seam tracking with line structured light vision is feasible and has good versatility.

Fast Learning Mapping Schemes for Robotic Hand–Eye Coordination

In aiming for advanced robotic systems that autonomously and permanently readapt to changing and uncertain environments, we introduce a scheme of fast learning and readaptation of robotic sensorimotor mappings based on biological mechanisms underpinning the development and maintenance of accurate human reaching. The study presents a range of experiments, using two distinct computational architectures, on both learning and realignment of robotic hand–eye coordination. Analysis of the results provide insights into the putative parameters and mechanisms required for fast readaptation and generalization from both a robotic and biological perspective.

Convergence analysis for the uncalibrated robotic hand–eye coordination based on the unmodeled dynamics observer

SUMMARYThe uncalibrated robotic hand–eye coordination problem is firstly modeled by a dynamic system, where the unknown hand–eye relationship is regarded as the system's unmodeled dynamics. A state observer is then designed to estimate impacts of this modeling error together with the system's external disturbances. With the estimation results as the compensation, the system control is thus accomplished based on a nonlinear combination of the system state errors. Convergence analysis of the whole system under the proposed control scheme is emphasized. Simulations and experiment results are presented to verify the performance of the proposed approach.

A hand–eye calibration algorithm based on screw motions

SUMMARYWhen computer vision technique is used in robotics, robotic hand–eye calibration is a very important research task. Many algorithms have been proposed for hand–eye calibration. Based on these algorithms, we introduce a new hand–eye calibration algorithm in this paper, which employs the screw motion theory to establish a hand–eye matrix equation by using quaternion and gets a simultaneous result for rotation and translation by solving linear equations. The algorithm proposed in this paper has high accuracy and stable computational efficiency and can be understood easily. Both simulations and real experiments show the superiority of our algorithm over the comparative algorithms.