Eye Calibration Research Articles

Hand–Eye Calibration Algorithm Based on an Optimized Neural Network

A robot can identify the position of a target and complete a grasping based on the hand–eye calibration algorithm, through which the relationship between the robot coordinate system and the camera coordinate system can be established. The accuracy of the hand–eye calibration algorithm affects the real-time performance of the visual servo system and the robot manipulation. The traditional calibration technique is based on a perfect mathematical model AX = XB, in which the X represents the relationship of (A) the camera coordinate system and (B) the robot coordinate system. The traditional solution to the transformation matrix has a certain extent of limitation and instability. To solve this problem, an optimized neural-network-based hand–eye calibration method was developed to establish a non-linear relationship between robotic coordinates and pixel coordinates that can compensate for the nonlinear distortion of the camera lens. The learning process of the hand–eye calibration model can be interpreted as B=fA, which is the coordinate transformation relationship trained by the neural network. An accurate hand–eye calibration model can finally be obtained by continuously optimizing the network structure and parameters via training. Finally, the accuracy and stability of the method were verified by experiments on a robot grasping system.

A General Approach to Hand–Eye Calibration Through the Optimization of Atomic Transformations

This article proposes a general approach to solve the hand–eye calibration problem. The system is general since it is able to calibrate any number of cameras and, moreover, is able to simultaneously perform the calibration of several instances of the two common hand–eye calibration use cases: eye-on-hand and eye-to-base. The calibration is solved with a nonlinear least squares method, and the reprojection error is used as a metric to guide the optimization procedure. Our approach is seamlessly integrated with the robot operating system framework and allows for the interactive positioning of sensors and labeling of data, facilitating both the data acquisition and labeling and the calibration procedures. Results show that the proposed approach is able to handle any calibration use case with a minimal initial configuration. The approach is compared with several other state-of-the-art hand–eye calibration algorithms. Results show that the proposed approach produces very accurate calibrations when compared to the state of the art.

Research on the Hand-eye calibration Method Based on Monocular Robot

The measurement system is composed of a single camera and a manipulator, and the conversion relationship between the manipulator tool coordinate system and the camera coordinate system is one of the key technologies of the system. This paper proposes a unit octet hand-eye calibration algorithm, which has better accuracy and anti-noise ability. This algorithm combines the characteristics of data on SO(4) on the basis of traditional dual quaternion, and establishes the conversion equation of AX=XB in combination with robot kinematics, and realizes the eye calibration of the manipulator. Compared with the traditional dual quaternion algorithm, the speed is faster, the robustness is good, and it has better stability and practicability.

Simultaneous Hand–Eye and Intrinsic Calibration of a Laser Profilometer Mounted on a Robot Arm

A method for simultaneous laser profilometer and hand–eye calibration in relation to an industrial robot as well as its implementation is presented. In contrast to other methods, the new calibration procedure requires the measurement of only one reference geometry to calculate all the transformation parameters. The reference geometry is measured with a laser profilometer from 15 different poses. The intrinsic parameters of the profilometer, as well as the extrinsic (hand–eye) parameters, are then numerically optimized to achieve the minimum deviation between the reference and the measured geometry. The method was characterized with experiments that revealed a standard deviation of the displacements between the reference geometry after the calibration of less than 0.105 mm in the case of using the robot-arm actuator and 0.046 mm in case of using a 5-axis CNC milling machine. The entire procedure, including measurement and calculation, can be completely automated and lasts less than 10 min. This opens up possibilities for regular on-site recalibration of the entire system.

A fast and accurate new algorithm for hand–eye calibration on SO(3)[formula omitted

In this paper, we propose a new solution for an old problem in robotics related to the calibration of the hand, i.e., the robot’s end-effector, and the eye, i.e., a stereo vision camera. The problem is formulated as a point set matching problem and a nonlinear estimator on manifold SO(3)×R3 used for obtaining the solution. The main advantage of the proposed approach is that it allows to decouple the error associated with the rotational estimation on Lie group SO(3) from the error of the translational estimation on R3, which subsequently allows to tune the learning rates for the rotation and translation estimation, separately. This will result in significant increase in the convergence speed of the proposed approach. To show the advantages of our approach, we compare the results with those obtained from other conventional hand–eye calibration solutions as well as those based on point set matching. The experimental results will demonstrate that our proposed hand–eye calibration approach outperforms other approaches in terms of accuracy and computational speed.

Flexible Extrinsic Parameter Calibration for Multicameras With Nonoverlapping Field of View

In many measurement applications using multicameras, there is often a nonoverlapping field of view (nFOV) between multicameras. It benefits greatly from having all of the cameras unified in a single coordinate frame by means of common spatial constraints. However, it is very difficult to establish common spatial constraints for accurate and quick calibration of the extrinsic parameters between multicameras in the case of long working distance, asymmetric working angle, or limited working space. To overcome these issues, we propose a flexible calibration method using a camera rig, which can be easily modeled as a hand–eye calibration problem and solved by only several sets of calibration images. To further improve the calibration accuracy, the camera intrinsic parameters, lens distortion coefficients, and extrinsic parameters are optimized simultaneously, similar to binocular calibration. However, this causes calculation instability of multiparameters. We extend the epipolar constraint to two images with an nFOV and add it into the optimization objective function together with the reprojection error, which ensures the basic consistency of intrinsic parameters before and after optimization, and reduce the fluctuation caused by the number of images involved in calibration. Experiments and application have proved that the proposed new methods are feasible and effective.

Easy Pose-Error Calibration for Articulated Serial Robot Based on Three-Closed-Loop Transformations

Most of the existing robot calibration methods have high technical requirements for the installation of end tools and detection sensors as well as other calibration equipments involved, which makes the calibration cumbersome and expensive. This article proposes a novel and easy pose-error calibration method for articulated serial robots based on three-closed-loop transformations. First, the principle of the three-closed-loop transformations of the coordinate systems is introduced, where the loops can be mutually verified to improve the accuracy of the calibration. Then, a hand–eye calibration method that can directly obtain the coordinate conversion between the camera and the tool without need to consider the installation or transformation error of the tool at the robot end is proposed. Furthermore, the entire calibration process is economical without using any expensive special instruments, and the use conditions of the calibration equipments are loose. Finally, experimental results show that the pose error of the robot can be precisely detected and verified by different closed loops.

A Calibration Optimization Method for a Welding Robot Laser Vision System Based on Generative Adversarial Network

The accuracy and efficiency of a calibration process have a great influence on the robot locating accuracy. In the welding robot laser vision system discussed in this article, traditional structured light and hand–eye calibration methods are complex and time consuming. They require many manually collected labeled data, which leads to low calibration efficiency and accuracy. To solve the above problems, a calibration optimization method for the welding robot laser vision system based on the generative adversarial network is proposed in this article. By establishing an accurate calibration model, the calibration process is transformed into an optimization problem for the calibration parameters. An efficient and intelligent collection method is adopted to provide accurate, diverse, and sufficient calibration data for the calibration optimization method. The experimental results verify that the proposed method greatly improves the efficiency of calibration data collection. The reference point positioning experiment demonstrates that our method has low uncertainty in point positioning. The tracking experiments prove that our method keeps good accuracy and robustness in the tracking system, especially when the welding robot’s pose changes greatly and the tracking trajectory is complex and changeable.

A Hybrid Hand–Eye Calibration Method for Multilink Cable-Driven Hyper-Redundant Manipulators

Multilink cable-driven hyper-redundant manipulators (MCDHMs) have a slender body and flexible movement characteristics, which are very suitable for complex unstructured environments. Since MCDHMs have multiple coupling between active cables, linkage cables, joints, and the end-effector, the mathematical modeling of the segmented linkage MCDHM becomes more complicated, and the uncertainty of its model makes the motion accuracy of the end-effector become low. Traditional calibration methods that rely solely on eye-in-hand or eye-to-hand calibration method to calibrate the MCDHM model will produce relatively large errors. This article proposes a hybrid hand–eye calibration method, which eliminates the errors introduced by inaccurate kinematics models through the fusion modeling of eye-in-hand and eye-to-hand. First, for unstructured application scenarios, the hybrid calibration framework under the condition of large model errors is introduced, and then the multiple kinematics model of the MCDHM is derived. Then, two different types of hybrid hand–eye calibration equations are derived and solved by the recursive iteration method. Furthermore, according to the hand–eye calibration result, the MCDHM kinematic parameters are calibrated by the particle swarm optimization (PSO) method. Finally, in order to demonstrate the feasibility and superiority of the hybrid calibration method, experiments are carried out on the real MCDHM to further verify the proposed method. The experimental results show that the proposed hybrid calibration method has higher calibration accuracy.

Vision-based hand-eye calibration for robot-assisted minimally invasive surgery.

The knowledge of laparoscope vision can greatly improve the surgical operation room (OR) efficiency. For the vision-based computer-assisted surgery, the hand-eye calibration establishes the coordinate relationship between laparoscope and robot slave arm. While significant advances have been made for hand-eye calibration in recent years, efficient algorithm for minimally invasive surgical robot is still a major challenge. Removing the external calibration object in abdominal environment to estimate the hand-eye transformation is still a critical problem. We propose a novel hand-eye calibration algorithm to tackle the problem which relies purely on surgical instrument already in the operating scenario for robot-assisted minimally invasive surgery (RMIS). Our model is formed by the geometry information of the surgical instrument and the remote center-of-motion (RCM) constraint. We also enhance the algorithm with stereo laparoscope model. Promising validation of synthetic simulation and experimental surgical robot system have been conducted to evaluate the proposed method. We report results that the proposed method can exhibit the hand-eye calibration without calibration object. Vision-based hand-eye calibration is developed. We demonstrate the feasibility to perform hand-eye calibration by taking advantage of the components of surgical robot system, leading to the efficiency of surgical OR.

Hand–eye calibration and grasping pose calculation with motion error compensation and vertical-component correction for 4-R(2-SS) parallel robot

Due to motion constraint of 4-R(2-SS) parallel robot, it is difficult to calculate the translation component of hand–eye calibration based on the existing model solving method accurately. Additionally, the camera calibration error, robot motion error, and invalid calibration motion poses make it difficult to achieve fast and accurate online hand–eye calibration. Therefore, we propose a hand–eye calibration method with motion error compensation and vertical-component correction for 4-R(2-SS) parallel robot by improving the existing eye-to-hand model and solving method. Firstly, the eye-to-hand model of single camera is improved and the robot motion error in the improved model is compensated to reduce the influence of camera calibration error and robot motion error on model accuracy. Secondly, the vertical-component of hand–eye calibration is corrected based on vertical constraint between calibration plate and end effector in parallel robot to calculate the pose and motion error in calibration of 4-R(2-SS) parallel robot accurately. Thirdly, the nontrivial solution constraint of eye-to-hand model is constructed and adopted to remove invalid calibration motion poses and plan calibration motion. Finally, the proposed method was verified by experiments with a fruit sorting system based on 4-R(2-SS) parallel robot. Compared with random motion, the existing model, and solving method, the average time of online calibration based on planned motion decreases by 29.773 s and the average error of calibration based on the improved model and solving method decreases by 151.293. The proposed method can improve the accuracy and efficiency of hand–eye calibration of 4-R(2-SS) parallel robot effectively and further realize accurate and fast grasping.

A New Formulation for Hand–Eye Calibrations as Point-Set Matching

Conventional methods formulate the calibration problem between a robot hand and a camera, also known as hand–eye-calibration problem, as AX = XB . However, in practice, these methods have limited accuracy, as will be shown in our experimental results. In this article, we formulated the hand–eye calibration problem as a point-set-matching problem and proposed a new approach to solve this problem. The proposed approach is particularly suitable for robotic applications and offers good accuracy. We obtain a solution for the said problem using the gradient-descent (GD) technique on the special Euclidean group SE(3). We call this approach GD-SE(3). To prove the validity of the proposed approach and to demonstrate its advantages, experimental results are provided, where we compare the performance of GD-SE(3) with both conventional solutions for the hand–eye calibration problem as well as those based on point-set matching. The results show that the accuracy of GD-SE(3) is comparable with those based on other point-set-matching algorithms. Yet, it outperforms conventional formulations based on AX = XB while offering a more suitable approach for the hand–eye calibration problem of robot manipulators.

Simultaneous Optimization of Patient-Image Registration and Hand-Eye Calibration for Accurate Augmented Reality in Surgery.

Augmented reality (AR) navigation using a position sensor in endoscopic surgeries relies on the quality of patient-image registration and hand-eye calibration. Conventional methods collect the necessary data to compute two output transformation matrices separately. However, the AR display setting during surgery generally differs from that during preoperative processes. Although conventional methods can identify optimal solutions under initial conditions, AR display errors are unavoidable during surgery owing to the inherent computational complexity of AR processes, such as error accumulation over successive matrix multiplications, and tracking errors of position sensor. We propose the simultaneous optimization of patient-image registration and hand-eye calibration in an AR environment before surgery. The relationship between the endoscope and a virtual object to overlay is first calculated using an endoscopic image, which also functions as a reference during optimization. After including the tracking information from the position sensor, patient-image registration and hand-eye calibration are optimized in terms of least-squares. Experiments with synthetic data verify that the proposed method is less sensitive to computation and tracking errors. A phantom experiment with a position sensor is also conducted. The accuracy of theproposed method is significantly higher than that of the conventional method. The AR accuracy of the proposed method is compared with those of the conventional ones, and the superiority of the proposed method is verified. This study demonstrates that the proposed method exhibits substantial potential for improving AR navigation accuracy.

Error Correctable Hand-Eye Calibration for Stripe-Laser Vision-Guided Robotics

The stripe-laser vision guidance technology has received more and more attention in industry and academia. The hand–eye calibration technology plays an important role in the applications of the stripe-laser vision-guided robotics. Most existing hand–eye calibration methods impose orthogonality constraints. However, the orthogonality in hand–eye relationship may not be true when the stripe-laser sensor has errors. This makes the final guiding error inevitable. In this article, an error correctable (EC) hand–eye calibration method based on the linear properties of the central projection for the stripe-laser vision sensor guidance system is proposed. In this method, the measurements of a vision sensor with errors can be corrected to the right tool center point coordinates. Experiments involving synthetic and real data confirm that the proposed hand–eye calibration method is robust to the vision sensor errors.

Robust and Accurate Hand-Eye Calibration Method Based on Schur Matric Decomposition.

To improve the accuracy and robustness of hand–eye calibration, a hand–eye calibration method based on Schur matric decomposition is proposed in this paper. The accuracy of these methods strongly depends on the quality of observation data. Therefore, preprocessing observation data is essential. As with traditional two-step hand–eye calibration methods, we first solve the rotation parameters and then the translation vector can be immediately determined. A general solution was obtained from one observation through Schur matric decomposition and then the degrees of freedom were decreased from three to two. Observation data preprocessing is one of the basic unresolved problems with hand–eye calibration methods. A discriminant equation to delete outliers was deduced based on Schur matric decomposition. Finally, the basic problem of observation data preprocessing was solved using outlier detection, which significantly improved robustness. The proposed method was validated by both simulations and experiments. The results show that the prediction error of rotation and translation was 0.06 arcmin and 1.01 mm respectively, and the proposed method performed much better in outlier detection. A minimal configuration for the unique solution was proven from a new perspective.

Research on Intelligent Grasping System of Monocular Vision Guided Manipulator

In order to solve the problem of how a cooperative robot can quickly and accurately locate and grasp the specified target work piece and avoid obstacles autonomously under the condition of man-machine cooperation and the variable position of the object to be grasped, a new method of robot arm grasping under the visual guidance of ROS system was proposed. Using ROS framework and tools, the color threshold algorithm in ROS-Opencv is used to realize the rapid recognition and positioning of grasping targets. Use the ROS-Moveit! The kinematics calculation, motion planning and obstacle avoidance of the manipulator are completed the relationship between the image coordinate system and the robot coordinate system is established by hand eye calibration technology. In addition, ROS tools can be used to obtain information such as the position and speed of each joint during the movement of the mechanical arm. Through experimental testing and data analysis, it is shown that the hand-eye system is accurate in calibration, object positioning, and the robot arm can accurately grasp the target object, which has certain practical application value.

A "eye-in-body" integrated surgery robot system for stereotactic surgery.

Current stereotactic surgical robots system relies on cumbersome operations such as calibration, tracking and registration to establish the accurate intraoperative coordinate transformation chain, which makes the system not easy to use. To overcome this problem, a novel stereotactic surgical robot system has been proposed and validated. First, a hand-eye integrated scheme is proposed to avoid the intraoperative calibration between robot arm and motion tracking system. Second, a special reference-tool-based patient registration and tracking method is developed to avoid intraoperative registration. Third, a model-free visual servo method is used to reduce the accuracy requirement of hand-eye relationship and robot kinematic model. Finally, a prototype of the system is constructed and performance tests and a pedicle screw drilling experiment are performed. The results show that the proposed system has acceptable accuracy. The target positioning error in the plane was - 0.68 ± 0.52mm and 0.06 ± 0.41mm. The orientation error was 0.43 ± 0.25°. The pedicle screw drilling experiment shows that the system can complete accurate stereotactic surgery. The stereotactic surgical robot system described in this paper can perform stereotactic surgery without the intraoperative hand-eye calibration and nor manual registration and can achieve an acceptable position and orientation accuracy while tolerating the errors in the hand-eye coordinate transformation error and the robot kinematics model error.

Methods for Simultaneous Robot-World-Hand-Eye Calibration: A Comparative Study.

In this paper, we propose two novel methods for robot-world-hand–eye calibration and provide a comparative analysis against six state-of-the-art methods. We examine the calibration problem from two alternative geometrical interpretations, called ‘hand–eye’ and ‘robot-world-hand–eye’, respectively. The study analyses the effects of specifying the objective function as pose error or reprojection error minimization problem. We provide three real and three simulated datasets with rendered images as part of the study. In addition, we propose a robotic arm error modeling approach to be used along with the simulated datasets for generating a realistic response. The tests on simulated data are performed in both ideal cases and with pseudo-realistic robotic arm pose and visual noise. Our methods show significant improvement and robustness on many metrics in various scenarios compared to state-of-the-art methods.

IMU-Aided High-Frequency Lidar Odometry for Autonomous Driving

For autonomous driving, it is important to obtain precise and high-frequency localization information. This paper proposes a novel method in which the Inertial Measurement Unit (IMU), wheel encoder, and lidar odometry are utilized together to estimate the ego-motion of an unmanned ground vehicle. The IMU is fused with the wheel encoder to obtain the motion prior, and it is involved in three levels of the lidar odometry: Firstly, we use the IMU information to rectify the intra-frame distortion of the lidar scan, which is caused by the vehicle’s own movement; secondly, the IMU provides a better initial guess for the lidar odometry; and thirdly, the IMU is fused with the lidar odometry in an Extended Kalman filter framework. In addition, an efficient method for hand–eye calibration between the IMU and the lidar is proposed. To evaluate the performance of our method, extensive experiments are performed and our system can output stable, accurate, and high-frequency localization results in diverse environment without any prior information.

General Hand–Eye Calibration Based on Reprojection Error Minimization

This letter describes a novel hand–eye calibration technique based on reprojection error minimization. In contrast to traditional hand–eye calibration methods, the proposed method directly takes images of the calibration pattern and does not require to explicitly estimate the camera pose for each input image. The proposed method is implemented as a pose graph optimization problem, so that it can solve the estimation problem efficiently and robustly, and it can be easily extended for different projection models. It can deal with different camera models (e.g, X-ray cameras with a source-detector projection model) by changing the projection model. Through simulations, we validated that the proposed method shows a good estimation accuracy, and it can be applied to hand–eye calibration with a source-detector camera model. The experimental results with real robots show that the proposed method is applicable to real environments, and it improves the quality of a task that requires accurate hand–eye estimation, such as three-dimensional reconstruction.