Hand-eye Calibration Research Articles

Robot grasping system based on deep learning target detection

The traditional robot grasping system often uses fixed point grasping or demonstrative grasping, but with the increase in the diversity of grasping targets and the randomness of poses, the traditional grasping method is no longer sufficient. A robot grasping method based on deep learning target detection is proposed for a high error rate of target recognition and low success rate of grasping in the robot grasping process. The method investigates the robotic arm hand-eye calibration and the deep learning-based target detection and poses estimation algorithm. The Basler camera is used as the visual perception tool of the robot arm, the AUBO i10 robot arm is used as the main body of the experiment, and the PP-YOLO deep learning algorithm performs target detection and poses estimation on the object. Through the collection of experimental data, several grasping experiments were conducted on the diversity of targets randomly placed in the poses under real scenes. The results showed that the success rate of grasping target detection was 94.93% and the robot grasping success rate was 93.37%.

Evaluation of a calibration rig for stereo laparoscopes.

Accurate camera and hand-eye calibration are essential to ensure high-quality results in image-guided surgery applications. The process must also be able to be undertaken by a nonexpert user in a surgicalsetting. This work seeks to identify a suitable method for tracked stereo laparoscope calibration within theater. A custom calibration rig, to enable rapid calibration in a surgical setting, was designed. The rig was compared against freehand calibration. Stereo reprojection, stereo reconstruction, tracked stereo reprojection, and tracked stereo reconstruction error metrics were used to evaluate calibrationquality. Use of the calibration rig reduced mean errors: reprojection (1.47 mm [SD 0.13] vs. 3.14 mm [SD 2.11], p-value 1e-8), reconstruction (1.37 px [SD 0.10] vs. 10.10 px [SD 4.54], p-value 6e-7), and tracked reconstruction (1.38 mm [SD 0.10] vs. 12.64 mm [SD 4.34], p-value 1e-6) compared with freehand calibration. The use of a ChArUco pattern yielded slightly lower reprojection errors, while a dot grid produced lower reconstruction errors and was more robust under strong globalillumination. The use of the calibration rig results in a statistically significant decrease in calibration error metrics, versus freehand calibration, and represents the preferred approach for use in the operatingtheater.

IABC: An iteratively automatic machine learning boosted hand-eye calibration method for laser displacement sensor

An on-machine measuring (OMM) system with a laser displacement sensor (LDS) is designed for measuring free-form surfaces of hypersonic aircraft’s radomes. To improve the measurement accuracy of the OMM system, a novel Iteratively Automatic machine learning Boosted hand-eye Calibration (IABC) method is proposed. Both the hand-eye relationship and LDS measurement errors can be calibrated in one calibration process without any hardware changes via IABC. Firstly, a new objective function is derived, containing analytical parameters of the hand-eye relationship and LDS errors. Then, a hybrid calibration model composed of two kernels is proposed to solve the objective function. One kernel is the analytical kernel designed for solving analytical parameters. Another kernel is the automatic machine learning (AutoML) kernel designed to model LDS errors. The two kernels are connected with stepwise iterations to find the best calibration results. Compared with traditional methods, hand-eye experiments show that IABC reduces the calibration RMSE by about 50%. Verification experiments show that IABC reduces the measurement deviations by about 25%-50% and RMSEs within 40%. Even when the training data are obviously less than the test data, IABC performs well. Experiments demonstrate that IABC is more accurate than traditional hand-eye methods.

Development of Integrated Automatic System of Laser Cladding for Repairing of Polycrystalline Diamond Compact Bits

In order to improve the automatic level for laser-cladding repair of high value industrial equipment, such as polycrystalline diamond compact bit (PDC bit) applied in oil industry, a universal scheme of integrated automatic system for repairing is proposed in this paper, and the basic functional modules together with the executing order according to which each module runs are defined. There are two main technical points, i.e.,inspection and repairing, that need to be realized for such integrated automatic system. Therefore, according to the proposed scheme and the existing instruments, a dual-robot system, which includes two KUKA industrial robots, is adopted as the technological implementation, where one robot is used to carry a 3D scanner to reconstruct the PDC bit to realize inspection while the other is used to hold the laser to melt the special powder flowing to the damaged region of the bit to complete cladding. To realize automatic running of the whole integrated system, a hand-eye calibration method, namely three-point calibration, is then proposed, by which coordinates of point cloud of the damaged PDC bit detected by 3D scanner can be transformed to those of the coordinate system of the robot with the laser, so that the cladding path planned via cutting slice of the damaged region of the PDC bit in the upper computer software, the key of the integrated system developed by QT programming tool, can be tracked by laser head and then the damaged part of the PDC bit can be repaired. Finally, a laser-cladding experiment for repairing PDC bit is carried out and the feasibility of the proposed scheme of the integrated automatic system and the effectiveness of the dual-robot system implemented via KUKA robots are verified, According to existing literature, no papers about such integrated system for automatic laser cladding repair have been published.

Hand–Eye Calibration Using a Tablet Computer

Many approaches have been developed to solve the hand–eye calibration problem. The traditional approach involves a precise mathematical model, which has advantages and disadvantages. For example, mathematical representations can provide numerical and quantitative results to users and researchers. Thus, it is possible to explain and understand the calibration results. However, information about the end-effector, such as the position attached to the robot and its dimensions, is not considered in the calibration process. If there is no CAD model, additional calibration is required for accurate manipulation, especially for a handmade end-effector. A neural network-based method is used as the solution to this problem. By training a neural network model using data created via the attached end-effector, additional calibration can be avoided. Moreover, it is not necessary to develop a precise and complex mathematical model. However, it is difficult to provide quantitative information because a neural network is a black box. Hence, a method with both advantages is proposed in this study. A mathematical model was developed and optimized using the data created by the attached end-effector. To acquire accurate data and evaluate the calibration results, a tablet computer was utilized. The established method achieved a mean positioning error of 1.0 mm.

A novel hand-eye calibration method of picking robot based on TOF camera.

Aiming at the stability of hand-eye calibration in fruit picking scene, a simple hand-eye calibration method for picking robot based on optimization combined with TOF (Time of Flight) camera is proposed. This method needs to fix the TOF depth camera at actual and calculated coordinates of the peach the end of the robot, operate the robot to take pictures of the calibration board from different poses, and record the current photographing poses to ensure that each group of pictures is clear and complete, so as to use the TOF depth camera to image the calibration board. Obtain multiple sets of calibration board depth maps and corresponding point cloud data, that is, "eye" data. Through the circle center extraction and positioning algorithm, the circle center points on each group of calibration plates are extracted, and a circle center sorting method based on the vector angle and the center of mass coordinates is designed to solve the circle center caused by factors such as mirror distortion, uneven illumination and different photographing poses. And through the tool center point of the actuator, the coordinate value of the circle center point on the four corners of each group of calibration plates in the robot end coordinate system is located in turn, and the "hand" data is obtained. Combined with the SVD method, And according to the obtained point residuals, the weight coefficients of the marker points are redistributed, and the hand-eye parameters are iteratively optimized, which improves the accuracy and stability of the hand-eye calibration. the method proposed in this paper has a better ability to locate the gross error under the environment of large gross errors. In order to verify the feasibility of the hand-eye calibration method, the indoor picking experiment was simulated, and the peaches were identified and positioned by combining deep learning and 3D vision to verify the proposed hand-eye calibration method. The JAKA six-axis robot and TuYang depth camera are used to build the experimental platform. The experimental results show that the method is simple to operate, has good stability, and the calibration plate is easy to manufacture and low in cost. work accuracy requirements.

A Cross-Line Structured Light Scanning System Based on a Measuring Arm

The measurement system proposed in this paper, using a measuring arm and line structured light, has a wide range of applications. To improve the scanning efficiency, the system outlined in this paper uses two single-line structured lights to form crosshair structured light, which we combine with a measuring arm to form a comprehensive scanning measurement system. The calibration method of Zhengyou Zhang and a calibration board are used to complete parameter calibration of the sensors and cameras, as well as hand–eye calibration of the measuring arm. For complex curved-surface objects, this system extracts the cross-line structured light optical center location, which suffers from ambiguity. Therefore, we introduce the use of periodic control of the two line structured light sources in order to resolve the light extraction polysemy. Our experimental results indicate that the proposed system can effectively satisfy the function of crosshair structured light scanning of large, complex surfaces.

TDPPL-Net: A Lightweight Real-Time Tomato Detection and Picking Point Localization Model for Harvesting Robots

Existing target detection models are large and have multiple network parameters, which can severely slow down the detection speed when deployed on small, low-cost GPU-free Industrial Personal Computers (IPC). As a result, this study proposes a lightweight real-time tomato detection and point-picking integrated network model based on YOLOv5 (TDPPL-Net). Firstly, the algorithm replaces the YOLOv5 backbone with a four-group lightweight downsampling model consisting of Ghost Conv and Ghost Bottleneck to reduce the model size, while adding the attention mechanism SimAM module to improve detection accuracy after each scale’s feature map. Secondly, the Spatial Pyramid Pooling-Fast(SPPF) network structure is used and the convolutional layers in the (Feature Pyramid Network and Path Aggregation Network)FPN+PAN structure are replaced with a depth-separable convolution to reduce the computational effort. Finally, the center of the bounding box is used as the picking point, and the corresponding depth information is obtained in combination with the Intel RealSense D435 camera, which is converted into 3D coordinates under the robot arm coordinate system after hand-eye calibration. The experimental results show that TDPPL-Net reduces the number of parameters by 59.84% compared with the original YOLOv5, the model volume is only 40% of the original, the mAP is 93.36, and the real-time detection speed on the IPC without GPU acceleration is 31.41 FPS, which is 170.31% higher than YOLOv5. The TDPPL-Net increases detection speed on low-performance equipment without compromising detection accuracy. It can detect and locate tomato picking points in real-time in the complex natural environment, which can meet the working requirements of harvesting robots.

A Novel One-Click Hand-Eye Calibration Method for Robot Manipulation with Learning-Based 3d Vision

A Novel One-Click Hand-Eye Calibration Method for Robot Manipulation with Learning-Based 3d Vision

Hand-Eye Calibration Method of Line Structured Light Vision Sensor Robot Based on Planar Target

Hand-Eye Calibration Method of Line Structured Light Vision Sensor Robot Based on Planar Target

Modified Hand–Eye Calibration Using Dual Quaternions

This paper presents a modified model for hand–eye calibration based on dual quaternion algebra. By using dual quaternions to represent the rotations and translations of a rigid body simultaneously in the task space, the formulation is elegant for the analysis of the hand–eye equation. The hand–eye transformation derived in this study is represented in a compact manner, which uses a combination of the dual part and the real part of the dual quaternion. Although the hand–eye equation can be solved by using six elements of a dual quaternion without using its scalar parts, the scaler numbers in both the real and dual parts of a dual quaternion contain part of the pose information. The originality is based on the derivation of the construct of the identification algorithm of external parameters of the camera by using all eight elements of a dual quaternion. Then, the data transformation between the cameras of the dual-arm hand–eye robot system is presented. The corresponding results demonstrate that the proposed hand–eye calibration algorithm can process measurement data with noise and can also improve the identification accuracy to verify its efficiency.

Ultrasound Probe and Hand-Eye Calibrations for Robot-Assisted Needle Biopsy

In robot-assisted ultrasound-guided needle biopsy, it is essential to conduct calibration of the ultrasound probe and to perform hand-eye calibration of the robot in order to establish a link between intra-operatively acquired ultrasound images and robot-assisted needle insertion. Based on a high-precision optical tracking system, novel methods for ultrasound probe and robot hand-eye calibration are proposed. Specifically, we first fix optically trackable markers to the ultrasound probe and to the robot, respectively. We then design a five-wire phantom to calibrate the ultrasound probe. Finally, an effective method taking advantage of steady movement of the robot but without an additional calibration frame or the need to solve the AX=XB equation is proposed for hand-eye calibration. After calibrations, our system allows for in situ definition of target lesions and aiming trajectories from intra-operatively acquired ultrasound images in order to align the robot for precise needle biopsy. Comprehensive experiments were conducted to evaluate accuracy of different components of our system as well as the overall system accuracy. Experiment results demonstrated the efficacy of the proposed methods.

Hand-eye Calibration Method of Articulated Arm and Linear Structured Light 3D Camera Based on the Plane

Hand-eye Calibration Method of Articulated Arm and Linear Structured Light 3D Camera Based on the Plane

A Novel Point Set Registration-Based Hand-Eye Calibration Method for Robot-Assisted Surgery.

Pedicle screw insertion with robot assistance dramatically improves surgical accuracy and safety when compared with manual implantation. In developing such a system, hand-eye calibration is an essential component that aims to determine the transformation between a position tracking and robot-arm systems. In this paper, we propose an effective hand-eye calibration method, namely registration-based hand-eye calibration (RHC), which estimates the calibration transformation via point set registration without the need to solve the AX=XB equation. Our hand-eye calibration method consists of tool-tip pivot calibrations in two-coordinate systems, in addition to paired-point matching, where the point pairs are generated via the steady movement of the robot arm in space. After calibration, our system allows for robot-assisted, image-guided pedicle screw insertion. Comprehensive experiments are conducted to verify the efficacy of the proposed hand-eye calibration method. A mean distance deviation of 0.70 mm and a mean angular deviation of 0.68° are achieved by our system when the proposed hand-eye calibration method is used. Further experiments on drilling trajectories are conducted on plastic vertebrae as well as pig vertebrae. A mean distance deviation of 1.01 mm and a mean angular deviation of 1.11° are observed when the drilled trajectories are compared with the planned trajectories on the pig vertebrae.

Robot-world and hand–eye calibration based on quaternion: A new method and an extension of classic methods, with their comparisons

As the two main rigid body transformations in robotic vision systems, the problems of robot-world calibration and hand–eye calibration were raised. Classic quaternion as an advantageous representation of rotation exerts its advantage for the separable methods of the hand–eye calibration problems. While the impact of rotation propagation errors on the translation estimate in separable methods was often ignored. In this paper, after a summarization of quaternion-based methods in SO(3) and SE(3), a dual equations model using adjoint transformation was derived which converts the transformation of rigid body motion into a line transformation. Then, a closed-form rotation solution based on quaternion conjugate was proposed, and two translation closed-form solutions with different propagation error paths were given respectively. Based on these, the impact of rotation propagation errors of the four methods on translation estimate was analyzed, and an assessment of when each method works most and least applicable was given. In addition, simulations and experiments were presented to assess all methods and verify the accuracy and efficacy of the proposed method against uncalibrated commercial robot.

Accuracy evaluation of hand-eye calibration techniques for vision-guided robots.

Hand-eye calibration is an important step in controlling a vision-guided robot in applications like part assembly, bin picking and inspection operations etc. Many methods for estimating hand-eye transformations have been proposed in literature with varying degrees of complexity and accuracy. However, the success of a vision-guided application is highly impacted by the accuracy the hand-eye calibration of the vision system with the robot. The level of this accuracy depends on several factors such as rotation and translation noise, rotation and translation motion range that must be considered during calibration. Previous studies and benchmarking of the proposed algorithms have largely been focused on the combined effect of rotation and translation noise. This study provides insight on the impact of rotation and translation noise acting in isolation on the hand-eye calibration accuracy. This deviates from the most common method of assessing hand-eye calibration accuracy based on pose noise (combined rotation and translation noise). We also evaluated the impact of the robot motion range used during the hand-eye calibration operation which is rarely considered. We provide quantitative evaluation of our study using six commonly used algorithms from an implementation perspective. We comparatively analyse the performance of these algorithms through simulation case studies and experimental validation using the Universal Robot’s UR5e physical robots. Our results show that these different algorithms perform differently when the noise conditions vary rather than following a general trend. For example, the simultaneous methods are more resistant to rotation noise, whereas the separate methods are better at dealing with translation noise. Additionally, while increasing the robot rotation motion span during calibration enhances the accuracy of the separate methods, it has a negative effect on the simultaneous methods. Conversely, increasing the translation motion range improves the accuracy of simultaneous methods but degrades the accuracy of the separate methods. These findings suggest that those conditions should be considered when benchmarking algorithms or performing a calibration process for enhanced accuracy.

Robot-world and hand–eye calibration based on motion tensor with applications in uncalibrated robot

Accurate estimation of robot-world and hand–eye transformations is crucial for robotic vision system. Although as existing methods provide accurate calibration results, which method is best for uncalibrated robot remains a challenging question. This study applied the motion tensor to the dual equations as an operator that performs coordinate frame change, resulting in the most efficient and elegant representation of dual equations. The analytical solution and the numerical solution were given, respectively. The analytical solution requires only four independent variables involving the screw axis, since the pitch and magnitude of a general screw motion are coordinate invariants. The numerical solution used the closed-form rotation solution based on the Kronecker product as an initial value, and followed the optimization with Gauss–Newton algorithm. The superiority with applications in uncalibrated robot was further tested through simulated and experimental tests compared to the existing methods.

Standard Dual Quaternion Optimization and Its Applications in Hand-Eye Calibration and SLAM

Several common dual quaternion functions, such as the power function, the magnitude function, the 2-norm function, and the kth largest eigenvalue of a dual quaternion Hermitian matrix, are standard dual quaternion functions, i.e., the standard parts of their function values depend upon only the standard parts of their dual quaternion variables. Furthermore, the sum, product, minimum, maximum, and composite functions of two standard dual functions, the logarithm and the exponential of standard unit dual quaternion functions, are still standard dual quaternion functions. On the other hand, the dual quaternion optimization problem, where objective and constraint function values are dual numbers but variables are dual quaternions, naturally arises from applications. We show that to solve an equality constrained dual quaternion optimization (EQDQO) problem, we only need to solve two quaternion optimization problems. If the involved dual quaternion functions are all standard, the optimization problem is called a standard dual quaternion optimization problem, and some better results hold. Then, we show that the dual quaternion optimization problems arising from the hand-eye calibration problem and the simultaneous localization and mapping (SLAM) problem are equality constrained standard dual quaternion optimization problems.

A Spatiotemporal Calibration Algorithm for IMU-LiDAR Navigation System Based on Similarity of Motion Trajectories.

The fusion of light detection and ranging (LiDAR) and inertial measurement unit (IMU) sensing information can effectively improve the environment modeling and localization accuracy of navigation systems. To realize the spatiotemporal unification of data collected by the IMU and the LiDAR, a two-step spatiotemporal calibration method combining coarse and fine is proposed. The method mainly includes two aspects: (1) Modeling continuous-time trajectories of IMU attitude motion using B-spline basis functions; the motion of the LiDAR is estimated by using the normal distributions transform (NDT) point cloud registration algorithm, taking the Hausdorff distance between the local trajectories as the cost function and combining it with the hand–eye calibration method to solve the initial value of the spatiotemporal relationship between the two sensors’ coordinate systems, and then using the measurement data of the IMU to correct the LiDAR distortion. (2) According to the IMU preintegration, and the point, line, and plane features of the lidar point cloud, the corresponding nonlinear optimization objective function is constructed. Combined with the corrected LiDAR data and the initial value of the spatiotemporal calibration of the coordinate systems, the target is optimized under the nonlinear graph optimization framework. The rationality, accuracy, and robustness of the proposed algorithm are verified by simulation analysis and actual test experiments. The results show that the accuracy of the proposed algorithm in the spatial coordinate system relationship calibration was better than 0.08° (3δ) and 5 mm (3δ), respectively, and the time deviation calibration accuracy was better than 0.1 ms and had strong environmental adaptability. This can meet the high-precision calibration requirements of multisensor spatiotemporal parameters of field robot navigation systems.

Hand-eye calibration for an unfocused light-field camera.

A calibration framework is established for an unfocused light-field camera and a robotic arm. With Gaussian optics and light-field imaging principles, the mapping relationship between a point light source and its corresponding plenoptic disc feature is established, and the intrinsic and extrinsic parameters of the unfocused light-field camera are calculated through nonlinear optimization. Transformation matrices for eye-to-hand and eye-in-hand configurations are subsequently solved and are validated by applying them to an industrial light-field camera-robotic arm system. With the proposed calibration method, 3D reconstruction for calibration board in different poses is demonstrated and calibration uncertainty is discussed in detail.