Hand-eye Calibration Research Articles

Application of combinatorial optimization algorithm in industrial robot hand eye calibration

In order to improve the measurement accuracy of laser profiler and industrial robot measurement system and reduce the influence of random error in calibration process, a hand eye calibration method based on combinatorial optimization is proposed. Firstly, the laser profiler is positioned by the robot to scan a standard sphere from different postures. Then, combinations from all the measurement data are selected to calculate several transformation matrices and the corresponding profile datasets in base coordinate system can be derived consequently. Since all the profile data come from the same stationary standard sphere, sphere fitting is applied to each dataset and the optimal hand eye transformation matrix can be derived which corresponds to the minimum fitting error. Experimental results show that the novel method can effectively improve the calibration accuracy and sphere loss which is weighted sum of sphere fitting error and radius error is reduced from 1.976 to 0.539 after combinatorial optimization.

A high-efficiency and high-precision automatic 3D scanning system for industrial parts based on a scanning path planning algorithm

To achieve high-precision and high-efficiency 3D scanning of industrial parts, this research proposes an automated 3D scanning system based on path planning algorithms, in which the 3D scanning sensor is realized based on the principle of raster projection. The proposed automated scanning system is composed of a robotic arm, a 3D scanner, and a turntable to achieve a complete 3D scanning of the object to be measured. More precisely, this paper proposes a virtual 3D scanning principle, which can accurately determine whether a certain point on the CAD model can be scanned by a viewpoint. Further, this paper proposes a path planning algorithm based on the virtual 3D scanning principle, which can calculate the optimal scanning path required to scan the object according to the CAD model of the scanned object. Furthermore, to ensure the accuracy of the system, a high-precision system calibration method is proposed to establish the accurate positional relationship between various parts of the system. Firstly, this research proposes a camera calibration algorithm based on close-range photogrammetry, which can calculate the high-precision interior and exterior orientation parameters of the camera, and apply the method of calculating camera external parameters to the hand-eye calibration, making the accuracy of hand-eye calibration is also improved accordingly. In addition, this research proposes a photogrammetry-based projector calibration algorithm to determine the positional relationship between the projector and the camera in the 3D scanner, and a turntable calibration method to determine the positional relationship between the rotation center of the turntable and the camera. Eventually, according to different usage requirements, two 3D scanning data splicing methods are proposed. One is a high-efficient automatic splicing method based on the results of system calibration without pasting mark points, and the other is a high-precision data splicing method based on photogrammetry. Experimental results demonstrate that this system has higher accuracy and better performance in system calibration, scanning path calculation, and multi-view scanning data splicing.

Significance of Camera Pixel Error in the Calibration Process of a Robotic Vision System

Although robotic vision systems offer a promising technology solution for rapid and reconfigurable in-process 3D inspection of complex and large parts in contemporary manufacturing, measurement accuracy poses a challenge for its wide deployment. One of the key issues in adopting a robotic vision system is to understand the extent of its measurement errors which are directly correlated with the calibration process. In this paper, a possible source of practical and inherent measurement uncertainties involved in the calibration process of a robotic vision system are discussed. The system considered in this work consists of an image sensor mounted on an industrial robot manipulator with six degrees of freedom. Based on a series of experimental tests and computer simulations, the paper gives a comprehensive performance comparison of different calibration approaches and shows the impact of measurement uncertainties on the calibration process. It has been found from the error sensitivity analysis that minor uncertainties in the calibration process can significantly affect the accuracy of the robotic vision system. Further investigations suggest that inducing errors in image calibration patterns can have an adverse effect on the hand–eye calibration process compared to the angular errors in the robot joints.

Hand-eye Calibration of Dual Mechanical Arms of Explosive Ordnance Disposal Robot

Hand-eye Calibration of Dual Mechanical Arms of Explosive Ordnance Disposal Robot

Research of Online Hand-Eye Calibration Method Based on ChArUco Board.

To solve the problem of inflexibility of offline hand–eye calibration in “eye-in-hand” modes, an online hand–eye calibration method based on the ChArUco board is proposed in this paper. Firstly, a hand–eye calibration model based on the ChArUco board is established, by analyzing the mathematical model of hand–eye calibration, and the image features of the ChArUco board. According to the advantages of the ChArUco board, with both the checkerboard and the ArUco marker, an online hand–eye calibration algorithm based on the ChArUco board is designed. Then, the online hand–eye calibration algorithm, based on the ChArUco board, is used to realize the dynamic adjustment of the hand–eye position relationship. Finally, the hand–eye calibration experiment is carried out to verify the accuracy of the hand–eye calibration based on the ChArUco board. The robustness and accuracy of the proposed method are verified by online hand–eye calibration experiments. The experimental results show that the accuracy of the online hand–eye calibration method proposed in this paper is between 0.4 mm and 0.6 mm, which is almost the same as the offline hand–eye calibration accuracy. The method in this paper utilizes the advantages of the ChArUco board to realize online hand–eye calibration, which improves the flexibility and robustness of hand–eye calibration.

Research on the Hand–Eye Calibration Method of Variable Height and Analysis of Experimental Results Based on Rigid Transformation

In view of the phenomenon that camera imaging will appear large up close and small from afar in the eye-to-hand hand-calibration system, one hand–eye calibration is carried out. The manipulator is only suitable for grasping objects of the same height, and the calibration results cannot be applied to grasping products with variable height. Based on the study of the pinhole camera model and the rigid transformation model between coordinate systems, the introduction of the calibration height parameters, the relationship between parameters of the rigid transformation matrix between image the coordinate system and the robot coordinate system, and sampling height are established. In the experiment, firstly, through the calibration of camera parameters, the influence of camera distortion on imaging quality is eliminated, and the influence of calibration height is ignored. Then, the machine coordinate system and image coordinate system of the calibration plate at different heights are calibrated using the four-point calibration method. The parameters of the rigid transformation matrix at different heights (H) are calculated. Finally, through experimental analysis, the high linear relationship between the parameters of the rigid transformation matrix from the image coordinate system to the robot coordinate system and the calibration height is fitted. By analyzing the random error of the experiment, the linear relationship between calibration height and pixel density is further established, and the systematic error of the experimental process is deeply analyzed. The experimental results show that the hand–eye calibration system based on this linear relationship is precise and suitable for grabbing products of any height, and the positioning error is less than 0.08%.

Increasing optical pose estimation accuracy via freeform design and its application to hand-eye calibration.

For robot-assisted assembly of complex optical systems, the alignment is facilitated by an accurate pose estimation of its components. However, wavefront-based pose estimation is typically ill-conditioned due to the inherent geometry of conventional industrially manufactured optical components. Therefore, we propose a novel approach in this paper to increase wavefront-based pose estimation accuracy via the design of freeform optics. For this purpose, an optimization problem is derived that parameterizes the component's surfaces by a predetermined freeform surface model. To show the efficacy of our approach, we provide simulation results to compare the pose estimation accuracy for a variety of optical designs. As an application example for the resulting improved pose estimation, a hand-eye calibration of a wavefront sensor is performed. This calibration originates from the field of robotics and represents the identification of a sensor coordinate system with respect to a global reference frame. For quantitative evaluation, the calibrating results are first presented with the aid of simulation data. Finally, the practical feasibility is demonstrated using a conventional industrial robot and additively manufactured freeform lenses.

Hand-Eye Calibration Made Easy Through a Closed-Form Two-Stage Method

An analysis of the existing hand-eye calibration methods reveals that most of them are far from trivial. And, what is worse, their intrinsic complexity makes it difficult to elucidate under which circumstances they fail to provide an accurate solution. Thus, although it might seem that the hand-eye calibration problem is uninspiring because it is assumed to be well-solved, we show in this letter that there was still room for improvement, both in terms of simplicity and robustness. After reviewing the most representative methods, we analyze the situations in which they fail, and we introduce a simpler closed-form alternative that accurately solves the problem in all the identified critical circumstances. Its performance is evaluated using simulated and real experimental data.

A Factor Graph Approach to Simultaneous Robot-World and Hand-Eye Calibration Based on Lie Group

The existing robot-world and hand-eye calibration algorithms always treated the robot pose as a deterministic value. Its uncertainty is not taken into account, which affects the calibration accuracy when the robot’s position accuracy is low. In this paper, we proposed a factor graph approach to simultaneous robot-world and hand-eye calibration, which considers the measurement error of the robot joint encoder. And then, the nonlinear optimization of reprojection error minimization based on a product of exponentials (POE)-based model is implemented to solve the calibration problem based on lie groups. Through simulation and experiment, we validated that the proposed algorithm can get more accurate calibration results compared with the state algorithm.

Hand-eye calibration for 2D laser profile scanners using straight edges of common objects

This paper presents a method for calibrating a 2D profile laser scanner mounted on an industrial articulated robot; a task also known as the hand-eye calibration problem. The challenge of recovering the transformation matrix, from the robot's flange coordinate system to the scanner's coordinate system, lies in the lack of sufficient 3D information, as only 2D data is available. The task is typically performed using precision calibration specimens such as spheres, disks, and planes or using additional external devices such as cameras and 3D sensors. Here, we present an approach based on detecting straight edges found in common objects. Points extracted from the same edge, under various robot poses, are used to solve the calibration problem using a two-phase least-squares strategy, where rotation is recovered first, followed by translation. The process is semi-autonomous, requires minimal laborious and error-prone manual operations; its setup effort is small, because common objects can be used instead of costly precision gauges or external devices; it does not require large number of samples and it is simple to reason about, implement and compute.

Optimal robot-world and hand-eye calibration with rotation and translation coupling

AbstractA classic hand-eye system involves hand-eye calibration and robot-world and hand-eye calibration. Insofar as hand-eye calibration can solve only hand-eye transformation, this study aims to determine the robot-world and hand-eye transformations simultaneously based on the robot-world and hand-eye equation. According to whether the rotation part and the translation part of the equation are decoupled, the methods can be divided into separable solutions and simultaneous solutions. The separable solutions solve the rotation part before solving the translation part, so the estimated errors of the rotation will be transferred to the translation. In this study, a method was proposed for calculation with rotation and translation coupling; a closed-form solution based on Kronecker product and an iterative solution based on the Gauss–Newton algorithm were involved. The feasibility was further tested using simulated data and real data, and the superiority was verified by comparison with the results obtained by the available method. Finally, we improved a method that can solve the singularity problem caused by the parameterization of the rotation matrix, which can be widely used in the robot-world and hand-eye calibration. The results show that the prediction errors of rotation and translation based on the proposed method be reduced to $0.26^\circ$ and $1.67$ mm, respectively.

Research of Hand–Eye System with 3D Vision towards Flexible Assembly Application

In order to improve industrial production efficiency, a hand–eye system based on 3D vision is proposed and the proposed system is applied to the assembly task of workpieces. First, a hand–eye calibration optimization algorithm based on data filtering is proposed in this paper. This method ensures the accuracy required for hand–eye calibration by filtering out part of the improper data. Furthermore, the improved U-net is adopted for image segmentation and SAC-IA coarse registration ICP fine registration method is adopted for point cloud registration. This method ensures that the 6D pose estimation of the object is more accurate. Through the hand–eye calibration method based on data filtering, the average error of hand–eye calibration is reduced by 0.42 mm to 0.08 mm. Compared with other models, the improved U-net proposed in this paper has higher accuracy for depth image segmentation, and the Acc coefficient and Dice coefficient achieve 0.961 and 0.876, respectively. The average translation error, average rotation error and average time-consuming of the object recognition and pose estimation methods proposed in this paper are 1.19 mm, 1.27°, and 7.5 s, respectively. The experimental results show that the proposed system in this paper can complete high-precision assembly tasks.

A novel singular-free solution based on principle of transfer theory for the hand-eye calibration problem

For the Endoscope-assisted Surgical Robot system, the success of the surgery can be heavily determined by the accuracy of the unknown hand-eye matrix between the manipulator and the laparoscopic camera. Although many analytical solutions algorithms have been proposed, it is found that the hand-eye matrix estimated by some existing analytical solutions algorithms will deviate from its true value in some cases. Therefore, the singularity analysis for the solution of the hand-eye calibration based on orthogonal dual tensor is firstly pointed out in this paper. To avoid the above singular phenomenon, a novel singular-free solution based on Principle of Transfer theory is proposed in this paper, which can be verified by the corresponding simulation and experimental results.

Hand–eye calibration method based on three-dimensional visual measurement in robotic high-precision machining

Hand–eye calibration method based on three-dimensional visual measurement in robotic high-precision machining

Active hand-eye calibration via online accuracy-driven next-best-view selection

Active hand-eye calibration via online accuracy-driven next-best-view selection

Hand-Eye Calibration of EOD Robot by Solving the AXB = YCZD Problem

The explosive ordnance disposal (EOD) robots work in a special environment, which requires dual robotic arms to work together for removing the bomb. Therefore, machine vision is very important to locating bombs, especially the accuracy of the hand-eye calibration technology. A basic problem encountered with the collaborative work of dual robotic arms is to solve the unknown homogeneous transformation matrix, including: hand-eye of robotic arm 1, base-base, and camera-end effector’s robotic arm 2. In this article, the hand-eye calibration problem of the dual robotic arm system is expressed as two matrix equations. A new method of simultaneously solving the unknowns in the matrix equation is proposed. This method consists of a closed-form method based on the Kronecker product and an iterative method that transforms a nonlinear problem with a convex function optimization problem. The closed-form method is used to quickly obtain the initial value of the iteration method to improve the efficiency and accuracy of the iteration. In addition, we propose a hand-eye calibration method based on the re-projection error of the RGB-D camera. In order to prove the feasibility and superiority in the proposed iterative method, we conducted simulation and actual experiments and compared them with the other two calibration methods. The comparison results verify the superiority in the proposed method in terms of accuracy.

Model-Free and Uncalibrated Eye-in-Hand Visual Servoing Approach for Concentric-Tube Robots

This article proposes a model-free and uncalibrated eye-in-hand visual servoing (EiH-VS) approach for controlling concentric-tube robots (CTRs) in minimally invasive surgery (MIS). Traditionally, a closed-loop EiH-VS controller requires an accurate robot kinematic model and hand–eye calibration. However, it is difficult to model CTR accurately when considering torsion, shear, friction, interactive force, and nonlinear constitutive behavior. In this article, we map image deviations to robot actuation variables with numerically calculated image Jacobian. The estimation of dynamic image Jacobian is based on a modified adaptive square-root unscented Kalman filter (MASR-UKF). A customized measurement matrix is constructed to describe the transformation between state and observation vectors. Moreover, an adaptive gain controller is designed to accelerate convergence. As a result, no prior knowledge of the CTR kinematic model and hand–eye calibration is needed in visual servoing tasks. Simulations and experiments have been conducted. The results validated the efficacy and efficiency of the proposed methods.

Fluoroscopic Navigation for a Surgical Robotic System Including a Continuum Manipulator.

We present an image-based navigation solution for a surgical robotic system with a Continuum Manipulator (CM). Our navigation system uses only fluoroscopic images from a mobile C-arm to estimate the CM shape and pose with respect to the bone anatomy. The CM pose and shape estimation is achieved using image intensity-based 2D/3D registration. A learning-based framework is used to automatically detect the CM in X-ray images, identifying landmark features that are used to initialize and regularize image registration. We also propose a modified hand-eye calibration method that numerically optimizes the hand-eye matrix during image registration. The proposed navigation system for CM positioning was tested in simulation and cadaveric studies. In simulation, the proposed registration achieved a mean error of 1.10±0.72 mm between the CM tip and a target entry point on the femur. In cadaveric experiments, the mean CM tip position error was 2.86±0.80 mm after registration and repositioning of the CM. The results suggest that our proposed fluoroscopic navigation is feasible to guide the CM in orthopedic applications.

Flatness Measurement of Platform Screen System Welding Assembly Using Stereo Vision and Grid Pattern Projector

The welding assemblies are the main components of the platform screen door system, which can offer the subway with safety, economy, and silence. The deformed welding assemblies can reduce the carrying capacity of the platform screen door system, so the flatness needs to be measured before installation. In this study, a flatness measuring robot based on stereo vision and grid pattern projector measuring system is developed to online measure the welding assembly flatness in workshop. The developed robot is capable of measuring the flatness by incorporating the following solutions: 1) measuring system consisting of stereo vision and grid pattern projector is designed; 2) grid intersection location method using grid intersection probability calculation, candidate preselection, grid intersection reexamination and sub-pixel refinement is proposed, which can consider both false positives and false negatives by using preselection-and-reexamination; 3) 3D coordinates of sampling points (grid intersection) are calculated based on stereo camera calibration and robot hand-eye calibration; 4) flatness measuring method based on the least squares plan is employed to calculate the flatness of welding assembly. The developed flatness measuring robot is robust, accurate, and efficient, as demonstrated by the following performances: an average measuring bias of 0.10 mm in actual working conditions, and an average measurement time of 4.1 s for single welding assembly, meeting the engineering requirements. It is foreseeable that the developed flatness measuring robot has great application market in the online measurement with a middling accuracy and high efficiency.

A Robotic Milling System Based on 3D Point Cloud

Industrial robots have advantages in the processing of large-scale components in the aerospace industry. Compared to CNC machine tools, robot arms are cheaper and easier to deploy. However, due to the poor consistency of incoming materials, large-scale and lightweight components make it difficult to automate robotic machining. In addition, the stiffness of the tandem structure is quite low. Therefore, the stability of the milling process is always a concern. In this paper, the robotic milling research is carried out for the welding pre-processing technology of large-scale components. In order to realize the automatic production of low-conformity parts, the on-site measurement–planning–processing method is adopted with the laser profiler. On the one hand, the laser profiler hand–eye calibration method is optimized to improve the measurement accuracy. On the other hand, the stiffness of the robot’s processing posture is optimized, combined with the angle of the fixture turntable. Finally, the experiment shows the feasibility of the on-site measurement–planning–processing method and verifies the correctness of the stiffness model.