Hand-eye Calibration Research Articles

Enhancing surgical navigation: a robust hand-eye calibration method for the Microsoft HoloLens 2.

Optical-see-through head-mounted displays have the ability to seamlessly integrate virtual content with the real world through a transparent lens and an optical combiner. Although their potential for use in surgical settings has been explored, their clinical translation is sparse in the current literature, largely due to their limited tracking capabilities and the need for manual alignment of virtual representations of objects with their real-world counterparts. We propose a simple and robust hand-eye calibration process for the depth camera of the Microsoft HoloLens2, utilizing a tracked surgical stylus fitted with infrared reflective spheres as the calibration tool. Using a Monte Carlo simulation and a paired-fiducial registration algorithm, we show that a calibration accuracy of 1.65mm can be achieved with as little as 6 fiducial points. We also present heuristics for optimizing the accuracy of the calibration. The ability to use our calibration method in a clinical setting is validated through a user study, with users achieving a mean calibration accuracy of 1.67mm in an average time of 42s. This work enables real-time hand-eye calibration for the Microsoft HoloLens2, without any need for a manual alignment process. Using this framework, existing surgical navigation systems employing optical or electromagnetic tracking can easily be incorporated into an augmented reality environment with a high degree of accuracy.

Automatic Robot Hand-Eye Calibration Enabled by Learning-Based 3D Vision

Hand-eye calibration, a fundamental task in vision-based robotic systems, is commonly equipped with collaborative robots, especially for robotic applications in small and medium-sized enterprises (SMEs). Most approaches to hand-eye calibration rely on external markers or human assistance. We proposed a novel methodology that addresses the hand-eye calibration problem using the robot base as a reference, eliminating the need for external calibration objects or human intervention. Using point clouds of the robot base, a transformation matrix from the coordinate frame of the camera to the robot base is established as “I=AXB.” To this end, we exploit learning-based 3D detection and registration algorithms to estimate the location and orientation of the robot base. The robustness and accuracy of the method are quantified by ground-truth-based evaluation, and the accuracy result is compared with other 3D vision-based calibration methods. To assess the feasibility of our methodology, we carried out experiments utilizing a low-cost structured light scanner across varying joint configurations and groups of experiments. The proposed hand-eye calibration method achieved a translation deviation of 0.930 mm and a rotation deviation of 0.265 degrees according to the experimental results. Additionally, the 3D reconstruction experiments demonstrated a rotation error of 0.994 degrees and a position error of 1.697 mm. Moreover, our method offers the potential to be completed in 1 second, which is the fastest compared to other 3D hand-eye calibration methods. We conduct indoor 3D reconstruction and robotic grasping experiments based on our hand-eye calibration method. Related code is released at https://github.com/leihui6/LRBO.

One-step Solving the Robot-world and Hand-Eye Calibration Based On the Principle of Transference

One-step Solving the Robot-world and Hand-Eye Calibration Based On the Principle of Transference

Research on visual servo strategy for the uncalibrated non-standard manipulator

Abstract Aiming at the problem that the hand-eye calibration of the uncalibrated non-standard manipulator cannot be addressed by the general calibration method, i.e., the eye-in-hand and eye-to-hand calibration method of the visual servo system was proposed to guide the manipulator without calibration. First, a camera was installed on the manipulator to complete the eye-in-hand calibration and achieve transformation from the end coordinate system to the camera coordinate system. Second, a second camera was used to achieve eye-to-hand calibration and obtain the transformation relationship from the end coordinate system to the calibration board. The entire calibration process only requires the collection of images from two scenarios, i.e., the docking and the completion of the docking processes. The implementation results show that the rotation error between groups is less than 1.5°, and the translation error between groups is less than 8 mm. Therefore, the proposed visual servo strategy can accurately guide the uncalibrated non-standard manipulator.

On flange-based 3D hand–eye calibration for soft robotic tactile welding

This paper investigates the direct application of standardized designs on the robot for conducting robot hand–eye calibration by employing 3D scanners with collaborative robots. The well-established geometric features of the robot flange are exploited by directly capturing its point cloud data. In particular, an iterative method is proposed to facilitate point cloud processing towards a refined calibration outcome. Several extensive experiments are conducted over a range of collaborative robots, including Universal Robots UR5 & UR10 e-series, Franka Emika, and AUBO i5 using an industrial-grade 3D scanner Photoneo Phoxi S & M and a commercial-grade 3D scanner Microsoft Azure Kinect DK. Experimental results show that translational and rotational errors converge efficiently to less than 0.28 mm and 0.25 degrees, respectively, achieving a hand–eye calibration accuracy as high as the camera’s resolution, probing the hardware limit. A welding seam tracking system is presented, combining the flange-based calibration method with soft tactile sensing. The experiment results show that the system enables the robot to adjust its motion in real-time, ensuring consistent weld quality and paving the way for more efficient and adaptable manufacturing processes.

Latent Space Representations for Marker-Less Realtime Hand-Eye Calibration.

Marker-less hand-eye calibration permits the acquisition of an accurate transformation between an optical sensor and a robot in unstructured environments. Single monocular cameras, despite their low cost and modest computation requirements, present difficulties for this purpose due to their incomplete correspondence of projected coordinates. In this work, we introduce a hand-eye calibration procedure based on the rotation representations inferred by an augmented autoencoder neural network. Learning-based models that attempt to directly regress the spatial transform of objects such as the links of robotic manipulators perform poorly in the orientation domain, but this can be overcome through the analysis of the latent space vectors constructed in the autoencoding process. This technique is computationally inexpensive and can be run in real time in markedly varied lighting and occlusion conditions. To evaluate the procedure, we use a color-depth camera and perform a registration step between the predicted and the captured point clouds to measure translation and orientation errors and compare the results to a baseline based on traditional checkerboard markers.

Self-correcting gripping robotic arm for QR code based on hand-eye calibrationv

During the operation of a robotic arm, the inaccuracy of the end-effector position can seriously affect the overall performance of the robot. In order to significantly reduce the positional error of the actuator with minimal cost, we adopt a 2D code-assisted positioning method based on the principle of hand-eye calibration to negatively feedback correct the coordinates of the end-effector. This method establishes a robotic arm-camera system, which improves the performance and accuracy of the robotic arm.

Continual learning approaches to hand–eye calibration in robots

This study addresses the problem of hand–eye calibration in robotic systems by developing Continual Learning (CL)-based approaches. Traditionally, robots require explicit models to transfer knowledge from camera observations to their hands or base. However, this poses limitations, as the hand–eye calibration parameters are typically valid only for the current camera configuration. We, therefore, propose a flexible and autonomous hand–eye calibration system that can adapt to changes in camera pose over time. Three CL-based approaches are introduced: the naive CL approach, the reservoir rehearsal approach, and the hybrid approach combining reservoir sampling with new data evaluation. The naive CL approach suffers from catastrophic forgetting, while the reservoir rehearsal approach mitigates this issue by sampling uniformly from past data. The hybrid approach further enhances performance by incorporating reservoir sampling and assessing new data for novelty. Experiments conducted in simulated and real-world environments demonstrate that the CL-based approaches, except for the naive approach, achieve competitive performance compared to traditional batch learning-based methods. This suggests that treating hand–eye calibration as a time sequence problem enables the extension of the learned space without complete retraining. The adaptability of the CL-based approaches facilitates accommodating changes in camera pose, leading to an improved hand–eye calibration system.

Grounding rod hanging and removing robot with hand-eye self-calibration capability in substation

In this paper, a robot system for hanging and removing grounding rods is designed by integrating 3D recognition technology, hand-eye self-calibration technology, and automatic operation technology. Specifically, a novel hand-eye self-calibration algorithm is proposed that only uses common objects in the actual scene, which differs from traditional robot hand-eye calibration in that it requires a dedicated calibration board to assist with offline completion. Addressing the problem that existing self-calibration methods cannot be optimized as a whole, resulting in low accuracy and instability of the solution, a multi-stage objective function optimization self-calibration algorithm is proposed. An optimization method based on the minimization of re-projection error is designed to compensate for the results, which uses an efficient Oriented fast and rotated brief (ORB) feature extraction algorithm and introduces a scoring mechanism to retain more correct matching points in the feature matching stage. Experimental verifications are conducted using both public dataset and practical robot system. In the dataset experiment, our method demonstrates superior accuracy and robustness compared to existing self-calibration methods. Furthermore, the practical robot platform experiment confirms the feasibility and efficacy of our approach across various wind speeds and lighting conditions.

A High-Precision Hand-Eye Coordination Localization Method under Convex Relaxation Optimization.

Traditional switching operations require on-site work, and the high voltage generated by arc discharges can pose a risk of injury to the operator. Therefore, a combination of visual servo and robot control is used to localize the switching operation and construct hand-eye calibration equations. The solution to the hand-eye calibration equations is coupled with the rotation matrix and translation vectors, and it depends on the initial value determination. This article presents a convex relaxation global optimization hand-eye calibration algorithm based on dual quaternions. Firstly, the problem model is simplified using the mathematical tools of dual quaternions, and then the linear matrix inequality convex optimization method is used to obtain a rotation matrix with higher accuracy. Afterwards, the calibration equations of the translation vectors are rewritten, and a new objective function is established to solve the coupling influence between them, maintaining positioning precision at approximately 2.9 mm. Considering the impact of noise on the calibration process, Gaussian noise is added to the solutions of the rotation matrix and translation vector to make the data more closely resemble the real scene in order to evaluate the performance of different hand-eye calibration algorithms. Eventually, an experiment comparing different hand-eye calibration methods proves that the proposed algorithm is better than other hand-eye calibration algorithms in terms of calibration accuracy, robustness to noise, and stability, satisfying the accuracy requirements of switching operations.

A new novel recognition and positioning system of black light-absorbing volute for automation depalletizing development

Abstract In the application of vision measurement, the black light-absorbing object is difficult to reflect the structured light from infrared emitter of the RGB-D camera. Therefore, an image recognition algorithm based on reference environment information is proposed to acquire the spatial positioning information of black volutes in the depalletizing system. The hardware system of the depalletizing system is mainly constructed of an upper computer, a six-axis industrial robot, an RGB-D camera and an end adsorption device. Firstly, the horizontal position information of each volute placed on the cardboard is obtained by the depth differences between the cardboard and the volute. Then, the depth information of the volute is obtained by the upper cardboard depth through collecting the position of the end vacuum suction cup triggered by feedback signal from vacuum generator. Secondly, a regional planar hand-eye calibration method is developed to improve the calibration accuracy in two-dimensional coordinates. The regional calibration method divides the robot working area into four regions: upper left, lower left, upper right, and lower right. The transformation matrix of each region is calculated separately. Finally, the depalletizing experiment is conducted on the three types of volutes. It is concluded that the average positioning error of the grasping center point of each volute obtained by our method is 3.795 mm, and its standard deviation is 1.769 mm. The average value of regional planar hand-eye calibration error is 4.044 mm, and its standard deviation is 1.501 mm. Under a stack of materials with dimensions of 1350 mm × 1350 mm × 1500 mm, the maximum error is controlled within 15 mm. Additionally, when combined with the end feedback compensation mechanism, the success rate for grasping all three volutes reaches 100%.

Research and Implementation of Robotic Arm Positioning and Grasping Based on Visual Guidance

Abstract This paper proposes a robotic arm vision guidance system based on the ROS platform, which combines the Hikrobot Robotics 2D vision sensor with the Xinsong GCR5 robotic arm to successfully realise vision-guided robotic arm positioning and grasping. Camera calibration, hand-eye calibration, and deep learning techniques are used to establish an accurate transformation relationship between the base coordinate system of the robotic arm and the camera coordinate system to achieve efficient target goal recognition and localisation. Adopting the MoveIt framework and RRT-connect algorithm, the robotic arm can flexibly plan the movement path and successfully complete the target grasping, which provides the theoretical basis and practical experience for robotic arm intelligent grasping.

Camera-IMU extrinsic calibration method based on intermittent sampling and RANSAC optimization

Accurately calibrate the extrinsic parameters between a camera and an Inertial Measurement Unit (IMU) is a prerequisite for achieving sensor data fusion in visual-inertial navigation systems. Due to delays introduced by triggering, transmission, and other factors, the sampled times of the camera and IMU do not align with the system timestamps, leading to a decrease in the accuracy of extrinsic calibration. Existing calibration methods are unable to avoid the temporal misalignment between the camera and IMU. This paper introduces a novel intermittent calibration sampling method to address temporal misalignment challenges. The approach entails detecting motion-stillness transition points in the IMU data as keyframe segmentation criteria, thereby decoupling data acquisition from system timestamps and aligning it with the sensor’s motion process. Subsequently, the robot hand-eye calibration method is applied to the extrinsic calibration of the Camera-IMU, combining the Random Sample Consensus algorithm to filter the poses of the camera and IMU. This process achieves precise calibration of extrinsic parameters. Experimental validation shows that the algorithm proposed in this paper enhances calibration accuracy when compared to the widely used Kalibr Camera-IMU calibration toolbox. The rotational extrinsic parameter accuracy increased by 181.25%, and the translational extrinsic parameter accuracy improved by 168.33%. These findings provide effective technical means for precise measurement of Camera-IMU extrinsic parameters.

Hand-eye calibration of line structured-light sensor based on double-sphere constraints

Aiming at the complex problem of solving the positional relationship between the line laser sensor at the end of the robot wrist and the coordinate system of the end flange, a line laser hand-eye calibration method based on a double ball fixed target is proposed. Firstly, a double standard ball with known radius and center distance is used as a reference tool, and the point cloud data of a single contour line is collected several times, and the coordinates of the two ball centers under the camera coordinate system can be obtained according to the center coordinates of the circle obtained from the fitted circle combined with the Pythagoras theorem. Then, according to the principle that the two spherical centers are fixed under the robot base coordinate system, the rigid transformation matrix between the line laser sensor and the robot is successively solved. When solving the rotation matrix, the constraint of the distance between the spherical centers of the two balls is introduced to obtain a solution with better accuracy. Experimental results show that the method has better overall performance compared with the traditional single-ball hand-eye calibration method.

Multi-camera joint calibration algorithm for guiding machining robot positioning considering ambient light and error uncertainty

Multi-camera guided robotic machining is widely used in the removal of flash and burrs on the complex automotive castings and forgings. The multi-camera calibration process, however, is easily affected by the ambient light and uncertain error, thereby resulting in the undesired robot positioning accuracy. To overcome the challenging problem, this paper proposes a multi-camera joint calibration (MCJC) algorithm by considering both the ambient light and error uncertainty to optimize and compensate the positioning error of the image-guided machining robot. The high dynamic range imaging is used at first to obtain stable calibration image data for countering the effects of ambient light changes. Then the uncertainty error in camera self-calibration process is removed by the confidence threshold filtering. The hand-eye calibration matrix is corrected finally using the deviation feedback optimization for positioning accuracy enhancement of the image-guided machining robot. Taking the automotive flywheel shell as the experimental object, the results demonstrate that the average absolute positioning error is controlled within 1.3 mm, the optimum can reach 0.43 mm, and the average relative positioning error is smaller than 0.5 mm. Compared with the existing state-of-the-art algorithms, the proposed algorithm shows high positioning accuracy and strong algorithm robustness, and can meet the requirements for the subsequent image-guided robotic machining.

Automatic Hand-Eye Calibration Method of Welding Robot Based on Linear Structured Light

Automatic Hand-Eye Calibration Method of Welding Robot Based on Linear Structured Light

Development of a vision system integrated with industrial robots for online weld seam tracking

This paper develops a laser-structured light vision system for the industrial welding manipulator. The designed structured light vision system consisting of a line laser projector and an industrial camera is a decent 3D object data acquisition platform. By applying the principle of triangulation, the depth value of the object can be obtained via the laser plane in the captured image. According to hand-eye calibration and the principle of triangulation, the 3D position of the objects in the robot coordinate system is calculated. Based on the discriminated convolution tracker and deep reinforcement learning that increases the accuracy and processing time of the tracker, allowing simultaneous scanning for seam data acquisition and welding along the weld path, a real-time seam tracking algorithm is developed to handle the square-groove butt joint with small gaps (i.e., <1 mm). Experiments are undertaken using a Yaskawa industrial welding robot integrating the designed laser-structured light vision system. Experiment results show the significant performance of the developed system in welding square-groove butt joints with curved profiles and small gaps, i.e., the welding absolute mean error is 0.31 mm, whereas in the case of using convolution tracker only, the welding absolute mean error 1.7 mm.

Hand-eye calibration of line structured-light sensor by scanning and reconstruction of a free-placed standard cylindrical target

This paper proposed a hand-eye calibration method for a line structured-light (LSL) scanner mounted on industrial robot. Unlike widely used spherical targets, the target used in our work is a cylinder, which is inexpensive and easy to be manufactured in larger size to cover a wider calibration space. In order to address the profile self-occlusion and drastically shape changing of conic section during calibration, an unconstrained dimensionality reduction method is proposed to fit the scanned profiles. Based on fitted ellipse centers under various robot poses, the hand-eye calibration model is developed based on the difference between cylinder radius and the distance from the profile point to the cylinder axis. Simulation and actual experiments have shown that the proposed ellipse fitting method is more accurate and robust, and the calibration method achieves higher reconstruction accuracy under the same robot positioning error, need less robot operations to obtain more accurate hand-eye parameters.

Investigation of Multi-Stage Visual Servoing in the context of autonomous assembly

This paper introduces a novel strategy, Multi-Stage Based Visual Servoing (MSBVS), to tackle the significant challenge of planning complex trajectories in robotic systems without hand–eye calibration, particularly in some assembly tasks. The MSBVS strategy decomposes a complete visual servoing process into three distinct phases, each characterized by unique functionalities. By assigning specific image features and control objectives to each phase, the strategy indirectly facilitates the planning and tracking complex trajectories. Furthermore, a novel image feature, the vanishing angle, is introduced, enhancing the precision in representing an object’s spatial position. Subsequently, a robotic vision system is established, and real-time simulations and experiments are performed to verify the efficacy of the MSBVS strategy. The results show that MSBVS efficiently plans and executes complex trajectories in robotic assembly tasks.

One-Step Solving the Hand–Eye Calibration by Dual Kronecker Product

Abstract Hand–eye calibration is a typical research direction in robotics applications. The current methods can be divided into two categories according to whether the rotational and translational equations are decoupled for computation: two-step methods and one-step methods. Both one-step and two-step methods generally convert such problems to linear null space computations, which are implemented by the corresponding computational operators. Owing to the booming development of the rotation operators, the two-step methods have been more fully researched. However, due to the limitations of the research on computational operators integrating rotation and translation, the one-step methods still have much scope for research. Dual algebra, as effective mathematical entities for screws and wrenches, provides the theoretical basis for the development of the one-step methods for hand–eye calibration. In this paper, a computational operator for the dual matrices computation was first proposed, i.e., dual Kronecker product. Subsequently, a hand–eye calibration framework was proposed based on the dual Kronecker product, which allowed the screw motion to be represented as multiple dual vectors. Furthermore, the equivalence of this framework with the orthogonal-dual-tensor-based approach was derived, providing a more intuitive computational representation. The feasibility and superiority of the proposed computational framework were experimentally verified.