Eye Calibration Research Articles

Hand-eye calibration for rigid laparoscopes using an invariant point.

PurposeLaparoscopic liver resection has significant advantages over open surgery due to less patient trauma and faster recovery times, yet it can be difficult due to the restricted field of view and lack of haptic feedback. Image guidance provides a potential solution but one current challenge is in accurate “hand–eye” calibration, which determines the position and orientation of the laparoscope camera relative to the tracking markers.MethodsIn this paper, we propose a simple and clinically feasible calibration method based on a single invariant point. The method requires no additional hardware, can be constructed by theatre staff during surgical setup, requires minimal image processing and can be visualised in real time. Real-time visualisation allows the surgical team to assess the calibration accuracy before use in surgery. In addition, in the laboratory, we have developed a laparoscope with an electromagnetic tracking sensor attached to the camera end and an optical tracking marker attached to the distal end. This enables a comparison of tracking performance.ResultsWe have evaluated our method in the laboratory and compared it to two widely used methods, “Tsai’s method” and “direct” calibration. The new method is of comparable accuracy to existing methods, and we show RMS projected error due to calibration of 1.95 mm for optical tracking and 0.85 mm for EM tracking, versus 4.13 and 1.00 mm respectively, using existing methods. The new method has also been shown to be workable under sterile conditions in the operating room.ConclusionWe have proposed a new method of hand–eye calibration, based on a single invariant point. Initial experience has shown that the method provides visual feedback, satisfactory accuracy and can be performed during surgery. We also show that an EM sensor placed near the camera would provide significantly improved image overlay accuracy.

Hand-Eye Calibration in Visually-Guided Robot Grinding.

Visually-guided robot grinding is a novel and promising automation technique for blade manufacturing. One common problem encountered in robot grinding is hand-eye calibration, which establishes the pose relationship between the end effector (hand) and the scanning sensor (eye). This paper proposes a new calibration approach for robot belt grinding. The main contribution of this paper is its consideration of both joint parameter errors and pose parameter errors in a hand-eye calibration equation. The objective function of the hand-eye calibration is built and solved, from which 30 compensated values (corresponding to 24 joint parameters and six pose parameters) are easily calculated in a closed solution. The proposed approach is economic and simple because only a criterion sphere is used to calculate the calibration parameters, avoiding the need for an expensive and complicated tracking process using a laser tracker. The effectiveness of this method is verified using a calibration experiment and a blade grinding experiment. The code used in this approach is attached in the Appendix.

In-field self-calibration of robotic manipulator using stereo camera: application to Humanitarian Demining Robot

A robotic manipulator using a stereo camera mounted on one of its links requires a precise kinematic transformation calibration between the manipulator and the camera coordinate frames, the so-called hand–eye calibration, to achieve high-accuracy end-effector positioning. This paper introduces a new method that performs simultaneous joint angle and hand–eye calibration, based on a traditional method that uses a sequence of pure rotations of the manipulator links. The new method considers an additional joint angle constraint, which improves the calibration accuracy when the circular arc that can be measured by the stereo camera is very limited. Experimental results using a manipulator developed for humanitarian demining demonstrate that relative errors between the end effector and the external points mapped by the stereo camera are greatly reduced compared to traditional methods.

Development and calibration of an integrated 3D scanning system for high-accuracy large-scale metrology

Quality control in advanced manufacturing requires automated and high-accuracy large-scale 3D measurement. This paper proposes a high-accuracy, low-cost 3D scanning system by integrating industrial robot with precise linear rail and laser sensor. The measuring principle and system construction of the integrated system are introduced in detail. A mathematical model is established for mapping the change of the laser sensor frame while it scans along the linear rail and a sphere-based algorithm for rail orientation calibration is introduced. Subsequently, taking the robot positioning error into consideration, an enhanced hand–eye calibration method is proposed to determine the relationship between robot end-effector and rail scanning frame. Validation experiments were performed, a maximum distance error of 0.071mm was detected within the rail range and a mean/maximum distance error of 0.309/0.604mm was detected in the robot volume. A large-scale scanning instance also shows that integrated robotic scanning system features high-efficiency and high-accuracy.

A flexible method combining camera calibration and hand–eye calibration

SUMMARYWe consider the conventional techniques of vision robot system calibration where camera parameters and robot hand–eye parameters are computed separately, i.e., first performing camera calibration and then carrying out hand–eye calibration based on the calibrated parameters of cameras. In this paper we propose a joint algorithm that combines the camera calibration and the hand–eye calibration together. The proposed algorithm gives the solutions of the cameras' parameters and the hand–eye parameters simultaneously by using nonlinear optimization. Both simulations and real experiments show the superiority of our algorithm. We also apply our algorithm in the real application of the robot-assisted surgical system, and very good results have been obtained.

Hand–eye calibration with epipolar constraints: Application to endoscopy

In this paper, we propose a new calibration method for a hand–eye system equipped with a camera undergoing radial distortion as a rigid endoscope. While classic methods propose either a separated estimation of the camera intrinsics and the hand–eye transform or a mixed non-linear estimation of both hand–eye and camera intrinsics assuming a pin-hole model, the proposed approach enables a simultaneous refinement of the hand–eye and the camera parameters including the distortion factor with only three frames of the calibrated pattern. We propose and compare two criteria to minimize: (i) the first one is based on the epipolar constraint between pairs of frames and (ii) the second one expresses the camera extrinsic with respect to hand–eye and world-grid transforms in the single frame reprojection error. We run bundle-adjustment on each criterion with respect to the distortion parameters, the camera intrinsics and the hand–eye transform. Our method allows us to recover more accurate 3D shapes when compared to state-of-the-art non-linear methods.

Error Modeling and Compensation Approach for Three-dimensional Surface Scanning Robot

For the three-dimensional surface scanning robotic system,the system measuring accuracy mainly depends on the following aspects: Robot positioning accuracy,line structured light vision sensor measuring accuracy,hand to eye calibration accuracy.The three aspects mentioned above can be considered as part ones.Whatever how much error will these three aspects be,the final error will be reflected in the obtained measuring data.However,the error model of the system is difficult to be established via mathematical models.As to enhance the system measuring accuracy,the error model is constructed via particle swarm optimization-radial basis function neural network(PSO-RBFNN).The input of PSO-RBFNN is chosen as the image coordinates of laser stripes,while the output of PSO-RBFNN is chosen as the errors between measured points and closest points after iterative closest point technique.The centers and widths of RBFNN are optimized by particle swarm optimization(PSO) to find more suitable parameters and thus reducing the number of neurons in the hidden layer.Experimental results demonstrate that the system measuring accuracy is improved after the error compensation scheme via constructed error network model.

Hand–eye calibration and positioning for a robot drilling system

In the aircraft manufacturing, drilling large amount of assembly holes in aircraft board is one of the key bottlenecks of production efficiency. To enhance the efficiency and quality of assembly holes' manufacturing, robot drilling system replacing manual operation becomes more and more urgent. Normally, a robot system needs accurate mathematical models of the manufactured object and the environment when it's working; as a matter of fact, because of the manufacturing error, the homogeneity between aircraft board and its mathematical model is dissatisfied. So a hand–eye vision system is introduced to realize the positioning of the end effector in order to improve the flexibility and robustness of a robot drilling system. The paper discusses the calibration and positioning of a hand–eye vision system for a robotic aircraft board drilling system. Because the drill must be vertical and keep a fixed distance to the aircraft board surface before drilling, the depth information of hand–eye relationship is neglected and by defining an intermediate scene coordinate system the hand–eye relationship between the robot coordinate system and the vision coordinate system is established. Then the position of target point can be described in the robot coordinate system by using the calibrated hand–eye relationship, and thus the navigation information for the robot drilling system can be provided. Experimental results of the calibration and positioning of the hand–eye vision of a robot drilling system is provided, and the main factors that affect the positioning error are analyzed.

A Method for Sustaining Consistent Sensory–Motor Coordination under Body Property Changes Including Tool Grasp/Release

The body's physical and spatial properties change because of unexpected accidents or objects it holds (e.g., tools). To maintain stable and consistent behaviors in acting time, it is necessary to adapt body representation to the unexpected changes. This adaptation is achieved when a robot sustains its coordination between sensation and body motion, i.e., observation and identification of the discrepancy between its knowledge and actually obtained sensory feedback. For sustainability, the robot should autonomously judge the reliability and the tolerance of the identification. Based on basic mathematics, we propose a method with an indirect but clear criterion of the reliability for the sustainable coordinations. We also implement the method on visual–motor coordination and on kinesthetic–motor coordination. Remarkably, our method achieves a marker-free, easily convergent and sufficiently accurate (i.e., easily applicable) hand–eye calibration method with irrelevant objects in view. The evaluation of the method is provided with experiments in a real robot. Our experimental results show the novelty of the concept of sustainable coordination and the availability of our method for the concept. We hope this paper be a powerful approach for building autonomous robots.

A hand–eye calibration algorithm based on screw motions

SUMMARYWhen computer vision technique is used in robotics, robotic hand–eye calibration is a very important research task. Many algorithms have been proposed for hand–eye calibration. Based on these algorithms, we introduce a new hand–eye calibration algorithm in this paper, which employs the screw motion theory to establish a hand–eye matrix equation by using quaternion and gets a simultaneous result for rotation and translation by solving linear equations. The algorithm proposed in this paper has high accuracy and stable computational efficiency and can be understood easily. Both simulations and real experiments show the superiority of our algorithm over the comparative algorithms.

Hand–Eye Calibration Applied to Viewpoint Selection for Robotic Vision

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> Viewpoint calibration is a method to manipulate hand–eye for generating calibration parameters for active viewpoint control and object grasping. In robot vision applications, accurate vision sensor calibration and robust vision-based robot control are essential for developing an intelligent and autonomous robotic system. This paper presents a new approach to hand–eye robotic calibration for vision-based object modeling and grasping. Our method provides a 1.0-pixel level of image registration accuracy when a standard Puma/Kawasaki robot generates an arbitrary viewpoint. To attain this accuracy, our new formalism of hand–eye calibration deals with a lens distortion model of a vision sensor. Our most distinguished approach of optimizing intrinsic parameters is to utilize a new parameter estimation algorithm using an extended Kalman filter. Most previous approaches did not even consider the optimal estimates of the intrinsic and extrinsic camera parameters, or chose one of the estimates obtained from multiple solutions, which caused a large amount of estimation error in hand–eye calibration. We demonstrate the power of this new method for: 1) generating 3-D object models using an interactive 3-D modeling editor; 2) recognizing 3-D objects using stereovision systems; and 3) grasping 3-D objects using a manipulator. Experimental results using Puma and Kawasaki robots are shown. </para>

Adaptive motion selection for online hand–eye calibration

SUMMARYWhile a robot moves, online hand–eye calibration to determine the relative pose between the robot gripper/end-effector and the sensors mounted on it is very important in a vision-guided robot system. During online hand–eye calibration, it is impossible to perform motion planning to avoid degenerate motions and small rotations, which may lead to unreliable calibration results. This paper proposes an adaptive motion selection algorithm for online hand–eye calibration, featured by dynamic threshold determination for motion selection and getting reliable hand–eye calibration results. Simulation and real experiments demonstrate the effectiveness of our method.

Binocular Coordination of the Eyes during Reading

Saccadic eye movements and fixations are the behavioral means by which we visually sample text during reading. Human oculomotor control is governed by a complex neurophysiological system involving the brain stem, superior colliculus, and several cortical areas. A very widely held belief among researchers investigating primate vision is that the oculomotor system serves to orient the visual axes of both eyes to fixate the same target point in space. It is argued that such precise positioning of the eyes is necessary to place images on corresponding retinal locations, such that on each fixation a single, nondiplopic, visual representation is perceived. Vision works actively through a continual sampling process involving saccades and fixations. Here we report that during normal reading, the eyes do not always fixate the same letter within a word. We also demonstrate that saccadic targeting is yoked and based on a unified cyclopean percept of a whole word since it is unaffected if different word parts are delivered exclusively to each eye via a dichoptic presentation technique. These two findings together suggest that the visual signal from each eye is fused at a very early stage in the visual pathway, even when the fixation disparity is greater than one character (0.29 deg), and that saccade metrics for each eye are computed on the basis of that fused signal.

Conformal Geometric Algebra for Robotic Vision

In this paper the authors introduce the conformal geometric algebra in the field of visually guided robotics. This mathematical system keeps our intuitions and insight of the geometry of the problem at hand and it helps us to reduce considerably the computational burden of the problems. As opposite to the standard projective geometry, in conformal geometric algebra we can deal simultaneously with incidence algebra operations (meet and join) and conformal transformations represented effectively using spinors. In this regard, this framework appears promising for dealing with kinematics, dynamics and projective geometry problems without the need to resort to different mathematical systems (as most current approaches do). This paper presents real tasks of perception and action, treated in a very elegant and efficient way: body---eye calibration, 3D reconstruction and robot navigation, the computation of 3D kinematics of a robot arm in terms of spheres, visually guided 3D object grasping making use of the directed distance and intersections of lines, planes and spheres both involving conformal transformations. We strongly believe that the framework of conformal geometric algebra can be, in general, of great advantage for applications using stereo vision, range data, laser, omnidirectional and odometry based systems.

Recovering the X-ray projection geometry for three-dimensional tomographic reconstruction with additional sensors: Attached camera versus external navigation system

Three-dimensional tomographic reconstruction using intra-operative mobile C-arms could provide physicians with new and exciting tools for image-guided surgery. Recovery of the projection geometry of mobile X-ray systems is a crucial step for such reconstruction procedures. Recent work on medical imaging describes the use of optical or electro-magnetic sensor systems in order to navigate surgical instruments. These systems can also be used for the estimation of C-arm motion, and therefore for the recovery of the projection geometry of the X-ray C-arm. In this case, the mathematical problem that needs to be solved is equivalent to the hand–eye calibration well studied by both the computer vision and robotics community. We first study the recovery of the motion and projection geometry using five different hand–eye calibration methods proposed in the literature. The optical navigation system POLARIS from Northern Digital Inc. was used in our experiments. The results of the estimated motion and projection geometry using the five hand–eye calibration methods are compared with the same results obtained using an off-the-shelf CCD camera attached to the mobile C-arm. The experimental results include three-dimensional tomographic reconstruction results using our mobile C-arm. We show that even though the motion of the C-arm is more precisely recovered using the navigation system, the projection geometry is better estimated using the attached CCD camera.

Motor algebra approach for visually guided robotics

This paper presents the formulation of the 3D kinematics in the geometric algebra framework. We show that this approach is extremely useful for solving problems in the field of visual guided robotics. In this algebraic system the 3D Euclidean motion of points, lines and planes can be advantageously represented using the algebra of motors. The computational complexity of the direct and indirect kinematics and other problems concerning robot manipulators depends on its degrees of freedom as well on its geometric characteristics. Our approach makes possible a direct algebraic formulation of the concrete problem in such a way that it reflects the underlying geometric structure. This is achieved by switching to a description of parts of the problem based on motor representations of points, lines and planes where necessary. The first robotics task this paper deals with is the formulation and computation of closed-form solutions of the direct and indirect kinematics of standard robot manipulators and a simple example of a grasping task. The flexible method presented here is new and it widens the current standard point or line representation based approaches for the treatment of problems related to robot manipulators. The second challenging task presented in this paper is the solution of the hand–eye calibration problem when cameras are attached to robot arms. The solution of this problem in the motor algebra framework turns to be linear. Both tasks are crucial for visual guided robotics, that is why we are strongly motivated to present them to show how useful it is to apply motor algebra for solving geometric problems in visual guided robotics.

A neuro-fuzzy control model for fine-positioning of manipulators

We propose an approach for learning the fine-positioning of a parallel-jaw gripper on a robot arm using visual sensor data as controller input. The first component of the model can be viewed as a perceptron network that projects high-dimensional input data into a low-dimensional eigenspace. The dimension reduction is efficient if the movements achieving optimal positioning are constrained to a local scenario. The second component is an adaptive fuzzy controller serving as an interpolator whose input space is the eigenspace and whose outputs are the motion parameters. Instead of undergoing cumbersome hand–eye calibration processes, our system is trained in a self-supervised learning procedure using systematic perturbation motion around the optimal position. The approach is applied to tasks of three degrees of freedom, e.g. translating the gripper in the x– y-plane and rotating it about the z-axis.