Hand-eye Relationship Research Articles

IHUVS: Infinite Homography-Based Uncalibrated Methodology for Robotic Visual Servoing

The construction of task functions in robotic manipulation is of paramount importance for uncalibrated visual servoing. The existing methods generally use image information as control variables and estimate the image Jacobian matrix online, thus possessing issues relating to convergence, and image Jacobian matrix singularities. Therefore, this work proposes a novel methodology dubbed infinite homography-based uncalibrated visual servoing (IHUVS), in which the visual control of the robot end-effector pose is decomposed into its rotational and translational components. The corresponding rotational controller designs the visual servoing task function using the relationship between the infinite homography matrix and rotation matrix, and employs the Kronecker product to derive linear equations for rotational control, as well as to conduct the associated task error analysis. Meanwhile, the translational controller utilizes Kalman filtering for online estimation of the Jacobian matrix that is required by the proportional control scheme. The robot end-effector motion in Cartesian space is generated via the IHUVS method, without knowing the camera's intrinsic parameters and the robot hand-eye relationship. A simulation analysis is carried out to assess the algorithm's numerical performance, while robotic visual servoing experiments are also conducted to verify the accuracy and efficacy of the proposed IHUVS method.

Visual Servo Control for Workspace Navigation of Nanorobot End-Effector Inside SEM

Despite of the promising advancements in the last decade, the majority of scanning electron microscope (SEM)-based nanomanipulation tasks remain manually performed. Even in automated tasks, human intervention is required, at least during the task preparatory stage, where both the object of interest and robot end-effector are adequately enclosed in the field of view (FOV) of SEM at moderate magnification. This paper proposes a fully automated visual servo control method for workspace navigation of nanorobot end-effector that actively maintains the end-effector position offset from the center of FOV during passive working scene zooming and translation operations. We also propose using the scaling image Jacobian matrix theory to adaptively establish the hand-eye relationship of the nanorobotic system at uncalibrated magnifications, without the need for hardware regulation. This proposed method for workspace navigation is applicable to almost all commercial nanomanipulation systems. To the authors’ best knowledge, there has been no dedicated research on this problem in the literature. Experiments show that the proposed method significantly improves workspace navigation efficiency by about two-thirds, even for a skilled operator. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —Workspace navigation refers to the process of moving the FOV to visually enclose the object or feature of interest at a moderate magnification, which is prerequisite for initializing either an automated or manual nanomanipulation task. This process involves consecutive zooming of the SEM in and out, as well as translating the sample stage, which can cause the end-effector to move out of the FOV. Therefore, it is necessary to move the robot end-effector with visual assistance along workspace navigation to avoid unexpected collisions. Typically, skilled operators perform this process manually in almost all nanomanipulation tasks, which is tedious and time-consuming. Therefore, developing an automated visual servo control method for workspace navigation of end-effector will significantly contribute to the nanomanipulation community.

Simultaneous calibration of hand-eye and kinematics for industrial robot using line-structured light sensor

For visually guided industrial robots, it is necessary to calibrate the robot's hand-eye relationship to accurately guide the robot's operations with the measurement data of the visual coordinate system. In most cases, the calibration process of hand-eye error does not consider the impact of kinematic errors, which accumulates the error in the calibration process and reduces the calibration accuracy. To enhance the calibration accuracy of industrial robots, a simultaneous calibration method of robot hand-eye relationship and kinematics using a line-structured light sensor is proposed. A simultaneous calibration model is established by using the differential kinematics principle. Afterward, the line-structured light sensor and two standard balls with known center distances are used in the calibration and accuracy verification experiments. After calibration and compensation with the proposed method, the average distance error of the robot declined from 3.967 mm to 1.001 mm. The proposed simultaneous calibration method for hand-eye relationship and kinematics dramatically improves the robot's positioning accuracy.

Reconstruction-Based Hand–Eye Calibration Using Arbitrary Objects

This paper introduces a flexible hand-eye calibration technique for 3D sensor from reconstruction perspective, with no need for specialized and accurate calibration rig. Our intention is to find the hand-eye relation which simultaneously aligns multi-view point clouds of a common scene into the robot base frame, namely simultaneous calibration and reconstruction. To achieve this goal, a novel variant of iterative closest point (ICP) algorithm based on Gauss-Newton method and Lie algebra is proposed, which iteratively transforms multi-view point clouds into robot base frame, estimates point-to-point correspondences between point clouds then refines the hand-eye relation to minimize the Euclidean distance between corresponding points. In addition to the calibration result, it returns a preliminary reconstruction as byproduct. Cases of degeneracy and applicable condition are given and proved. Using arbitrary daily objects with no prior information and real robotic eye-in-hand system, we verify our method feasible and effective.

IABC: An iteratively automatic machine learning boosted hand-eye calibration method for laser displacement sensor

An on-machine measuring (OMM) system with a laser displacement sensor (LDS) is designed for measuring free-form surfaces of hypersonic aircraft’s radomes. To improve the measurement accuracy of the OMM system, a novel Iteratively Automatic machine learning Boosted hand-eye Calibration (IABC) method is proposed. Both the hand-eye relationship and LDS measurement errors can be calibrated in one calibration process without any hardware changes via IABC. Firstly, a new objective function is derived, containing analytical parameters of the hand-eye relationship and LDS errors. Then, a hybrid calibration model composed of two kernels is proposed to solve the objective function. One kernel is the analytical kernel designed for solving analytical parameters. Another kernel is the automatic machine learning (AutoML) kernel designed to model LDS errors. The two kernels are connected with stepwise iterations to find the best calibration results. Compared with traditional methods, hand-eye experiments show that IABC reduces the calibration RMSE by about 50%. Verification experiments show that IABC reduces the measurement deviations by about 25%-50% and RMSEs within 40%. Even when the training data are obviously less than the test data, IABC performs well. Experiments demonstrate that IABC is more accurate than traditional hand-eye methods.

Adaptive finite-time model estimation and control for manipulator visual servoing using sliding mode control and neural networks

ABSTRACT The image-based visual servoing without models of the system is challenging since it is hard to fetch an accurate estimation of hand-eye relationship via merely visual measurement. Whereas, the accuracy of the estimated hand-eye relationship expressed in local linear format with the Jacobian matrix is important to the whole system's performance. In this article, we proposed a finite-time controller as well as a Jacobian matrix estimator in a combination of online and offline ways. The local linear formulation is formulated first. Then, we use a combination of online and offline methods to boost the estimation of the highly coupled and nonlinear hand-eye relationship with data collected via depth camera. A neural network (NN) is pre-trained to give a relative reasonable initial estimation of the Jacobian matrix. Then, an online updating method is carried out to modify the offline trained NN for a more accurate estimation. Moreover, a sliding mode control algorithm is introduced to realize a finite-time controller. Compared with previous methods, our algorithm possesses better convergence speed. The proposed estimator possesses excellent performance in the accuracy of initial estimation and powerful tracking capabilities for time-varying estimation for the Jacobian matrix compared with other data-driven estimators. The proposed scheme acquires the combination of neural network and finite-time control effect which drives a faster convergence speed compared with the exponentially converge ones. Another main feature of our algorithm is that the state signals in the system are proven to be semi-global practical finite-time stable. Several experiments are carried out to validate the proposed algorithm's performance.

Zooming-Free Hand-Eye Self-Calibration for Nanorobotic Manipulation Inside SEM

Nanorobotic manipulation inside Scanning Electron Microscope (SEM) is widely used in nanomaterial characterization and assembly tasks. However, tedious and time-consuming hardware regulations, like axis alignment, are still critical and mandatory to keep the functionality of nanorobot systems. This paper proposes a novel zooming-free hand-eye self-calibration method for nanorobotic manipulation inside SEM to circumvent the hardware regulation problem, which is applicable to a wide range of system configurations. The hand-eye relationship calibration method is based on the active incremental motion of the nanorobot without referring to expensive and complicated calibration patterns. Both perspective projection at low magnification and affine projection at high magnification are modelled, and unified by the image Jacobian matrix technique. With several times active calibration at different magnification factors in advance, the hand-eye relationship can be, thereafter, determined from the model at arbitrary status. The calibration model can be applied to both accurate SEM visual measurement and precision visual servo control. The corresponding methods regarding two-dimensional (2D) measurement, in-plane positioning, and constrained three-dimension (3D) positioning based on the proposed hand-eye relationship model are also presented in this paper. Experiments show that practical contact between robot end-effector and silicon nanowire under 1500× is well established with standard deviation less than one pixel.

Towards Accurate Solution of Robot Hand-eye Relationship Based on the LMI-SDP Optimization

Towards Accurate Solution of Robot Hand-eye Relationship Based on the LMI-SDP Optimization

Flexible Scanning Method by Integrating Laser Line Sensors with Articulated Arm Coordinate Measuring Machines

Measuring and reconstructing the shape of workpieces have been considered as a fundamental step in both reverse engineering and product quality control. Owing to increasing structural complexity of recent products, measurements from multiple directions are typically required in current scanning techniques. Specifically, the plane structured light can be applied to measure one area of a part at a time, with an additional algorithm required to merge the collected data of each area. Alternatively, the line structured light sensor integrated on CNC machines or CMMs could also realize multi-view measurement. However, the system needs to be repeatedly calibrated at each new direction. This paper presents a flexible scanning method by integrating laser line sensors with articulated arm coordinate measuring machines (AACMM). Since the output of the laser line sensor is 2D raw data in the laser plane, our system model introduces an explicit transformation from the 2D sensor coordinate frame to the 3D base coordinate frame of the AACMM (i.e., the translation and rotation the of the 2D sensor coordinate in the sixth coordinate system of AACMM). To solve the model, the “conjugate pairs” are proposed and identified by measuring a fixed point (e.g., a sphere center). Moreover, a search algorithm is adopted to find the optimal solution, which noticeably boosts the model accuracy. The experimental results show that the error of the system is about 0.2 mm, which is caused by the error of the AACMM, the sensor error and the calibration error. By measuring a complicated part, the proposed system is proved to be flexible and facilitate, with the ability to measure a part expediently from any necessary direction. Furthermore, the proposed calibration method can also be used for robot hand-eye relationship calibration.

A Point Cloud-Based Feature Recognition and Path Planning Method

Based on the research background of in situ automatic ultrasonic phased array inspection of irregular porous castings, due to the limited in situ inspection stations, complex shape of irregular porous castings, and extreme multireflection structural features, it is necessary to identify the positioning inspection features among multiple features to be inspected and plan the optimal inspection path. This research is interested in the porous location recognition and its detection path planning of irregular porous castings. For this, a point cloud-based multifeature contour recognition and location algorithm were proposed to simultaneously extract and locate the hole feature and cylindrical feature from an irregular porous castings. Furthermore, a detection path planning method was put forward to search the shortest robot’s detection path based on the above acquired features by visual recognition and positioning technology. First, through the calibration of the industrial robot tool coordinate system and the internal and external parameters of the camera, the “EyeinHand” hand-eye conversion relationship was established. Second, the robot vision system collects the point cloud information of the area to be inspected and performs point cloud splicing, the accuracy of the original data, the removal of noise such as invalid points, outliers and internal noise points, on this basis, the boundary curve of the hole to be inspected was extracted, the cylindrical equation was fitted, its geometric center was calculated, and the central coordinates and axis direction of the contour of the hole to be inspected were obtained. Finally, all the detection paths were traversed through the multibranch tree to obtain the optimal detection path of the detection points of multiple targets. The experimental results show that the positioning accuracy of the feature of the hole to be inspected by the vision system is 0.107 mm, the aperture extraction accuracy is 0.002 mm, the cylinder fitting accuracy is 0.04 mm, and the calculation accuracy of the angle between the two axes is within 0.4. When the number of features to be inspected is different, the average moving distance can be saved by 10.7% by using the end effector after path optimization. The feasibility of in situ automatic ultrasonic phased array detection for irregular porous castings using by visual positioning is verified.

Robot line structured light vision measurement system: light strip center extraction and system calibration

We investigate the light strip center extraction and system calibration method of a robot line structured light vision measurement system to realize fast on-line measurement of complex objects. First, the model of a robot vision measurement system is established to analyze the key technologies involved for improving the measurement accuracy of the system. Second, on the basis of the traditional gray gravity method, a gray gravity method based on the normal direction is proposed to extract the center of the laser stripe. Considering the influence of camera distortion, the accuracy of extracting the center stripe can be further improved. Third, a two-dimensional checkerboard is applied to synchronously calibrate the camera parameters, laser plane parameters, and hand-eye relationship of the robot, thus improving the efficiency and accuracy of system calibration. Finally, three-dimensional topography and the geometric dimensions of a complex part are measured by the constructed system with high accuracy. The experimental results show that the average measurement error is <0.2 mm and the measurement accuracy is <1.3 % , indicating that the system can be effectively used in actual industrial measurement.

Learning to Calibrate - Estimating the Hand-eye Transformation Without Calibration Objects

Hand-eye calibration is a method to determine the transformation linking between the robot and camera coordinate systems. Conventional calibration algorithms use a calibration grid to determine camera poses, corresponding to the robot poses, both of which are used in the main calibration procedure. Although such methods yield good calibration accuracy and are suitable for offline applications, they are not applicable in a dynamic environment such as robotic-assisted minimally invasive surgery (RMIS) because changes in the setup can be disruptive and time-consuming to the workflow as it requires yet another calibration procedure. In this paper, we propose a neural network-based hand-eye calibration method that does not require camera poses from a calibration grid but only uses the motion from surgical instruments in a camera frame and their corresponding robot poses as input to recover the hand-eye matrix. The advantages of using neural network are that the method is not limited by a single rigid transformation alignment and can learn dynamic changes correlated with kinematics and tool motion/interactions. Its loss function is derived from the original hand-eye transformation, the re-projection error and also the pose error in comparison to the remote centre of motion. The proposed method is validated with data from da Vinci Si and the results indicate that the designed network architecture can extract the relevant information and estimate the hand-eye matrix. Unlike the conventional hand-eye approaches, it does not require camera pose estimations which significantly simplifies the hand-eye problem in RMIS context as updating the hand-eye relationship can be done with a trained network and sequence of images. This introduces a potential of creating a hand-eye calibration approach that is capable of accurately updating the hand-eye matrix according to the changes in RMIS setup.

Simultaneous identification of joint stiffness, kinematic and hand-eye parameters of measurement system integrated with serial robot and 3D camera

Purpose This paper aims to propose a measurement principle and a calibration method of measurement system integrated with serial robot and 3D camera to identify its parameters conveniently and achieve high measurement accuracy. Design/methodology/approach A stiffness and kinematic measurement principle of the integrated system is proposed, which considers the influence of robot weight and load weight on measurement accuracy. Then an error model is derived based on the principle that the coordinate of sphere center is invariant, which can simultaneously identify the parameters of joint stiffness, kinematic and hand-eye relationship. Further, considering the errors of the parameters to be calibrated and the measurement error of 3D camera, a method to generate calibration observation data is proposed to validate both calibration accuracy and parameter identification accuracy of calibration method. Findings Comparative simulations and experiments of conventional kinematic calibration method and the stiffness and kinematic calibration method proposed in this paper are conducted. The results of the simulations show that the proposed method is more accurate, and the identified values of angle parameters in modified Denavit and Hartenberg model are closer to their real values. Compared with the conventional calibration method in experiments, the proposed method decreases the maximum and mean errors by 19.9% and 13.4%, respectively. Originality/value A new measurement principle and a novel calibration method are proposed. The proposed method can simultaneously identify joint stiffness, kinematic and hand-eye parameters and obtain not only higher measurement accuracy but also higher parameter identification accuracy, which is suitable for on-site calibration.

A Visual Servo Control Method for Tomato Cluster-Picking Manipulators Based on a T-S Fuzzy Neural Network

HighlightsA T-S fuzzy neural network was applied to the visual servo control system of a tomato picking manipulator.The T-S fuzzy neural network structure was designed, and collected data were used to train the neural network model.A visual servo control system for the picking manipulator based on the neural network was designed and tested.The T-S fuzzy neural network was superior to a BP neural network in visual servo control of the picking manipulator.Abstract. To reduce the computational load of image Jacobian matrix estimation and to avoid the appearance of singularity of a Jacobian matrix in the visual servo control of a picking manipulator, a T-S fuzzy neural network algorithm is proposed to replace the image Jacobian matrix. This better fits the hand-eye relationship by combining the knowledge structure of fuzzy reasoning with the self-learning ability of a neural network. The T-S fuzzy neural network was trained and tested by collecting the variation data of image features and joint angles; after training, the T-S fuzzy neural network was used to predict the joint angles of the picking manipulator. Simulation results show that the square sum of training errors and testing errors were 0.017 and 0.032, respectively, after training the T-S fuzzy neural network. A T-S fuzzy neural network controller was applied to the visual servo system of the picking robot, and the test results show that the average difference between the end-effector and the ultimate target location of the visual servo system based on the T-S fuzzy neural network controller was 0.0037 m, which was 79.44% less than that of the visual servo system based on a BP neural network. The final average error of image features was between 0.52 and 3.25 pixels, which was 74.932% less than that of the visual servo system based on the BP neural network. Keywords: Picking manipulator, Tomato clusters, T-S fuzzy neural network, Visual servoing.

Accurate positioning of a drilling and riveting cell for aircraft assembly

Modern aircraft assembly demands assembly cells or machines with higher machining efficiency and accuracy. Thus, a dual-machine drilling and riveting cell is developed in this paper. We firstly discuss its physical design, as well as the automatic drilling and riveting process. With the automatic drilling and riveting cell, drilling and riveting production line of aircraft panels can be expected. The frame chain of the drilling and riveting cell is constructed to link the assembly cell to its task space, which is the kinematics base. System calibrations, including task space calibration, the sensor calibration of an orientation alignment unit, the floating calibration of the implicit hand-eye relationship, are explored. For high positioning accuracy, a multi-sensor servoing method is proposed for cell positioning. An orientation-based laser servoing strategy, which uses the feedback of the orientation errors measured by laser displacement sensors, is used to align drilling direction and camera shooting direction. Besides, A single-camera-based visual servoing is applied to align the tool center point (TCP) to reference holes, to obtain their coordinates for drilling position modification. Experiments of multi-sensor servoing for cell positioning are performed on an automatic drilling and riveting machine developed for the panel assembly of an aircraft in China. With the cell positioning method, the automatic drilling and riveting cell can approximately achieve an accuracy of 0.05 mm, which can adequately fulfill the requirement for the assembly of the aircraft.

Hybrid calibration and detection approach for mobile robotic manufacturing systems

PurposeThe purpose of this paper is to present a visual detection approach to predict the poses of target objects placed in arbitrary positions before completing the corresponding tasks in mobile robotic manufacturing systems.Design/methodology/approachA hybrid visual detection approach that combines monocular vision and laser ranging is proposed based on an eye-in-hand vision system. The laser displacement sensor is adopted to achieve normal alignment for an arbitrary plane and obtain depth information. The monocular camera measures the two-dimensional image information. In addition, a robot hand-eye relationship calibration method is presented in this paper.FindingsFirst, a hybrid visual detection approach for mobile robotic manufacturing systems is proposed. This detection approach is based on an eye-in-hand vision system consisting of one monocular camera and three laser displacement sensors and it can achieve normal alignment for an arbitrary plane and spatial positioning of the workpiece. Second, based on this vision system, a robot hand-eye relationship calibration method is presented and it was successfully applied to a mobile robotic manufacturing system designed by the authors’ team. As a result, the relationship between the workpiece coordinate system and the end-effector coordinate system could be established accurately.Practical implicationsThis approach can quickly and accurately establish the relationship between the coordinate system of the workpiece and that of the end-effector. The normal alignment accuracy of the hand-eye vision system was less than 0.5° and the spatial positioning accuracy could reach 0.5 mm.Originality/valueThis approach can achieve normal alignment for arbitrary planes and spatial positioning of the workpiece and it can quickly establish the pose relationship between the workpiece and end-effector coordinate systems. Moreover, the proposed approach can significantly improve the work efficiency, flexibility and intelligence of mobile robotic manufacturing systems.

Calibration Method of Roof Weld Coating Robot System Based on Plane-to-Plane Intersection Model

In order to solve the problems of traditional calibration methods, such as assigning internal and external parameters for camera and hand-eye relation matrices, a new method to calibrate weld coating robots based on plane-to-plane intersection model is developed in this paper. The mathematical model of the system is established by the homogeneous transformation theory, and then used to calibrate the line structured light sensor (LSLS). By adjusting the robot, the coordinate systems of the LSLS and the workpiece are superposed to solve the hand-eye relation matrix of the car roof weld coating robot. Finally, the calibrated experimental process and results towards car roof weld coating robot system are analyzed. The results show that this method is simple and high accuracy in roof weld coating robot system, and can be applied to the field calibration of car roof weld coating robot system. Furthermore, it can provide guiding significance for calibration of other robot systems.

Error Correctable Hand-Eye Calibration for Stripe-Laser Vision-Guided Robotics

The stripe-laser vision guidance technology has received more and more attention in industry and academia. The hand–eye calibration technology plays an important role in the applications of the stripe-laser vision-guided robotics. Most existing hand–eye calibration methods impose orthogonality constraints. However, the orthogonality in hand–eye relationship may not be true when the stripe-laser sensor has errors. This makes the final guiding error inevitable. In this article, an error correctable (EC) hand–eye calibration method based on the linear properties of the central projection for the stripe-laser vision sensor guidance system is proposed. In this method, the measurements of a vision sensor with errors can be corrected to the right tool center point coordinates. Experiments involving synthetic and real data confirm that the proposed hand–eye calibration method is robust to the vision sensor errors.

Calibration of 6-DOF industrial robots based on line structured light

To improve the accuracy of an industrial 6-DOF robot, this paper proposes a new calibration method based on a line structured light measurement system. The line structured light measurement system mounted on the end-effector is used as a tool to acquire the three-dimensional coordinate information of the centre of a standard sphere. Hand-eye calibration is used to establish the transformation between the robot and the measurement system. A combined kinematic model that includes both the kinematic parameters of the robot and the initial hand-eye relationship is established by applying the modified Denavit and Hartenberg (MDH) model. An error model is derived based on the principle that the coordinate of the sphere centre is invariant in the base frame of the robot. Consequently, the errors in the kinematic parameters of the robot and the hand-eye relationship can be identified by substituting the measurement results and the original parameters of the robot into the error model. The results of the simulations and experiments show that the accuracy of the robot is significantly improved by implementing the proposed calibration method. The proposed method is effective, simple, time-saving and low-cost and is therefore a suitable candidate for on-site calibration of industrial robots.

An Uncalibrated Visual Servo Method Based on Projective Homography

An uncalibrated visual servo method based on projective homography, denoted as Projective Homography based Uncalibrated Visual Servoing (PHUVS), is proposed in this paper, in which a novel task function based on the element of projective homography is devised to realize visual servo without a prior knowledge of the camera intrinsic parameters and hand-eye relationships. The main advantage of this method is that it is not only suitable for totally uncalibrated scenarios but also cheap in computation costs when compared with classical image-based uncalibrated visual servoing methods. Numerical experiments are performed and the results confirm that the new approach is capable of both static positioning and dynamic tracking tasks, and presents competitive computational efficiency and accuracy performance. Note to Practitioners —Vision guided robot has been widely used in industries. However the industrial state-of-art robot visual servo systems usually require system calibration, including camera calibration, hand-eye calibration, and robot calibration. These calibration process are time consuming and requires some expertise, which is rare for ordinary workers. Moreover, when hand-eye configuration changed due to certain reasons such as adjustment of production line layout or displacement due to machining force or switch of robot end effector, the calibrated parameters are no longer accurate. This paper offers a plug and play approach for eye-in-hand robot visual servo tasks. When a robot-camera system is set up, there is no need for the operator to calibrate the system. With reference (goal) image and the current images obtained by the moving camera, the robot can autonomously achieve the goal pose which is described simply by the reference camera. With our novel task function, the computation cost is reduced, which will benefit the real-time application. The accuracy and some dynamic performance in both static positioning and dynamic target tracking are evaluated by robotic toolbox. This approach has the potential to improve the flexibility and robustness of robot system in robotic manipulation tasks such as assemble and painting, or robotic machining tasks such as matching the origin point of operating trajectory in large part manufacturing. Furthermore, this approach is generic, and can be extended to mobile robot visual servo control tasks.