Uncalibrated Visual Servoing Research Articles

Uncalibrated Visual Servoing for Underwater Vehicle Manipulator Systems with an Eye in Hand Configuration Camera.

This paper presents an uncalibrated visual servoing scheme for underwater vehicle manipulator systems (UVMSs) with an eye-in-hand camera under uncertainties. These uncertainties contain vision sensor parameters, UVMS kinematics and feature position information. At first, a linear separation approach is addressed to collect these uncertainties into vectors, and this approach can also be utilized in other free-floating based manipulator systems. Secondly, a novel nonlinear adaptive controller is proposed to achieve image error convergence by estimating these vectors, the gradient projection method is utilized to optimize the restoring moments. Thirdly, a high order disturbance observer is addressed to deal with time-varying disturbances, and the convergence of the image errors is proved under the Lyapunov theory. Finally, in order to illustrate the effectiveness of the proposed method, numerical simulations based on a 9 degrees of freedom (DOFs) UVMS with an eye-in-hand camera are conducted. In simulations, the UVMS is expected to track a circle trajectory on the image plane, meanwhile, time-varying disturbances are exerted on the system. The proposed scheme can achieve accurate and smooth tracking results during simulations.

Unscented Particle Filter for Online Total Image Jacobian Matrix Estimation in Robot Visual Servoing

The main purpose of visual servoing is to control the motion of a robot system based on visual information provided by one or more cameras. It is an important research topic in the robotics community. In uncalibrated visual servoing, the image Jacobian matrix estimation is of great importance to the success of visual servoing control. This paper addresses the online estimation of the total Jacobian matrix for robot visual servoing using the unscented particle filter. We first give the definition of the total Jacobian matrix and formulate the total Jacobian matrix estimation problem into Bayesian filtering framework. Then, we propose to estimate the total Jacobian matrix using the unscented particle filter. Each particle is propagated and updated using the unscented Kalman filter equations. Such an update can make full use of the image feature measurements and consequently generate more accurate estimation results. The simulation results on a 2DOF robot visual servoing platform show that the proposed method provides accurate and reliable performance in the object tracking task.

Visual Servoing of Nonholonomic Mobile Robots: A Review and a Novel Perspective

Mobile robots combined with visual feedback have been an indispensable role of the society, including in industry and in service, which have attracted extensive researchers over the past years. In this paper, we provide an overview of the visual servoing for nonholonomic mobile robots and introduce two feasible system modeling methods in perspective of active disturbance rejection control (ADRC). Firstly, the visual servoing systems of mobile robots are classified into independent type and coupling type based on the robot and camera configuration. Then three mainstream strategies are discussed according to the different requirements for camera calibration, and the efficiency of which are presented from existing works. In order to cope with the uncalibrated visual servoing of nonholonomic mobile robots, we give two feasible attempts to convert the robot system model into the canonical integrator chain of the ADRC by using the input-state scaling technique and flat output. Finally, future work is outlined accompanied with the challenges.

Intelligence evolution for service robot: An ADRC perspective

The strategy of active disturbance rejection control (ADRC) and its applications in intelligence evolution for service robot are summarized. It is also shown that the philosophy of ADRC is consistent with the essential characteristics of intelligence evolution. Most importantly, we concentrate on five core issues which will be encountered when applying ADRC to deal with intelligence evolution for service robot, that is, how to eliminate the impact of unknown composite disturbances, how to handle the nonholonomic constraints in uncalibrated visual servoing, how to realize eye-hand-torque coordination, how to deal with the disturbance in simultaneous localization and mapping (SLAM), and how to reject the imperfections induced by network in human-robot interaction. The main purpose of this paper is to clarify the challenges encountered on intelligence evolution for service robot when one applies ADRC to, hoping that more and more researchers can give some suggestions or work together to deal with these problems, and flourishing results of ADRC from both theory and applications.

Delay-Dependent Stability in Uncalibrated Image-Based Dynamic Visual Servoing Robotic System

This paper addresses the stability problem of uncalibrated image-based visual servoing robotic systems. Both the visual feedback delay and the uncalibrated visual parameters can be the sources of instability for visual servoing robotic systems. To eliminate the negative effects caused by kinematic uncertainties and delays, we propose an adaptive controller including the delay-affected Jacobian matrix and design an adaptive law accordingly. Besides, the delay-dependent stability conditions are provided to show the relationship between the system stability and the delayed time in order to obtain less conservative results. A Lyapunov-Krasovskii functional is constructed, and a rigorously mathematic proof is given. Finally, the simulation results are presented to show the effectiveness of the proposed control scheme.

Visual servoing tracking control of uncalibrated manipulators with a moving feature point

ABSTRACTThe paper is concerned with the problem of uncalibrated visual servoing robots tracking a dynamic feature point along with the desired trajectory. A nonlinear observer and a nonlinear controller are proposed, which allow the considered uncalibrated visual servoing robotic system to fulfil the desired tracking task. Based on this novel control method, a dynamic feature point with unknown motion parameters can be tracked effectively along with the desired trajectory, even with multiple uncertainties existing in the camera, the kinematics and the manipulator dynamics. By the Lyapunov theory, asymptotic convergence of the image errors to zero with the proposed control scheme is rigorously proven. Simulations have been conducted to verify the performance of the proposed control scheme. The results demonstrated good convergence of the image errors.

Parking objects by pushing using uncalibrated visual servoing

Pushing is one of the strategies to perform robotic manipulation when the object is too large or too heavy. Motivated by this, we address the problem of how to push an object on a plane to a target pose with two cooperating robots. The main contribution is a new uncalibrated image-based control scheme that computes the required motion of the object to reach the target pose. Then, as an application of this control scheme, we study the conditions that allow performing the task of pushing the object with two robots. The setup consists of a fixed external uncalibrated camera looking at the workspace where the object and the robots stand. The task is defined with a target image of the object in the desired pose. The proposed control scheme computes the motion commands of the pusher robots and, as a result, they translate and rotate the object by imposing non-holonomic velocity constraints. This yields smooth, continuous and efficient trajectories. The stability of the control scheme is also proven. Experiments illustrate the performance of the control scheme.

An Uncalibrated Visual Servo Method Based on Projective Homography

An uncalibrated visual servo method based on projective homography, denoted as Projective Homography based Uncalibrated Visual Servoing (PHUVS), is proposed in this paper, in which a novel task function based on the element of projective homography is devised to realize visual servo without a prior knowledge of the camera intrinsic parameters and hand-eye relationships. The main advantage of this method is that it is not only suitable for totally uncalibrated scenarios but also cheap in computation costs when compared with classical image-based uncalibrated visual servoing methods. Numerical experiments are performed and the results confirm that the new approach is capable of both static positioning and dynamic tracking tasks, and presents competitive computational efficiency and accuracy performance. Note to Practitioners —Vision guided robot has been widely used in industries. However the industrial state-of-art robot visual servo systems usually require system calibration, including camera calibration, hand-eye calibration, and robot calibration. These calibration process are time consuming and requires some expertise, which is rare for ordinary workers. Moreover, when hand-eye configuration changed due to certain reasons such as adjustment of production line layout or displacement due to machining force or switch of robot end effector, the calibrated parameters are no longer accurate. This paper offers a plug and play approach for eye-in-hand robot visual servo tasks. When a robot-camera system is set up, there is no need for the operator to calibrate the system. With reference (goal) image and the current images obtained by the moving camera, the robot can autonomously achieve the goal pose which is described simply by the reference camera. With our novel task function, the computation cost is reduced, which will benefit the real-time application. The accuracy and some dynamic performance in both static positioning and dynamic target tracking are evaluated by robotic toolbox. This approach has the potential to improve the flexibility and robustness of robot system in robotic manipulation tasks such as assemble and painting, or robotic machining tasks such as matching the origin point of operating trajectory in large part manufacturing. Furthermore, this approach is generic, and can be extended to mobile robot visual servo control tasks.

Development on a magnetic anchoring robot system based on visual servo control for laparoendoscopic single-site surgery.

Surgical robot systems have been used in laparoendoscopic single-site surgery (LESS) to improve patient outcomes. A magnetic anchoring surgical robot system for LESS can effectively extend the operation space. A robot system based on visual servo control for LESS is proposed. It includes a magnetic anchoring robot and a control subsystem, in which an uncalibrated visual servo control method obtains an accurate positioning capability of the robot for LESS. The results of the simulation and the tissue experiment show that the robot system can successfully accomplish the expected control functionalities for LESS. The average positioning error of the proposed system is 1.622mm. The magnetic anchoring robot system is able to implement the autonomous positioning of its end-effector through the proposed control approach according to experimental results.

Virtual exoskeleton-driven uncalibrated visual servoing control for mobile robotic manipulators based on human–robot–robot cooperation

This paper presents an uncalibrated visual servoing control system based on the human–robot–robot cooperation (HRRC). In case of malfunctions of the joint sensors of a robotic manipulator, the proposed system enables the mobile robot to continue operating the manipulator to complete the task that requires careful handling. With the aid of a virtual exoskeleton, an operator may use a human–computer interaction (HCI) device to guide the malfunctioning manipulator. During the guiding process, the virtual exoskeleton serves as a connector between the HCI device and the manipulator. However, when using the HCI device to guide the virtual exoskeleton, there could be a risk of a large-residual problem at any time caused by non-uniform guiding. To solve this problem, a residual switching algorithm (RSA) has been proposed that can identify whether the residual should be calculated based on the motion characteristics of the artificial guiding, reducing the computational cost and ensuring the tracking stability. To enhance the virtual exoskeleton’s ability to drive the manipulator, a multi-joint fuzzy driving controller has been proposed, which can drive the corresponding joint of the manipulator in accordance with an offset vector between the virtual exoskeleton and the manipulator. Lastly, the guiding experiments have verified that, compared with the contrast algorithm, the proposed RSA has a better tracking performance. A peg-in-hole assembly experiment has shown that the proposed control system can assist the operator to control efficiently the robotic manipulator with malfunctioning joint sensors.

Visual Servo Regulation of Wheeled Mobile Robots With an Uncalibrated Onboard Camera

In this paper, a visual servo regulation strategy is designed for an uncalibrated camera system mounted on a wheeled mobile robot subject to nonholonomic motion constraint, which can drive the mobile robot to the target pose with exponential convergence. Unlike existing methods, the proposed approach can work well without the camera intrinsic parameters being known in advance. Specifically, a novel fundamental matrix-based algorithm is first proposed to rotate the robot to point toward the desired position, with the camera intrinsic parameters estimated simultaneously by employing the fundamental matrix and a projection homography matrix. Subsequently, by utilizing the obtained camera intrinsic parameters, a straight-line motion controller is developed to drive the robot to the desired position, with the orientation of the robot always facing the target position. Another pure rotation controller is finally adopted to correct the orientation error. The exponentially convergent properties of the visual servo errors are proven with mathematical analysis. The performance of the proposed uncalibrated visual servo regulation method is further validated by both simulation and experimental results.

Orthogonal Image Features for Visual Servoing of a 6-DOF Manipulator With Uncalibrated Stereo Cameras

We present an approach to control a 6-degree-of-freedom (DOF) manipulator using an uncalibrated visual servoing (VS) approach that addresses the challenges of choosing proper image features for target objects and designing a VS controller to enhance the tracking performance. The main contribution of this paper is the definition of a new virtual visual space (image space). A novel stereo camera model employing virtual orthogonal cameras is used to map 6-D poses from Cartesian space to this virtual visual space . Each component of the 6-D pose vector defined in this virtual visual space is linearly independent, leading to a full-rank $6\times 6$ image Jacobian matrix , which allows avoiding classical problems, such as image space singularities and local minima. Furthermore, the control for rotational and translational motion of robot is decoupled due to the diagonal image Jacobian. Finally, simulation results with an eye-to-hand robotic system confirm the improvement in controller stability and motion performance with respect to conventional VS approaches. Experimental results on a 6-DOF industrial robot are provided to illustrate the effectiveness of the proposed method and the feasibility of using this method in practical scenarios.

Robust uncalibrated visual servoing control based on disturbance observer

In this paper, an uncalibrated visual servoing scheme with optimal disturbance rejection performance is proposed based on disturbance observer (DOB). Comparing with traditional uncalibrated methods, which estimate online the hand-eye relationship characterized by varying image Jacobian, the uncertainty of image Jacobian is eliminated via DOB to approach a given nominal model in this paper. By solving a constrained optimization problem transformed into an H∞ control framework, the disturbance rejection performance is optimized while ensuring the robust stability of the closed-loop visual servoing system. The controller is based on the nominal image Jacobian matrix, thus avoiding singularities and local minima. Simulations and experiments show that the proposed scheme performs better in tracking an object than traditional algorithms. The disturbance and image noise rejection performance is verified.

Kalman Filter-Based Robot Manipulator Five-Degrees of Freedom Uncalibrated Vision Positioning

Robot manipulator position and posture control is a popular topic in the field of uncalibrated visual servoing, this paper presents a kalman filter-based robot manipulator five-degrees of freedom uncalibrated vision positioning method. In the case of the fixed binocular cameras and manipulator parameters are unknown; firstly, the specific point and angle image features information in the camera image space were selected in order to describe the relative pose relationship between robot manipulator ends and goals. Then, the kalman filter online estimation algorithm was applied to calculate image Jacobian matrix which is mapping relationship between image space to cartesian mission space, and vision controller was designed in the image plane realized robot manipulator five-degrees of freedom uncalibrated vision positioning control. Finally, Six-degrees of freedom robot manipulator’s five-degrees of freedom uncalibrated visual positioning Simulink model established in the Matlab environment, and the simulation result show that kalman filter online estimation method made the robot manipulator rapid convergence to the desired position and posture with high accuracy.

A Simulation Research on the Visual Servo Based on Pseudo-Inverse of Image Jacobian Matrix for Robot

The image Jacobian Matrix must obtain during the course of uncalibrated visual servo for classic algorithms firstly. Then the inverse of image Jacobian Matrix or pseudo-inverse of image Jacobian Matrix can be taken. But when the inverse of image Jacobian Matrix is not exist or pseudo-inverse of image Jacobian Matrix is not easy to get, the uncalibrated visual servo for robot can not realize. In this paper, a research is carried on by simulation between the classic method for uncalibrared visual servo and the strategy by computing pseudo-inverse of image Jacobian Matrix. It is conclusion that the latter not only has advantage of the performance for tracking, but also reduces computational complexity for control.

Uncalibrated Visual Servoing through the Efficient Estimation of the Image Jacobian for Large Residual

An uncalibrated visual servo control method for tracking a target is presented. We define the robot-positioning problem as an unconstrained optimization problem to minimize the image error between the target feature and the robot end-effector feature. We propose a method to find the residual term for more precise modeling using the secant approximation method. The composite image Jacobian is estimated by the proper method for eye-to-hand configuration without knowledge of the kinematic structure, imaging geometry and intrinsic parameter of camera. This method is independent of the motion of a target feature. The algorithm for regulation of the joint velocity for safety and stability is presented using the cost function. Adaptive regulation for visibility constraints is proposed using the adaptive parameter.

Image Jacobian Matrix Estimation Based on Online Support Vector Regression

Research into robotics visual servoing is an important area in the field of robotics. It has proven difficult to achieve successful results for machine vision and robotics in unstructured environments without using any a priori camera or kinematic models. In uncalibrated visual servoing, image Jacobian matrix estimation methods can be divided into two groups: the online method and the offline method. The offline method is not appropriate for most natural environments. The online method is robust but rough. Moreover, if the images feature configuration changes, it needs to restart the approximating procedure. A novel approach based on an online support vector regression (OL-SVR) algorithm is proposed which overcomes the drawbacks and combines the virtues just mentioned.

Robust feature tracking on the beating heart for a robotic-guided endoscope

Visualization during minimally invasive bypass surgery on the beating heart can be enhanced by using a robotic-guided endoscope and visual servoing from the endoscopic images. In order to achieve these objectives, this work has focused on developing and testing algorithms for accurate, robust and real-time motion tracking of features on the beating heart, using marker-less approaches and an uncalibrated endoscope. Lucas-Kanade pyramidal optical flow-based algorithms and speeded-up robust features (SURF)-based methods have been extensively evaluated, using a range of developed metrics, in order to quantify accuracy, robustness and drift under a variety of circumstances. Three sets of experiments are reported: the first set compared the two tracking methods, using a beating-heart phantom and a static endoscope; the second set evaluated the methods when images were taken using a moving robotic-guided endoscope; and finally, the Lucas-Kanade optical flow algorithm was extensively tested in a visual servoing application, using a robotic endoscope. The combination of a Lucas-Kanade tracking algorithm and a SURF-based feature detection method gave the best performance in terms of accuracy and robustness of tracking, while preserving real-time computation requirements. The optimal parameters consist of a window size of 51 × 51 pixels and an interframe motion threshold of 20 pixels. Feature tracking was successfully integrated into uncalibrated visual servoing or a robotic-guided endoscope. Robust feature tracking on a beating heart with endoscopic video can be achieved in real-time and may facilitate robotically-assisted, minimally invasive bypass surgery and conventional laparoscopic surgery.

SVM-based identification and un-calibrated visual servoing for micro-manipulation

This paper presents an improved support vector machine (SVM) algorithm, which employs invariant moments-based edge extraction to obtain feature attribute. A heuristic attribute reduction algorithm based on rough set’s discernible matrix is proposed to identify and classify micro-targets. To avoid the complicated calibration for intrinsic parameters of camera, an improved Broyden’s method is proposed to estimate the image Jacobian matrix which employs Chebyshev polynomial to construct a cost function to approximate the optimization value. Finally, a visual controller is designed for a robotic micromanipulation system. The experiment results of micro-parts assembly show that the proposed methods and algorithms are effective and feasible.

Homography-Based Visual Servo Control With Imperfect Camera Calibration

In this technical note, a robust adaptive uncalibrated visual servo controller is proposed to asymptotically regulate a robot end-effector to a desired pose. A homography-based visual servo control approach is used to address the six degrees-of-freedom regulation problem. A high-gain robust controller is developed to asymptotically stabilize the rotation error, and an adaptive controller is developed to stabilize the translation error while compensating for the unknown depth information and intrinsic camera calibration parameters. A Lyapunov-based analysis is used to examine the stability of the developed controller.