Depth-independent Interaction Matrix Research Articles

Resilient adaptive trajectory tracking control for uncalibrated visual servoing systems with unknown actuator failures

A resilient adaptive tracking control scheme for uncalibrated visual servoing system with unknown actuator failures is proposed, where the camera parameters, the robotic physical parameters, and the position information of end-effector cannot be measured accurately. For our scheme, the nonlinear composite Jacobian matrix involving unknown parameters is estimated by an adaptive algorithm based on the depth-independent interaction matrix frame so that the image feature with time-varying depth can be tackled effectively. With the use of projection technique in the adaptive algorithm, the possible singularity of the estimated composite Jacobian matrix can be well circumvented without any stringent assumption, which is critical for establishing the closed-loop stability. Moreover, a novel adaptive resilient control scheme based on dynamic decoupling is developed to separate the design of visual servoing controller and fault compensation, improving the visual servoing system resilience against unknown actuator failures in uncalibrated environment. Through a rigorous Lyapunov argument, the asymptotic convergence of image tracking error to zero is established successfully regardless of the occurrence of unknown actuator failures. The effectiveness of the proposed visual tracking control scheme is confirmed by simulation results.

Adaptive neural network control for image‐based visual servoing of robot manipulators

This paper presents a novel adaptive neural network control strategy for image-based visual servoing (IBVS) of robotic manipulators with both eye-in-hand and eye-to-hand camera configurations in the presence of unknown dynamics and external disturbances. The IBVS method is combined with the adaptive neural network to construct the proposed adaptive neural network controller to solve the visual servoing control problem of robots. The adaptive neural network based IBVS controller is designed based on the depth-independent interaction matrix, which can be trained on-line to identify the visual servoing robotic system modeling errors. Moreover, the proposed method can approach the unknown nonlinear dynamics for both eye-in-hand and eye-to-hand camera configurations without requiring the robot dynamics to be linearly parameterizable, and the exact knowledge of the robot structure is not needed. On the basis of the nonlinear robot dynamics, the Lyapunov stability analysis is given to prove the asymptotical convergence of the image position and velocity errors. Simulation results for both camera configurations are provided to demonstrate the performance of the proposed adaptive neural network based approach.

Disturbance observer based adaptive model predictive control for uncalibrated visual servoing in constrained environments

This paper presents an adaptive model predictive control (MPC) method based on disturbance observer (DOB) to improve the disturbance rejection performance of the image-based visual servoing (IBVS) system. The proposed control method is developed based on the depth-independent interaction matrix, which can simultaneously handle unknown camera intrinsic and extrinsic parameters, unknown depth parameters, system constraints, as well as external disturbances. The proposed control scheme includes two parts which are the feedback regulation part based on the adaptive MPC and the feedforward compensation part based on the modified DOB. Unlike the traditional DOB that is based on the fixed nominal plant model, the modified DOB here is based on the estimated plant model. The adaptive MPC controller consists of an iterative identification algorithm, which not only can provide the model parameters for both the controller and the modified DOB, but also can be used to control plant dynamics and to minimize the effects of DOB. Simulations for both the eye-in-hand and eye-to-hand camera configurations are conducted to illustrate the effectiveness of the proposed method.

Quality Inspection of Remote Radio Units Using Depth-Free Image-Based Visual Servo With Acceleration Command

In this paper, the problem of quality inspection of remote radio units (RRUs) is approached using the image-based visual servo control. A novel computer vision pipeline is designed that recognizes the power port of RRU and tracks it from the stream of images. For the control part, a new depth-independent interaction matrix is designed that relates the depth information with the area of the region of interest surrounding the power port. Based on this, an acceleration command is generated to drive the robot's trajectories. Furthermore, a propotional-derivative (PD)-type controller is designed based on the idea of sliding surface in variable structure control. This reduces the number of design parameters to a single parameter. The designed controllers are proven to be stable using the Lyapunov stability analysis. Simulation results and experimental validations are provided to support the research arguments.

Model Predictive Control for Uncalibrated and Constrained Image-Based Visual Servoing Without Joint Velocity Measurements

This paper presents a novel scheme for image-based visual servoing (IBVS) of a robot manipulator by considering robot dynamics without using joint velocity measurements in the presence of constraints, uncalibrated camera intrinsic and extrinsic parameters and unknown feature position parameters. An approach to design model predictive control (MPC) method based on identification algorithm and sliding mode observer has been proposed. Based on the MPC method, the IBVS tasks can be considered as a nonlinear optimization problem while the constraints due to the visibility constraint and the torque constraint can be explicitly taken into account. By using the depth-independent interaction matrix framework, the identification algorithm can be used to update the unknown parameters and the prediction model. In addition, many existing controllers require the joint velocity measurements which can be contaminated by noises, thus resulting in the IBVS performance degradation. To overcome the problem without joint velocity measurements, the sliding mode observer is designed to estimate the joint velocities of the IBVS system. The simulation results for both eye-in-hand and eye-to-hand camera configurations are presented to verify the effectiveness of the proposed control method.

Adaptive Robust Visual Servoing/Force Control for Robot Manipulator With Dead-Zone Input

In this paper, we investigate the adaptive robust visual servoing/force control problem for the uncertain robot manipulator. The robot is considered to work with an unknown constraint surface and the unknown depth of feature point in image plane is assumed to be time-varying. The unknown constraint surface is linearly parameterized, and new adaptive laws are designed to estimate the unknown parameters online. In image based visual servoing control, the unknown time-varying depth plays a special role as it appears nonlinearly in the overall Jacobian matrix and cannot be adapted together with other uncertain kinematic parameters. To handle this problem, a compensatory depth-independent interaction matrix framework and the corresponding adaptive laws are proposed to compensate for the depth in the closed-loop dynamics. Moreover, the dead-zone input constraint is considered and handled by designing the robust compensatory terms and adaptive laws. With the proposed control scheme, it is proved that the image position and force tracking errors converge to zero asymptotically. Finally, numerical simulation and experimental results illustrate the effectiveness of the proposed approach.

A unified design method for adaptive visual tracking control of robots with eye-in-hand/fixed camera configuration

A unified design method is proposed in this paper to handle adaptive visual tracking control problem of robots. In the proposed scheme, the robot dynamic parameters, camera intrinsic and extrinsic parameters and position parameters of feature points are assumed to be uncertain. A unified kinematics model is presented to simultaneously cope with kinematics modeling problem of robots with the eye-in-hand or fixed camera configuration. Based on the unified kinematics, a unified design method is proposed to solve the visual tracking control problem of robots with the eye-in-hand or fixed camera configuration. By using the depth-independent interaction matrix framework, adaptive laws are derived to handle the unknown parameters. Lyapunov stability analysis is provided to show asymptotical convergence of image position and velocity tracking errors. To show the effectiveness of the proposed unified design method, experimental results for both camera configurations are also given.

Adaptive visual servoing using common image features with unknown geometric parameters

This paper generalizes the concept of the depth-independent interaction matrix, developed for point and line features in our early work, to generalized image features. We derive the conditions under which the depth-independent interaction matrix can be linearly parameterized by the geometric parameters of the generalized image features, and propose an adaptive visual servo controller for robot manipulators using the generalized image features whose geometric parameters are unknown. To estimate the unknown parameters on-line, we propose new error functions that are linear to estimation errors of the parameters and an algorithm that minimizes the error functions using multiple images. The Lyapunov theory is used to prove asymptotic stability of the proposed controller based on the nonlinear dynamics of the manipulator. It is also shown that in addition to points and lines, other common image features like distances, angles, areas, and centroids all satisfy the conditions for the linear parameterization. Experiments have been conducted to validate the proposed control method.

Visual tracking of robots in uncalibrated environments

This paper presents a new adaptive controller for visual tracking control of a robot manipulator in 3D general motion with a fixed camera whose intrinsic and extrinsic parameters are uncalibrated. In addition to camera parameters, the feature positions are also assumed unknown. Based on the fact that the unknown parameters appears linearly in the closed-loop dynamics of the system if the depth-independent interaction matrix is adopted to map the image errors onto the joint space of the manipulator, a new adaptive algorithm was developed to estimated the unknown parameters on-line. With a full consideration of dynamic responses of the robot manipulator, the Lyapunov method is employed to prove asymptotic convergence of the image errors. Simulation and experiment results are used to demonstrate the performance of the proposed approach.

Dynamic visual servoing of a small scale autonomous helicopter in uncalibrated environments

This paper presents a novel adaptive controller for image-based visual servoing of a small autonomous helicopter to cope with uncalibrated camera parameters and unknown 3D geometry of the feature points. The controller is based on the back-stepping technique, but its design has two new features. First, it incorporates the visual feedback into the last step of the backstepping procedure, while existing backsteppingbased methods employ the visual feedback at the early steps. Second, the controller maps the image errors onto the actuator space via a depth-independent interaction matrix to avoid estimation the depths of the feature points. The new design method makes it possible to linearly parameterize the closed-loop dynamics by the unknown camera parameters and coordinates of the feature points in the 3D space so that an adaptive algorithm can be developed to estimate the unknown parameters and coordinates on-line. Two potential functions are introduced in the controller to guarantee convergence of the image errors and to avoid trivial solutions of the estimated parameters. The Lyapunov method is used to prove the asymptotic stability of the proposed controller based on the nonlinear dynamics of the helicopter. Simulations have been also conducted to demonstrate the performance of the proposed method.

Improved Stability Results for Visual Tracking of Robotic Manipulators Based on the Depth-Independent Interaction Matrix

As we know, the dynamic visual-tracking control, which is based on the depth-independent interaction matrix, has been proposed to cope with the general 3-D motion of robot manipulators. To deal with the unknown camera parameters, an adaptive law has been designed. It is noted, however, that the designed adaptive law uses a vector signal defined by the true depths of feature points, which are unavailable when the camera parameters are unknown. Nevertheless, such an adaptive law can still be implemented by replacing the exact depths with the estimated depths, since the estimated depths can be readily calculated by using the estimated values of the camera parameters. In this case, however, we cannot definitely say whether or not the robot system can be guaranteed to achieve asymptotical stability or even stable behavior. To overcome this problem, in this paper, we redefine the mentioned vector signal using the estimated depths and show that the design based on the new vector signal can make the robot system asymptotically stable. Additionally, the existing tracking control design based on the concept of depth-independent interaction matrix is initial-state dependent. Thus, in this paper, we also modify the existing design to obtain an initial-state-independent result. To show the performance of the proposed designs, simulation results based on a two-link planar manipulator are presented. In addition, preliminary experimental results using an industrial manipulator are also given.

Adaptive Visual Servoing Using Point and Line Features With an Uncalibrated Eye-in-Hand Camera

This paper presents a novel approach for image-based visual servoing of a robot manipulator with an eye-in-hand camera when the camera parameters are not calibrated and the 3-D coordinates of the features are not known. Both point and line features are considered. This paper extends the concept of depth-independent interaction (or image Jacobian) matrix, developed in earlier work for visual servoing using point features and fixed cameras, to the problem using eye-in-hand cameras and point and line features. By using the depth-independent interaction matrix, it is possible to linearly parameterize, by the unknown camera parameters and the unknown coordinates of the features, the closed-loop dynamics of the system. A new algorithm is developed to estimate unknown parameters online by combining the Slotine-Li method with the idea of structure from motion in computer vision. By minimizing the errors between the real and estimated projections of the feature on multiple images captured during motion of the robot, this new adaptive algorithm can guarantee the convergence of the estimated parameters to the real values up to a scale. On the basis of the nonlinear robot dynamics, we proved asymptotic convergence of the image errors by the Lyapunov theory. Experiments have been conducted to demonstrate the performance of the proposed controller.

Dynamic Visual Tracking for Manipulators Using an Uncalibrated Fixed Camera

This paper presents a new controller for controlling a number of feature points on a robot manipulator to trace desired trajectories specified on the image plane of a fixed camera. It is assumed that the intrinsic and extrinsic parameters of the camera are not calibrated. A new adaptive algorithm is developed to estimate the unknown parameters online, based on three original ideas. First, we use the pseudoinverse of the depth-independent interaction matrix to map the image errors onto the joint space of the manipulator. By eliminating the depths in the interaction matrix, we can linearly parameterize the closed-loop dynamics of the manipulator. Second, to guarantee the existence of the pseudoinverse, the adaptive algorithm introduces a potential force to drive the estimated parameters away from the values that result in a singular Jacobian matrix. Third, to ensure that the estimated parameters are convergent to their true values up to a scale, we combine the Slotine-Li method with an online algorithm for minimizing the error between the estimated projections and real image coordinates of the feature points. We have proved asymptotic convergence of the image errors to zero by the Lyapunov theory based on the nonlinear robot dynamics. Experiments have been carried out to verify the performance of the proposed controller.

Uncalibrated visual servoing of robots using a depth-independent interaction matrix

This paper presents a new adaptive controller for image-based dynamic control of a robot manipulator using a fixed camera whose intrinsic and extrinsic parameters are not known. To map the visual signals onto the joints of the robot manipulator, this paper proposes a depth-independent interaction matrix, which differs from the traditional interaction matrix in that it does not depend on the depths of the feature points. Using the depth-independent interaction matrix makes the unknown camera parameters appear linearly in the closed-loop dynamics so that a new algorithm is developed to estimate their values on-line. This adaptive algorithm combines the Slotine-Li method with on-line minimization of the errors between the real and estimated projections of the feature points on the image plane. Based on the nonlinear robot dynamics, we prove asymptotic convergence of the image errors to zero by the Lyapunov theory. Experiments have been conducted to verify the performance of the proposed controller. The results demonstrated good convergence of the image errors