Homography-based Visual Servo Research Articles

Homography-Based Visual Servoing of Robot Pose Under an Uncalibrated Eye-to-Hand Camera

Homography-Based Visual Servoing of Robot Pose Under an Uncalibrated Eye-to-Hand Camera

A geometric approach for homography-based visual servo control of underactuated UAVs

This paper proposes a new geometric control method for homography-based visual servo control of Underactuated UAVs. In order to solve the application difficulties of geometric control in HBVS and explore a visual servo control technology that can be applied to aerial detection operations, this paper integrates the geometric control into the visual servoing framework and design a new homography-based geometric visual servoing controller. The outer loop is used as feedback information using the virtual homography matrix between the two images. The inner loop controls the orientation of the UAVs through geometric control. The stability of the proposed controller is proved based on Lyapunov’s theory. The proposed method has better transient performance and dynamic performance than the conventional visual servo method. The excellent performance of the controller has been proven by a large number of experiments. In addition, the application of the controller on an unmanned aerial manipulator is demonstrated.

Homography-Based Visual Servoing of Eye-in-Hand Robots With Exact Depth Estimation

Visual servoing can effectively control robots using visual feedback to improve their intelligence and reliability. For a feature point detected by a monocular camera, the time-varying depth appearing nonlinearly in the Jacobian matrix is difficult to be measured without the prior geometry knowledge of the observed object. Hence, the depth of the feature point is one of the major uncertain parameters in visual servoing. Considering unknown Cartesian feature positions, this study presents a dynamics-based homography-based visual servoing (HBVS) controller for the three-dimensional (3-D) pose regulation of eye-in-hand robot arms with monocular cameras. The uncertain depth is represented as a linear form of its Cartesian feature position, and a composite learning law is applied to estimate position parameters accurately, resulting in exact depth estimation. Compared to existing adaptive HBVS methods, the distinctive feature of the proposed method is that it is a dynamics-based design and guarantees exact depth estimation under a much weaker condition termed interval excitation than persistent excitation. Simulations and experiments on a collaborative robot with 7 degrees of freedom named Franka Emika Panda have verified the effectiveness of the proposed method.

A Homography-Based Visual Servo Control Approach for an Underactuated Unmanned Aerial Vehicle in GPS-Denied Environments

This paper proposes a homography-based visual servo control approach for the stabilization of underactuated unmanned aerial vehicles in GPS-denied environments. A virtual homography matrix is proposed, and a new visual error vector is defined to obtain the passivity-like visual error equation. In addition, a homography-based online translation velocity estimator and a nonlinear backstepping visual servo control approach are addressed to promote fast maneuvering without linear velocity feedback. Unlike conventional homography-based visual servo control solutions that only guarantee the local stability in the equilibrium, the proposed control scheme can get the global stability in large offset. Simulation and experimental tests with a realistic quadrotor subject to the measurement noise are carried out to verify the robustness and performance of the control approach. In addition, the autonomous tracking experiment when the target speed is unpredictable are also reported.

Robust homography-based visual servo control for a quadrotor UAV tracking a moving target

In this study, a new robust homography-based visual tracking control approach for the quadrotor unmanned aerial vehicle (UAV) is developed. Specifically, employing the homography matrix as feedback, a hierarchical homography-based visual servoing (HBVS) scheme with a new command attitude extraction method to account for the underactuation of UAV is proposed. On this basis, a smooth hyperbolic tangent function is fulfilled as an augmented part of the backstepping control scheme, which guarantees the non-negative total thrust and avoid singularity. Additionally, a cascaded filter-based estimator and adaptive laws with integrable functions are embedded to counteract uncertainties including external perturbations, unknown acceleration of the moving target, and unknown image depth, and to facilitate the system’s asymptotic stability simultaneously. The theoretical analysis testifies that the whole close-loop system is asymptotically stable. Simulations further verify that the proposed HBVS controller can realize the visual tracking with a superior performance.

Homography-Based Visual Servoing for Underactuated VTOL UAVs Tracking a 6-DOF Moving Ship

This paper develops a novel homography-based visual servo control method for the vertical take-off and landing (VTOL) unmanned aerial vehicle (UAV) tracking the trajectory of a 6 degrees of freedom (6-DOF) moving ship. Different from the classical homography-based visual servoing which is only suitable for the trajectory tracking of a planar moving target, an extended homography-based visual servoing framework is proposed to track the trajectory of a 6-DOF moving target regardless of its time-varying rotational motion. Taking entries in the homography matrix as feedback, the visual dynamics of UAV is established and decoupled into a translational motion sub-dynamics and an attitude sub-dynamics based on the hierarchical control strategy. In the translational motion subsystem, the robust controller is designed by embedding the nonlinear differentiator, the adaptive method, and a smooth saturation model in the backstepping control scheme. The saturation model is introduced to generate the constrained control inputs to ensure the nonsingular attitude extraction and help the visual points remain in the camera’s field of view (FoV) simultaneously. In the attitude subsystem, a robust state-constrained control method is proposed by utilizing a robust control barrier function (RCBF), the quadratic programming (QP) and a saturation model to guarantee the visibility, where RCBF can accommodate unknown dynamics. The theoretical analysis demonstrates the asymptotic stability of the closed-loop system. Simulations are carried out to further validate the controller performance.

Homography-based Visual Servoing with Remote Center of Motion for Semi-autonomous Robotic Endoscope Manipulation.

The dominant visual servoing approaches in Minimally Invasive Surgery (MIS) follow single points or adapt the endoscope's field of view based on the surgical tools' distance. These methods rely on point positions with respect to the camera frame to infer a control policy. Deviating from the dominant methods, we formulate a robotic controller that allows for image-based visual servoing that requires neither explicit tool and camera positions nor any explicit image depth information. The proposed method relies on homography-based image registration, which changes the automation paradigm from point-centric towards surgical-scene-centric approach. It simultaneously respects a programmable Remote Center of Motion (RCM). Our approach allows a surgeon to build a graph of desired views, from which, once built, views can be manually selected and automatically servoed to irrespective of robot-patient frame transformation changes. We evaluate our method on an abdominal phantom and provide an open source ROS Moveit integration for use with any serial manipulator. A video is provided.

Adaptive homography-based visual servo for micro unmanned surface vehicles

In this paper, a novel adaptive homography-based visual servo (AHBVS) scheme is proposed to regulate a micro unmanned surface vehicle (MUSV) to the desired pose in the presence of both unknown image depth and unmatched dynamics. By virtue of homography decomposition technique, the scaled pose errors are directly retrieved from the live and desired images which are captured by a monocular camera. On the basis of the MUSV characteristics, a completely new visual servo system is firstly derived from visual measurements including both kinematics and dynamics. Different from kinematics solutions, the dynamics-level AHBVS controllers adapting to unknown image depth and compensating unmatched dynamics are developed by incorporating backstepping technique and Lyapunov synthesis, and thereby facilitating practical implementations. Lyapunov analysis proves that the proposed AHBVS scheme renders the closed-loop visual servo system globally uniformly asymptotically stable (GUAS). Simulation results demonstrate remarkable performance on a prototype MUSV.

An Inertial-Aided Homography-Based Visual Servo Control Approach for (Almost) Fully Actuated Autonomous Underwater Vehicles

A nonlinear inertial-aided image-based visual servo control approach for the stabilization of (almost) fully actuated autonomous underwater vehicles (AUVs) is proposed. It makes use of the homography matrix between two images of a planar scene as feedback information while the system dynamics are exploited in a cascade manner in a control design: An outer-loop control defines a reference setpoint based on the homography matrix and an inner-loop control ensures the stabilization of the setpoint by assigning the thrust and torque controls. Unlike conventional solutions that only consider the system kinematics, the proposed control scheme is novel in considering the full system dynamics (incorporating all degrees of freedom, nonlinearities, and couplings, as well as interactions with the surrounding fluid) and in not requiring information of the relative depth and normal vector of the observed scene. Augmented with integral corrections, the proposed controller is robust with respect to model uncertainties and disturbances. The almost global asymptotic stability of the closed-loop system is demonstrated, which is the largest domain of attraction one can achieve by means of continuous feedback control. Simulation results illustrating these properties on a realistic AUV model subjected to a sea current are presented and finally experimental results on a real AUV are reported.

Visual servo walking control for humanoids with finite-time convergence and smooth robot velocities

ABSTRACTIn this paper, we address the problem of humanoid locomotion guided from information of a monocular camera. The goal of the robot is to reach a desired location defined in terms of a target image, i.e., a positioning task. The proposed approach allows us to introduce a desired time to complete the positioning task, which is advantageous in contrast to the classical exponential convergence. In particular, finite-time convergence is achieved while generating smooth robot velocities and considering the omnidirectional waking capability of the robot. In addition, we propose a hierarchical task-based control scheme, which can simultaneously handle the visual positioning and the obstacle avoidance tasks without affecting the desired time of convergence. The controller is able to activate or inactivate the obstacle avoidance task without generating discontinuous velocity references while the humanoid is walking. Stability of the closed loop for the two task-based control is demonstrated theoretically even during the transitions between the tasks. The proposed approach is generic in the sense that different visual control schemes are supported. We evaluate a homography-based visual servoing for position-based and image-based modalities, as well as for eye-in-hand and eye-to-hand configurations. The experimental evaluation is performed with the humanoid robot NAO.

Homography-based Visual Servoing for Autonomous Underwater Vehicles

A nonlinear visual servoing approach is proposed for the stabilisation of fully-actuated autonomous underwater vehicles (AUVs) by exploiting the homography matrix between the two images of a planar scene. In a cascade manner, an outer-loop control defines a reference setpoint based on the homography matrix, and an inner-loop control ensures the stabilisation of the setpoint by assigning thrust and torque controls. In contrast with conventional kinematic solution, the proposed controller deals with the high nonlinearity and coupling of the system dynamics and ensures almost-global asymptotical stability. In addition, the interactions of the AUV with the surrounding fluid (e.g., added mass and drag effects) are often difficult to model precisely whereas they may significantly perturb its motion. The proposed controller –augmented with an effective integral action– allows for the compensation of model uncertainties and for robust performance against such perturbations. Simulation results illustrating these properties on a realistic AUV model subject to sea current are reported.

GPU Acceleration in a Visual Servo System

In this paper we present our novel work of using the Graphic Processing Unit (GPU) to improve the performance of a homography-based visual servo system. We propose a GPU accelerated Efficient Second-order Minimization (GPU-ESM) algorithm to ensure a fast and stable homography solution, approximately 20 times faster than its CPU implementation. To enhance the system stability, we adopt a GPU accelerated Scale Invariant Feature Transform (SIFT) algorithm to deal with those cases where GPU-ESM algorithm performs poor, such as large image differences, occlusion and so on. The combination of both GPU accelerated algorithms is described in detail. The effectiveness of our GPU accelerated system is evaluated with experimental data. The key optimization techniques in our GPU applications are presented as a reference for other researchers.

Adaptive Homography-Based Visual Servo Tracking Control via a Quaternion Formulation

In this paper, an adaptive homography-based visual servo tracking controller is developed for the camera-in-hand problem using a quaternion formulation to represent rotation tracking error. The desired trajectory in the tracking problem is encoded by a sequence of images (e.g., a video sequence), and Lyapunov methods are employed to facilitate the control design and the stability analysis. An adaptive estimation law is designed to compensate for the lack of unknown depth information. Experimental results are provided to demonstrate the performance of the visual servo tracking controller.

Homography-Based Visual Servo Control With Imperfect Camera Calibration

In this technical note, a robust adaptive uncalibrated visual servo controller is proposed to asymptotically regulate a robot end-effector to a desired pose. A homography-based visual servo control approach is used to address the six degrees-of-freedom regulation problem. A high-gain robust controller is developed to asymptotically stabilize the rotation error, and an adaptive controller is developed to stabilize the translation error while compensating for the unknown depth information and intrinsic camera calibration parameters. A Lyapunov-based analysis is used to examine the stability of the developed controller.

Navigation function-based visual servo control

In this paper, the mapping between the desired camera feature vector and the desired camera pose (i.e., the position and orientation) is investigated to develop a measurable image Jacobian-like matrix. An image-space path planner is then proposed to generate a desired image trajectory based on this measurable image Jacobian-like matrix and an image-space navigation function (NF) (i.e., a special potential field function) while satisfying rigid body constraints. An adaptive, homography-based visual servo tracking controller is then developed to navigate the position and orientation of a camera held by the end-effector of a robot manipulator to a goal position and orientation along the desired image-space trajectory while ensuring the target points remain visible (i.e., the target points avoid self-occlusion and remain in the field-of-view (FOV)) under certain technical restrictions. Due to the inherent nonlinear nature of the problem and the lack of depth information from a monocular system, a Lyapunov-based analysis is used to analyze the path planner and the adaptive controller. Simulation results are provided to illustrate the performance of the proposed approach.

Homography-based visual servo tracking control of a wheeled mobile robot

A visual servo tracking controller is developed in this paper for a monocular camera system mounted on an underactuated wheeled mobile robot (WMR) subject to nonholonomic motion constraints (i.e., the camera-in-hand problem). A prerecorded image sequence (e.g., a video) of three target points is used to define a desired trajectory for the WMR. By comparing the target points from a stationary reference image with the corresponding target points in the live image and the prerecorded sequence of images, projective geometric relationships are exploited to construct Euclidean homographies. The information obtained by decomposing the Euclidean homography is used to develop a kinematic controller. A Lyapunov-based analysis is used to develop an adaptive update law to actively compensate for the lack of depth information required for the translation error system. Experimental results are provided to demonstrate the control design.

Adaptive homography-based visual servo tracking for a fixed camera configuration with a camera-in-hand extension

In this brief, a homography-based adaptive visual servo controller is developed to enable a robot end-effector to track a desired Euclidean trajectory as determined by a sequence of images for both the camera-in-hand and fixed-camera configurations. To achieve the objectives, a Lyapunov-based adaptive control strategy is employed to actively compensate for the lack of unknown depth measurements and the lack of an object model. The error systems are constructed as a hybrid of pixel information and reconstructed Euclidean variables obtained by comparing the images and decomposing a homographic relationship. Simulation results are provided to demonstrate the performance of the developed controller for the fixed camera configuration.