Adaptive Visual Servo Controller Research Articles

Towards Semi-Autonomous Colon Screening Using an Electromagnetically Actuated Soft-Tethered Colonoscope Based on Visual Servo Control.

Conventional colonoscopy using a flexible colonoscope remains two major limitations, including patient discomfort and difficult manipulations for surgeons. Robotic colonoscopes have been developed to conduct colonoscopy in a patient-friendly manner. However, most robotic colonoscopes still maintain nonintuitive and difficult manipulations, which limits their clinical applications. In this paper, we demonstrated visual servo-based semi-autonomous manipulations of an electromagnetic actuated soft-tethered (EAST) colonoscope, which aims to lower difficulties of robotic colonoscope manipulations. Kinematic modeling of EAST colonoscope is conducted, with an adaptive visual servo controller established. Template matching method and a lumen and polyp detection model are developed to enable semi-autonomous manipulations, including region-of-interest automatic tracking and autonomous navigation with automatic polyp detection. The EAST colonoscope demonstrates visual servoing with an average convergence time of around 2.5 s and performs disturbance rejection within 3.0 s. Semi-autonomous manipulations were conducted in both a commercialized colonoscopy simulator and an ex-vivo porcine colon to show the efficacy of reducing the user workload compared to manual control. The EAST colonoscope can perform visual servoing and semi-autonomous manipulations with the developed methods in both laboratory and ex-vivo environments. The proposed solutions and techniques improve the autonomy level of robotic colonoscopes and reduce user workloads, which promotes the development and clinical translation of robotic colonoscopy.

Uncalibrated Adaptive Visual Servoing of Robotic Manipulators with Uncertainties in Kinematics and Dynamics

In the study, we propose a novel adaptive visual servoing control scheme for robotic manipulators with kinematic and dynamic uncertainties, where the camera used is uncalibrated, which implies that its intrinsic and extrinsic parameters are unavailable for measurement. For our scheme, a depth-independent composite Jacobian matrix is constructed to make visual parameters and robotic physical parameters appear linearly in a parametrized uniform form so that an adaptive algorithm can be developed to estimate their values. With the raised adaptive algorithm, the potential singularity of the Jacobian matrix can be well circumvented by updating estimated parameters in an appropriate tiny range of actual values. With our scheme, the asymptotic convergence of the image tracking error to zero is established successfully, in addition to the signal boundedness of the closed-loop system. The effectiveness of the proposed scheme is confirmed by simulation results based on a 6-DOF PUMA manipulator.

Adaptive visual servo control law for finite-time tracking to land quadrotor on moving platform using virtual reticle algorithm

This paper proposed an image-based visual servoing (IBVS) control law for a quadrotor that is equipped with a single monocular camera attached to its bottom. For control purposes virtual reticle plane (VRP) algorithm is used to track the relative 3D position of the quadrotor to the tilting and moving target landing platform within the range of the camera’s field of view (FOV). In this article, the landing platform’s tilting motion is considered sinusoidal type oscillatory motion for the overhead camera. For control purposes, an adaptive finite-time control (AFTC) is proposed, based on the finite-time control (FTC) and adaptive approximation of uncertainties. A constructive combination of FTC and adaptive approximation inherits benefits of both to overcome each other’s limitations. The task of the controller is to regulate the position error to zero in time calculated by VRP. The experimental results confirmed the effectiveness of the VRP algorithm to track the desired parameters of the moving target. Finally, simulations are performed to illustrate the effectiveness and improved performance of the proposed AFTC in response time, robustness, and tracking accuracy.

Adaptive Visually Servoed Tracking Control for Wheeled Mobile Robot with Uncertain Model Parameters in Complex Environment

This paper investigates the stabilization and trajectory tracking problem of wheeled mobile robot with a ceiling-mounted camera in complex environment. First, an adaptive visual servoing controller is proposed based on the uncalibrated kinematic model due to the complex operation environment. Then, an adaptive controller is derived to provide a solution of uncertain dynamic control for a wheeled mobile robot subject to parametric uncertainties. Furthermore, the proposed controllers can be applied to a more general situation where the parallelism requirement between the image plane and operation plane is no more needed. The overparameterization of regressor matrices is avoided by exploring the structure of the camera-robot system, and thus, the computational complexity of the controller can be simplified. The Lyapunov method is employed to testify the stability of a closed-loop system. Finally, simulation results are presented to demonstrate the performance of the suggested control.

Research on Aerial Robot Based on Visual Servo

Towards target grasping by an aerial manipulator, an aerial manipulator system is presented, for which a separated control strategy is adopted. In this paper, a suitable adaptive visual servo controller is proposed based on the original aerial manipulator system, and a visual servo controller is designed based on the depth independent image Jacobian matrix. Thus, the grasping motion of the manipulator is controlled by the visual servo of the monocular camera, which can greatly enhance the functionality and speed of the aerial manipulator system. Furthermore, a unique single degree of freedom joint collocation camera is designed to enable the camera to lock the target object at any time during the motion of aircraft. Finally, this paper designs a unique state machine system, which can make the control system switch into any state.

Adaptive Visual Servoing Control for Hoisting Positioning Under Disturbance Condition

This paper proposes a visual servo scheme for hoisting positioning under disturbance conditions. In actual hoisting work, disturbances such as equipment and load vibration are inevitable, which brings challenges to the development of a visual servo for hoisting positioning. The main problems are as follows: (1) the correlation between visual error and disturbance is not considered or well resolved; (2) the disturbance has a great influence on the control stability, but it is difficult to model. At present, there is no detailed research on the above problems. In this paper, the visual error is defined by the image error of the feedback signal based on dynamic equations containing disturbances. An adaptive sliding mode control algorithm is employed to decrease the influence of external disturbance, and the coefficient of the slide surface is established based on the adaptive gain. In view of the belief that it is difficult to model disturbance terms, a nonlinear disturbance observer is introduced to obtain equivalent disturbance. On this basis, an adaptive control algorithm with disturbance compensation is proposed to improve the robustness of the visual servo system. We use Lyapunov’s method to analyze the stability conditions of the system. Compared with the other state-of-the-art methods, the simulation results show that our method has superior performance in convergence, accuracy, and restraining disturbance. Finally, the proposed algorithm is applied to the hoisting platform for experimental research, which proves the effectiveness of the controller.

Collaboration and Task Planning of Turtle-Inspired Multiple Amphibious Spherical Robots.

Amphibious Spherical Robots (ASRs) use an electric field to communicate and collaborate effectively in a turbid water of confined spaces where other mode communication modalities failed. This paper proposes an embedded architecture formation strategy for a group of turtle-inspired amphibious robots to maintain a long distance-parameterized path based on dynamic visual servoing. Inspired by this biological phenomenon, we design an artificial multi-robot cooperative mode and explore an electronic communication and collaborate devices, the control method is based in particular on underwater environment and also conduct a detailed analysis of control motion module. The objectives of control strategies are divided into four categories: The first strategy is that the leader robot controls the action of the overall robots to maintain collaborate together during motion along a desired geometric path and to follow a timing law that the communication efficiency and the arrival times to assigned sites. Furthermore, we design an adaptive visual servoing controller for trajectory tracking task, taking into account system dynamics with environment interactions. After that, the third strategy is a centralized optimization algorithm for the redistribution of target mission changes. Finally, this paper also proposes a new method of control strategies in order to guarantee that each robot in the team moves together according to the preset target toward its location in the group formation based on communication and stability modules.

Adaptive Visual Servoing for an Underwater Soft Robot Considering Refraction Effects

Robots inspired from marine organisms are tremendously developed for applications of underwater exploration, rescuing, navigation, etc. In this article, we implement an uncalibrated visual servoing scheme to achieve accurate positioning performance of an octopus-tentacle-like soft robot arm. The image-based adaptive visual servoing controller is designed based on the underwater dynamic model of the robot system. An adaptive mechanism to solve the tedious camera calibration problem is also extremely complicated in underwater environment due to the refraction effect resulting in changes of optical condition. In this article, this effect is analogous to the radial distortion. The presented algorithm can linearize the distortion model, and then online iteratively estimate the unknown image mapping model based on the classical Slotine-Li adaptive algorithm. The intrinsic and extrinsic parameters can also be estimated in real time. The presented adaptive controller is verified both theoretically using the Lyapunov stability analysis to prove the stability of the dynamical system, and experimentally to prove the accuracy and rapid convergence to the target image position.

Underwater Dynamic Visual Servoing for a Soft Robot Arm With Online Distortion Correction

Bioinspired soft robots have generated increasing attentions due to their optimal performance in specific environments stemming from their unique morphology and sensorimotor capabilities. This also raises new challenges in modeling and control due to their unconventional soft mechanism. In this paper, we aim to improve the controllability of an octopus tentacle-like soft robot arm operating in an underwater environment and to reproduce the perception-guided tracking performance of its biological counterpart when preying on and pursuing a target. To this end, we design an adaptive visual servoing controller for trajectory tracking tasks, taking into account system dynamics with environment interactions. Unlike on-the-ground visual servoing controller, the underwater environment exerts multirefraction effects on the image projection process, increasing difficulties in obtaining an accurate camera projection model. This paper thus proposes an adaptive method that can online correct image distortion caused by the deflection of light rays passing through different mediums to compensate for refraction effects. The adaptive algorithm can simultaneously estimate unknown camera parameters to completely avoid complex offline underwater camera calibration. The proposed adaptive controller validates the trajectory tracking performance experimentally and position and image velocity.

Adaptive visual servoing control for an underwater soft robot

PurposeSoft robotics, regarded as a new research branch of robotics, has generated increasing interests in this decade and has demonstrated its outperformance in addressing safety issues when cooperating with human beings. However, there is still lack of accurate close-loop control because of the difficulty in acquiring feedback information and accurately modeling the system, especially in interactive environments. To this end, this paper aims to improve the controllability of the soft robot working in specific underwater environment. The system dynamics, which takes complicated hydrodynamics into account, is solved using Kane’s method. The dynamics-based adaptive visual servoing controller is proposed to realize accurate sensorimotor control.Design/methodology/approachThis paper presents an image-based visual servoing control scheme for a cable-driven soft robot with a fixed camera observing the motions. The intrinsic and extrinsic parameters of the camera can be adapted online so that tedious camera calibration work can be eliminated. It is acknowledged that kinematics-based control can be only applied into tasks in the free space and has limitation in accelerating the motion speed of robot arms. That is, one must consider the unneglectable interaction effects generated from the environment and objectives when operating soft robots in such interactive control tasks. To extend the application of soft robots into underwater environment, the study models system dynamics considering complicated hydrodynamic effects. With the pre-knowledge of the external effects, the performance of the robot can be further improved by adding the compensation term into the controller.FindingsThe proposed controller has theoretically proved its convergence of image error, adaptive estimation error and the stability of the dynamical system based on Lyapunov’s analysis. The authors also validate the performance of the controller in positioning control task in an underwater environment. The controller shows its capacity of rapid convergence to and accurate tracking performance of a static image target in a physical experiment.Originality/valueTo the best of the authors’ knowledge, there is no such research before that has developed dynamics-based visual servoing controller which takes into account the environment interactions. This work can thus improve the control accuracy and enhance the applicability of soft robotics when operating in complicated environments.

Adaptive-Repetitive Visual-Servo Control of Low-Flying Aerial Robots via Uncalibrated High-Flying Cameras

This paper presents the design and implementation of an adaptive-repetitive visual-servo control system for a moving high-flying vehicle (HFV) with an uncalibrated camera to monitor, track, and precisely control the movements of a low-flying vehicle (LFV) or mobile ground robot. Applications of this control strategy include the use of high-flying unmanned aerial vehicles (UAVs) with computer vision for monitoring, controlling, and coordinating the movements of lower altitude agents in areas, for example, where GPS signals may be unreliable or nonexistent. When deployed, a remote operator of the HFV defines the desired trajectory for the LFV in the HFV’s camera frame. Due to the circular motion of the HFV, the resulting motion trajectory of the LFV in the image frame can be periodic in time, thus an adaptive-repetitive control system is exploited for regulation and/or trajectory tracking. The adaptive control law is able to handle uncertainties in the camera’s intrinsic and extrinsic parameters. The design and stability analysis of the closed-loop control system is presented, where Lyapunov stability is shown. Simulation and experimental results are presented to demonstrate the effectiveness of the method for controlling the movement of a low-flying quadcopter, demonstrating the capabilities of the visual-servo control system for localization (i.e.,, motion capturing) and trajectory tracking control. In fact, results show that the LFV can be commanded to hover in place as well as track a user-defined flower-shaped closed trajectory, while the HFV and camera system circulates above with constant angular velocity. On average, the proposed adaptive-repetitive visual-servo control system reduces the average RMS tracking error by over 77% in the image plane and over 71% in the world frame compared to using just the adaptive visual-servo control law.

Visual Servoing of Soft Robot Manipulator in Constrained Environments With an Adaptive Controller

It is unavoidable for a soft manipulator to interact with environments during some tasks. These interactions may affect the soft manipulator and make the kinematic model different from the one in free space, e.g., the soft manipulator's effective length and the target positions might change. In order to apply the soft manipulator to constrained environments, an adaptive visual servo controller based on piecewise-constant curvature kinematic, without knowing the true values of the manipulator's length and the target positions, is proposed in this paper. Experimental results in the free space, constrained environment, and the gravity-influenced environment, demonstrate the convergence of the image errors under the proposed controller.

Adaptive Visual Servo Control of Robotic Harvesting Systems

Harvesting efficiency is a critical factor in determining whether robotic harvesting can be economically superior to hand picking. One practical challenge in robotic harvesting is to grasp a moving fruit, where unsuccessful pick cycles may result in lower harvesting efficiency. The fruit motion can be due to exogenous disturbances such as wind gust, canopy unloading, and particularly, fruit detachment forces. In this paper, an adaptive visual servo control law is presented to estimate and subsequently compensate the unknown time-varying fruit motion. The fruit motion in the image plane and along the optical axis is modeled as a second-order spring-mass system. The unknown parameters of fruit motion are identified using an adaptive parameter update law, and a robust feedback term is included in the control law to account for modeling uncertainties. Lyapunov-based stability analysis guarantees uniformly ultimately bounded regulation of the robot to the fruit position. Simulation results are provided to verify the feasibility of the developed controller.

Vision-Based Tracking Control of Underactuated Water Surface Robots Without Direct Position Measurement

Over the past decade, the trajectory tracking of underactuated water surface robots (or boats, surface vessels, etc.) has been an attractive topic in the control and automation community, and numerous controllers are proposed for this challenging problem. However, most, if not all, of the existing trajectory tracking controllers of the underactuated water surface robots assume the global positions of the robots can be accurately measured. In the practical applications, the global position measurement systems are sometimes unstable or even unavailable in the working environments of the robots. To avoid the direct position measurement, this brief presents a new controller for the trajectory tracking of underactuated water surface robots using monocular visual feedback. This controller works on the basis of a novel adaptive algorithm for estimating global position of the robot online using visual feature tracking from a monocular camera, and its orientation and velocity measured by the attitude and heading reference system sensor and a velocity observer. It is proved by Lyapunov theory that the proposed adaptive visual servo controller gives rise to asymptotic tracking of a desired trajectory and convergence of the position estimation to the actual position. Experiments are conducted to validate the effectiveness and robust performance of the proposed controller.

Visual Servoing Trajectory Tracking of Nonholonomic Mobile Robots Without Direct Position Measurement

Localization is one of the most difficult and costly problems in mobile robotics. To avoid this problem, this paper presents a new controller for the trajectory tracking of nonholonomic mobile robots using visual feedback without direct position measurement. This controller works on the basis of a novel adaptive algorithm for estimating the global position of the mobile robot online using natural visual features measured by a vision system and its orientation and velocity measured by odometry and Attitude and Heading Reference System (IMU&Compass) sensors. The nonholonomic motion constraint of mobile robots is fully taken into account, compared with most of the existing visual servo controllers for mobile robots. The Lyapunov theory is used to prove that the proposed adaptive visual servo controller gives rise to asymptotic tracking of a desired trajectory and convergence of the position estimation to the actual position. A graphical processing unit is adopted to implement the proposed adaptive controller in parallel to achieve real-time detection and tracking of visual features. Experiments on a mobile robot are conducted to validate the effectiveness and robust performance of the proposed controller.

Dynamic Modeling and Image-based Adaptive Visual Servoing of Cable-driven Soft Robotic Manipulator

This paper focus on dynamic visual seroving of a cable-driven soft robotic manipulator system. The soft robotic manipulator has no rigid structure. Based on Lagrange mechanics, kinetic energy, elastic potential energy and gravitational potential energy of each segment are analyzed, thus general dynamic equation of the soft robotic manipulator is obtained. On this basis, a depth-independent image Jacobian matrix is presented and an image-based visual servo controller is designed. Applied by adaptive algorithm, the controller could estimate unknown 3D feature positions online, and Lyapunov method is involved to prove the stability of the system. Experiments are conducted to demonstrate reasonableness and validity of dynamic model of the soft robotic manipulator and image-based adaptive visual servo controller.

Adaptive visual servoing using common image features with unknown geometric parameters

This paper generalizes the concept of the depth-independent interaction matrix, developed for point and line features in our early work, to generalized image features. We derive the conditions under which the depth-independent interaction matrix can be linearly parameterized by the geometric parameters of the generalized image features, and propose an adaptive visual servo controller for robot manipulators using the generalized image features whose geometric parameters are unknown. To estimate the unknown parameters on-line, we propose new error functions that are linear to estimation errors of the parameters and an algorithm that minimizes the error functions using multiple images. The Lyapunov theory is used to prove asymptotic stability of the proposed controller based on the nonlinear dynamics of the manipulator. It is also shown that in addition to points and lines, other common image features like distances, angles, areas, and centroids all satisfy the conditions for the linear parameterization. Experiments have been conducted to validate the proposed control method.

Adaptive visual servo control to simultaneously stabilize image and pose error

A novel, Lyapunov-based visual servo controller is presented that stabilizes both the entire image and pose error vectors simultaneously, rather than a subset of the errors. Furthermore, the controller uses adaptive depth estimation to eliminate the need to measure depth or obtain knowledge of the scene. A stability proof is presented. Simulation and experimental results compare the performance of the proposed method to PBVS, IBVS and 2.5D VS approaches.

Hybrid Adaptive Vision—Force Control for Robot Manipulators Interacting with Unknown Surfaces

A hybrid control scheme based on adaptive visual servoing and direct force control is proposed for robot manipulators to perform interaction tasks on smooth surfaces. The camera parameters, as well as the constraint surface, are considered to be uncertain. A fixed camera with optical axis non-perpendicular to the robot workspace is used for position control, while a force sensor mounted on the robot wrist is used for force regulation. In order to solve the interaction problem on unknown surfaces, a method is developed to estimate the constraint geometry and keep the end-effector orthogonal to the surface at the contact point, during the task execution. Experimental results are presented to illustrate the performance and feasibility of the proposed scheme.

Adaptive homography-based visual servo tracking for a fixed camera configuration with a camera-in-hand extension

In this brief, a homography-based adaptive visual servo controller is developed to enable a robot end-effector to track a desired Euclidean trajectory as determined by a sequence of images for both the camera-in-hand and fixed-camera configurations. To achieve the objectives, a Lyapunov-based adaptive control strategy is employed to actively compensate for the lack of unknown depth measurements and the lack of an object model. The error systems are constructed as a hybrid of pixel information and reconstructed Euclidean variables obtained by comparing the images and decomposing a homographic relationship. Simulation results are provided to demonstrate the performance of the developed controller for the fixed camera configuration.