Image-based Visual Servo Control Research Articles

Adaptive Switch Image-based Visual Servoing for Industrial Robots

In this paper, an adaptive switch image-based visual servoing (IBVS) controller for industrial robots is presented. The proposed control algorithm decouples the rotational and translational camera motions and decomposes the IBVS control into three separate stages with different gains. This method can increase the system response speed and improve the tracking performance of IBVS while the proposed adaptive law deals with the uncertainties of the monocular camera in eye-in-hand configuration. The stability of the designed controller is proved using Lyapunov method. Experimental results on a 6 degree of freedom (DOF) robot show the significant enhancement of the control performance over other IBVS methods, in terms of the response time and tracking performance. Also the designed visual servoing controller demonstrates its capability to overcome some of the inherent drawbacks of IBVS, such its inability to perform a 180° camera rotation around its center.

Quality Inspection of Remote Radio Units Using Depth-Free Image-Based Visual Servo With Acceleration Command

In this paper, the problem of quality inspection of remote radio units (RRUs) is approached using the image-based visual servo control. A novel computer vision pipeline is designed that recognizes the power port of RRU and tracks it from the stream of images. For the control part, a new depth-independent interaction matrix is designed that relates the depth information with the area of the region of interest surrounding the power port. Based on this, an acceleration command is generated to drive the robot's trajectories. Furthermore, a propotional-derivative (PD)-type controller is designed based on the idea of sliding surface in variable structure control. This reduces the number of design parameters to a single parameter. The designed controllers are proven to be stable using the Lyapunov stability analysis. Simulation results and experimental validations are provided to support the research arguments.

Enhanced Switch Image-Based Visual Servoing Dealing with FeaturesLoss

In this paper, an enhanced switch image-based visual servoing controller for a six-degree-of-freedom (DOF) robot with a monocular eye-in-hand camera configuration is presented. The switch control algorithm separates the rotating and translational camera motions and divides the image-based visual servoing (IBVS) control into three distinct stages with different gains. In the proposed method, an image feature reconstruction algorithm based on the Kalman filter is proposed to handle the situation where the image features go outside the camera’s field of view (FOV). The combination of the switch controller and the feature reconstruction algorithm improves the system response speed and tracking performance of IBVS, while ensuring the success of servoing in the case of the feature loss. Extensive simulation and experimental tests are carried out on a 6-DOF robot to verify the effectiveness of the proposed method.

Robust Kalman filtering with long short-term memory for image-based visual servo control

Visual servo control systems based on Kalman filter (KF) is susceptible to noise interference, the initialization of the Jacobi matrix is difficult, and the observation value of the Jacobian matrix is not accurate. In order to address these problems, we proposed a robust KF algorithm with long short-term memory (LSTM) for an image-based visual servo control system and applied the system to an uncalibrated image-based visual servo (IBVS) control system to estimate the filtering gain error, state estimation error, and the observation error, which were then used for online training in LSTM. The visual servo control system controls the motion of the manipulator, and simultaneously updates the LSTM network. Therefore, the Jacobian matrix obtained using LSTM was employed to estimate the state volume of the robust KF, which constitutes a circulatory system, and the complementary effect was realized. The method was applied to a six-degrees-of-freedom manipulator of the eye-in-hand model to conduct experiments. The simulation results indicate that the proposed visual servo control system has strong anti-noise interference capability. Furthermore, it facilitates Jacobian matrix initialization and has high observation accuracy for the Jacobian matrix.

Robust adaptive visual tracking control for uncertain robotic systems with unknown dead-zone inputs

This paper is concerned with the image-based visual servoing (IBVS) control for uncalibrated camera-robot system with unknown dead-zone constraint, where the uncertain kinematics and dynamics are also considered. The control implementation is achieved by constructing a smooth inverse model for dead-zone-input to eliminate the nonlinear effect resulting from the actuator constraint. A novel adaptive algorithm, which does not require a priori knowledge of the parameter intervals of dead-zone model, is proposed to update the parameter values online, and the dead-zone slopes are not required the same. Furthermore, to accommodate the uncertainties of uncalibrated camera-robot system, adaptation laws are developed to estimate the uncertain parameters, simultaneously avoiding singularity of the image Jacobian matrix. With the full consideration of unknown dead-zone constraint and system uncertainties, an adaptive robust visual tracking control scheme together with dead-zone compensation is subsequently established such that the image tracking error converges to the origin. Based on a 3-DOF manipulator, simulations are conducted to verify the tracking performance of the proposed controller.

A Novel Geometric Hierarchical Approach for Dynamic Visual Servoing of Quadrotors

This paper proposes a novel nonlinear geometric hierarchical dynamic visual servoing approach to drive a quadrotor to the desired pose defined by a previously captured image of a planar target. Different from existing works, the key novelty is to extend the positionbased nonlinear hierarchical control to image-based nonlinear hierarchical control. More specifically, by seamlessly integrating the nonlinear hierarchical control with the geometric control, and taking full advantage of the cascade property of the system, the proposed visual servoing strategy does not require the thrust force or its derivative to be measurable when compared with the existing backstepping methods, which brings much convenience for practical applications. For the attitude loop, the axis-angle rotation representation is adopted to design a tracking control law on the vector space s (3). In the outer loop, perspective image moments in the virtual image plane are employed as image feedback to construct the outer-loop image-based visual servoing controller with geometric control and backstepping techniques. Based on Lyapunov techniques and the theory of cascade systems, it is rigorously proven that the proposed image-based controller achieves asymptotic stability. Comparative experiments are conducted to show that the proposed approach has advantages of better transient performance, better steady-state performance, and stronger robustness.

Adaptive Image-Based Visual Servoing for Hovering Control of Quad-Rotor

Image-based visual servoing (IBVS) achieves precise positioning and motion control for a relatively stationary target by visual feedback, but problems persist with convergence and stability. Appropriate servoing gains for the IBVS are critical to the convergence and stability, but this control gain is heuristically a constant for most IBVS applications. This paper proposes an integrated method that allows adaptive adjustment of the servoing gain by reinforcement learning (RL) for IBVS control. The proposed method learns a policy to determine the value of the servoing gain on the fly. To ensure rapid convergence for the RL, truncating ${Q}$ -learning (TQL) with faster convergence is used as learning algorithm, which uses truncated temporal differences (TDs) to update the TD. A nonuniform state space partitioning as a state encoder for RL allows more efficient policy. A strategy that uses the Metropolis derived from the simulated annealing is introduced for selecting the action, in order to balance exploration and exploitation so as to accelerate the learning speed. The integrated IBVS control system is tested using experiments involving a quad-rotor helicopter hovering control. The results of simulation and experiment show that the integrated IBVS method increases stability and ensures more rapid convergence than other methods.

Adaptive Image-Based Visual Servoing With Temporary Loss of the Visual Signal

Image-based visual servoing (IBVS) can reach a desired position for a relatively stationary target using continuous visual feedback. Proper feature extraction and appropriate servoing control laws are essential to performance for IBVS. IBVS control can be interrupted or interfered abruptly if no features are extracted when the observed object is occluded. To address the problem of missing feature points in current images during a visual navigation task, a homography method that uses a priori visual information is proposed to predict all of the missing feature points and to ensure the execution of IBVS. The mixture parameter for the image Jacobian matrix can also affect the control of IBVS. The settings for the mixture parameter are heuristic so there is no a systematic approach for most IBVS applications. An adaptive control approach is proposed to determine the mixture parameter. The proposed method uses a reinforcement learning (RL) method to adaptively adjust the mixture parameter during the robot movement, which allows more efficient control than a constant parameter. A logarithmic interval state-space partition for RL is used to ensure efficient learning. The integrated visual servoing control system is validated by several experiments that involve wheeled mobile robots reaching a target with a desired configuration. The results for simulation and experiment demonstrate that the proposed method has a faster convergence rate than other methods.

An approach integrating planning and image-based visual servo control for road following and moving obstacles avoidance

ABSTRACT This paper proposes an approach that integrates planning and image based visual servo control for road following and moving obstacle avoidance. One main objective of this article is to represent a robot's general plan or strategy in the form of a finite state machine or automaton. This automaton is designed previously to execution of the plan and then it is used for any instance of the robot's task. The visual servo control is used to regulate the robot's velocity according to the visual target (task specification) which depends on the state in the automaton. All the algorithms and control laws have been implemented and simulation results and experiments with a real scale-size car-like robot are presented and discussed.

An Optimized Image-Based Visual Servo Control for Fixed-Wing Unmanned Aerial Vehicle Target Tracking With Fixed Camera

This paper presents an optimized image-based visual servo (IBVS) control scheme for tracking a ground target by using a fixed-wing unmanned aerial vehicle (UAV) with a monocular camera fixed on it. Unlike the widely used rotor UAVs, the fixed-wing UAV has much more dynamical constraints such as it cannot move omni-direction and its minimum speed is limited by the stalling speed. This makes the target tracking problem more challenging. The proposed scheme leverages the image Jacobian matrix to build a connection between the velocity of the feature point and that of the UAV. Afterward, considering the camera is fixed on the body of the UAV, an “ideal camera” model is proposed to compensate the shifts of the feature point caused by the changes of the UAV's attitude. Then, an optimized control law without solving the pseudo-inverse of the image Jacobian matrix is proposed with the aid of the least square method from the target center in the image coordinate system. This control law takes the velocity of the feature point as inputs and the yaw angular velocity of the UAV as outputs. The stability of the proposed law is analyzed with the Lyapunov method, showing that the UAV will circle around the ground target asymptotically. Finally, the proposed scheme is evaluated by a hardware-in-the-loop (HIL) simulation based on the Gazebo simulator and the off-the-shelf autopilot hardware.

Model Predictive Visual Servoing of Fully-Actuated Underwater Vehicles With a Sliding Mode Disturbance Observer

This paper presents a sliding-mode observer-based model predictive control (SMO-MPC) strategy for image-based visual servoing (IBVS) of fully-actuated underwater vehicles subject to field of view and actuator constraints and model uncertainties. In the proposed SMO-MPC controller, the visual system model and the approximate underwater vehicle model are used to predict the future trajectories from the current states driven by input candidates over a certain horizon. With the consideration of system uncertainties, including external disturbances and unknown dynamic parameters, a sliding-mode observer is designed to estimate the modeling mismatch, which is feedforward to the dynamic model in MPC. The actual control signals are generated at each step by minimizing a cost function of predicted trajectories under system constraints. The effectiveness of the proposed SMO-MPC IBVS controller is verified by comparative simulations using a fully-actuated underwater vehicle with different control configurations.

Desired compensation RISE-based IBVS control of quadrotors for tracking a moving target

To achieve accurate moving target tracking in the presence of multiple disturbances, this paper presents a desired compensation robust integral of the sign of the error (DCRISE) feedback control strategy for quadrotor unmanned aerial vehicles under image-based visual servo (IBVS) framework. For the purpose of effectively regulating the pose and attitude of quadrotor, proper perspective image moments extracted from the defined virtual image plane are chosen as the image features. Then, a DCRISE feedback controller is constructed for the quadrotor, in which a robust integral of the sign of the error method is employed for suppressing the multiple disturbances and a desired compensation-based feedforward control is introduced to address the nonlinearity nature of the system and circumvent the requirements on the availability of linear velocity. It is theoretically proved by the Lyapunov analysis that all the closed-loop tracking errors under the proposed DCRISE-based IBVS control structure are convergent to a small neighborhood around the origin in spite of multiple disturbances. The validity and advantages of the present approach are confirmed through extensive simulations and comparisons.

Adaptive neural networks-based visual servoing control for manipulator with visibility constraint and dead-zone input

This paper proposed an online image-based visual servoing (IBVS) controller for manipulator systems with dead-zone input. The adaptive neural networks (NNs) are used to approximate the unknown nonlinear dynamics. The Barrier Lyapunov Function (BLF) is constructed to overcome the visibility constraint problem, in which both the constant symmetric barriers and time-varying asymmetric barriers are considered. With the proposed control method, it is proved that all the signals in the closed-loop system are semi-globally uniformly bounded and the image error is remained in a bounded compact set. Finally, simulation examples are given to illustrate the effectiveness of the proposed control method.

Pipeline following by visual servoing for Autonomous Underwater Vehicles

A nonlinear image-based visual servo control approach for pipeline following of fully-actuated Autonomous Underwater Vehicles (AUV) is proposed. It makes use of the binormalized Plücker coordinates of the pipeline borders detected in the image plane as feedback information while the system dynamics are exploited in a cascade manner in the control design. Unlike conventional solutions that consider only the system kinematics, the proposed control scheme accounts for the full system dynamics in order to obtain an enlarged provable stability domain. Control robustness with respect to model uncertainties and external disturbances is reinforced using integral corrections. Robustness and efficiency of the proposed approach are illustrated via both realistic simulations and experimental results on a real AUV.

Adaptive visual servoing control for an underwater soft robot

PurposeSoft robotics, regarded as a new research branch of robotics, has generated increasing interests in this decade and has demonstrated its outperformance in addressing safety issues when cooperating with human beings. However, there is still lack of accurate close-loop control because of the difficulty in acquiring feedback information and accurately modeling the system, especially in interactive environments. To this end, this paper aims to improve the controllability of the soft robot working in specific underwater environment. The system dynamics, which takes complicated hydrodynamics into account, is solved using Kane’s method. The dynamics-based adaptive visual servoing controller is proposed to realize accurate sensorimotor control.Design/methodology/approachThis paper presents an image-based visual servoing control scheme for a cable-driven soft robot with a fixed camera observing the motions. The intrinsic and extrinsic parameters of the camera can be adapted online so that tedious camera calibration work can be eliminated. It is acknowledged that kinematics-based control can be only applied into tasks in the free space and has limitation in accelerating the motion speed of robot arms. That is, one must consider the unneglectable interaction effects generated from the environment and objectives when operating soft robots in such interactive control tasks. To extend the application of soft robots into underwater environment, the study models system dynamics considering complicated hydrodynamic effects. With the pre-knowledge of the external effects, the performance of the robot can be further improved by adding the compensation term into the controller.FindingsThe proposed controller has theoretically proved its convergence of image error, adaptive estimation error and the stability of the dynamical system based on Lyapunov’s analysis. The authors also validate the performance of the controller in positioning control task in an underwater environment. The controller shows its capacity of rapid convergence to and accurate tracking performance of a static image target in a physical experiment.Originality/valueTo the best of the authors’ knowledge, there is no such research before that has developed dynamics-based visual servoing controller which takes into account the environment interactions. This work can thus improve the control accuracy and enhance the applicability of soft robotics when operating in complicated environments.

Ego-Motion Estimation of a Quadrotor Based on Nonlinear Observer

In this paper, an ego-motion estimation problem of a quadrotor unmanned aerial vehicle is studied. A nonlinear observer is proposed to simultaneously estimate the depth of the point features and the translational velocity of the quadrotor using visual feedback from an on-board monocular camera. Image moments defined on a rotated image plane are chosen as image features, and a decoupled image feature dynamics is achieved to facilitate the observer design. Based on the image feature dynamics and the dynamic model of the quadrotor, an observer is proposed to estimate the ego-motion of the quadrotor. Only several point features need to be tracked in consecutive acquired images, which makes the observer computationally efficient and robust. The stability of the observer is proved by means of Lyapunov analysis. Finally, computer simulations and experimental tests, including closed loop image-based visual servoing control and comparison with extended-Kalman-filter based observer, are conducted to verify the validity of the observer.

An Enhanced IBVS Controller of a 6DOF Manipulator Using Hybrid PD-SMC Method

The accuracy and stability are two fundamental concerns of the visual servoing control system. This paper presents an enhanced image based visual servoing (IBVS) method for increasing the accuracy of a 6DOF manipulator. The controller is designed to combine proportional derivative (PD) control with sliding mode control (SMC) on a 6DOF manipulator. The properly tuned PD controller can ensure the fast tracking performance and SMC can deal with the external disturbance and uncertainties due to the depth. The enhanced IBVS controller benefits from simple structure and easy implementation of PD control and good robustness to uncertainties of SMC. The stability of the proposed method is proven by using Lyapunov method. Simulation and experimental results are used to demonstrate the effectiveness of the proposed controller.

A Neural Controller for Image-Based Visual Servoing of Manipulators With Physical Constraints.

Main issues in visual servoing of manipulators mainly include rapid convergence of feature errors to zero and the safety of joints regarding joint physical limits. To address the two issues, in this paper, an image-based visual servoing scheme is proposed for manipulators with an eye-in-hand configuration. Compared with existing schemes, the proposed one does not require performing pseudoinversion for the image Jacobian matrix or inversion for the Jacobian matrix associated with the forward kinematics of the manipulators. Theoretical analysis shows that the proposed scheme not only guarantees the asymptotic convergence of feature errors to zero but also the compliance with joint angle and velocity limits of the manipulators. Besides, simulation results based on a PUMA560 manipulator with a camera mounted on the end effector verify the theoretical conclusions and the efficacy of the proposed scheme.

An intelligent visual servo control system for quadrotors

While quadrotors are becoming more popular, control of these unmanned air vehicles should be improved. In this study, a new intelligent image-based visual servo control system is proposed for the flight guidance control of quadrotors. Features are essential for visual servoing and the proposed system utilizes the features of a shape that provide a clear sight of the landing site instead of point features. Furthermore, the system focuses on three problems of visual servo control: finding an appropriate gain value under velocity limits, keeping the shape features in the field of view and tracking a moving target. As a solution to the first problem, a fuzzy logic unit that uses feature error and error derivative norms as inputs are deployed to assign the gain, adaptively. The second problem is solved by defining safe and risky regions in the image plane to take precautions before field of view missing. Another fuzzy logic unit is activated when the shape passes through a risky region to provide counter velocity in x or y direction and to drag the shape through the safe region. As the last stage, Kalman filtering with Potter’s square root update is added to the proposed system to increase the feature tracking performance. This update also promises divergence avoidance. To show the performance of the proposed system, simulation results for fixed and moving targets under feature disturbance are presented for a quadrotor. The results verify that the proposed system is capable of handling visual servoing problems.

Path planning for visual servoing with search algorithm

To solve the visual servoing tasks in complex environment, a path planning method based on improved rapidly exploring random trees algorithm is proposed. First, the improved rapidly exploring random trees planning method is adopted, which keeps the observed feature points in the field of view. The start node and the desired node are initialized as roots of multi-trees which grow harmoniously to plan path of the robot. Then, the planned path is used to project the three-dimensional target feature points into the image space and obtain the feature trajectory for the image-based visual servoing controller. Finally, the feature trajectory is tracked by the image-based visual servoing controller. The proposed visual servoing design method takes field of view constraints, camera retreat problem, and obstacle avoidance into consideration, which can significantly improve the ability of the robotic manipulator, especially in the narrow space. Simulation and experiment on 6-degree-of-freedom robot are conducted. The results present the effectiveness of the proposed algorithm.