Image-based Visual Servo Control Research Articles

Visual Servoing of a Cable-Driven Soft Robot Manipulator With Shape Feature

Soft continuum robot is of superiority in the applications in unstructured environments due to its high flexibility and safe interaction ability. Accurate shape control is regarded as one of the prerequisites to improve its practicability. Considering the situation where the 3D position signals are not available, this letter investigates the vision-based shape control scheme of a soft robot manipulator. The curvature features used to depict the reference image shape of the robot backbone curve are solved through nonlinear optimization with given constraints of two endpoints in pixel coordinates. Thereafter, an adaptive image-based visual servoing controller is designed to track the reference image shape features with an uncalibrated monocular camera. The proposed algorithm was tested on an eight-cable driving soft robot prototype and proved its validity in convergence to the desired image shape.

A fuzzy CMAC learning approach to image based visual servoing system

This study presents a fuzzy robotic joint controller using a cerebellar model articulation controller (CMAC) integrating a Takagi-Sugeno (T-S) framework with an online compensator for an articulated manipulator. The proposed controller is applied to image-based visual servoing (IBVS), including closed-loop feedback control and the kinematic Jacobian calculation. This approach learns a mapping from image feature errors for each joint’s velocity instead of the classical kinematics, thereby reducing the computational complexity and improving the self-regulation ability of the control system. These connecting weights of the cerebellar model learn offline, and an online compensator that uses reinforcement learning is developed to resolve system noise and uncertainties in an unknown environment. Compared with the classical inverse kinematics model, this approach does not need an excessive computational expense so that this proportional controller can be implemented in general scenarios with an eye-in-hand configuration. Experimental results show the proposed method can outperform the classical IBVS controller.

Image-Based Indoor Topological Navigation with Collision Avoidance for Resource-Constrained Mobile Robots

This paper presents a complete topological navigation system for a resource-constrained mobile robot like Pepper, based on image memory and the teach-and-repeat paradigm. Image memory is constructed from a set of reference images that are acquired during a prior mapping phase and arranged topologically. A* search is used to find the optimal path between the current location and the destination. The images from the robot’s RGB camera are used to localize within the topological graph, and an Image-Based Visual Servoing (IBVS) control scheme drives the robot to the next node in the graph. Depth images update a local egocentric occupancy grid, and another IBVS controller navigates local free-space. The output of the two IBVS controllers is fused to form the final control command for the robot. We demonstrate real-time navigation for the Pepper robot in an indoor open-plan office environment without the need for accurate mapping and localization. Our core navigation module can run completely onboard the robot (which has quite limited computing capabilities) at 5 Hz without requiring any external computing resources. We have successfully performed navigation trials over 15 days, visiting more than 50 destinations and traveling more than 1200m with a success rate of over 80%. We discuss remaining challenges and openly share our software.

Quadrotor going through a window and landing: An image-based visual servo control approach

This paper considers the problem of controlling a quadrotor to go through a window and land on a planar target, the landing pad, using an Image-Based Visual Servo (IBVS) controller that relies on sensing information from two on-board cameras and an IMU. The maneuver is divided into two stages: crossing the window and landing on the pad. For the first stage, a control law is proposed that guarantees that the vehicle will not collide with the wall containing the window and will go through the window with non-zero velocity along the direction orthogonal to the window, keeping at all times a safety distance with respect to the window edges. For the landing stage, the proposed control law ensures that the vehicle achieves a smooth touchdown, keeping at all time a positive height above the plane containing the landing pad. For control purposes, the centroid vectors provided by the combination of the spherical image measurements of a collection of landmarks (corners) for both the window and the landing pad are used as position measurement. The translational optical flow relative to the wall, window edges, and landing plane is used as velocity cue. To achieve the proposed objective, no direct measurements nor explicit estimate of position or velocity are required. Simulation and experimental results are provided to illustrate the performance of the presented controller.

Hierarchical image-based visual serving of underwater vehicle manipulator systems based on model predictive control and active disturbance rejection control

The image-based visual servoing (IBVS) problem of a redundant underwater vehicle manipulator system (UVMS) with an eye-in-hand camera is investigated in this paper. With consideration of system constraints, dynamic uncertainties, and the absence of vehicle velocity sensors for a typical UVMS, we propose a hierarchical control architecture, which is composed of an unscented Kalman filtering (UKF)-based vehicle motion estimator, a kinematic model predictive IBVS controller, and two decoupled dynamic velocity controllers for the vehicle and the manipulator, respectively. In details, the underwater vehicle velocities are estimated by UKF with visual measurements based on the UVMS kinematic model. With these estimates, a nonlinear model predictive controller (MPC) is designed to solve the IBVS problem of UVMSs and generate velocity commands for the underwater vehicle and the manipulator, simultaneously. The visual kinematic model of UVMSs is used to predict the future trajectories, and the field-of-view, manipulator joint angle, and UVMS velocity constraints are handled when solving the optimization problem. To reduce the system complexity of UVMSs, the high-dimensional UVMS’s dynamic model is decoupled into two low-dimensional models of the vehicle and the manipulator, and a dynamic inversion-based active disturbance rejection control (DI-ADRC) method is developed for dynamic controller design. The physical interaction effects between the vehicle and manipulator are considered as external disturbances acting on each other, which are estimated by extended state observers (ESO) in ADRC. The simulation experiments with a typical UVMS are performed to demonstrate the effectiveness of the proposed hierarchical IBVS controller in dealing with system redundancy, constraints and uncertainties.

Adaptive visual servo control law for finite-time tracking to land quadrotor on moving platform using virtual reticle algorithm

This paper proposed an image-based visual servoing (IBVS) control law for a quadrotor that is equipped with a single monocular camera attached to its bottom. For control purposes virtual reticle plane (VRP) algorithm is used to track the relative 3D position of the quadrotor to the tilting and moving target landing platform within the range of the camera’s field of view (FOV). In this article, the landing platform’s tilting motion is considered sinusoidal type oscillatory motion for the overhead camera. For control purposes, an adaptive finite-time control (AFTC) is proposed, based on the finite-time control (FTC) and adaptive approximation of uncertainties. A constructive combination of FTC and adaptive approximation inherits benefits of both to overcome each other’s limitations. The task of the controller is to regulate the position error to zero in time calculated by VRP. The experimental results confirmed the effectiveness of the VRP algorithm to track the desired parameters of the moving target. Finally, simulations are performed to illustrate the effectiveness and improved performance of the proposed AFTC in response time, robustness, and tracking accuracy.

Linear Velocity-Free Visual Servoing Control for Unmanned Helicopter Landing on a Ship With Visibility Constraint

In this article, a constrained image-based visual servoing control method for the shipboard landing problem of unmanned helicopters is proposed. First, the pitch and roll motion of ship are predicted by an autoregressive (AR) model to determine an appropriate period for landing. Subsequently, a novel robust sliding mode controller without linear velocity measurements is developed on the basis of the perspective image feature in a virtual image plane. Meanwhile, a modified Chebyshev neural network (CNN) is proposed to estimate the uncertainties, including the linear acceleration of ship and translational perturbation, while an adaptive law is employed to compensate the influence of rotational disturbances. The whole controller only requires the measurements feedback of a vision sensor and an inertial measurement unit (IMU). Ulteriorly, to prevent the visual target on the ship from going beyond the field of view of camera, the constrained controller is developed by a control barrier function and a quadratic programming, where the unknown relative velocity is estimated by a velocity observer. Finally, simulations are implemented to substantiate the capability of the presented shipboard landing control method.

Adaptive IBVS and Force Control for Uncertain Robotic System with Unknown Dead-zone Inputs

This article introduces a novel control strategy for the uncertain eye-to-hand system, which is considered to work with unknown model of constraint surface and uncalibrated camera model. Besides, the uncertain dynamics and kinematics are also included in the system. In order to be closer to the real robot system, we also consider it with dead-zone inputs situation. So the parameter intervals and slopes of the dead-zone model is also unknown. Hence, a novel adaptive image-based visual servoing (IBVS) and force control approach is put forward. The control method of unknown force and uncalibrated camera model is achieved by adaptive control. The solution of unknown dead-zone inputs is completed by designing a inverse smooth model of dead-zone inputs to offset the nonlinear affect due to the actuator constraint, and the whole system is proved that the force tracking control and image position converge to zero asymptotically. Finally, the MATLAB simulation is set up and the experiment shows the validity of the proposed scheme.

Image-based Visual Servoing Control of a Quadcopter Air Vehicle

ABSTRACT This article proposes two optimal controllers that stabilize the altitude, attitude, heading, and position of the quadcopter in space. The first optimal control method consists of computing the proportional controller gain by minimizing the error between measured and the desired feature vector values, as opposed to the classical visual servoing (VS) scheme that requires a known value of the proportional controller gain which is often chosen empirically. The second control method relies on the optimal tuning of PD controller gains applied on the altitude, attitude, and position loops. This, however, may lead to a convergence problem, which is overcome by the use of an optimization technique, the bat algorithm (BA). This latter is an efficient metaheuristic algorithm that has been successfully used in many optimization problems. Several simulations are run in MATLAB, in which visual servo control of the Vertical Takeof and Landing (VTOL) type of Unmanned Aerial Vehicles (UAVs), known as the quadcopter, is implemented. The obtained results show that the Bat algorithm renders the classical PD controller adaptive and provides a high flexibility in tracking the UAV trajectory even in the presence of disturbances. It also guarantees the stability and accuracy of the optimal, minimal time, vision-based controller.

Automated three-dimensional image processing for 2-blastomere and 4-blastomere embryo surgical applications

The surgical tool and embryo blastomere locations are crucial to perform cell surgery. This paper presents an automated algorithm to compute blastomere centroids and surgical tool coordinate. The algorithm utilises the camera to procure Z-stack images of the workspace. It uses a three-dimensional (3D) canny filter and k-means clustering to segment and compute 3D centroid of each blastomere. For surgical tool tip location, region of interest (ROI) is established via template matching. 2D canny filter is applied on the ROI to produce segmentation results. These results are compiled together to estimate the 3D tool tip location. All computations are conducted on image plane which provide convenience to implement image-based visual servoing (IBVS) control in the future. With the overall obtained computation variations for 2-blastomere (2B) and 4-blastomere (4B) embryo, the proposed algorithm is suitable for blastomere aspiration alike procedures.

Vision-Based Formation Control of Mobile Robots With FOV Constraints and Unknown Feature Depth

In this the problem of leader-following tracking control of wheeled mobile robots with implicit communication through onboard visual sensing is addressed. New image-based visual servoing (IBVS) control schemes are proposed to ensure prescribed performance and the field-of-view (FOV) constraints in the absence of the feature depth information. First, the error transformation with predefined performance specifications is introduced for handling the visibility constraints. Then, in the setting of control of the uncertain system with unknown control coefficients, a Nussbaum gain adaptive controller and a static nonlinear gain controller are proposed without the need of feature depth information. Finally, both numerical simulations and physical experiments are presented to demonstrate the applicability and performance of the proposed control schemes.

An acceleration-level visual servoing scheme for robot manipulator with IoT and sensors using recurrent neural network

With the rapid evolution of Internet of Things (IoT) and sensor technology, robot manipulators are able to acquire a great deal of necessary information to fulfill various tricky tasks. Based on the recurrent neural network method, this paper investigates the robot manipulator with visual sensors for the image-based visual servoing (IBVS) control at the acceleration level. In consideration of multiple-level joint constraints, the newly proposed acceleration-level IBVS scheme can drive the corresponding feature of the end-effector to the desired fixed feature with theoretical convergence proved. In order to verify the correctness of the theoretical analyses, simulations are conducted on different robot manipulators for the visual servoing task with excellent performance. In addition, comparison results between the proposed acceleration-level IBVS scheme and the existing method highlight the efficiency and the superiority of the proposed method. In short, the proposed method extends the application scenarios of the traditional visual servoing schemes by using acceleration commands, which also improves the safety of the visual servoing control of the redundant robot manipulator.

Vision‐based tip position tracking control of two‐link flexible manipulator

Owing to the non-collocation of actuators and sensors in a flexible-link manipulator (FLM) it becomes difficult to achieve accurate tip position tracking. To resolve this issue, a vision sensor is used for direct measurement of the tip position instead of employing the traditional mechanical sensors. Among the different visual servoing (VS) control schemes, image-based VS (IBVS) is more effective. However, there are many challenges in the IBVS scheme such as singularities in the interaction matrix and local minima in trajectories that affect the system performance in real-time applications. In this study, the moment-based new visual feature is selected to address the aforesaid issues that arise in the IBVS scheme. Furthermore, a new two-time scale IBVS controller is developed for addressing the tip-tracking control problem of the two-link flexible manipulator (TLFM). In the proposed control scheme, the dynamics of the FLM is decomposed into two-time scale models, namely a slow subsystem and a fast subsystem. The performance and robustness of the proposed new two-time scale IBVS controller for TLFM are verified by pursuing simulation studies. It is observed from the obtained results that the proposed controller effectively stabilises the oscillatory dynamics and tracks the reference trajectory accurately.

Adaptive Image-Based Leader–Follower Formation Control of Mobile Robots With Visibility Constraints

This article addresses the problem of leader-follower formation control of mobile robots using only onboard monocular cameras that subjected to visibility constraints. An adaptive image-based visual servoing control strategy was proposed following the prescribed performance control methodology. First, the leader-follower visual kinematics in the image plane and an error transformation with predefined performance specifications are presented. Then, an adaptive control law with online estimating the inverse height between the optical center of camera and the single feature point attached to leader is designed to ensure the global stability of the closed-loop system. Finally, the applicability and performance of the proposed control scheme are demonstrated by the numerical simulations and hardware experiments. Compared with other formation control schemes, our solution relies only on onboard visual sensors without communication, since it does not need the relative angle/distance between the robots, or the velocity of the leader. Moreover, it guarantees the prescribed transient and the steady-state performance besides the visibility constraints.

Natural Feature-based Visual Servoing for Grasping Target with an Aerial Manipulator

Aerial transportation and manipulation have attracted increasing attention in the unmanned aerial vehicle field, and visual servoing methodology is widely used to achieve the autonomous aerial grasping of a target object. However, the existing marker-based solutions pose a challenge to the practical application of target grasping owing to the difficulty in attaching markers on targets. To address this problem, this study proposes a novel image-based visual servoing controller based on natural features instead of artificial markers. The natural features are extracted from the target images and further processed to provide servoing feature points. A six degree-of-freedom (6-DoF) aerial manipulator system is proposed with differential kinematics deduced to achieve aerial grasping. Furthermore, a controller is designed when the target object is outside a manipulator’s workspace by utilizing both the degrees-of-freedom of unmanned aerial vehicle and manipulator joints. Thereafter, a weight matrix is used as basis to develop a multi-tasking visual servoing framework to integrate the controllers inside and outside the manipulator’s workspace. Lastly, experimental results are provided to verify the effectiveness of the proposed approach.

Image-based visual servoing improvement through utilization of adaptive control gain and pseudo-inverse of the weighted mean of the Jacobians

The use of information acquired by stationary or onboard vision sensors for feedback control of pose and motion of a robot is known as visual servoing. In this paper, an image-based visual servoing controller with improved rate and domain of convergence is presented using adaptive control gain and pseudo-inverse of the weighted mean of the Jacobians. Furthermore, the stability of the proposed controller is proved by means of Lyapunov criteria, and simulations results are provided to demonstrate the validity and advantages of this control scheme. In comparison with the pseudo-inverse of the mean of the Jacobians scheme, the proposed controller increased the convergence rate of the servoing system up to seven times when used in complex motion tasks, and widened the convergence domain of the system up to 179° initial camera yaw angle when used in pure rotation tasks.

Robust online model predictive control for image-based visual servoing in polar coordinates

In this paper, a novel robust online model predictive control (RMPC) method for image-based visual servoing (IBVS) in polar coordinates is proposed. First, the Jacobian matrix is transformed into a weighted combination of vertex matrices of convex polytopic by tensor product (TP) model transformation method. Then, a new IBVS control design condition for 6-DOF manipulator submitting to robot physical limitations and visibility constraints is obtained in polar coordinates by using RMPC technique. The optimal value of the control signal can be solved online when carrying out the convex optimization problem. The proposed control strategy can effectively improve the trajectory in the case that involves translation and rotation with fast response while averting the pseudo-inverse of the image Jacobian matrix. Conclusively, the effectiveness of the proposed scheme is validated by simulations and experiments on a 6-DOF manipulator.

A new image-based visual servoing method with velocity direction control

In image-based visual servoing (IBVS), for more satisfying trajectories, the target is always expected to move in straight lines to the desired positions. In this paper, a new IBVS controller for trajectory adjustment is obtained by optimizing a multi-objective function about adjusting a direction in the image plane. By applying the control law based on the optimized parameters, less redundant motion can be achieved in contrast to the traditional methods. Meanwhile, it will be beneficial for reducing the risk of the target leaving the field of view (FOV) and the robustness to the uncertainties of depth and camera internal parameters remains. The experimental results of the 6 degrees of freedom (DOF) robot with eye-in-hand configuration demonstrate the effectiveness and practicability of the proposed method.

Uncalibrated Image-Based Visual Servoing Control with Maximum Correntropy Kalman Filter

Abstract A major challenge in solving robot visual servoing problems with unstructured environments is to obtain image Jacobian matrix, especially non-Gaussian noise always exist in the whole process. However, the standard Kalman Filter (KF) is exhausted to find the optimal value under Gaussian noise assumption. In this paper, a Kalman filter which adopted the maximum correntropy criterion (MCC) instead of the minimum mean square error (MMSE) criterion is proposed to solve the approximation issue of the image Jacobian. The simulation and experiment results using a conventional 6R manipulator are conducted to verify the effectiveness of the proposed method.

Image-Based Visual Servoing Control of Robot Manipulators Using Hybrid Algorithm With Feature Constraints

The challenge in addressing uncalibrated visual servoing (VS) control of robot manipulators with unstructured environments is to obtain appropriate interaction matrix and keep the image features in the field of view (FOV), especially when the non-Gaussian noise disturbance exists in the VS process. In this article, a hybrid control algorithm which combines bidirectional extreme learning machine (B-ELM) with smooth variable structure filter (SVSF) is proposed to estimate interaction matrix and tackle visibility constraints. For VS, the nonlinear mapping between image features and interaction matrix is approximated using the B-ELM learning. To increase the capability of anti-interference, the SVSF is employed to re-estimate interaction matrix. A constraint function presenting feature coordinates and region boundaries is given and added to the velocity controller, which drags image features away from the restricted region and ensures the smoothness of the velocities. Since the camera and robot model parameters are not required in developing the control strategy, the servoing task can be fulfilled flexibly and simply. Simulation and experimental results on a conventional 6-degree-of-freedom manipulator verify the effectiveness of the proposed method.