Image-based Visual Servoing Research Articles

Image-Feature-Based Integrated Path Planning and Control for Spacecraft Rendezvous Operations

This paper develops an integrated path planning and control framework for a specific problem of image-feature-based spacecraft rendezvous with an uncooperative target. For visibility concerns, the chasing spacecraft is required to approach and synchronize with the tumbling target, as well as to retain the target within the camera’s field of view. To this end, an offline path planner is designed by using the gradient of a so-called navigation function to generate a set of desired image trajectories. A novel formulation of the spacecraft relative dynamic model is presented based on current image features and the generated desired ones. To track the desired image trajectories, a robust image-based visual servoing controller is designed via passivity theorem. The stability of the closed-loop system is analyzed using the Lyapunov method. In addition, a hyperbolic tangent function is introduced to cope with the chattering phenomenon caused by the signum function. A scenario of rendezvousing with the defunct Envisat is demonstrated to validate the effectiveness and advantages of the proposed control scheme. Comparative studies with the conventional image-based and pose-based control schemes are performed through numerical simulations, in order to evaluate the target visibility, control accuracy, and fuel consumption under the proposed control scheme.

Singularity Analysis for the Perspective-Four and Five-Line Problems

This paper deals with image-based visual servoing and pose estimation by observing four and five lines. Our main interest is to determine the relative configurations of the camera and the observed lines that lead to problems in control and stability. Since it is equivalent to finding the singularities of the corresponding Jacobian matrix, we use tools from computational algebraic geometry to seek configurations such that all of its minors vanish simultaneously. By choosing a suitable basis for this matrix, we revisit the problem in the case of three lines to show that one type of the singularities is when the camera lies on the hyperboloid of one sheet uniquely defined by the lines. This result is further exploited to prove that the one-dimensional singularities, if any, in the case of n lines appear when the camera lies on the transversals to the observed lines. Thus, by forcing the transversals to be complex, we can avoid the aforementioned type of singularities in the case of four lines although the algebra shows that there can always be up to 10 inevitable singular locations of the camera for the other type of singularity. For five lines, we find out that there are no singularities in the generic case. The singularities are also characterized for four and five lines with orthogonality and parallelism constraints. Furthermore, a visual servoing library is used to conduct some simulated experiments to substantiate the theoretical results. As expected, we observe problems in control in the vicinity of a singularity as well as increased errors in pose estimation.

Adaptive neural network control for image‐based visual servoing of robot manipulators

This paper presents a novel adaptive neural network control strategy for image-based visual servoing (IBVS) of robotic manipulators with both eye-in-hand and eye-to-hand camera configurations in the presence of unknown dynamics and external disturbances. The IBVS method is combined with the adaptive neural network to construct the proposed adaptive neural network controller to solve the visual servoing control problem of robots. The adaptive neural network based IBVS controller is designed based on the depth-independent interaction matrix, which can be trained on-line to identify the visual servoing robotic system modeling errors. Moreover, the proposed method can approach the unknown nonlinear dynamics for both eye-in-hand and eye-to-hand camera configurations without requiring the robot dynamics to be linearly parameterizable, and the exact knowledge of the robot structure is not needed. On the basis of the nonlinear robot dynamics, the Lyapunov stability analysis is given to prove the asymptotical convergence of the image position and velocity errors. Simulation results for both camera configurations are provided to demonstrate the performance of the proposed adaptive neural network based approach.

전방향 모바일 매니퓰레이터의 강인 영상기반 비주얼 서보잉

This paper proposes robust image based visual servoing of an omnidirectional mobile manipulator using integral sliding mode control to compensate for the uncertainty that may occur while performing the image-based visual servoing. The stability of the overall omnidirecitonal mobile manipulator system using the proposed method is proven by the Lyapunov stability analysis by showing that the error stays on the sliding surface always and asymptotically converges to zero. In order to solve the singularity problem that can occur when controlling the omnidirectional mobile robot and the manipulator at the same time, a switching technique is proposed in such a way that the omnidirectional mobile robot and manipulator operate sequentially to prevent the abnormal operation of the manipulator and solve the singularity problem. The validity of the proposed robust image based visual servoing method is verified through simulation results.

Mix Frame Visual Servo Control Framework for Autonomous Assistive Robotic Arms.

Assistive robotic arms (ARAs) that provide care to the elderly and people with disabilities, are a significant part of Human-Robot Interaction (HRI). Presently available ARAs provide non-intuitive interfaces such as joysticks for control and thus, lacks the autonomy to perform daily activities. This study proposes that, for inducing autonomous behavior in ARAs, visual sensors integration is vital, and visual servoing in the direct Cartesian control mode is the preferred method. Generally, ARAs are designed in a configuration where its end-effector’s position is defined in the fixed base frame while orientation is expressed in the end-effector frame. We denoted this configuration as ‘mixed frame robotic arms’. Consequently, conventional visual servo controllers which operate in a single frame of reference are incompatible with mixed frame ARAs. Therefore, we propose a mixed-frame visual servo control framework for ARAs. Moreover, we enlightened the task space kinematics of a mixed frame ARAs, which led us to the development of a novel “mixed frame Jacobian matrix”. The proposed framework was validated on a mixed frame JACO-2 7 DoF ARA using an adaptive proportional derivative controller for achieving image-based visual servoing (IBVS), which showed a significant increase of 31% in the convergence rate, outperforming conventional IBVS joint controllers, especially in the outstretched arm positions and near the base frame. Our Results determine the need for the mixed frame controller for deploying visual servo control on modern ARAs, that can inherently cater to the robotic arm’s joint limits, singularities, and self-collision problems.

A new decoupled control law for image-based visual servoing control of robot manipulators

A new decoupled control law for image-based visual servoing control of robot manipulators

Image space trajectory tracking of 6-DOF robot manipulator in assisting visual servoing

As vision is a versatile sensor, vision-based control of robot is becoming more important in industrial applications. The control signal generated using the traditional control algorithms leads to undesirable movement of the end-effector during the positioning task. This movement may sometimes cause task failure due to visibility loss. In this paper, a sliding mode controller (SMC) is designed to track 2D image features in an image-based visual servoing task. The feature trajectory tracking helps to keep the image features always in the camera field of view and thereby ensures the shortest trajectory of the end-effector. SMC is the right choice to handle the depth uncertainties associated with translational motion. Stability of the closed-loop system with the proposed controller is proved by the Lyapunov method. Three feature trajectories are generated to test the efficacy of the proposed method. Simulation tests are conducted and the superiority of the proposed method over a Proportional Derivative – Sliding Mode Controller (PD-SMC) in terms of settling time and distance travelled by the end-effector is established in the presence and absence of depth uncertainties. The proposed controller is also tested in real-time by integrating the visual servoing system with a 6-DOF industrial robot manipulator, ABB IRB 1200.

Completely Distributed Time-Varying Formation Target Tracking for Quadrotor Team via Image-Based Visual Servoing

In this article, the completely distributed adaptive time-varying formation target tracking control problem for quadrotor team via image-based visual servoing is addressed. An adaptive cascade formation controller is proposed, which consists of a trajectory tracking controller, an attitude controller, and an image-based visual servoing controller. All the controllers consist of a nominal part to achieve the desired performance and a robust part to retrain the influence of parameter uncertainties and external disturbance. The robust properties of the closed-loop control system are proven via the theoretical analysis. Moreover, the simulation results validate the effectiveness of the proposed formation tracking control protocol in this paper.

Implement contour following task of objects with unknown geometric models by using combination of two visual servoing techniques

In this paper, two visual servoing techniques are combined to implement the contour following task of objects with unknown geometric models and the contour plane can be anywhere in the workspace. The scaled Euclidean position and orientation of the contour plane are defined based on decomposing the homography matrix. The shortest path visual servoing (SPVS) technique is applied to control the robot end-effector or the camera to the position parallel to the contour plane. Then, the image-based visual servoing (IBVS) technique is applied to conduct the contour following task. The desired feature command points are extracted from the contour using image processing algorithms. Based on the feature command point, four virtual feature points are generated to form the interaction matrix used in IBVS. The effectiveness of the proposed approach is verified based on the simulation result by using a camera attached at the end-effector of the six-DOF PUMA 560 manipulator.

Optimal Image-Based Task-Sequence/Path Planning and Robust Hybrid Vision/Force Control of Industrial Robots

In this paper, a novel image-based task-sequence/path planning scheme coupled with a robust vision/force control method is suggested for solving the multi-task operation problem of an eye-in-hand (EIH) industrial robot interacting with a workpiece. The proposed method suggests an optimal task sequence planning scheme to perform all the tasks and an optimal path planning method to generate a collision-free path between the tasks when the robot performs free-motion. To this end, a new method is presented which solves both problems simultaneously. A novel deadlock-free modified artificial potential field (MAPF) based on rotational potential force is developed for generating the collision-free path betweentasks in the three-dimensional (3D) environment. The parameters of the MAPF and the sequence of the tasks are found by an optimizer simultaneously. This problem can be considered as a MAPF-constrained-generalized-traveling-salesman-problem (MAPF-CGTSP), which is a mix-integer optimization problem. The mix-integer version of multi-tracker optimization algorithm (MTOA) is developed to solve the problem. However, since image-based visual servoing (IBVS) is used for motion control, the planning is conducted in the image space. Integrated with the proposed planning method, a novel chattering-free filtered quasi sliding mode controller (FQSMC) is specially designed for robust vision/force control of the robot. FQSMC exploits a novel variable-gain orthogonal-sliding-manifold (VGOSM)which enables the robot to switch between free-motion mode and interaction mode. FQSMC overcomes large uncertainties and filters out the existing noises by exploiting an intrinsic filter within its control law. Experimental results show the superiority of the proposed approach to other state-of-the-art methods.

Constrained Visual Servoing of Quadrotors Based on Model Predictive Control

One of the main issues of visual servoing schemes occurs when the target objects leave out the field of view (FOV) of the camera, which causes failure or poor performance of the controller. Solving this problem can be a challenge due to traditional controllers cannot include system's constraints. This work presents model predictive control (MPC) for constrained image-based visual servoing (IBVS) applied in quadrotors, considering constraints in FOV and restrictions in the control actions. To handle with image constraints the MPC considers: (1) the target objects stay only in camera's FOV and this work converts these restrictions in state constraints, (2) merge image instantaneous kinematics and the dynamic of commercial quadrotors (Mavic Pro 2) in a general mathematical model in order to satisfy the bounded control actions and image constraints. Due to commercial quadrotors allow velocities like control inputs, this work considers the reduced dynamic model in general velocities space and it was identified using Dynamic Mode Decomposition with control (DMDc) algorithm. This work uses Webots to evaluate the performance of the proposed controller. Finally the controller is compared with a classical IBVS scheme in order to verify the efficacy of the proposed controller and systematically evaluate the performance considering the system constraints.

Image-Based Visual Servoing System for Components Alignment Using Point and Line Features

The design of visual servoing system for robotic high-precision assembly is of great challenge. For the difficulty of insufficient accuracy of target object feature extraction, a deep neural network combined with feature pyramid network (FPN) structure is proposed. This lightweight network requires only a small amount of labeled data to achieve significant segmentation results. The control laws of translation and orientation for component alignment are separately designed. The translation is controlled based on the interaction matrix of point features. The orientation is controlled based on the interaction matrix of line features. The relations between the cameras’ motion and the end-effector of the manipulator are calibrated via the manipulator’s active movements, which are <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$3 \times $ </tex-math></inline-formula> 3 transformation matrices. The depth information of feature points is integrated into the transformation matrix. The alignment pose error estimation is realized with the interaction matrices, transformation matrices, and point and line features. A robotic assembly system is developed to assemble two aviation circular connectors with six degree-of-freedoms (DOFs) in high precision in 3-D space. The experimental results verify the effectiveness of the proposed method.

Image-based visual servoing with depth estimation

For the depth estimation problem in the image-based visual servoing (IBVS) control, this paper proposes a new observer structure based on Kalman filter (KF) to recover the feature depth in real time. First, according to the number of states, two different mathematical models of the system are established. The first one is to extract the depth information from the Jacobian matrix as the state vector of the system. The other is to use the depth information and the coordinate point information of the two-dimensional image plane as the state vector of the system. The KF is used to estimate the unknown depth information of the system in real time. And an IBVS controller gain adjustment method for 6-degree-of-freedom (6-DOF) manipulator is obtained using fuzzy controller. This method can obtain the gain matrix by taking the depth and error information as the input of the fuzzy controller. Compared with the existing works, the proposed observer has less redundant motion while solving the Jacobian matrix depth estimation problem. At the same time, it will also be beneficial to reducing the time for the camera to reach the target. Conclusively, the experimental results of the 6-DOF robot with eye-in-hand configuration demonstrate the effectiveness and practicability of the proposed method.

The Estimation of Image Jacobian Matrix with Time-Delay Compensation

Time delay exists in image-based visual servo system, which will have a certain impact on the system control. To solve the impact of time delay, the time delay compensation of the object feature point image and the image Jacobian matrix is discussed in this paper. Some work is done in this paper: The estimation of the object feature point image under time delay is based on a proposed robust decorrelation Kalman filtering model, for the measurement vectors which cannot be obtained during time delay in the robust Kalman filtering model, a polynomial fitting method is proposed in which the selection of the polynomial includes the position, velocity and acceleration of the object feature point which impact the feature point trajectory, then the more accurate object feature point image can be obtained. From the estimated object feature point image under time delay, the more accurate image Jacobian matrix under time delay can be obtained. Simulation and experimental results verify the feasibility and superiority of this paper method.

Homography-based Visual Servoing with Remote Center of Motion for Semi-autonomous Robotic Endoscope Manipulation.

The dominant visual servoing approaches in Minimally Invasive Surgery (MIS) follow single points or adapt the endoscope's field of view based on the surgical tools' distance. These methods rely on point positions with respect to the camera frame to infer a control policy. Deviating from the dominant methods, we formulate a robotic controller that allows for image-based visual servoing that requires neither explicit tool and camera positions nor any explicit image depth information. The proposed method relies on homography-based image registration, which changes the automation paradigm from point-centric towards surgical-scene-centric approach. It simultaneously respects a programmable Remote Center of Motion (RCM). Our approach allows a surgeon to build a graph of desired views, from which, once built, views can be manually selected and automatically servoed to irrespective of robot-patient frame transformation changes. We evaluate our method on an abdominal phantom and provide an open source ROS Moveit integration for use with any serial manipulator. A video is provided.

A Switching Image-Based Visual Servoing Method for Cooperative Continuum Robots

A Switching Image-Based Visual Servoing Method for Cooperative Continuum Robots

Image-Based Visual Servoing of Rotorcrafts to Planar Visual Targets of Arbitrary Orientation

This letter for the first time extends the virtual camera image-based visual servoing (IBVS) scheme to enable an underactuated rotorcraft UAV to regulate its translational motion and heading relative to a planar visual target of arbitrary orientation. The conversion from real camera images to virtual camera images of visual targets are proposed based on a set of rotation matrices. Hence, image moment features can be reused due to the simplicity and decoupled structure of the interaction matrix, and satisfactory 3D Cartesian trajectory of UAVs. In the design of the IBVS control law, the external disturbance and model uncertainties are estimated by an integral-based filter. In addition, to enable tracking of a moving visual target, a velocity estimator is developed. The global asymptotic stability of the error dynamics is proven. Both of the simulation and experimental results of tracking of a tilted moving planar target are provided to show the efficacy of the proposed IBVS scheme.

Robotic grasping method of bolster spring based on image-based visual servoing with YOLOv3 object detection algorithm

In this paper, to address the problem of automatic positioning and grasping of bolster spring with complex geometric features and cluttered background, a novel image-based visual servoing (IBVS) control method based on the corner points features of YOLOv3 object detection bounding box is proposed and applied to the robotic grasping system of bolster spring. The YOLOv3 object detection model is used to detect and position the bolster spring and then based on the corner points features of the bolster spring bounding box, the IBVS controller is designed to drive the end effector of the robot to the desired pose. This method adopts the approach-align-grasp control strategy to achieve the grasping of the bolster spring, which is very robust to the calibration parameter errors of the camera and the robot model. With the help of Robotics and Machine Vision Toolboxes in Matlab, IBVS simulation analysis based on feature points is carried out. The results show that it is reasonable to use the corner points of the object detection bounding box as image features under the initial pose where the image plane is parallel to the object plane. The positioning and grasping experiments are conducted on the robotic grasping platform of bolster spring. The results show that this method is effective for automatic positioning and grasping of bolster spring with complex geometric features and cluttered background, and it has strong robustness to the change of illumination.

Visual Servoing of a Moving Target by an Unmanned Aerial Vehicle.

To track moving targets undergoing unknown translational and rotational motions, a tracking controller is developed for unmanned aerial vehicles (UAVs). The main challenges are to control both the relative position and orientation between the target and the UAV to within desired values, and to guarantee that the generated control input to the UAV is feasible (i.e., below its motion capability). Moreover, the UAV is controlled to ensure that the target always remains within the field of view of the onboard camera. These control objectives were achieved by developing a nonlinear-model predictive controller, in which the future motion of the target is predicted by quadratic programming (QP). Since constraints of the feature vector and the control input are considered when solving the optimal control problem, the control inputs can be bounded and the target can remain inside the image. Three simulations were performed to compare the efficacy and performance of the developed controller with a traditional image-based visual servoing controller.

Constrained visual predictive control of tendon-driven continuum robots

Due to their compliance, continuum robots (CRs) hold great potential for many applications. However, despite intensive recent research, their control poses significant challenges. The nonlinear kinematic behavior, limited actuation channels, and physical and environmental constraints typically associated with CRs hinder the development of effective control strategies. In this paper, a visual predictive position control method for tendon-driven continuum robots is proposed. The developed control approach integrates the advantages of image-based visual servoing and model predictive control techniques to enable direct end-point control in the presence of constraints and improve the control robustness to system uncertainties, sensing noise, and modeling errors. Both simulation and experimental results demonstrate the effectiveness of the method.