Image-based Visual Servoing Scheme Research Articles

A data-driven acceleration-level scheme for image-based visual servoing of manipulators with unknown structure.

The research on acceleration-level visual servoing of manipulators is crucial yet insufficient, which restricts the potential application range of visual servoing. To address this issue, this paper proposes a quadratic programming-based acceleration-level image-based visual servoing (AIVS) scheme, which considers joint constraints. Besides, aiming to address the unknown problems in visual servoing systems, a data-driven learning algorithm is proposed to facilitate estimating structural information. Building upon this foundation, a data-driven acceleration-level image-based visual servoing (DAIVS) scheme is proposed, integrating learning and control capabilities. Subsequently, a recurrent neural network (RNN) is developed to tackle the DAIVS scheme, followed by theoretical analyses substantiating its stability. Afterwards, simulations and experiments on a Franka Emika Panda manipulator with eye-in-hand structure and comparisons among the existing methods are provided. The obtained results demonstrate the feasibility and practicality of the proposed schemes and highlight the superior learning and control ability of the proposed RNN. This method is particularly well-suited for visual servoing applications of manipulators with unknown structure.

Logarithmic Observation of Feature Depth for Image-Based Visual Servoing

Due to the robustness to robot modeling and camera calibration errors and avoidance of complete target geometry, image-based visual servoing has always been an important topic in the fields such as robotics, computer vision and so forth. When the image information obtained by the camera is mapped to the robotic task space to design the servoing control law, the resulting interaction matrix, which links the spatial velocity of the camera to the temporal variation of the selected image features, depends on the unknown feature depths. The use of inaccurate feature depths may influence the stability and robustness of the controller, and even cause the failure of the task. In this article, based on the perspective camera model, by employing the principle of reduced order observer, a novel logarithmic observer is presented for on-line recovery of feature depth. Compared with the typical observers now available, the presented observer offers several advantages: global convergence, a faster convergence rate of error structure than exponential error structure, a less restricted observability condition and greater robustness against measurements with noise. The comparison results of numerical simulations indicate the superiority of the presented observer, and real experiments with Kinect v2 sensor further validate the effectiveness of the presented observer in practical situation. Note to Practitioners—This article was motivated by the depth problem in the image-based visual servoing scheme, but it can also be used in other situations where the image depth information is needed, such as 3D reconstruction, robot navigation, etc. The existing depth acquisition methods include TOF sensors, stereo vision, depth observers and so on. However, TOF sensors are sensitive to light conditions, and the mounting space of stereo vision is slightly large, and there is contradiction between observation performance and computational complexity in most existing observers. In this article, a novel structure of logarithmic reduced order observer is described in detail, which can be utilized to estimate the depth information of images easily. The simulations and experiments verify the good performance of the observer. The limitations of the given observer are that the estimation accuracy is not very good under weak excitation, and the camera needs to be calibrated in advance. Future work will focus on overcoming these two limitations.

Image-Based Visual Servoing of Rotorcrafts to Planar Visual Targets of Arbitrary Orientation

This letter for the first time extends the virtual camera image-based visual servoing (IBVS) scheme to enable an underactuated rotorcraft UAV to regulate its translational motion and heading relative to a planar visual target of arbitrary orientation. The conversion from real camera images to virtual camera images of visual targets are proposed based on a set of rotation matrices. Hence, image moment features can be reused due to the simplicity and decoupled structure of the interaction matrix, and satisfactory 3D Cartesian trajectory of UAVs. In the design of the IBVS control law, the external disturbance and model uncertainties are estimated by an integral-based filter. In addition, to enable tracking of a moving visual target, a velocity estimator is developed. The global asymptotic stability of the error dynamics is proven. Both of the simulation and experimental results of tracking of a tilted moving planar target are provided to show the efficacy of the proposed IBVS scheme.

Image-based control for rendezvous and synchronization with a tumbling space debris

This paper investigates the problem of concurrent rendezvous and attitude synchronization with a tumbling space debris. A novel formulation of the system dynamics model is first presented, based on image features and their rates, for the 6-degree-of-freedom relative pose tracking, which does not require an accurate estimation of the relative motions. Two image-based visual servoing schemes are then developed, which integrate visual navigation into the control law by defining state errors directly in the two-dimensional image space. The first scheme, i.e., observer-based controller, estimates the image rates without any need for relative velocity measurements. The second scheme, i.e., adaptive controller, updates the control gains by estimating the uncertainty terms in real time, hence eliminates the need for assigning their upper bounds a priori. The stability analysis of both controllers guarantees that the state errors remain uniformly ultimately bounded and exponentially converge to an arbitrarily small compact set containing the equilibrium point during the rendezvous and attitude synchronization mission. Numerical simulations and a comparative study with PD-like, image-based visual servoing are presented to validate the effectiveness and advantages of the proposed control schemes.

Vision‐based tip position tracking control of two‐link flexible manipulator

Owing to the non-collocation of actuators and sensors in a flexible-link manipulator (FLM) it becomes difficult to achieve accurate tip position tracking. To resolve this issue, a vision sensor is used for direct measurement of the tip position instead of employing the traditional mechanical sensors. Among the different visual servoing (VS) control schemes, image-based VS (IBVS) is more effective. However, there are many challenges in the IBVS scheme such as singularities in the interaction matrix and local minima in trajectories that affect the system performance in real-time applications. In this study, the moment-based new visual feature is selected to address the aforesaid issues that arise in the IBVS scheme. Furthermore, a new two-time scale IBVS controller is developed for addressing the tip-tracking control problem of the two-link flexible manipulator (TLFM). In the proposed control scheme, the dynamics of the FLM is decomposed into two-time scale models, namely a slow subsystem and a fast subsystem. The performance and robustness of the proposed new two-time scale IBVS controller for TLFM are verified by pursuing simulation studies. It is observed from the obtained results that the proposed controller effectively stabilises the oscillatory dynamics and tracks the reference trajectory accurately.

Robust uncalibrated visual stabilization for nonholonomic mobile robots

ABSTRACT This paper proposes a new image-based visual servoing scheme for stabilizing the nonholonomic mobile robot equipped with a monocular vision system to the desired configuration. In the vision system, neither intrinsic nor extrinsic camera parameters are calibrated. In addition, the lack of depth information should be tackled when using a monocular camera. The main difficulty encountered lies in the simultaneous existence of nonholonomic constraints, uncalibrated camera parameters and unknown depth. To deal with these problems, a dual-loop control structure is utilized. For the inner loop, an extended state observer is constructed to observe the system uncertainties caused by the uncalibrated camera parameters and unknown depth. For the outer loop, a switching controller is synthesized based on the compensated system model to maneuver the mobile robot to the desired configuration. Simulations and experimental results illustrate the effectiveness and robustness of the proposed control scheme.

A Neural Controller for Image-Based Visual Servoing of Manipulators With Physical Constraints.

Main issues in visual servoing of manipulators mainly include rapid convergence of feature errors to zero and the safety of joints regarding joint physical limits. To address the two issues, in this paper, an image-based visual servoing scheme is proposed for manipulators with an eye-in-hand configuration. Compared with existing schemes, the proposed one does not require performing pseudoinversion for the image Jacobian matrix or inversion for the Jacobian matrix associated with the forward kinematics of the manipulators. Theoretical analysis shows that the proposed scheme not only guarantees the asymptotic convergence of feature errors to zero but also the compliance with joint angle and velocity limits of the manipulators. Besides, simulation results based on a PUMA560 manipulator with a camera mounted on the end effector verify the theoretical conclusions and the efficacy of the proposed scheme.

목표물의 거리 및 특징점 불확실성 추정을 통한 매니퓰레이터의 영상기반 비주얼 서보잉

This paper proposes a robust image-based visual servoing scheme using a nonlinear observer for a monocular eye-in-hand manipulator. The proposed control method is divided into a range estimation phase and a target-tracking phase. In the range estimation phase, the range from the camera to the target is estimated under the non-moving target condition to solve the uncertainty of an interaction matrix. Then, in the target-tracking phase, the feature point uncertainty caused by the unknown motion of the target is estimated and feature point errors converge sufficiently near to zero through compensation for the feature point uncertainty.

Applying BAT Evolutionary Optimization to Image-Based Visual Servoing

This paper presents a predictive control strategy for an image-based visual servoing scheme that employs evolutionary optimization. The visual control task is approached as a nonlinear optimization problem that naturally handles relevant visual servoing constraints such as workspace limitations and visibility restrictions. As the predictive scheme requires a reliable model, this paper uses a local model that is based on the visual interaction matrix and a global model that employs 3D trajectory data extracted from a quaternion-based interpolator. The work assumes a free-flying camera with 6-DOF simulation whose results support the discussion on the constraint handling and the image prediction scheme.

Randomized Kinodynamic Planning for Robust Visual Servoing

We incorporate a randomized kinodynamic path planning approach with image-based control of a robotic arm equipped with an in-hand camera. The proposed approach yields continuously differentiable camera trajectories by taking camera dynamics into account, while accounting for a critical set of image and physical constraints at the planning stage. The proposed planner explores the camera state space for permissible trajectories by iteratively extending a search tree in this space and simultaneously tracking these trajectories in the robot configuration space. The planned camera trajectories are projected into the image space to obtain desired feature trajectories which are then tracked using an image-based visual servoing scheme. We validate the effectiveness of the proposed framework in incorporating the aforementioned constraints through a number of visual servoing experiments on a six-degree-of-freedom robotic arm. We also provide empirical results that demonstrate its performance in the presence of uncertainties, and accordingly suggest additional planning strategies to increase robustness with respect to possible deviations from planned trajectories.

Landing of an Airliner Using Image Based Visual Servoing

In this paper, an image-based visual servoing scheme is presented in order to control a transport aircraft in final approach phase by overcoming the need for external information (e.g. ILS or GPS systems) and runway knowledge. Based on three decoupled visual features and inertial data, the guidance scheme is designed and validated on a simplified model and tested on a realistic nonlinear simulator. Finally simulation results are presented in order to validate the choice of the visual features and the guidance laws to land the aircraft.

Visual Control of Planar Parallel Robots Without Using Velocity Measurements

This work proposes an image-based visual servoing scheme applied to a class of overactuated planar parallel robots with revolute joints. A Proportional Derivative algorithm computes torques for the robot active joints. The Derivative action operates at the joint level and the Proportional action works at the visual level. A linear filter provides velocity estimates from active joint position measurements, and Lyapunov Stability theory allows concluding closed loop stability. Experimental results on a laboratory prototype show that the proposed approach permits using higher gains compared with a control law relying on velocity estimates generated using visual measurements; moreover, these experiments also show that the closed loop system is robust in face of kinematic uncertainty.

On visual servoing based on efficient second order minimization

This paper deals with the Efficient Second order Minimization ( ESM) and the image-based visual servoing schemes. In other words, it deals with the minimization based on the pseudo-inverse of the mean of the Jacobians or on the mean of Jacobian Pseudo-inverses. Chronologically, it has been noticed in Tahri and Chaumette (2003) [22] that ESM generally improves the system behavior when compared with the system in which only the simple Jacobian Pseudo-inverses are used. Subsequently, a mathematical explanation has been given in Malis (2004) [12]. In this paper, the proofs given by Malis are discussed and it will be shown that there is a limitation to the validity of the ESM. We will also show that the use of ESM does not necessarily ensure a better system behavior, especially in the situations where large rotational motions are involved. Further, a new appropriate formula of the ESM is proposed and validated using several kinds of features.

Image-Based Visual Servoing for Vanishing Features and Ground Lines Tracking: Application to a UAV Automatic Landing

A unified modeling framework is defined and an image-based visual servoing scheme is introduced to evaluate the feasibility of a vision-based guidance for a UAV model during survey missions and for automatic landing. Relevant image features are selected to allow for the proposed objective, namely decoupling rotation and translation in the image, and allowing to respect the natural separation between the lateral and longitudinal motions of the aircraft. The controller design is adapted to include the aircraft dynamics and the vision output in an linear optimal control approach. Simulation results are presented in order to validate the proposed solution and allow for a critical evaluation.

Image-based visual servoing schemes for nonholonomic mobile manipulators

SUMMARYWe consider the task-oriented modeling of the differential kinematics of nonholonomic mobile manipulators (NMMs). A suitable NMM Jacobian is defined that relates the available input commands to the time derivative of the task variables, and can be used to formulate and solve kinematic control problems. When the NMM is redundant with respect to the given task, we provide an extension of two well-known redundancy resolution methods for fixed-base manipulators (Projected Gradient and Task Priority) and introduce a novel technique (Task Sequencing) aimed at improving performance, e.g., avoiding singularities. The proposed methods are applied then to the specific case of image-based visual servoing, where the NMM image Jacobian combines the interaction matrix and the kinematic model of the mobile manipulator. Comparative numerical results are presented for two case studies.