Visual Servo Control Research Articles

DIJE: Dense Image Jacobian Estimation and its Application to Visual Servo Control

DIJE: Dense Image Jacobian Estimation and its Application to Visual Servo Control

Direct visual servoing and interaction control for a two-arms on-orbit servicing spacecraft

A direct visual-servoing algorithm for control of a space-based two-arm manipulator is proposed in this paper. The algorithm can be utilized in a two-arm manipulators configuration, where one of the arms performs the manipulation and the second arm is dedicated to the observation of the target zone of manipulation. The algorithm utilizes both visual features extracted from onboard cameras and force and torque measured at the manipulator's end-effector to control the movements of the manipulator during on-orbit servicing operations. The algorithm takes into account the relative dynamics of the bodies involved, it relies on images taken independently from de-localized cameras, e.g. at the end-effector of a second manipulator, and it integrates an impedance control for the compensation of eventual contact reactions when the end effector touches and operates the target body. The analytical derivations demonstrate the stability of the algorithm and incorporate an impedance compliance strategy into an optimal framework formulation. Simulations results in two different scenarios have been presented to show the adequate behavior of the presented approach in on-orbit-servicing operations.

Image-based visual servoing with depth estimation

For the depth estimation problem in the image-based visual servoing (IBVS) control, this paper proposes a new observer structure based on Kalman filter (KF) to recover the feature depth in real time. First, according to the number of states, two different mathematical models of the system are established. The first one is to extract the depth information from the Jacobian matrix as the state vector of the system. The other is to use the depth information and the coordinate point information of the two-dimensional image plane as the state vector of the system. The KF is used to estimate the unknown depth information of the system in real time. And an IBVS controller gain adjustment method for 6-degree-of-freedom (6-DOF) manipulator is obtained using fuzzy controller. This method can obtain the gain matrix by taking the depth and error information as the input of the fuzzy controller. Compared with the existing works, the proposed observer has less redundant motion while solving the Jacobian matrix depth estimation problem. At the same time, it will also be beneficial to reducing the time for the camera to reach the target. Conclusively, the experimental results of the 6-DOF robot with eye-in-hand configuration demonstrate the effectiveness and practicability of the proposed method.

Sampled-data robust visual servoing control for moving target tracking of an inertially stabilized platform with a measurement delay

The vision based moving target tracking is a key technology for unmanned systems in complex environments, which is mainly limited by the visual measurement delay and kinematic uncertainties. In this paper, a time-delay disturbance observer based sampled-data control approach is developed for the visual servoing of an inertially stabilized platform. As a camera is employed to capture the position of the target, the measurement delay is generated by the acquisition and the processing of the image information. Besides, kinematic uncertainties arising from the variable feature depth and inner-loop tracking errors of given reference signals are also unavoidable. To this end, the model of the perturbed and delayed system is established firstly. Due to the measurement delay, it is quite difficult to get the current system state. As such, a new time-delay disturbance observer is developed to estimate the previous disturbance and its differences, which facilitates predictions of the current state and the current disturbance. Based on above predictions, a sampled-data robust controller is designed with the rigorous stability analysis of the closed-loop system. Experiments on tracking a target are performed to validate promising qualities of the proposed method.

Design of Robot Vision Servo Control System Based on Image

Visual servo is a closed-loop control system of robot, which takes the image information obtained by visual sensor as feedback. Generally speaking, visual servo plays an important role in robot control, which is one of the main research directions in the field of robot control and plays a decisive role in the development of intelligent robots. In order to make the robot competent for more complex tasks and work more intelligently, autonomously and reliably, it is necessary not only to improve the control system of the robot, but also to obtain more and better information about the working environment of the robot. This paper introduces the principle and basic realization method of robot visual servo based on image, and expounds the problems and solutions in image feature extraction and visual servo controller design. In order to further expand the application field of robots and improve the operation performance of robots, robots must have higher intelligence level and stronger adaptability to the environment, so as to manufacture intelligent robots that can replace human labor.

Automated Rust Removal: Rust Detection and Visual Servo Control

Rust removal is one of the most hazardous and difficult tasks in the restoration of steel structures such as bridges and towers. Robotic systems provide an alternative technology for performing the task safely and efficiently. The capability to automatically remove rust considering the condition of the rusted area is a significant function of robotic systems. In this paper, we propose an approach for identifying rust as a key technology in the rust removal process. Rust detection can be performed automatically by processing a sequence of camera images. For the utilization of the approach in practical applications, it is designed to be robust to variations in the conditions of rusted areas caused by the presence of working tools or rust powders in the images. A fuzzy force controller that replicates the human behavior during rust removal is designed to construct a visual servo control framework for the rust removal process. The proposed approach is validated using experiments conducted on a rust-grinding robotic prototype.

A Robot Dynamic Target Grasping Method Based on Affine Group Improved Gaussian Resampling Particle Filter

Tracking and grasping a moving target is currently a challenging topic in the field of robotics. The current visual servo grasping method is still inadequate, as the real-time performance and robustness of target tracking both need to be improved. A target tracking method is proposed based on improved geometric particle filtering (IGPF). Following the geometric particle filtering (GPF) tracking framework, affine groups are proposed as state particles. Resampling is improved by incorporating an improved conventional Gaussian resampling algorithm. It addresses the problem of particle diversity loss and improves tracking performance. Additionally, the OTB2015 dataset and typical evaluation indicators in target tracking are adopted. Comparative experiments are performed using PF, GPF and the proposed IGPF algorithm. A dynamic target tracking and grasping method for the robot is proposed. It combines an improved Gaussian resampling particle filter algorithm based on affine groups and the positional visual servo control of the robot. Finally, the robot conducts simulation and experiments on capturing dynamic targets in the simulation environment and actual environment. It verifies the effectiveness of the method proposed in this paper.

Robotic grasping method of bolster spring based on image-based visual servoing with YOLOv3 object detection algorithm

In this paper, to address the problem of automatic positioning and grasping of bolster spring with complex geometric features and cluttered background, a novel image-based visual servoing (IBVS) control method based on the corner points features of YOLOv3 object detection bounding box is proposed and applied to the robotic grasping system of bolster spring. The YOLOv3 object detection model is used to detect and position the bolster spring and then based on the corner points features of the bolster spring bounding box, the IBVS controller is designed to drive the end effector of the robot to the desired pose. This method adopts the approach-align-grasp control strategy to achieve the grasping of the bolster spring, which is very robust to the calibration parameter errors of the camera and the robot model. With the help of Robotics and Machine Vision Toolboxes in Matlab, IBVS simulation analysis based on feature points is carried out. The results show that it is reasonable to use the corner points of the object detection bounding box as image features under the initial pose where the image plane is parallel to the object plane. The positioning and grasping experiments are conducted on the robotic grasping platform of bolster spring. The results show that this method is effective for automatic positioning and grasping of bolster spring with complex geometric features and cluttered background, and it has strong robustness to the change of illumination.

Visual Servoing of a Moving Target by an Unmanned Aerial Vehicle.

To track moving targets undergoing unknown translational and rotational motions, a tracking controller is developed for unmanned aerial vehicles (UAVs). The main challenges are to control both the relative position and orientation between the target and the UAV to within desired values, and to guarantee that the generated control input to the UAV is feasible (i.e., below its motion capability). Moreover, the UAV is controlled to ensure that the target always remains within the field of view of the onboard camera. These control objectives were achieved by developing a nonlinear-model predictive controller, in which the future motion of the target is predicted by quadratic programming (QP). Since constraints of the feature vector and the control input are considered when solving the optimal control problem, the control inputs can be bounded and the target can remain inside the image. Three simulations were performed to compare the efficacy and performance of the developed controller with a traditional image-based visual servoing controller.

Recent Advances on Robot Visual Servo Control Methods

Background: In order to enable robots to complete complex tasks, robots are required to have higher intelligence and adaptability. RVSC (Robot Visual Servo Control) is an organic combination of machine vision and servo control and is a complex system with strong coupling and nonlinearity, which improves the performance of robots and provides feasible solutions for solving practical problems in many fields. Objective: To review the latest development of RVSC methods and provide references for researchers in related fields. Methods: RVSC related patents and literature were analyzed from the aspects of the algorithm, innovation and application. In addition, the typical RVSC methods are summarized. Results: This paper analyzes the advantages and disadvantages of various methods in many patents and literature. Moreover, the typical applications, potential problems and future development directions of this field is put forward. Conclusion: RVSC has better performance than traditional control methods. In addition, it can also be combined with some algorithms or knowledge to further improve performance. But, the further improvements of RVSC are needed in the following areas: the convenience of operation, the optimization of trajectory planning, the combination of multiple sensors, etc. Furthermore, more patents of the RVSC also should be invented.

Constrained visual predictive control of tendon-driven continuum robots

Due to their compliance, continuum robots (CRs) hold great potential for many applications. However, despite intensive recent research, their control poses significant challenges. The nonlinear kinematic behavior, limited actuation channels, and physical and environmental constraints typically associated with CRs hinder the development of effective control strategies. In this paper, a visual predictive position control method for tendon-driven continuum robots is proposed. The developed control approach integrates the advantages of image-based visual servoing and model predictive control techniques to enable direct end-point control in the presence of constraints and improve the control robustness to system uncertainties, sensing noise, and modeling errors. Both simulation and experimental results demonstrate the effectiveness of the method.

Neural Network Based Visual Servo Control Under the Condition of Heavy Loading

Neural Network Based Visual Servo Control Under the Condition of Heavy Loading

Adaptive 3D Visual Servoing of a Scara Robot Manipulator with Unknown Dynamic and Vision System Parameters

In the present work, we develop an adaptive dynamic controller based on monocular vision for the tracking of objects with a three-degrees of freedom (DOF) Scara robot manipulator. The main characteristic of the proposed control scheme is that it considers the robot dynamics, the depth of the moving object, and the mounting of the fixed camera to be unknown. The design of the control algorithm is based on an adaptive kinematic visual servo controller whose objective is the tracking of moving objects even with uncertainties in the parameters of the camera and its mounting. The design also includes a dynamic controller in cascade with the former one whose objective is to compensate the dynamics of the manipulator by generating the final control actions to the robot even with uncertainties in the parameters of its dynamic model. Using Lyapunov’s theory, we analyze the two proposed adaptive controllers for stability properties, and, through simulations, the performance of the complete control scheme is shown.

Visual Servoing of a Cable-Driven Soft Robot Manipulator With Shape Feature

Soft continuum robot is of superiority in the applications in unstructured environments due to its high flexibility and safe interaction ability. Accurate shape control is regarded as one of the prerequisites to improve its practicability. Considering the situation where the 3D position signals are not available, this letter investigates the vision-based shape control scheme of a soft robot manipulator. The curvature features used to depict the reference image shape of the robot backbone curve are solved through nonlinear optimization with given constraints of two endpoints in pixel coordinates. Thereafter, an adaptive image-based visual servoing controller is designed to track the reference image shape features with an uncalibrated monocular camera. The proposed algorithm was tested on an eight-cable driving soft robot prototype and proved its validity in convergence to the desired image shape.

Telerobotic Operation of Intensive Care Unit Ventilators.

Since the first reports of a novel coronavirus (SARS-CoV-2) in December 2019, over 33 million people have been infected worldwide and approximately 1 million people worldwide have died from the disease caused by this virus, COVID-19. In the United States alone, there have been approximately 7 million cases and over 200,000 deaths. This outbreak has placed an enormous strain on healthcare systems and workers. Severe cases require hospital care, and 8.5% of patients require mechanical ventilation in an intensive care unit (ICU). One major challenge is the necessity for clinical care personnel to don and doff cumbersome personal protective equipment (PPE) in order to enter an ICU unit to make simple adjustments to ventilator settings. Although future ventilators and other ICU equipment may be controllable remotely through computer networks, the enormous installed base of existing ventilators do not have this capability. This paper reports the development of a simple, low cost telerobotic system that permits adjustment of ventilator settings from outside the ICU. The system consists of a small Cartesian robot capable of operating a ventilator touch screen with camera vision control via a wirelessly connected tablet master device located outside the room. Engineering system tests demonstrated that the open-loop mechanical repeatability of the device was 7.5 mm, and that the average positioning error of the robotic finger under visual servoing control was 5.94 mm. Successful usability tests in a simulated ICU environment were carried out and are reported. In addition to enabling a significant reduction in PPE consumption, the prototype system has been shown in a preliminary evaluation to significantly reduce the total time required for a respiratory therapist to perform typical setting adjustments on a commercial ventilator, including donning and doffing PPE, from 271 to 109 s.

Image-Based Indoor Topological Navigation with Collision Avoidance for Resource-Constrained Mobile Robots

This paper presents a complete topological navigation system for a resource-constrained mobile robot like Pepper, based on image memory and the teach-and-repeat paradigm. Image memory is constructed from a set of reference images that are acquired during a prior mapping phase and arranged topologically. A* search is used to find the optimal path between the current location and the destination. The images from the robot’s RGB camera are used to localize within the topological graph, and an Image-Based Visual Servoing (IBVS) control scheme drives the robot to the next node in the graph. Depth images update a local egocentric occupancy grid, and another IBVS controller navigates local free-space. The output of the two IBVS controllers is fused to form the final control command for the robot. We demonstrate real-time navigation for the Pepper robot in an indoor open-plan office environment without the need for accurate mapping and localization. Our core navigation module can run completely onboard the robot (which has quite limited computing capabilities) at 5 Hz without requiring any external computing resources. We have successfully performed navigation trials over 15 days, visiting more than 50 destinations and traveling more than 1200m with a success rate of over 80%. We discuss remaining challenges and openly share our software.

Dynamic Visual Servoing of A 6-RSS Parallel Robot Based on Optical CMM

The parallel robots exhibit some outstanding properties on the repeatability, stiffness and force-to-weight ratio compared with serial robots. 6-DOF parallel robots have been utilized in various applications and the research on control design has attracted the attentions of researchers. However, the current Cartesian space path tracking performance of the parallel robots cannot meet the growing requirements from industry. In this paper, a dynamic sliding mode control (DSMC) scheme combined with the position-based visual servoing (PBVS) method is proposed to improve the tracking performance of the 6-Revolute-Spherical-Spherical (6-RSS) parallel robot based on the measurements from the optical coordinate measuring machine (CMM) sensor. By employing the CMM sensor, the pose of the parallel robot in Cartesian space can be estimated and incorporated in a closed-loop visual servoing control scheme in real time. The stability of the proposed DSMC has been proved by using Lyapunov theorem. The real-time experiment tests on a 6-RSS parallel robot demonstrate that the complex 6-dimension trajectory tracking can be achieved with high-accuracy. Compared with the classical kinematic level controllers, the proposed DSMC exhibits the superiority in terms of tracking performance and robustness.

Quadrotor going through a window and landing: An image-based visual servo control approach

This paper considers the problem of controlling a quadrotor to go through a window and land on a planar target, the landing pad, using an Image-Based Visual Servo (IBVS) controller that relies on sensing information from two on-board cameras and an IMU. The maneuver is divided into two stages: crossing the window and landing on the pad. For the first stage, a control law is proposed that guarantees that the vehicle will not collide with the wall containing the window and will go through the window with non-zero velocity along the direction orthogonal to the window, keeping at all times a safety distance with respect to the window edges. For the landing stage, the proposed control law ensures that the vehicle achieves a smooth touchdown, keeping at all time a positive height above the plane containing the landing pad. For control purposes, the centroid vectors provided by the combination of the spherical image measurements of a collection of landmarks (corners) for both the window and the landing pad are used as position measurement. The translational optical flow relative to the wall, window edges, and landing plane is used as velocity cue. To achieve the proposed objective, no direct measurements nor explicit estimate of position or velocity are required. Simulation and experimental results are provided to illustrate the performance of the presented controller.

Single landmark feedback-based time optimal navigation for a differential drive robot

In this paper, we propose a feedback-based control approach to execute the time optimal motion trajectories for a differential drive robot. These trajectories are composed of straight lines and rotations in place. We show that the evolution of the position of a single landmark over time, in a local reference frame, makes it possible to track a prescribed time-optimal robot’s trajectory, based on feedback of the landmark’s position. We also show that the closed-loop system is an exponentially stable one with a nonvanishing perturbation, and that globally uniformly ultimately boundedness of the tracking errors can be achieved. The two main results of this work are: 1) Our approach leverages visual servo control type of methods with tools from optimal control for executing time-optimal trajectories in the state space based on feedback information. 2) The approach is able to work with the minimum number of landmarks–only one–this represents a necessary and sufficientcondition for landmark-based navigation. Experiments in a physical robot, a nonholonomic differential drive system equipped with an omnidirectional laser sensor, are shown, which validate the proposed theoretical modelling.

Towards Fully Autonomous Ultrasound Scanning Robot With Imitation Learning Based on Clinical Protocols

Ultrasound scanning plays an important role in modern clinical examinations. Thanks to its small footprint, low cost, and popularity, it has been widely used in annual physical examinations and many other diagnosis and intervention procedures. However, the scanning results depend heavily on the clinician operators' skills, causing inconsistency and even false detection. A fully autonomous ultrasound scanning robot can be a promising solution, enabling consistent and accurate operation. In this letter, we propose a procedure-specified imitation learning framework based on clinical protocols to implement autonomous scanning tasks. Based on the targeted carotid artery examination procedure, we design a feature-based visual servo controller for both in-plane and out-of-plane feature tasks. A one-step-exploring (OSE) method is also designed to improve the robustness and transition smoothness in between tasks. The proposed methods are tested and validated with experiments on both medical phantom and human subjects. The results show that the proposed appraoch is able to significantly improve the success rate of procedure completion from 22.2% to 84.6% for the out-of-plane feature-based operation.