Visual Servoing Control Scheme Research Articles

Image-Based Indoor Topological Navigation with Collision Avoidance for Resource-Constrained Mobile Robots

This paper presents a complete topological navigation system for a resource-constrained mobile robot like Pepper, based on image memory and the teach-and-repeat paradigm. Image memory is constructed from a set of reference images that are acquired during a prior mapping phase and arranged topologically. A* search is used to find the optimal path between the current location and the destination. The images from the robot’s RGB camera are used to localize within the topological graph, and an Image-Based Visual Servoing (IBVS) control scheme drives the robot to the next node in the graph. Depth images update a local egocentric occupancy grid, and another IBVS controller navigates local free-space. The output of the two IBVS controllers is fused to form the final control command for the robot. We demonstrate real-time navigation for the Pepper robot in an indoor open-plan office environment without the need for accurate mapping and localization. Our core navigation module can run completely onboard the robot (which has quite limited computing capabilities) at 5 Hz without requiring any external computing resources. We have successfully performed navigation trials over 15 days, visiting more than 50 destinations and traveling more than 1200m with a success rate of over 80%. We discuss remaining challenges and openly share our software.

Dynamic Visual Servoing of A 6-RSS Parallel Robot Based on Optical CMM

The parallel robots exhibit some outstanding properties on the repeatability, stiffness and force-to-weight ratio compared with serial robots. 6-DOF parallel robots have been utilized in various applications and the research on control design has attracted the attentions of researchers. However, the current Cartesian space path tracking performance of the parallel robots cannot meet the growing requirements from industry. In this paper, a dynamic sliding mode control (DSMC) scheme combined with the position-based visual servoing (PBVS) method is proposed to improve the tracking performance of the 6-Revolute-Spherical-Spherical (6-RSS) parallel robot based on the measurements from the optical coordinate measuring machine (CMM) sensor. By employing the CMM sensor, the pose of the parallel robot in Cartesian space can be estimated and incorporated in a closed-loop visual servoing control scheme in real time. The stability of the proposed DSMC has been proved by using Lyapunov theorem. The real-time experiment tests on a 6-RSS parallel robot demonstrate that the complex 6-dimension trajectory tracking can be achieved with high-accuracy. Compared with the classical kinematic level controllers, the proposed DSMC exhibits the superiority in terms of tracking performance and robustness.

Saturated Visual-Servoing Control Strategy for Nonholonomic Mobile Robots With Experimental Evaluations

A novel saturated visual control strategy for nonholonomic mobile robots is presented. The principal focus is to deal with the stabilization problem. The saturation bounds for the control actions avoid exceeding the physical restrictions of the actuators. This feature is the main difference concerning related works. The closed-loop system is analyzed using the theory of non-autonomous cascade systems, and global uniform asymptotic stability is guaranteed. Experimental validation is carried out by using a unicycle-type mobile robot with an onboard camera. The experiment consists of stabilizing the robot in a preset pose using visual information. The experimental results are compared with a literature unsaturated visual servoing strategy. The results confirm the effectiveness of the proposed approach, completing the assigned task and bounding the control inputs.

Automated 3-D Deformation of a Soft Object Using a Continuum Robot

This study investigates the use of a tendon-driven continuum robot to deform a soft object, whereas the robot body is deformed into an arbitrary shape to adapt to a constrained environment. A dynamic estimator (DE) is developed to approximate the Jacobian matrix that associates the actuator input with the deformed output of the soft object. This helps solve the singularity problem and reduce the effects of noise. Then a visual predictive controller (VPC) with a reference trajectory is developed to ensure a smooth operation. A linear extended-state observer (ESO) is further designed to measure the robot states, such that the controller can compensate for the estimation error. Simulations and experiments are performed to verify the proposed control approach. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —The motivation of this article is to solve the problem of automatic deformation control of soft objects in restricted environments. The existing soft object deformation control is achieved using rigid robots in an open environment, but rigid robots are difficult to use in specific applications where the environment is restricted (e.g., natural orifice surgery). Flexible continuum robots with mechanical compliance can manipulate soft objects in narrow spaces. However, due to environmental constraints, the robot body may be deformed into any shape regardless of the input of the actuator. To solve the problem, this research provides a new visual servo control strategy that deforms soft objects using a continuum robot in a restricted environment. The proposed method can control a flexible robot to manipulate soft objects while taking into account the change in the robot configuration in a restricted environment.

Visual servo optimization stabilization of nonholonomic mobile robots based on control Lyapunov functions

This study considers the visual stabilization problem of nonholonomic mobile robots and proposes a novel optimization stabilization method for visual servo control of nonholonomic mobile robots with monocular cameras fixed onboard. The main idea of the method is to utilize control Lyapunov functions of discrete-time nonlinear systems to design a family of explicit stabilization control laws of the visual servo error system. The parameters of the control laws can indirectly reflect the performance of the visual servo controllers. Then taking account of visibility constraints and actuator limitations, a set of optimal parameters of the control laws is calculated by offline solving a constrained finite horizon optimal control problem. Moreover, the stabilization results on the optimal visual servo controller are established based on the properties of control Lyapunov functions. Finally, some simulation experiments are used to illustrate and evaluate the performance of the visual servo control scheme proposed here.

Adaptive Image-Based Leader–Follower Formation Control of Mobile Robots With Visibility Constraints

This article addresses the problem of leader-follower formation control of mobile robots using only onboard monocular cameras that subjected to visibility constraints. An adaptive image-based visual servoing control strategy was proposed following the prescribed performance control methodology. First, the leader-follower visual kinematics in the image plane and an error transformation with predefined performance specifications are presented. Then, an adaptive control law with online estimating the inverse height between the optical center of camera and the single feature point attached to leader is designed to ensure the global stability of the closed-loop system. Finally, the applicability and performance of the proposed control scheme are demonstrated by the numerical simulations and hardware experiments. Compared with other formation control schemes, our solution relies only on onboard visual sensors without communication, since it does not need the relative angle/distance between the robots, or the velocity of the leader. Moreover, it guarantees the prescribed transient and the steady-state performance besides the visibility constraints.

Catenary-based visual servoing for tether shape control between underwater vehicles

This paper deals with the shape control of a passive tether that links underwater vehicles. The pair of robots is composed of a leader that explores the environment, and a follower that is in charge of regulating the tether shape by moving appropriately. Both robots are equipped with standard low-cost sensors: cameras, depth sensor and inertial measurement unit (IMU). This work introduces an innovative visual servoing control scheme based on a parameterized catenary model of the tether, assuming that it is negatively buoyant. The tether is detected in the image through color segmentation and its 3-D shape parameters are estimated through a nonlinear optimization procedure thanks to an adapted Gauss–Newton algorithm. Simulations and experimental results show that the relative positioning of the robots can be achieved through simultaneous tether shape visual servoing in both leader and follower’s cameras, even in situations of tether partial visibility and strong disturbances, which proves the approach to be reliable.

Image based fixed time visual servoing control for the quadrotor UAV

In this study, the visual servoing control problem is investigated, and a fixed time based visual servoing control scheme is proposed for quadrotor unmanned aerial vehicles. First, based on the perspective projection principle and target points, effective image features are defined on an image plane called the virtual image plane. Then, the image feature dynamics and QUAV dynamics are combined to derive a coupled non-linear system, and the non-linear system is decoupled into two subsystems under inner-outer loop control framework. Furthermore, the novel sliding mode surfaces based fixed-time controllers are designed, and the resulting closed-loop systems are proved to be stable. Finally, the numerical simulations and experiments are conducted to validate the effectiveness of the proposed control scheme.

Dynamic Linear Predictive Optimization of Flexible Robot Profiling MFA Model

As the airbag of a flexible robot is affected by external environmental factors during the profiling process, there are many uncertainties in the process of deformation of the airbag. For this reason, the general nonlinear control strategy cannot obtain an accurate data model. In this paper, a flexible robot profiling MFA (Model-Free Adaptive) model based on adaptive predictive dynamic linear optimization is proposed. Firstly, the real-time thickness of the airbag is obtained through edge detection by using the image processing algorithm. Secondly, the airbag aerodynamic model is constructed by visual servo control strategy. Then, a nonlinear control system based on model-free adaptive control is established. Thirdly, the weighting factor is used to limit the variation range of the input quantity, and the deviation of the actual value and the expected value is corrected by the adaptive prediction mechanism. Finally, the servo control the airbag is completed. The experimental results show that the improved model proposed in this paper solves the overshoot phenomenon of the standard control model with less control error and higher robustness.

Vision-Based Navigation of Omnidirectional Mobile Robots

This letter considers the problem of collision-free navigation of omnidirectional mobile robots in environments with obstacles. Information from a monocular camera, encoders, and an inertial measurement unit is used to achieve the task. Three different visual servoing control schemes, compatible with the class of considered robot kinematics and sensor equipment, are analyzed and their robustness properties with respect to actuation inaccuracies discussed. Then, a controller is proposed with formal guarantee of convergence to the bisector of a corridor. The main controller components are a visual servoing control scheme and a velocity estimation algorithm integrating visual, kinematic, and inertial information. The behavior of all the considered algorithms is analyzed and illustrated through simulations both for a wheeled and a humanoid robot. The solution proposed as the most efficient and robust with respect to actuation inaccuracies is also validated experimentally on a real humanoid NAO.

Automated Laser Ablation of Motile Sperm for Immobilization

Automated manipulation of single cells is required in both biological and clinical applications. In clinical infertility treatments, a single motile sperm is immobilized and inserted into an egg cell for in vitro fertilization. Sperm immobilization is essential to ease the ensuing pick-up procedure, and importantly, it prevents the sperm tail from beating inside the egg cell, which causes a lower fertilization rate. For immobilizing a motile sperm, the sperm tail must be accurately positioned and aligned with the manipulation tool (e.g., laser spot). Manual immobilization using laser ablation has stringent skill requirements, and is not able to accurately position the sperm tail to the center of the laser spot for immobilization. This letter presents a visual servo system that is capable of accurately positioning the tail of a motile sperm relative to the laser spot for automated sperm immobilization. A visual servo control strategy was developed to estimate and compensate for the motion of the sperm tail. Experimental results showed that the visual servo controller achieved a positioning accuracy of 1.7 $\mu$ m, independent of sperm speed or swimming direction. By quantitatively evaluating the effect of laser energy on sperm immobilization, a consistent immobilization success rate of 100% was achieved (based on experiments on 900 sperms) with a throughput five times that of manual operation. Experimental results confirmed that this automated immobilization technique did not induce damage to sperm DNA.

An Optimized Image-Based Visual Servo Control for Fixed-Wing Unmanned Aerial Vehicle Target Tracking With Fixed Camera

This paper presents an optimized image-based visual servo (IBVS) control scheme for tracking a ground target by using a fixed-wing unmanned aerial vehicle (UAV) with a monocular camera fixed on it. Unlike the widely used rotor UAVs, the fixed-wing UAV has much more dynamical constraints such as it cannot move omni-direction and its minimum speed is limited by the stalling speed. This makes the target tracking problem more challenging. The proposed scheme leverages the image Jacobian matrix to build a connection between the velocity of the feature point and that of the UAV. Afterward, considering the camera is fixed on the body of the UAV, an “ideal camera” model is proposed to compensate the shifts of the feature point caused by the changes of the UAV's attitude. Then, an optimized control law without solving the pseudo-inverse of the image Jacobian matrix is proposed with the aid of the least square method from the target center in the image coordinate system. This control law takes the velocity of the feature point as inputs and the yaw angular velocity of the UAV as outputs. The stability of the proposed law is analyzed with the Lyapunov method, showing that the UAV will circle around the ground target asymptotically. Finally, the proposed scheme is evaluated by a hardware-in-the-loop (HIL) simulation based on the Gazebo simulator and the off-the-shelf autopilot hardware.

Adaptive visual servoing control for an underwater soft robot

PurposeSoft robotics, regarded as a new research branch of robotics, has generated increasing interests in this decade and has demonstrated its outperformance in addressing safety issues when cooperating with human beings. However, there is still lack of accurate close-loop control because of the difficulty in acquiring feedback information and accurately modeling the system, especially in interactive environments. To this end, this paper aims to improve the controllability of the soft robot working in specific underwater environment. The system dynamics, which takes complicated hydrodynamics into account, is solved using Kane’s method. The dynamics-based adaptive visual servoing controller is proposed to realize accurate sensorimotor control.Design/methodology/approachThis paper presents an image-based visual servoing control scheme for a cable-driven soft robot with a fixed camera observing the motions. The intrinsic and extrinsic parameters of the camera can be adapted online so that tedious camera calibration work can be eliminated. It is acknowledged that kinematics-based control can be only applied into tasks in the free space and has limitation in accelerating the motion speed of robot arms. That is, one must consider the unneglectable interaction effects generated from the environment and objectives when operating soft robots in such interactive control tasks. To extend the application of soft robots into underwater environment, the study models system dynamics considering complicated hydrodynamic effects. With the pre-knowledge of the external effects, the performance of the robot can be further improved by adding the compensation term into the controller.FindingsThe proposed controller has theoretically proved its convergence of image error, adaptive estimation error and the stability of the dynamical system based on Lyapunov’s analysis. The authors also validate the performance of the controller in positioning control task in an underwater environment. The controller shows its capacity of rapid convergence to and accurate tracking performance of a static image target in a physical experiment.Originality/valueTo the best of the authors’ knowledge, there is no such research before that has developed dynamics-based visual servoing controller which takes into account the environment interactions. This work can thus improve the control accuracy and enhance the applicability of soft robotics when operating in complicated environments.

Robotic Immobilization of Motile Sperm for Clinical Intracytoplasmic Sperm Injection.

In clinical intracytoplasmic sperm injection (ICSI), a motile sperm must be immobilized before insertion into an oocyte. This paper aims to develop a robotic system for automated tracking, orientation control, and immobilization of motile sperms for clinical ICSI applications. We adapt the probabilistic data association filter by adding sperm head orientation into state variables for robustly tracking the sperm head and estimating sperm tail positions under interfering conditions. The robotic system also utilizes a motorized rotational microscopy stage and a new visual servo control strategy that predicts and compensates for sperm movements to actively adjust sperm orientation for immobilizing a sperm swimming in any direction. The system robustly tracked sperm head with a tracking success rate of 96.0% and estimated sperm tail position with an accuracy of 1.08 μm under clinical conditions where the occlusion of the target sperm and interference from other sperms occur. Experimental results from robotic immobilization of 400 sperms confirmed that the system achieved a consistent immobilization success rate of 94.5%, independent of sperm velocity or swimming direction. Our adapted tracking algorithm effectively distinguishes the target sperm from interfering sperms. Predicting and compensating for sperm movements significantly reduce the positioning error during sperm orientation control. These features make the robotic system suitable for automated sperm immobilization. The robotic system eliminates stringent skill requirements in manual sperm immobilization. It is capable of manipulating sperms swimming in an arbitrary direction with a high success rate.

Image Based Visual Servoing for Omnidirectional Wheeled Mobile Robots in Voltage Mode

This paper presents a new image based visual servoing (IBVS) control scheme for omnidirectional wheeled mobile robots with four swedish wheels. The contribution is the proposal of a scheme that consider the overall dynamic of the system; this means, we put together mechanical and electrical dynamics. The actuators are direct current (DC) motors, which imply that the system input signals are armature voltage applied to DC motors. In our control scheme the PD control law and eye-to-hand camera configuration are used to compute the armature voltages and to measure system states, respectively. Stability proof is performed via Lypunov direct method and LaSalle's invariance principle. Simulation and experimental results were performed in order to validate the theoretical proposal and to show the good performance of the posture errors.

Fall monitoring and detection for at-risk persons using a UAV

We describe a demonstrator application that uses a UAV to monitor and detect falls of an at-risk person. The position and state (upright or fallen) of the person are determined with deep-learning-based computer vision, where existing network weights are used for position detection, while for fall detection the last layer is fine-tuned in additional training. A simple visual servoing control strategy keeps the person in view of the drone, and maintains the drone at a set distance from the person. In experiments, falls were reliably detected, and the algorithm was able to successfully track the person indoors.

Visual Servoing of Constrained Mobile Robots Based on Model Predictive Control

This paper develops an image-based visual servoing (IBVS) control strategy using model predictive control (MPC) to stabilize a physically constrained mobile robot. In IBVS strategy, ambiguity, and degeneracy problems of the homography and fundamental matrix-based algorithms can be avoided. Moreover, a synthetic error vector incorporating the advantages of IBVS and position-based visual servoing is defined that includes both the robot angle and image coordinates. By using linear system control theory, the kinematics of nonholonomic chained robotic systems can be transformed into a skew-symmetric form, and through introducing an exponential decay phase, the uncontrollable problem can be solved. Then, an MPC strategy is developed and, thereafter, iteratively transformed into a constrained quadratic programming (QP) problem. Subsequently, we utilize a primal-dual neural network (PDNN) to solve this QP problem. By using PDNN optimization, the cost function of MPC effectively converges to the exact optimal values. Finally, experimental studies on the actual robotic systems have been conducted to demonstrate the performance of the proposed approach.

Survey of Visual Servoing Control Schemes for Mobile Robot Navigation

In this paper, visual servoing control scheme for mobile robot navigation is discussed. The survey introduces a brief discussion of the subjects of the system under consideration. These subjects include mobile robot platform kinematics, vision systems, visual servoing, mapping, navigation, automatic control, and finally the avoidance techniques used to avoid obstacle during the navigation process. Different vision based navigation systems will be discussed and a comparative study will be given.

Uniform ultimate bounded robust model reference adaptive PID control scheme for visual servoing

This paper proposes a uniform ultimate bounded (UUB) controller framework using model reference adaptive control for visual servoing of the ball on plate system. To address the major challenges in designing a control scheme for visual servoing applications including inter-axis coupling, exogenous disturbances, and plant perturbations due to modelling errors, a robust model reference adaptive PID control scheme using e1 modification method is put forward. The key advantage of this methodology is its ability to yield asymptotic stability of the closed loop system without prior information on the plant perturbations. Moreover, exploiting the Erzberger׳s perfect model following condition, the algorithm obtains the pseudo inverse of the system to make the system track different test trajectories. The stability and convergence of the proposed scheme are proved using Schwarz׳s inequality condition and Frobenius norms. To evaluate the tracking performance, two test cases namely reference following during exogenous disturbance and tracking under plant perturbations are validated. Simulation results accentuate that the proposed scheme yields satisfactory tracking response even during plant perturbation and exogenous disturbances.

Autonomous robotic capture of non-cooperative target by adaptive extended Kalman filter based visual servo

This paper presents a real-time, vision-based algorithm for the pose and motion estimation of non-cooperative targets and its application in visual servo robotic manipulator to perform autonomous capture. A hybrid approach of adaptive extended Kalman filter and photogrammetry is developed for the real-time pose and motion estimation of non-cooperative targets. Based on the pose and motion estimates, the desired pose and trajectory of end-effector is defined and the corresponding desired joint angles of the robotic manipulator are derived by inverse kinematics. A close-loop visual servo control scheme is then developed for the robotic manipulator to track, approach and capture the target. Validating experiments are designed and performed on a custom-built six degrees of freedom robotic manipulator with an eye-in-hand configuration. The experimental results demonstrate the feasibility, effectiveness and robustness of the proposed adaptive extended Kalman filter enabled pose and motion estimation and visual servo strategy.