Visual Servo Control Research Articles

Tracking moving target for 6 degree-of-freedom robot manipulator with adaptive visual servoing based on deep reinforcement learning PID controller.

In this paper, an image-based visual servoing (IBVS) controller with a 6 degree-of-freedom robotic manipulator that tracks moving objects is investigated using the proposed Deep Q-Networks and proportional-integral-derivative (DQN-PID) controller. First, the classical IBVS controller and the problem of feature loss and large steady-state error for tracking moving targets are introduced. Then, a DQN-PID based IBVS method is proposed to solve the problem of feature loss and large steady-state error and improve the servo precision, as the existing methods are hard to use for solve the problems. Specifically, the IBVS method is inherited by our controller to build the tracking model, and a value-based reinforcement learning method is proposed as an adaptive law for dynamically tuning the PID parameters in the discrete space, which can track the moving target and keep the servo feature in the field of the camera. Finally, compared with the different existing methods, the DQN-PID based IBVS method has merits of higher accuracy and more stable tracking, or generalization.

Design of UAV target tracking controller based on visual servo

The task of tracking a moving target by the UAV has two major challenges: one is that the multi-angle and multi-scale changes caused by the target’s movement make the target detection more difficult, the other is that the stability and accurate of target tracking are difficult to guarantee. In order to solve these problems, using monocular vision to get the image information of the target and YOLOv4 is used to detect multi-angle and multi-scale moving targets. Then based on the image visual servo control method, the PID controller is designed, and the deviation between the target pixel and the ideal pixel is used as the input, the output of the controller is the desired pixel speed of the UAV in image coordinate system. the tracking speed of the UAV under the machine system is solved according to the image Jacobian matrix, and a tracking strategy is designed to improve the tracking accurate and stability. In the process of target tracking, the difference between the x-axis and Y-axis coordinates of the target center point and the image center point is less than 30 pixels. The results show that the proposed algorithm can keep the target in the center of the image.

A Surgeon Preference-Guided Autonomous Instrument Tracking Method With a Robotic Flexible Endoscope Based on dVRK Platform

In minimally invasive surgery, endoscopes serve as the eyes of surgeon. To avoid fatigue in manual endoscope steering, robotic endoscope holders have been developed. Unfortunately, existing robotic endoscope holders are not widely adopted due to the poor surgeon-robot cooperation. In this work, we developed an intelligent flexible endoscope system based on the da Vinci Research Kit. In the system, surgical instruments are detected and classified in real time with an oriented bounding box-based object detection method. A custom dataset of 6243 images is established to train the detection neural network. Then, a surgeon’s preference guided visual servoing control method is proposed for automatically tracking the detected instruments during minimally invasive surgery. In order to realize 3-degree of freedom control on the image plane, an image moment-based visual servoing control method is adopted. To improve the dynamic performance of the system control, a modified genetic algorithm is developed to select the optimal gains of the robot controller. Both simulation and experimental results show the feasibility of the proposed intelligent flexible endoscope system.

Design of a Gough–Stewart Platform Based on Visual Servoing Controller

Designing a robot with the best accuracy is always an attractive research direction in the robotics community. In order to create a Gough–Stewart platform with guaranteed accuracy performance for a dedicated controller, this paper describes a novel advanced optimal design methodology: control-based design methodology. This advanced optimal design method considers the controller positioning accuracy in the design process for getting the optimal geometric parameters of the robot. In this paper, three types of visual servoing controllers are applied to control the motions of the Gough–Stewart platform: leg-direction-based visual servoing, line-based visual servoing, and image moment visual servoing. Depending on these controllers, the positioning error models considering the camera observation error together with the controller singularities are analyzed. In the next step, the optimization problems are formulated in order to get the optimal geometric parameters of the robot and the placement of the camera for the Gough–Stewart platform for each type of controller. Then, we perform co-simulations on the three optimized Gough–Stewart platforms in order to test the positioning accuracy and the robustness with respect to the manufacturing errors. It turns out that the optimal control-based design methodology helps get both the optimum design parameters of the robot and the performance of the controller {robot + dedicated controller}.

Autonomous, Mobile Manipulation in a Wall-building Scenario: Team LARICS at MBZIRC 2020

In this paper we present our hardware design and control approaches for a mobile manipulation platform used in Challenge 2 of the MBZIRC 2020 competition. In this challenge, a team of UAVs and a single UGV collaborate in an autonomous, wall-building scenario, motivated by construction automation and large-scale robotic 3D printing. The robots must be able, autonomously, to detect, manipulate, and transport bricks in an unstructured, outdoor environment. Our control approach is based on a state machine that dictates which controllers are active at each stage of the Challenge. In the first stage our UGV uses visual servoing and local controllers to approach the target object without considering its orientation. The second stage consists of detecting the object’s global pose using OpenCV-based processing of RGB-D image and point-cloud data, and calculating an alignment goal within a global map. The map is built with Google Cartographer and is based on onboard LIDAR, IMU, and GPS data. Motion control in the second stage is realized using the ROS Move Base package with Time-Elastic Band trajectory optimization. Visual servo algorithms guide the vehicle in local object-approach movement and the arm in manipulating bricks. To ensure a stable grasp of the brick’s magnetic patch, we developed a passively-compliant, electromagnetic gripper with tactile feedback. Our fully-autonomous UGV performed well in Challenge 2 and in post-competition evaluations of its brick pick-and-place algorithms.

Autonomous Mobile Robot Implemented in LEGO EV3 Integrated with Raspberry Pi to Use Android-Based Vision Control Algorithms for Human-Machine Interaction

Robotic applications, such as educational programs, are well-known. Nonetheless, there are challenges to be implemented in other settings, e.g., mine detection, agriculture support, and tasks for industry 4.0. The main challenge consists of robotic operations supported by autonomous decision using sensed-based features extraction. A prototype of a robot assembled using mechanical parts of a LEGO MINDSTORMS Robotic Kit EV3 and a Raspberry Pi controlled through servo algorithms of 2D and 2D1/2 vision approaches was implemented to tackle this challenge. This design is supported by simulations based on image, position, and a hybrid scheme for visual servo controllers. Practical implementation is operated using navigation guided by running up image-based visual servo control algorithms embedded in a Raspberry Pi that uses a control criterion based on error evolution to compute the difference between a target and sensed image. Images are collected by a camera installed on a mobile robotic platform manually and automatically operated and controlled using the Raspberry Pi. An Android application to watch the images by video streaming is shown here, using a smartphone and a video related to the implemented robot’s operation. This kind of robot might be used to complete field reactive tasks in the settings mentioned above, since the detection and control approaches allow self-contained guidance.

Visual servoing with deep reinforcement learning for rotor unmanned helicopter

Visual servoing is a key approach to achieve visual control for the rotor unmanned helicopter. The challenges of the inaccurate matrix estimation and the target loss restrict the performance of the visual servoing control systems. This work proposes a novel visual servoing controller using the deep Q-network to achieve an efficient matrix estimation. A deep Q-network learning agent learns a policy estimating the interaction matrix for visual servoing of a rotor unmanned helicopter using continuous observation. The observation includes a combination of feature errors. The current matrix and the desired matrix constitute the action space. A well-designed reward guides the deep Q-network agent to get a policy to generate a time-varying linear combination between the current matrix and the desired matrix. Then, the interaction matrix is calculated by the linear combination. The potential mapping between the observation and the interaction matrix is learned by cascading the deep neural network layers. Experimental results show that the proposed method achieves faster convergence and lower target loss probability in tracking than the visual servoing methods with the fixed parameter.

Homography-Based Visual Servoing for Underactuated VTOL UAVs Tracking a 6-DOF Moving Ship

This paper develops a novel homography-based visual servo control method for the vertical take-off and landing (VTOL) unmanned aerial vehicle (UAV) tracking the trajectory of a 6 degrees of freedom (6-DOF) moving ship. Different from the classical homography-based visual servoing which is only suitable for the trajectory tracking of a planar moving target, an extended homography-based visual servoing framework is proposed to track the trajectory of a 6-DOF moving target regardless of its time-varying rotational motion. Taking entries in the homography matrix as feedback, the visual dynamics of UAV is established and decoupled into a translational motion sub-dynamics and an attitude sub-dynamics based on the hierarchical control strategy. In the translational motion subsystem, the robust controller is designed by embedding the nonlinear differentiator, the adaptive method, and a smooth saturation model in the backstepping control scheme. The saturation model is introduced to generate the constrained control inputs to ensure the nonsingular attitude extraction and help the visual points remain in the camera’s field of view (FoV) simultaneously. In the attitude subsystem, a robust state-constrained control method is proposed by utilizing a robust control barrier function (RCBF), the quadratic programming (QP) and a saturation model to guarantee the visibility, where RCBF can accommodate unknown dynamics. The theoretical analysis demonstrates the asymptotic stability of the closed-loop system. Simulations are carried out to further validate the controller performance.

Image-Feature-Based Integrated Path Planning and Control for Spacecraft Rendezvous Operations

This paper develops an integrated path planning and control framework for a specific problem of image-feature-based spacecraft rendezvous with an uncooperative target. For visibility concerns, the chasing spacecraft is required to approach and synchronize with the tumbling target, as well as to retain the target within the camera’s field of view. To this end, an offline path planner is designed by using the gradient of a so-called navigation function to generate a set of desired image trajectories. A novel formulation of the spacecraft relative dynamic model is presented based on current image features and the generated desired ones. To track the desired image trajectories, a robust image-based visual servoing controller is designed via passivity theorem. The stability of the closed-loop system is analyzed using the Lyapunov method. In addition, a hyperbolic tangent function is introduced to cope with the chattering phenomenon caused by the signum function. A scenario of rendezvousing with the defunct Envisat is demonstrated to validate the effectiveness and advantages of the proposed control scheme. Comparative studies with the conventional image-based and pose-based control schemes are performed through numerical simulations, in order to evaluate the target visibility, control accuracy, and fuel consumption under the proposed control scheme.

Adaptive neural network control for image‐based visual servoing of robot manipulators

This paper presents a novel adaptive neural network control strategy for image-based visual servoing (IBVS) of robotic manipulators with both eye-in-hand and eye-to-hand camera configurations in the presence of unknown dynamics and external disturbances. The IBVS method is combined with the adaptive neural network to construct the proposed adaptive neural network controller to solve the visual servoing control problem of robots. The adaptive neural network based IBVS controller is designed based on the depth-independent interaction matrix, which can be trained on-line to identify the visual servoing robotic system modeling errors. Moreover, the proposed method can approach the unknown nonlinear dynamics for both eye-in-hand and eye-to-hand camera configurations without requiring the robot dynamics to be linearly parameterizable, and the exact knowledge of the robot structure is not needed. On the basis of the nonlinear robot dynamics, the Lyapunov stability analysis is given to prove the asymptotical convergence of the image position and velocity errors. Simulation results for both camera configurations are provided to demonstrate the performance of the proposed adaptive neural network based approach.

Robotic manipulation based on 3-D visual servoing and deep neural networks

The critical challenge, for robot–object-interaction, is to estimate visually the pose of the target object in a 3D space and combine it into a vision-based control scheme in manipulation applications. This paper proposes a novel reliable framework for deep ConvNet combined with visual servoing using a single RGB camera. We introduce an extensive system called Deep-Visual-Servoing (DVS) that addresses an integration of: (I) training of deep-CNNs using synthetic dataset only and operates successfully in real-world scenario, (II) continuous 3 D object pose estimation as the sensing feedback in a 3D visual servoing control scheme, and (III) design, integration and experimentation of visual servoing approach based on Lyapunov’s theory. The proposed deep based learning approach, the kinematic modeling and controller design are experimentally verified and discussed using the 6 DOF UR5 manipulator.

An onboard-eye-to-hand visual servo and task coordination control for aerial manipulator based on a spherical model

Compared with the unmanned aerial vehicle (UAV), the aerial manipulator extends the ability to interact with the environment by using the onboard manipulator. The complex dynamic characteristics of the combined system increase the challenges of accurately control the aerial manipulator to approximate the target object. In this paper, an onboard-eye-to-hand visual servo and task coordination control strategy is proposed for the aerial manipulator to enhance the accurate manipulation and flight stability. By establishing a new spherical coordinate error equation, the flying platform (quadrotor) and onboard manipulator can be controlled simultaneously. Meanwhile, the multi-task coordinated control scheme is adopted to achieve precise positioning and grasping of the target object, including estimating the object pose, dynamically compensating the change of the center of gravity of the aerial manipulator, and avoiding the onboard manipulator joint limitations. The stability of the closed-loop system is analyzed by the Lyapunov method. Finally, the feasibility of the proposed visual servo method is verified in simulations and real-world outdoor experiments, respectively. The strengths of the proposed method are demonstrated by comparing it with the conventional visual servo method.

Mix Frame Visual Servo Control Framework for Autonomous Assistive Robotic Arms.

Assistive robotic arms (ARAs) that provide care to the elderly and people with disabilities, are a significant part of Human-Robot Interaction (HRI). Presently available ARAs provide non-intuitive interfaces such as joysticks for control and thus, lacks the autonomy to perform daily activities. This study proposes that, for inducing autonomous behavior in ARAs, visual sensors integration is vital, and visual servoing in the direct Cartesian control mode is the preferred method. Generally, ARAs are designed in a configuration where its end-effector’s position is defined in the fixed base frame while orientation is expressed in the end-effector frame. We denoted this configuration as ‘mixed frame robotic arms’. Consequently, conventional visual servo controllers which operate in a single frame of reference are incompatible with mixed frame ARAs. Therefore, we propose a mixed-frame visual servo control framework for ARAs. Moreover, we enlightened the task space kinematics of a mixed frame ARAs, which led us to the development of a novel “mixed frame Jacobian matrix”. The proposed framework was validated on a mixed frame JACO-2 7 DoF ARA using an adaptive proportional derivative controller for achieving image-based visual servoing (IBVS), which showed a significant increase of 31% in the convergence rate, outperforming conventional IBVS joint controllers, especially in the outstretched arm positions and near the base frame. Our Results determine the need for the mixed frame controller for deploying visual servo control on modern ARAs, that can inherently cater to the robotic arm’s joint limits, singularities, and self-collision problems.

A new decoupled control law for image-based visual servoing control of robot manipulators

A new decoupled control law for image-based visual servoing control of robot manipulators

Autonomous Landing System of a VTOL UAV on an Upward Docking Station Using Visual Servoing

In this paper, we proposed a novel autonomous system of unmanned aerial vehicles docking station that consists of electromagnet docking structure, auto-landing, and visual servo control. Comparing with the conventional downward docking system, we develop an upward docking system that could disregard ground effect and improve the utility of limited operating space. The design of visual servo control enables UAVs to maintain stability and perform docking only relying on the detection of the ArUco marker as position feedback. We show the hardware design of the electromagnet docking station and the experiment demonstrates the successful docking process in different flight modes and upward docking.

A Control Method for SMA Robotic Actuators

Soft robots show remarkable benefits over conventional rigid robots due to their high energy density and other factors. We propose a circular soft robot integrating the control system, actuating it with several shape memory alloy (SMA) actuators. Our research methodology involved connecting the DC voltage supply and L298N module to provide uninterrupted power to the actuator and thence control the actuator via Arduino Uno MCU and TOF camera. We designed the controller to simultaneously complete the positioning and manipulation tasks. A novel method utilizing visual servo and closed-loop control algorithm was proposed and integrated into the controller. This method involves the implementation of multi-gait locomotion using SMA actuators. Additionally, the development of closed-loop dynamic controllers for a continuous soft robot is also evaluated. The proposed control model is designed and simulated on the MATLAB tool. To verify the efficiency of the proposed forward-feedback controller, simulations and experiments were conducted in the current study. A new control method using PID control based on the Kalman filtering algorithm and visual servo for the SMA actuator designed in this research is introduced. We conclude that applying spike excitation voltage would benefit the actuating performance. Overall, the experimental results demonstrated a promising future for the purposed control method.

Completely Distributed Time-Varying Formation Target Tracking for Quadrotor Team via Image-Based Visual Servoing

In this article, the completely distributed adaptive time-varying formation target tracking control problem for quadrotor team via image-based visual servoing is addressed. An adaptive cascade formation controller is proposed, which consists of a trajectory tracking controller, an attitude controller, and an image-based visual servoing controller. All the controllers consist of a nominal part to achieve the desired performance and a robust part to retrain the influence of parameter uncertainties and external disturbance. The robust properties of the closed-loop control system are proven via the theoretical analysis. Moreover, the simulation results validate the effectiveness of the proposed formation tracking control protocol in this paper.

Visual Servo Control of Endoscope-Holding Robot Based on Multi-Objective Optimization: System Modeling and Instrument Tracking

Visual Servo Control of Endoscope-Holding Robot Based on Multi-Objective Optimization: System Modeling and Instrument Tracking

Automated real-time 3D visual servoing control of single cell surgery with application to microinjection processes

Automated real-time 3D visual servoing control of single cell surgery with application to microinjection processes

Automated real-time 3D visual servoing control of single cell surgery with application to microinjection processes

Automated real-time 3D visual servoing control of single cell surgery with application to microinjection processes