Image-based Visual Servo Control Research Articles

High-Precision Drilling by Anchor-Drilling Robot Based on Hybrid Visual Servo Control in Coal Mine

Rock bolting is a commonly used method for stabilizing the surrounding rock in coal-mine roadways. It involves installing rock bolts after drilling, which penetrate unstable rock layers, binding loose rocks together, enhancing the stability of the surrounding rock, and controlling its deformation. Although recent progress in drilling and anchoring equipment has significantly enhanced the efficiency of roof support in coal mines and improved safety measures, how to deal with drilling rigs’ misalignment with the through-hole center remains a big issue, which may potentially compromise the quality of drilling and consequently affect the effectiveness of bolt support or even result in failure. To address this challenge, this article presents a robotic teleoperation system alongside a hybrid visual servo control strategy. Addressing the demand for high precision and efficiency in aligning the drilling rigs with the center of the drilling hole, a hybrid control strategy is introduced combining position-based and image-based visual servo control. The former facilitates an effective approach to the target area, while the latter ensures high-precision alignment with the center of the drilling hole. The robot teleoperation system employs the binocular vision measurement system to accurately determine the position and orientation of the drilling-hole center, which serves as the designated target position for the drilling rig. Leveraging the displacement and angle sensor information installed on each joint of the manipulator, the system utilizes the kinematic model of the manipulator to compute the spatial position of the end-effector. It dynamically adjusts the spatial pose of the end-effector in real time, aligning it with the target position relative to its current location. Additionally, it utilizes monocular vision information to fine-tune the movement speed and direction of the end-effector, ensuring rapid and precise alignment with the target drilling-hole center. Experimental results demonstrate that this method can control the maximum alignment error within 7 mm, significantly enhancing the alignment accuracy compared to manual control. Compared with the manual control method, the average error of this method is reduced by 41.2%, and the average duration is reduced by 4.3 s. This study paves a new path for high-precision drilling and anchoring of tunnel roofs, thereby improving the quality and efficiency of roof support while mitigating the challenges associated with significant errors and compromised safety during manual control processes.

Uncalibrated and Unmodeled Image-Based Visual Servoing of Robot Manipulators Using Zeroing Neural Networks.

Neural networks have been widely investigated for the control of robot manipulators and recurrent neural network (RNN) is accepted as a powerful tool for visual servoing. Different from existing control schemes for robot-camera systems, this article proposes a novel image-based visual servoing (IBVS) control scheme for both the regulation and tracking control of robot manipulators in the framework of a special class of RNN, termed zeroing neural network (ZNN), which does not require prior knowledge about camera configuration and kinematic model parameters. The proposed control scheme is composed of a data-driven mapping estimator and a controller, both of which are designed based on ZNN. To facilitate the deployment of the proposed IBVS control scheme, a discrete-time version of the proposed control scheme is developed. Theoretical analysis for the proposed method is presented in terms of convergence, stability, and robustness. In addition, simulations and experiments are carried out based on different types of robot-camera systems to verify the efficacy and portability of the proposed control scheme for solving regulation and trajectory IBVS problems. Moreover, comparative studies are performed to reveal the merits of the proposed control scheme.

Vision-based UAV adaptive tracking control for moving targets with velocity observation

An adaptive image-based visual servoing (IBVS) controller is designed for a quadrotor unmanned aerial vehicle (UAV) to achieve robust tracking for moving targets, under underactuation and tight coupling constraints of UAV kinematics. Specifically, image features are selected from perspective image moments of a planar target to obtain virtual feature dynamics regarding UAV kinematics and dynamics. By constructing an auxiliary variable, a translational velocity observer for moving target is constructed by using virtual image features. An IBVS tracking controller is designed without target geometric information by combining UAV and visual feature dynamics. Designed controller and observer make the UAV robustly reach desired height and track the moving target, despite uncertainty of target movement. The controller has asymptotical convergence performance, and the target velocity is observed according to Lyapunov stability analysis. Simulation and experimental results show that the proposed method has smoother and more accurate performance in motion tracking and target velocity prediction under system uncertainty.

Learning an Image-Based Visual Servoing Controller for Object Grasping

Adaptive and cooperative control of arms and fingers for natural object reaching and grasping, without explicit 3D geometric pose information, is observed in humans. In this study, an image-based visual servoing controller, inspired by human grasping behavior, is proposed for an arm-gripper system. A large-scale dataset is constructed using Pybullet simulation, comprising paired images and arm-gripper control signals mimicking expert grasping behavior. Leveraging this dataset, a network is directly trained to derive a control policy that maps images to cooperative grasp control. Subsequently, the learned synergy grasping policy from the network is directly applied to a real robot with the same configuration. Experimental results demonstrate the effectiveness of the algorithm. Videos can be found at https://www.bilibili.com/video/BV1tg4y1b7Qe/ .

Image-based visual servoing with Kalman filter and swarm intelligence optimisation algorithm

The article proposes a new Kalman depth estimation and an improved swarm intelligence optimisation algorithm for adaptive tuning of servo gain for image-based visual servo control. First, a Kalman depth estimation model is established from the principle of image-based visual servoing, and two state equations are designed for depth estimation based on the number of state quantities. Second, the improved sparrow search algorithm is proposed to tune the servo gain adaptively to improve the convergence speed and stability. To verify the effectiveness of the proposed method, the conventional image-based visual servoing and conventional Kalman estimation are reproduced and compared with the proposed method, and the simulation is completed on the Simulink simulation platform for verification. Finally, the experiments are completed in the robotic arm experimental platform. Both the simulation and experimental results show the effectiveness of the proposed method, which reduces the redundancy of the camera and shortens the convergence time.

Neural Network-Based Robust Guaranteed Cost Control for Image-Based Visual Servoing of Quadrotor.

In this article, a neural network (NN)-based robust guaranteed cost control design is proposed for image-based visual servoing (IBVS) control of quadrotors. According to the dynamics of three subsystems (yaw, height, and lateral subsystems) derived from the quadrotor IBVS dynamic model, the main control design is to solve the robust control problem for the time-varying lateral subsystem with angle constraints and uncertain disturbances. Considering the system dynamics, a two-loop structure is conducted. The outer loop uses the linear quadratic regulator to solve the Riccati equation for the lateral image feature system, and the inner loop adopts the optimal robust guaranteed cost control to solve the lateral velocity system. For the lateral velocity system, the optimal robust control problem is transformed to solve the modified Hamilton-Jacobi-Bellman equation of the corresponding optimal control problem utilizing adaptive dynamic programming. The implementation is accomplished with the time-varying NN and the designed estimated weight update law. In addition, the stability and effectiveness are proved by the theoretic proof and simulations.

Data-Driven Model Predictive Control for Uncalibrated Visual Servoing

This paper addresses the image-based visual servoing (IBVS) control problem with an uncalibrated camera, unknown dynamics, and constraints. A novel data-driven uncalibrated IBVS (UIBVS) strategy is proposed, incorporated with the Koopman-based model predictive control (KMPC) algorithm and the adaptive robust Kalman filter (ARKF). First, to alleviate the need for calibration of the camera’s intrinsic and extrinsic parameters, the ARKF with an adaptive factor is utilized to estimate the image Jacobian matrix online, thereby eliminating the laborious camera calibration procedures and improving robustness against camera disturbances. Then, a data-driven MPC strategy is proposed, wherein the unknown nonlinear dynamic model is learned using the Koopman operator theory, resulting in a linear Koopman prediction model. Only input–output data are used to construct the prediction model, and hence, the proposed approach is robust against model uncertainties. Furthermore, with a symmetric quadratic cost function, the proposed approach solves the quadratic programming problem online, and visibility constraints as well as joint torque constraints are taken into account. As a result, the proposed KMPC scheme can be implemented in real time, and the UIBVS performance degradation which arises from the control torque constraints can be avoided. Simulations and comparisons for a 2-DOF robotic manipulator demonstrate the feasibility of the proposed approach. Simulation results further validate that the computation time of the proposed approach is comparable to the one of kinematic-based methods.

Regularized Maximum Correntropy Criterion Kalman Filter for Uncalibrated Visual Servoing in the Presence of Non-Gaussian Feature Tracking Noise.

Some advantages of using cameras as sensor devices on feedback systems are the flexibility of the data it represents, the possibility to extract real-time information, and the fact that it does not require contact to operate. However, in unstructured scenarios, Image-Based Visual Servoing (IBVS) robot tasks are challenging. Camera calibration and robot kinematics can approximate a jacobian that maps the image features space to the robot actuation space, but they can become error-prone or require online changes. Uncalibrated visual servoing (UVS) aims at executing visual servoing tasks without previous camera calibration or through camera model uncertainties. One way to accomplish that is through jacobian identification using environment information in an estimator, such as the Kalman filter. The Kalman filter is optimal with Gaussian noise, but unstructured environments may present target occlusion, reflection, and other characteristics that confuse feature extraction algorithms, generating outliers. This work proposes RMCKF, a correntropy-induced estimator based on the Kalman Filter and the Maximum Correntropy Criterion that can handle non-Gaussian feature extraction noise. Unlike other approaches, we designed RMCKF for particularities in UVS, to deal with independent features, the IBVS control action, and simulated annealing. We designed Monte Carlo experiments to test RMCKF with non-Gaussian Kalman Filter-based techniques. The results showed that the proposed technique could outperform its relatives, especially in impulsive noise scenarios and various starting configurations.

Visual-RRT: Integrating IBVS as a steering method in an RRT planner

In this paper, we present a new approach to robot motion planning that anticipates the use of vision-based feedback control during task execution. We accomplish this by incorporating an image-based visual servo (IBVS) controller directly into the steering function used by a Rapidly Exploring Random Tree (RRT) planner. Our approach requires a number of extensions to traditional RRT-style planning. First, we derive a new sampling strategy that augments the usual state information by including image features that will be used by the IBVS control law. These augmented samples are then used by our new IBVS steering function, which simulates an IBVS control law to generate local trajectories that extend the current tree. These trajectories must be validated to ensure that they are collision-free and that all image features remain unoccluded and within the camera field of view throughout the local trajectory. We also provide a formal proof showing that the proposed approach is probabilistically complete. We have applied our approach to the problem of planning trajectories for three different systems: a robotic arm, an unmanned aerial vehicle (UAV) and a car-like robot, which are equipped with an IBVS control law. We explore performance trade-offs in the control design via simulation studies and demonstrate real-world effectiveness via experiments in which a small-scale car-like robot uses IBVS to navigate a track that includes a number of obstacles and potential occlusions. By exploring performance trade-offs, we mean that several elements, such as the metric used to identify nearest neighbors in the RRT and the steering method used to generate nodes, are tested and compared.

Adaptive visual servoing for the robot manipulator with extreme learning machine and reinforcement learning

AbstractIn this study, a novel image‐based visual servo (IBVS) controller for robot manipulators is investigated using an optimized extreme learning machine (ELM) algorithm and an offline reinforcement learning (RL) algorithm. First of all, the classical IBVS method and its difficulties in accurately estimating the image interaction matrix and avoiding the singularity of pseudo‐inverse are introduced. Subsequently, an IBVS method based on ELM and RL is proposed to solve the problem of the singularity of the pseudo‐inverse solution and tune adaptive servo gain, improving the servo efficiency and stability. Specifically, the ELM algorithm optimized by particle swarm optimization (PSO) was used to approximate the pseudo‐inverse of the image interaction matrix to reduce the influence of camera calibration errors. Then, the RL algorithm was adopted to tune the adaptive visual servo gain in continuous space and improve the convergence speed. Finally, comparative simulation experiments on a 6‐DOF robot manipulator were conducted to verify the effectiveness of the proposed IBVS controller.

Image-Based Visual Servoing of Unmanned Aerial Manipulators for Tracking and Grasping a Moving Target

In this paper, an image-based visual servoing (IBVS) control strategy is proposed for the unmanned aerial manipulator (UAM) system to track and grasp a moving target. Specifically, a robust-adaptive velocity observer is designed to estimate the relative velocity between the tracked target and the UAM platform. Based on the velocity observer, an IBVS controller using onboard camera of the UAM platform is proposed for moving target tracking without velocity measurement. Then, the barrier Lyapunov function is introduced into the UAM platform IBVS controller to ensure the safety of target tracking. Besides, another virtual camera is constructed on manipulator end-effector to compensate for the tracking error of the UAM platform. As a benefit, the eye-to-hand onboard camera ensures the global view of the UAM, and the eye-in-hand virtual camera of the manipulator ensures the accuracy of the grasping task. Finally, the stability of the proposed IBVS control strategy is analyzed through Lyapunov theory. The comparative simulations are provided to illustrate the target tracking performance of the proposed method. The experimental results demonstrate that the proposed method can be applied to the UAM with a low-cost sensor suite to realize the tasks of tracking and grasping a moving target.

Perception-Aware Image-Based Visual Servoing of Aggressive Quadrotor UAVs

The maintenance of visual features within the sensor field of view poses a significant challenge for underactuated aerial vehicles such as quadrotors, especially during aggressive maneuvers. However, the existing image-based visual servo (IBVS) control methods rely on strict target visibility assumptions or impose excessive constraints on the quadrotor's agility to meet this requirement. Furthermore, the effectiveness of the visibility constraint defined in prior works remains unverified in aggressive flight tests. To address these issues, we present a robust IBVS scheme for quadrotors to perform aggressive maneuvers while ensuring target visibility. Based on the nonlinear model-predictive control framework, we propose a novel underactuation compensation scheme to eliminate the need for a virtual camera frame, which enables us to formulate the true target visibility constraint. We then introduce a quaternion-based representation of spherical visual features to handle the nonsmooth vector field problem on the 2-sphere and derive its corresponding image kinematics. We validate our method through three challenging visual servo tasks where agile maneuvers are desired: fast landing, aggressive long-distance flight, and dynamic object tracking. Extensive simulation and experiment show that our method consistently achieves a target-visible rate of 100% in all the image frames, even under a maximum pitch of 21.04 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^{\circ }$</tex-math></inline-formula> . The results validate the effectiveness of our visibility constraint under large robot rotations and underscore its importance in enabling robust and aggressive flights.

Hybrid Model Predictive Control with Penalty Factor Based on Image-Based Visual Servoing for Constrained Mobile Robots

For the constrained mobile robot automatic parking system, the hybrid model predictive control with a penalty factor based on image-based visual servoing (IBVS) is proposed to address the problem of feature point loss and emergency braking in dynamic obstacle scenarios caused by excessive target bias gain when using traditional IBVS control methods. The traditional IBVS control is transformed into an optimization problem with constraints in the finite time domain, by defining the optimization function based on the mobile robot’s positional deviation and image feature point deviation, while using actuator saturation and speed limit as constraints. Based on this, a convex optimization function with penalty factors is defined and combined with incremental model predictive control. This control strategy could ensure the emergency braking performance of the mobile robot when the image feature points are massively obscured by obstacles in dynamic scenes, while improving the accuracy and real-time of its trajectory tracking control. Finally, simulation comparisons are conducted to verify the effectiveness of the proposed control method.

Dynamic Target Tracking of Unmanned Aerial Vehicles Under Unpredictable Disturbances

This study proposes an image-based visual servoing (IBVS) method based on a velocity observer for an unmanned aerial vehicle (UAV) for tracking a dynamic target in Global Positioning System (GPS)-denied environments. The proposed method derives the simplified and decoupled image dynamics of underactuated UAVs using a constructed virtual camera and then considers the uncertainties caused by the unpredictable rotations and velocities of the dynamic target. A novel image depth model that extends the IBVS method to track a rotating target with arbitrary orientations is proposed. The depth model ensures image feature accuracy and image trajectory smoothness in rotating target tracking. The relative velocities of the UAV and the dynamic target are estimated using the proposed velocity observer. Thanks to the velocity observer, translational velocity measurements are not required, and the control chatter caused by noise-containing measurements is mitigated. An integral-based filter is proposed to compensate for unpredictable environmental disturbances in order to improve the anti-disturbance ability. The stability of the velocity observer and IBVS controller is analyzed using the Lyapunov method. Comparative simulations and multistage experiments are conducted to illustrate the tracking stability, anti-disturbance ability, and tracking robustness of the proposed method with a dynamic rotating target.

Hybrid Visual Servo Control of a Robotic Manipulator for Cherry Tomato Harvesting

This paper aims to develop a visual servo control of a robotic manipulator for cherry tomato harvesting. In the robotic manipulator, an RGB-depth camera was mounted to the end effector to acquire the poses of the target cherry tomatoes in space. The eye-in-hand-based visual servo controller guides the end effector to implement eye–hand coordination to harvest the target cherry tomatoes, in which a hybrid visual servo control method (HVSC) with the fuzzy dynamic control parameters was proposed by combining position-based visual servo (PBVS) control and image-based visual servo (IBVS) control for the tradeoff of both performances. In addition, a novel cutting and clipping integrated mechanism was designed to pick the target cherry tomatoes. The proposed tomato-harvesting robotic manipulator with HVSC was validated and evaluated in a laboratory testbed based on harvesting implementation. The results show that the developed robotic manipulator using HVSC has an average harvesting time of 9.40 s/per and an average harvesting success rate of 96.25% in picking cherry tomatoes.

Adaptive intelligent vision-based control of a flexible-link manipulator

Present space robots such as planetary robots and flexible robots have structural flexibility in their arms and joints that leads to an error in the tip positioning owing to tip deflection. The flexible-link manipulator (FLM) is a non-collocated system that has unstable and inaccurate system performance. Thus, tip-tracking of FLM possesses difficult control challenges. The purpose of this study is to design adaptive intelligent tip-tracking control strategy for FLMs to deal with this control challenges of FLM. A vision sensor is utilized in conjunction with a traditional mechanical sensor to directly measure tip-position in order to address the aforementioned problem. Image-based visual servoing (IBVS), one of several visual servoing control techniques, is more efficient. However, the IBVS scheme faces numerous difficulties that impair the system’s performance in real-time applications, including singularities in the interaction matrix, local minima in trajectory, visibility issues. To address the issues with the IBVS scheme, a novel adaptive intelligent IBVS (AI-IBVS) controller for tip-tracking control of a two-link flexible manipulator (TLFM) is designed in this study. In particular, this paper addresses the IBVS issues along-with retention of visual features in the field-of-view (FOV). First, in order to retain object within the camera FOV, an intelligent controller with off-policy reinforcement learning (RL) is proposed. Second, a composite controller for TLFM is developed to combine RL controller and IBVS controller. The simulation has been conducted to examine the effectiveness and robustness of the proposed controller. The obtained results show that the AI-IBVS controller developed here possesses the capabilities of self-learning and decision-making for robust tip-tracking control of TLFM. Further, a comparison with other similar approach is presented.

Robust Observer-Based Visual Servo Control for Quadrotors Tracking Unknown Moving Targets

This article addresses the image-based control problem of a quadrotor for tracking an unknown moving target under external disturbances. Based on the continuous observer design, a constructive image-based visual servoing (IBVS) geometric tracking control strategy is proposed and utilized. First, a higher order sliding mode observer is designed based on image dynamics for continuous estimation of unmeasurable linear velocities and time-varying disturbances. Furthermore, exploiting the linear quadratic regulator optimal design method, an outer loop IBVS controller is proposed such that the position of the quadrotor converges to the desired tracking pattern. Next, an inner loop geometric controller is designed using Lie algebra functions to regulate the attitudes. Based on the Lyapunov stability theory, it is rigorously proven that the proposed IBVS geometric controller achieves asymptotic stability. Finally, the effectiveness and robustness of the proposed approach against uncertain targets, time-varying disturbances, and large initial inclination angles are demonstrated through numerical simulations and experiments.

Stitching Dynamic Movement Primitives and Image-Based Visual Servo Control

Utilizing perception for feedback control in combination with dynamic movement primitive (DMP)-based motion generation for a robot’s end-effector control is a useful solution for many robotic manufacturing tasks. For instance, while performing an insertion task when the hole or the recipient part is not visible in the eye-in-hand camera, a learning-based movement primitive method can be used to generate the end-effector path. Once the recipient part is in the field of view (FOV), image-based visual servo (IBVS) can be used to control the motion of the robot. Inspired by such applications, this article presents a generalized control scheme that switches between motion generation using DMP and IBVS control. To facilitate the design, a common state-space representation for the DMP and the IBVS systems is first established. The stability analysis of the switched system using multiple Lyapunov functions shows that the state trajectories converge to a bound asymptotically. The developed method is validated by three real-world experiments using the eye-in-hand configuration of a Baxter research robot.

An image-based visual servoing control method for UAVs based on fuzzy logic

Visual servoing is a method to achieve precise positioning and motion control of objects by visual feedback, and it is widely applied in the fields of robotics and unmanned aerial vehicles (UAVs) in recent years. This paper presents a novel image-based visual servoing (IBVS) control method for UAVs based on fuzzy logic to effectively solve the problem under field of view constraint and improve the control efficiency. In this paper, a fuzzy logic of servo gain is designed for the control input of visual servoing, which solves the problem of feature loss in IBVS and improves the efficiency. Meanwhile, a deep computing method based on known data is proposed to solve the unknown depth of Jacobian matrix, which makes the control easier to converge. The effectiveness of the proposed method is verified by the simulation of a quadrotor UAV equipped with a monocular camera.

Robust Image-Based Landing Control of a Quadrotor on an Unpredictable Moving Vehicle Using Circle Features

This paper addresses the landing problem of a quadrotor on an unpredictable moving vehicle, using a robust image-based visual servoing (IBVS) method. The circle-based image moments are defined to construct image dynamics, and the passivity-like property of the circle features is preserved by reprojecting to the virtual image plane. The landing control system is decoupled into translation and rotation modules due to the rotation invariance of the proposed circle features. First, by exploiting the error transformation in the image space, a robust IBVS controller can overcome the lack of both the desired depth information of target features and the velocity feedback of the target. Next, an adaptive geometric attitude controller is developed directly using rotation matrices to avoid the singularities of Euler-angles and the ambiguity of quaternions. One benefit of the proposed scheme is that it can potentially improve the camera visibility, guarantee the transient and steady-state behaviors in image space, and be efficiently implemented on the low-cost quadrotor. Finally, The stability analysis is presented using Lyapunov stability theory on cascaded systems, and the effectiveness of the proposed control strategy is demonstrated through simulations and experiments. Note to Practitioners—The motivation of this paper is to investigate a practical control strategy for the image-based landing control of underactuated quadrotors on an unpredictable moving vehicle. In most of the existing image-based landing control schemes for underactuated quadrotors, having the prior predictive model of the moving landing vehicle to provide a feed-forward compensation during the landing maneuver. However, due to the fact that the landing environment and vehicle are primarily stochastic, resulting in no predictive models are valid in practice. Therefore, this paper suggests a robust image-based landing control strategy without the model or state of the moving landing vehicle. In particular, a novel virtual circle feature, possessing the characteristic of rotation invariance, is designed for the landing of underactuated quadrotors, which decouples the landing system and simplifies the control design. Moreover, the image feature errors are directly retained within prescribed performance funnels in the image space. As a result, the transient and steady-state landing behaviors can be implicitly guaranteed in Cartesian space. The stability and convergence of the system are analyzed mathematically and the experiment using quadrotors provides promising results. In ongoing research, we are addressing the issues of collision avoidances and unknown disturbances to provide a more realistic setup for the autonomous deployment and recovery of underactuated quadrotors in GPS-denied environments.