Image-based Visual Servoing Research Articles

The docking control system of an autonomous underwater vehicle combining intelligent object recognition and deep reinforcement learning

This study develops a visual-based docking system (VDS) for an autonomous underwater vehicle (AUV), significantly enhancing docking performance by integrating intelligent object recognition and deep reinforcement learning (DRL). The system overcomes traditional navigation limitations in complex and unpredictable environments by using a variable information dock (VID) for precise multi-sensor docking recognition in the AUV. Employing image-based visual servoing (IBVS) technology, the VDS efficiently converts 2D visual data into accurate 3D motion control commands. It integrates the YOLO (short for You Only Look Once) algorithm for object recognition and the deep deterministic policy gradient (DDPG) algorithm, improving continuous motion control, docking accuracy, and adaptability. Experimental validation at the National Cheng Kung University towing tank demonstrates that the VDS enhances control stability and operational reliability, reducing the mean absolute error (MAE) in depth control by 42.03% and pitch control by 98.02% compared to the previous method. These results confirm the VDS's reliability and its potential for transforming AUV docking.

Enhancing quadrotor robustness control using image-based visual servoing (IBVS) with fuzzy logic

Visual servoing is a commonly employed approach in robotics and unmanned aerial vehicles (UAVs) that facilitates accurate object positioning and movement control by utilizing visual feedback. As quadrotors gain popularity, more control methods have been developed. This article presents a method for enhancing quadrotor robustness control using image-based visual servoing (IBVS) with fuzzy logic. Unlike traditional visual servoing, which relies on a fixed gain and often encounters challenges with velocity convergence and maintaining the object in the field of view, this method is designed to enhance the visual servo control of quadrotors by dynamically adjusting the gain of the IBVS system through a fuzzy logic controller. This controller adaptively adjusts the servo gain in response to feature errors and the depth of the object's feature points. MATLAB simulations clearly demonstrate the superior performance of this fuzzy logic integrated method compared to classical approaches, showcasing enhanced control capabilities in challenging environments.

Visual Servoing for Aerial Vegetation Sampling Systems

This research describes a vision-based control strategy that employs deep learning for an aerial manipulation system developed for vegetation sampling in remote, dangerous environments. Vegetation sampling in such places presents considerable technical challenges such as equipment failures and exposure to hazardous elements. Controlling aerial manipulation in unstructured areas such as forests remains a significant challenge because of uncertainty, complex dynamics, and the possibility of collisions. To overcome these issues, we offer a new image-based visual servoing (IBVS) method that uses knowledge distillation to provide robust, accurate, and adaptive control of the aerial vegetation sampler. A convolutional neural network (CNN) from a previous study is used to detect the grasp point, giving critical feedback for the visual servoing process. The suggested method improves the precision of visual servoing for sampling by using a learning-based approach to grip point selection and camera calibration error handling. Simulation results indicate the system can track and sample tree branches with minimum error, demonstrating that it has the potential to improve the safety and efficiency of aerial vegetation sampling.

A Novel Fuzzy Image-Based UAV Landing Using RGBD Data and Visual SLAM

In this work, an innovative perception-guided approach is proposed for landing zone detection and realization of Unmanned Aerial Vehicles (UAVs) operating in unstructured environments ridden with obstacles. To accommodate secure landing, two well-established tools, namely fuzzy systems and visual Simultaneous Localization and Mapping (vSLAM), are implemented into the landing pipeline. Firstly, colored images and point clouds acquired by a visual sensory device are processed to serve as characterizing maps that acquire information about flatness, steepness, inclination, and depth variation. By leveraging these images, a novel fuzzy map infers the areas for risk-free landing on which the UAV can safely land. Subsequently, the vSLAM system is employed to estimate the platform’s pose and an additional set of point clouds. The vSLAM point clouds presented in the corresponding keyframe are projected back onto the image plane on which a threshold fuzzy landing score map is applied. In other words, this binary image serves as a mask for the re-projected vSLAM world points to identify the best subset for landing. Once these image points are identified, their corresponding world points are located, and among them, the center of the cluster with the largest area is chosen as the point to land. Depending on the UAV’s size, four synthesis points are added to the vSLAM point cloud to execute the image-based visual servoing landing using image moment features. The effectiveness of the landing package is assessed through the ROS Gazebo simulation environment, where comparisons are made with a state-of-the-art landing site detection method.

Adaptive positioning of large-scale aircraft components using a simplified image-based visual servoing via online measurement of ball-head center

In digital alignment systems, ensuring fast and accurate alignment between the ball head on large aircraft components and the ball socket of the positioner is the key to aircraft assembly efficiency and quality. Image-based visual servoing is one of the most advanced techniques with high efficiency and positioning accuracy, but it does not account for measuring the ball-head position. Moreover, variable target positions, similar feature interference, and poor lighting conditions pose great challenges to the fast and robust visual measurement of ball-head position. Hence, an online visual measurement method is introduced to measure the absolute position of the ball head in real time. An adaptive region of interest extraction algorithm and an improved edge following method are introduced to extract the elliptical contour from the ball-head image, and ellipse detection is achieved through a simple triplet selection algorithm. A geometric model based on the perspective projection of the sphere is established for estimating the ball-head center. Then, the simplified image-based visual servoing method is developed by constantizing the image Jacobian matrix. Next, an adaptive positioning method of large-scale aircraft components with multiple ball heads is proposed from a holistic perspective. Finally, experiments show that the final positioning accuracy of each ball head in the X/Y plane is less than ± 0.05 mm, with an efficiency almost ten times that of the conventional method. After adaptive adjustment, the position and angle errors of the simulated component are reduced within 0.6 mm and 0.3°, respectively.

A Data-Driven Image-Based Visual Servoing Scheme for Redundant Manipulators With Unknown Structure and Singularity Solution

A Data-Driven Image-Based Visual Servoing Scheme for Redundant Manipulators With Unknown Structure and Singularity Solution

Control barrier function based visual servoing for Mobile Manipulator Systems under functional limitations

This paper proposes a new control strategy for Mobile Manipulator Systems (MMSs) that integrates image-based visual servoing (IBVS) to address operational limitations and safety constraints. The proposed approach based on the concept of control barrier functions (CBFs), provides a solution to address various operational challenges including visibility constraints, manipulator joint limits, predefined system velocity bounds, and system dynamic uncertainties. The proposed control strategy is a two-tiered structure, wherein the first level, a CBF-IBVS controller calculates control commands, taking into account the Field of View (FoV) constraints. By leveraging null space techniques, these commands are transposed to the joint-level configuration of the MMS, while considering system operational limits. Subsequently, in the second level, a CBF velocity controller employed for the entire MMS undertakes the tracking of the commands at the joint level, ensuring compliance with the predefined system’s velocity limitations as well as the safety of the whole combined system dynamics. The proposed control strategy offers superior transient and steady-state responses and heightened resilience to disturbances and modeling uncertainties. Furthermore, due to its low computational complexity, it can be easily implemented on an onboard computing system, facilitating real-time operation. The proposed strategy’s effectiveness is illustrated via simulation outcomes, which reveal enhanced performance and system safety compared to conventional IBVS methods. The results indicate that the proposed approach is effective in addressing the challenging operational limitations and safety constraints of mobile manipulator systems, making it suitable for practical applications.

Image-Based Visual Servoing of Manipulators With Unknown Depth: A Recurrent Neural Network Approach.

The image-based visual servoing (IBVS) of manipulators is important for intelligent manipulation using visual feedbacks. While the traditional IBVS methods for manipulators require the knowledge of the depth information in the interaction matrix, in this article, we propose a novel IBVS method for manipulators without depth estimation by leveraging the property of the associated image Jacobian. Because of a novel transformation, the IBVS problem is converted into a convex optimization problem subject to the kinematic constraint, joint constraints, and other constraints that are not explicitly related to the depth information. The problem is then solved by developing a recurrent neural network of global asymptotic convergence, and a dynamic neural control law without depth estimation emerges for the IBVS of manipulators. The theoretical guarantee and simulation results are provided to show the efficacy of the proposed method.

Motion Prediction and Object Detection for Image-Based Visual Servoing Systems Using Deep Learning

This study primarily investigates advanced object detection and time series prediction methods in image-based visual servoing systems, aiming to capture targets better and predict the motion trajectory of robotic arms in advance, thereby enhancing the system’s performance and reliability. The research first implements object detection on the VOC2007 dataset using the Detection Transformer (DETR) and achieves ideal detection scores. The particle swarm optimization algorithm and 3-5-3 polynomial interpolation methods were utilized for trajectory planning, creating a unique dataset through simulation. This dataset contains randomly generated trajectories within the workspace, fully simulating actual working conditions. Significantly, the Bidirectional Long Short-Term Memory (BILSTM) model was improved by substituting its traditional Multilayer Perceptron (MLP) components with Kolmogorov–Arnold Networks (KANs). KANs, inspired by the K-A theorem, improve the network representation ability by placing learnable activation functions on fixed node activation functions. By implementing KANs, the model enhances parameter efficiency and interpretability, thus addressing the typical challenges of MLPs, such as the high parameter count and lack of transparency. The experiments achieved favorable predictive results, indicating that the KAN not only reduces the complexity of the model but also improves learning efficiency and prediction accuracy in dynamic visual servoing environments. Finally, Gazebo software was used in ROS to model and simulate the robotic arm, verify the effectiveness of the algorithm, and achieve visual servoing.

Neural Network-Based Robust Guaranteed Cost Control for Image-Based Visual Servoing of Quadrotor.

In this article, a neural network (NN)-based robust guaranteed cost control design is proposed for image-based visual servoing (IBVS) control of quadrotors. According to the dynamics of three subsystems (yaw, height, and lateral subsystems) derived from the quadrotor IBVS dynamic model, the main control design is to solve the robust control problem for the time-varying lateral subsystem with angle constraints and uncertain disturbances. Considering the system dynamics, a two-loop structure is conducted. The outer loop uses the linear quadratic regulator to solve the Riccati equation for the lateral image feature system, and the inner loop adopts the optimal robust guaranteed cost control to solve the lateral velocity system. For the lateral velocity system, the optimal robust control problem is transformed to solve the modified Hamilton-Jacobi-Bellman equation of the corresponding optimal control problem utilizing adaptive dynamic programming. The implementation is accomplished with the time-varying NN and the designed estimated weight update law. In addition, the stability and effectiveness are proved by the theoretic proof and simulations.

Discrete-Time Visual Servoing Control with Adaptive Image Feature Prediction Based on Manipulator Dynamics.

In this paper, a practical discrete-time control method with adaptive image feature prediction for the image-based visual servoing (IBVS) scheme is presented. In the discrete-time IBVS inner-loop/outer-loop control architecture, the time delay caused by image capture and computation is noticed. Considering the dynamic characteristics of a 6-DOF manipulator velocity input system, we propose a linear dynamic model to describe the motion of a robot end effector. Furthermore, for better estimation of image features and smoothing of the robot's velocity input, we propose an adaptive image feature prediction method that employs past image feature data and real robot velocity data to adopt the prediction parameters. The experimental results on a 6-DOF robotic arm demonstrate that the proposed method can ensure system stability and accelerate system convergence.

High-Precision Drilling by Anchor-Drilling Robot Based on Hybrid Visual Servo Control in Coal Mine

Rock bolting is a commonly used method for stabilizing the surrounding rock in coal-mine roadways. It involves installing rock bolts after drilling, which penetrate unstable rock layers, binding loose rocks together, enhancing the stability of the surrounding rock, and controlling its deformation. Although recent progress in drilling and anchoring equipment has significantly enhanced the efficiency of roof support in coal mines and improved safety measures, how to deal with drilling rigs’ misalignment with the through-hole center remains a big issue, which may potentially compromise the quality of drilling and consequently affect the effectiveness of bolt support or even result in failure. To address this challenge, this article presents a robotic teleoperation system alongside a hybrid visual servo control strategy. Addressing the demand for high precision and efficiency in aligning the drilling rigs with the center of the drilling hole, a hybrid control strategy is introduced combining position-based and image-based visual servo control. The former facilitates an effective approach to the target area, while the latter ensures high-precision alignment with the center of the drilling hole. The robot teleoperation system employs the binocular vision measurement system to accurately determine the position and orientation of the drilling-hole center, which serves as the designated target position for the drilling rig. Leveraging the displacement and angle sensor information installed on each joint of the manipulator, the system utilizes the kinematic model of the manipulator to compute the spatial position of the end-effector. It dynamically adjusts the spatial pose of the end-effector in real time, aligning it with the target position relative to its current location. Additionally, it utilizes monocular vision information to fine-tune the movement speed and direction of the end-effector, ensuring rapid and precise alignment with the target drilling-hole center. Experimental results demonstrate that this method can control the maximum alignment error within 7 mm, significantly enhancing the alignment accuracy compared to manual control. Compared with the manual control method, the average error of this method is reduced by 41.2%, and the average duration is reduced by 4.3 s. This study paves a new path for high-precision drilling and anchoring of tunnel roofs, thereby improving the quality and efficiency of roof support while mitigating the challenges associated with significant errors and compromised safety during manual control processes.

Image-Based Visual Servoing for Three Degree-of-Freedom Robotic Arm with Actuator Faults

Image-Based Visual Servoing for Three Degree-of-Freedom Robotic Arm with Actuator Faults

Fixed-Time Differentiator-Based Adaptive Nonsingular Fast Terminal Image-Based Visual Servoing for a Quadrotor UAV Subject to Turbulent Wind

Fixed-Time Differentiator-Based Adaptive Nonsingular Fast Terminal Image-Based Visual Servoing for a Quadrotor UAV Subject to Turbulent Wind

Model-free visual servoing based on active disturbance rejection control and adaptive estimator for robotic manipulation without calibration

PurposeVisual feedback control is a promising solution for robots work in unstructured environments, and this is accomplished by estimation of the time derivative relationship between the image features and the robot moving. While some of the drawbacks associated with most visual servoing (VS) approaches include the vision–motor mapping computation and the robots’ dynamic performance, the problem of designing optimal and more effective VS systems still remains challenging. Thus, the purpose of this paper is to propose and evaluate the VS method for robots in an unstructured environment.Design/methodology/approachThis paper presents a new model-free VS control of a robotic manipulator, for which an adaptive estimator aid by network learning is proposed using online estimation of the vision–motor mapping relationship in an environment without the knowledge of statistical noise. Based on the adaptive estimator, a model-free VS schema was constructed by introducing an active disturbance rejection control (ADRC). In our schema, the VS system was designed independently of the robot kinematic model.FindingsThe various simulations and experiments were conducted to verify the proposed approach by using an eye-in-hand robot manipulator without calibration and vision depth information, which can improve the autonomous maneuverability of the robot and also allow the robot to adapt its motion according to the image feature changes in real time. In the current method, the image feature trajectory was stable in the camera field range, and the robot’s end motion trajectory did not exhibit shock retreat. The results showed that the steady-state errors of image features was within 19.74 pixels, the robot positioning was stable within 1.53 mm and 0.0373 rad and the convergence rate of the control system was less than 7.21 s in real grasping tasks.Originality/valueCompared with traditional Kalman filtering for image-based VS and position-based VS methods, this paper adopts the model-free VS method based on the adaptive mapping estimator combination with the ADRC controller, which is effective for improving the dynamic performance of robot systems. The proposed model-free VS schema is suitable for robots’ grasping manipulation in unstructured environments.

An Advanced IBVS-Flatness Approach for Real-Time Quadrotor Navigation: A Full Control Scheme in the Image Plane

This article presents an innovative method for planning and tracking the trajectory in the image plane for the visual control of a quadrotor. The community of researchers working on 2D control widely recognizes this challenge as complex, because a trajectory defined in image space can lead to unpredictable movements of the robot in Cartesian space. While researchers have addressed this problem for mobile robots, quadrotors continue to face significant challenges. To tackle this issue, the adopted approach involves considering the separation of altitude control from the other variables, thus reducing the workspace. Furthermore, the movements of the quadrotor (pitch, roll, and yaw) are interdependent. Consequently, the connection between the inputs and outputs cannot be reversed. The task complexity becomes significant. To address this issue, we propose the following scenario: When the quadrotor is equipped with a downward-facing camera, flying at high altitude is sensible to spot a target. However, to minimize disturbances and conserve energy, the quadrotor needs to descend in altitude. This can result in the target being lost. The solution to this problem is a new methodology based on the principle of differential flatness, allowing the separation of altitude control from the other variables. The system first detects the target at high altitude, then plots a trajectory in the image coordinate system between the acquired image and the desired image. It is crucial to emphasize that this step is performed offline, ensuring that the image processing time does not affect the control frequency. Through the proposed trajectory planning, complying with the constraints of differential flatness, the quadrotor can follow the imposed dynamics. To ensure the tracking of the target while following the generated trajectory, the proposed control law takes the form of an Image Based Visual Servoing (IBVS) scheme. We validated this method using the RVCTOOLS environment in MATLAB. The DJI Phantom 1 quadrotor served as a testbed to evaluate, under real conditions, the effectiveness of the proposed control law. We specifically designed an electronic card to transfer calculated commands to the DJI Phantom 1 control joystick via Bluetooth. This card integrates a PIC18F2520 microcontroller, a DAC8564 digital-to-analogue converter, and an RN42 Bluetooth module. The experimental results demonstrate the effectiveness of this method, ensuring the precise tracking of the target as well as the accurate tracking of the path generated in the image coordinate system.

A Framework for IBVS Using Virtual Work

The visual servoing of manipulators is challenged by two main problems: the singularity of the inverse Jacobian and the physical constraints of a manipulator. In order to overcome the singularity issue, this paper presents a novel approach for image-based visual servoing (IBVS), which converts the propagation of errors in the image plane into the conduction of virtual forces using the principle of virtual work. This approach eliminates the need for Jacobian inversion computations and prevents matrix inversion singularity. To tackle physical constraints, reverse thinking is adopted to derive the function of the upper and lower bounds of the joint velocity on the joint angle. This enables the proposed method to configure the physical constraints of the robot in a more intuitive manner. To validate the effectiveness of the proposed method, an eye-in-hand system based on UR5 in VREP, as well as a physical robot, were established.

A Hierarchical Data-Driven Predictive Control of Image-Based Visual Servoing Systems With Unknown Dynamics.

In this article, a hierarchical predictive control (PC) algorithm is designed for visual servoing mobile robot systems. At the kinematic level, the image-based visual servoing model of a wheeled mobile robot is established. By defining the corresponding performance index of the PC, an iterative linear quadratic regulator (iLQR) is used to obtain the velocity controller and to provide reference velocity for dynamics. In dynamics, a data-driven PC controller based on the Gaussian process (GP) is proposed to obtain the torque controller with unknown dynamics. The input-to-state practical stability (ISpS) of the system based on the proposed data-driven PC method is proved by introducing reasonable assumptions. The corresponding theorem also analyzes the maximum upper bound of GP inference error. Finally, the effectiveness of the proposed hierarchical controller is verified by simulations and experiments.

Online Predictive Visual Servo Control for Constrained Target Tracking of Fixed-Wing Unmanned Aerial Vehicles

This paper proposes an online predictive control method for fixed-wing unmanned aerial vehicles (UAVs) with a pan-tilt camera in target tracking. It aims to achieve long-term tracking while concurrently maintaining the target near the image center. Particularly, this work takes the UAV and pan-tilt camera as an overall system and deals with the target tracking problem via joint optimization, so that the tracking ability of the UAV can be improved. The image captured by the pan-tilt camera is the unique input associated with the target, and model predictive control (MPC) is used to solve the optimization problem with constraints that cannot be performed by the classic image-based visual servoing (IBVS). In addition to the dynamic constraint of the UAV, the perception constraint of the camera is also taken into consideration, which is described by the maximum distance between the target and the camera. The accurate detection of the target depends on the amount of its feature information contained in the image, which is highly related to the relative distance between the target and the camera. Moreover, considering the real-time requirements of practical applications, an MPC strategy based on soft constraints and a warm start is presented. Furthermore, a switching-based approach is proposed to return the target back to the perception range quickly once it exceeds the range, and the exponential asymptotic stability of the switched controller is proven as well. Both numerical and hardware-in-the-loop (HITL) simulations are conducted to verify the effectiveness and superiority of the proposed method compared with the existing method.

Uncalibrated and Unmodeled Image-Based Visual Servoing of Robot Manipulators Using Zeroing Neural Networks.

Neural networks have been widely investigated for the control of robot manipulators and recurrent neural network (RNN) is accepted as a powerful tool for visual servoing. Different from existing control schemes for robot-camera systems, this article proposes a novel image-based visual servoing (IBVS) control scheme for both the regulation and tracking control of robot manipulators in the framework of a special class of RNN, termed zeroing neural network (ZNN), which does not require prior knowledge about camera configuration and kinematic model parameters. The proposed control scheme is composed of a data-driven mapping estimator and a controller, both of which are designed based on ZNN. To facilitate the deployment of the proposed IBVS control scheme, a discrete-time version of the proposed control scheme is developed. Theoretical analysis for the proposed method is presented in terms of convergence, stability, and robustness. In addition, simulations and experiments are carried out based on different types of robot-camera systems to verify the efficacy and portability of the proposed control scheme for solving regulation and trajectory IBVS problems. Moreover, comparative studies are performed to reveal the merits of the proposed control scheme.