Visual Servoing Method Research Articles

Autonomous Landing Guidance for Quad-UAVs Based on Visual Image and Altitude Estimation

In this paper, an autonomous landing guidance strategy is proposed for quad-UAVs, including landing marker detection, altitude estimation, and adaptive landing commands generation. A double-layered nested marker is designed to ensure that the marker can be captured both in high and low altitudes. A deep learning-based marker detection method is designed where the intersection of union is replaced by the normalized Wasserstein distance in the computation of non-maximum suppression to improve the detection accuracy. The UAV altitude measured by inertial measurement unit is fused with vision-based altitude estimation data to improve the accuracy during the landing process. An image-based visual servoing method is designed to guide the UAV approach to the landing marker. Both simulation and flight experiments are conducted to verify the proposed strategy.

Research on adaptive visual servo method for circular symmetrical objects

Research on adaptive visual servo method for circular symmetrical objects

Adaptive positioning of large-scale aircraft components using a simplified image-based visual servoing via online measurement of ball-head center

In digital alignment systems, ensuring fast and accurate alignment between the ball head on large aircraft components and the ball socket of the positioner is the key to aircraft assembly efficiency and quality. Image-based visual servoing is one of the most advanced techniques with high efficiency and positioning accuracy, but it does not account for measuring the ball-head position. Moreover, variable target positions, similar feature interference, and poor lighting conditions pose great challenges to the fast and robust visual measurement of ball-head position. Hence, an online visual measurement method is introduced to measure the absolute position of the ball head in real time. An adaptive region of interest extraction algorithm and an improved edge following method are introduced to extract the elliptical contour from the ball-head image, and ellipse detection is achieved through a simple triplet selection algorithm. A geometric model based on the perspective projection of the sphere is established for estimating the ball-head center. Then, the simplified image-based visual servoing method is developed by constantizing the image Jacobian matrix. Next, an adaptive positioning method of large-scale aircraft components with multiple ball heads is proposed from a holistic perspective. Finally, experiments show that the final positioning accuracy of each ball head in the X/Y plane is less than ± 0.05 mm, with an efficiency almost ten times that of the conventional method. After adaptive adjustment, the position and angle errors of the simulated component are reduced within 0.6 mm and 0.3°, respectively.

Visual Servoing for Aerial Vegetation Sampling Systems

This research describes a vision-based control strategy that employs deep learning for an aerial manipulation system developed for vegetation sampling in remote, dangerous environments. Vegetation sampling in such places presents considerable technical challenges such as equipment failures and exposure to hazardous elements. Controlling aerial manipulation in unstructured areas such as forests remains a significant challenge because of uncertainty, complex dynamics, and the possibility of collisions. To overcome these issues, we offer a new image-based visual servoing (IBVS) method that uses knowledge distillation to provide robust, accurate, and adaptive control of the aerial vegetation sampler. A convolutional neural network (CNN) from a previous study is used to detect the grasp point, giving critical feedback for the visual servoing process. The suggested method improves the precision of visual servoing for sampling by using a learning-based approach to grip point selection and camera calibration error handling. Simulation results indicate the system can track and sample tree branches with minimum error, demonstrating that it has the potential to improve the safety and efficiency of aerial vegetation sampling.

Model-less optimal visual control of tendon-driven continuum robots using recurrent neural network-based neurodynamic optimization

Tendon-driven continuum robots (TDCRs) have infinite degrees of freedom and high flexibility, posing challenges for accurate modeling and autonomous control, especially in confined environments. This paper presents a model-less optimal visual control (MLOVC) method using neurodynamic optimization to enable autonomous target tracking of TDCRs in confined environments. The TDCR’s kinematics are estimated online from sensory data, establishing a connection between the actuator input and visual features. An optimal visual servoing method based on quadratic programming (QP) is developed to ensure precise target tracking without violating the robot’s physical constraints. An inverse-free recurrent neural network (RNN)-based neurodynamic optimization method is designed to solve the complex QP problem. Comparative simulations and experiments demonstrate that the proposed method outperforms existing methods in target tracking accuracy and computational efficiency. The RNN-based controller successfully achieves target tracking within constraints in confined environments.

Deep learning approach for detecting tomato flowers and buds in greenhouses on 3P2R gantry robot

In recent years, significant advancements have been made in the field of smart greenhouses, particularly in the application of computer vision and robotics for pollinating flowers. Robotic pollination offers several benefits, including reduced labor requirements and preservation of costly pollen through artificial tomato pollination. However, previous studies have primarily focused on the labeling and detection of tomato flowers alone. Therefore, the objective of this study was to develop a comprehensive methodology for simultaneously labeling, training, and detecting tomato flowers specifically tailored for robotic pollination. To achieve this, transfer learning techniques were employed using well-known models, namely YOLOv5 and the recently introduced YOLOv8, for tomato flower detection. The performance of both models was evaluated using the same image dataset, and a comparison was made based on their Average Precision (AP) scores to determine the superior model. The results indicated that YOLOv8 achieved a higher mean AP (mAP) of 92.6% in tomato flower and bud detection, outperforming YOLOv5 with 91.2%. Notably, YOLOv8 also demonstrated an inference speed of 0.7 ms when considering an image size of 1920×1080\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$1920 \ imes 1080$$\\end{document} pixels resized to 640×640\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$640 \ imes 640$$\\end{document} pixels during detection. The image dataset was acquired during both morning and evening periods to minimize the impact of lighting conditions on the detection model. These findings highlight the potential of YOLOv8 for real-time detection of tomato flowers and buds, enabling further estimation of flower blooming peaks and facilitating robotic pollination. In the context of robotic pollination, the study also focuses on the deployment of the proposed detection model on the 3P2R gantry robot. The study introduces a kinematic model and a modified circuit for the gantry robot. The position-based visual servoing method is employed to approach the detected flower during the pollination process. The effectiveness of the proposed visual servoing approach is validated in both un-clustered and clustered plant environments in the laboratory setting. Additionally, this study provides valuable theoretical and practical insights for specialists in the field of greenhouse systems, particularly in the design of flower detection algorithms using computer vision and its deployment in robotic systems used in greenhouses.

Fully Autonomous Brick Pick and Place in Fields by Articulated Aerial Robot: Results in Various Outdoor Environments

Picking and placing of objects by aerial robots in fields is an important and challenging task for a range of outdoor activities, including industry and rescue operations. General strategies depend on the magnetic force to the pick object; however, this approach lacks both generality and robustness. Therefore, we focus on an articulated structure to grasp bricking. Another issue in performing pick-and-place tasks in fields is autonomous recognition using onboard sensors. In this article, we present a fully autonomous pick-and-place scheme in outdoor environments using articulated aerial robots. First, an articulated robot model with an actively tiltable sensor is developed to guarantee robustness in both state estimation and object detection. Second, object detection methods are designed according to the distance between the robot and target object. Third, a comprehensive motion strategy is developed to perform an autonomous object searching, picking, and placing sequence. In particular, a visual servoing method for robot position control is also proposed in this motion strategy to improve the robustness while approaching the target. Finally, we present the experimental results of autonomous brick picking and placing by our proposed methods in various outdoor environments, including a real robotics challenge, the 2020 Mohamed Bin Zayed International Robotics Challenge (MBZIRC 2020). To the best of our knowledge, our study represents the first successful attempt at fully autonomous object grasping by an articulated aerial robot in a field.

Model-free visual servoing based on active disturbance rejection control and adaptive estimator for robotic manipulation without calibration

PurposeVisual feedback control is a promising solution for robots work in unstructured environments, and this is accomplished by estimation of the time derivative relationship between the image features and the robot moving. While some of the drawbacks associated with most visual servoing (VS) approaches include the vision–motor mapping computation and the robots’ dynamic performance, the problem of designing optimal and more effective VS systems still remains challenging. Thus, the purpose of this paper is to propose and evaluate the VS method for robots in an unstructured environment.Design/methodology/approachThis paper presents a new model-free VS control of a robotic manipulator, for which an adaptive estimator aid by network learning is proposed using online estimation of the vision–motor mapping relationship in an environment without the knowledge of statistical noise. Based on the adaptive estimator, a model-free VS schema was constructed by introducing an active disturbance rejection control (ADRC). In our schema, the VS system was designed independently of the robot kinematic model.FindingsThe various simulations and experiments were conducted to verify the proposed approach by using an eye-in-hand robot manipulator without calibration and vision depth information, which can improve the autonomous maneuverability of the robot and also allow the robot to adapt its motion according to the image feature changes in real time. In the current method, the image feature trajectory was stable in the camera field range, and the robot’s end motion trajectory did not exhibit shock retreat. The results showed that the steady-state errors of image features was within 19.74 pixels, the robot positioning was stable within 1.53 mm and 0.0373 rad and the convergence rate of the control system was less than 7.21 s in real grasping tasks.Originality/valueCompared with traditional Kalman filtering for image-based VS and position-based VS methods, this paper adopts the model-free VS method based on the adaptive mapping estimator combination with the ADRC controller, which is effective for improving the dynamic performance of robot systems. The proposed model-free VS schema is suitable for robots’ grasping manipulation in unstructured environments.

Robust Hybrid Visual Servoing of Omnidirectional Mobile Manipulator With Kinematic Uncertainties Using a Single Camera.

A robust hybrid visual servoing (HVS) method for a single camera-mounted omnidirectional mobile manipulator (OMM) with kinematic uncertainties caused by slipping is proposed herein. Most existing studies pertaining to the visual servoing of mobile manipulators do not consider the kinematic uncertainties and the singularity of the manipulator that can occur during actual operations; furthermore, they required external sensors other than a single camera. In this study, the kinematics of an OMM are modeled considering kinematic uncertainties. Accordingly, an integral sliding-mode observer (ISMO) is designed to estimate the kinematic uncertainties. Subsequently, an integral sliding-mode control (ISMC) law is proposed to achieve robust visual servoing using the estimates of the ISMO. Additionally, an ISMO-ISMC-based HVS method is proposed to address the singularity issue of the manipulator; this method guarantees both robustness and finite-time stability in the presence of kinematic uncertainties. Overall, the entire visual servoing task is performed using only a single camera attached to the end effector without any other external sensors, unlike in previous studies. The stability and performance of the proposed method are verified numerically and experimentally in a slippery environment that generates kinematic uncertainties.

Electric Vehicle Battery Disassembly Using Interfacing Toolbox for Robotic Arms

This paper showcases the integration of the Interfacing Toolbox for Robotic Arms (ITRA) with our newly developed hybrid Visual Servoing (VS) methods to automate the disassembly of electric vehicle batteries, thereby advancing sustainability and fostering a circular economy. ITRA enhances collaboration between industrial robotic arms, server computers, sensors, and actuators, meeting the intricate demands of robotic disassembly, including the essential real-time tracking of components and robotic arms. We demonstrate the effectiveness of our hybrid VS approach, combined with ITRA, in the context of Electric Vehicle (EV) battery disassembly across two robotic testbeds. The first employs a KUKA KR10 robot for precision tasks, while the second utilizes a KUKA KR500 for operations needing higher payload capacity. Conducted in T1 (Manual Reduced Velocity) mode, our experiments underscore a swift communication protocol that links low-level and high-level control systems, thus enabling rapid object detection and tracking. This allows for the efficient completion of disassembly tasks, such as removing the EV battery’s top case in 27 s and disassembling a stack of modules in 32 s. The demonstrated success of our framework highlights its extensive applicability in robotic manufacturing sectors that demand precision and adaptability, including medical robotics, extreme environments, aerospace, and construction.

A geometric approach for homography-based visual servo control of underactuated UAVs

This paper proposes a new geometric control method for homography-based visual servo control of Underactuated UAVs. In order to solve the application difficulties of geometric control in HBVS and explore a visual servo control technology that can be applied to aerial detection operations, this paper integrates the geometric control into the visual servoing framework and design a new homography-based geometric visual servoing controller. The outer loop is used as feedback information using the virtual homography matrix between the two images. The inner loop controls the orientation of the UAVs through geometric control. The stability of the proposed controller is proved based on Lyapunov’s theory. The proposed method has better transient performance and dynamic performance than the conventional visual servo method. The excellent performance of the controller has been proven by a large number of experiments. In addition, the application of the controller on an unmanned aerial manipulator is demonstrated.

Fast and stable pedicel detection for robust visual servoing to harvest shaking fruits

This study introduces a fast and stable pedicel detection method for robust visual servoing (RVS), supplemented with video stabilization (ViS) and fast pedicel detection, to realize automated harvesting of shaking fruits. By incorporating ViS, the system effectively mitigates the effects of motion blur, thereby ensuring consistent and precise object detection. In addition, the Fourier spectrum-based band-stop filter (FSBF) is used to improve clarity. The proposed approach also leverages the fast point feature histogram (FPFH) for fast pedicel detection, achieving real-time detection rates of 15–37 fps. Furthermore, it incorporates 6D pose estimation, culminating in the implementation of a 6D pose-based robust visual servoing (6DRVS) system. The performance of this system is evaluated using standard metrics such as perception accuracy, approach accuracy, precision, recall, accuracy, and F1-score in both preliminary tests and on-site experiments at two cucumber farms in Korea. The 6DRVS, supplemented with fast pedicel detection and ViS, exhibited improvements across all evaluation metrics. It recorded 90.00% perception accuracy, 82.22% approach accuracy, 0.957 precision, 0.938 recall, 0.900 accuracy, and 0.947 F1-score, highlighting its essential role in ensuring precise and efficient harvesting.

A Leader-follower formation control of mobile robots by position-based visual servo method using fisheye camera

This paper presents a leader-follower formation control of multiple mobile robots by position-based method using a fisheye camera. A fisheye camera has a wide field of view and recognizes a wide range of objects. In this paper, the fisheye camera is first modeled on spherical coordinates and then a position estimation technique is proposed by using an AR marker based on the spherical model. This paper furthermore presents a method for estimating the velocity of a leader robot based on a disturbance observer using the obtained position information. The proposed techniques are combined with a formation control based on the virtual structure. In this paper, the formation controller and velocity estimator can be designed independently, and the stability analysis of the total system is performed by using Lyapunov theorem. The effectiveness of the proposed method is demonstrated by simulation and experiments using two real mobile robots.

An Optimized Control Strategy for Image-Based Visual Servoing Using Polar Coordinates

Image-based visual servoing is a crucial technique in robotic systems that enables accurate manipulation and control through visual feedback. In scenarios involving substantial rotational motion or changes in orientation, traditional Cartesian-based visual servoing methods may encounter limitations in terms of control performance. To address this challenge, we propose a novel approach that use coordinates for visual servoing control. By reparametrizing the control inputs in terms of polar radius and polar angle, our method offers a more intuitive and efficient way to handle rotations and angular variations. In addition, we further study the polar-based IBVS control by conducting experiments using a Proportional (P) controller as well as the Levenberg-Marquardt (LM) algorithm respectively, and the experimental results indicate that the proposed method demonstrates a faster convergence speed and better robustness.

An Innovative Collision-Free Image-Based Visual Servoing Method for Mobile Robot Navigation Based on the Path Planning in the Image Plan.

In this article, we present an innovative approach to 2D visual servoing (IBVS), aiming to guide an object to its destination while avoiding collisions with obstacles and keeping the target within the camera's field of view. A single monocular sensor's sole visual data serves as the basis for our method. The fundamental idea is to manage and control the dynamics associated with any trajectory generated in the image plane. We show that the differential flatness of the system's dynamics can be used to limit arbitrary paths based on the number of points on the object that need to be reached in the image plane. This creates a link between the current configuration and the desired configuration. The number of required points depends on the number of control inputs of the robot used and determines the dimension of the flat output of the system. For a two-wheeled mobile robot, for instance, the coordinates of a single point on the object in the image plane are sufficient, whereas, for a quadcopter with four rotating motors, the trajectory needs to be defined by the coordinates of two points in the image plane. By guaranteeing precise tracking of the chosen trajectory in the image plane, we ensure that problems of collision with obstacles and leaving the camera's field of view are avoided. Our approach is based on the principle of the inverse problem, meaning that when any point on the object is selected in the image plane, it will not be occluded by obstacles or leave the camera's field of view during movement. It is true that proposing any trajectory in the image plane can lead to non-intuitive movements (back and forth) in the Cartesian plane. In the case of backward motion, the robot may collide with obstacles as it navigates without direct vision. Therefore, it is essential to perform optimal trajectory planning that avoids backward movements. To assess the effectiveness of our method, our study focuses exclusively on the challenge of implementing the generated trajectory in the image plane within the specific context of a two-wheeled mobile robot. We use numerical simulations to illustrate the performance of the control strategy we have developed.

A Motion Planning Method for Visual Servoing Using Deep Reinforcement Learning in Autonomous Robotic Assembly

Assembly positioning by visual servoing (VS) is a basis for autonomous robotic assembly. In practice, VS control suffers potential stability and convergence problems due to image and physical constraints, e.g., field of view constraints, image local minima, obstacle collisions, and occlusion. Therefore, this article proposes a novel deep reinforcement learning-based hybrid visual servoing (DRL-HVS) controller for motion planning of VS tasks. DRL-HVS controller takes current observed image features and camera pose as inputs, and the core parameters of hybrid VS are dynamically optimized using a deep deterministic policy gradient (DDPG) algorithm to obtain an optimal motion scheme, considering image/physical constraints and robot motion performance. In addition, an adaptive exploration strategy is proposed to further improve the training efficiency by adaptively tuning the exploration noise parameters. In this way, the offline pretrained DRL-HVS controller in the virtual environment, where the DDPG actor–critic network is continuously optimized, can be quickly deployed to a real robot system for real-time control. Experiments based on an eye-in-hand VS system are conducted with a calibrated HIKVISION RGB camera mounted on the end-effector of a GSK-RB03A1 six degree-of-freedom (6-DoF) robot. Basic VS task experiments show that the proposed controller achieves better performance than the existing methods: the servoing time is 24% smaller than that of the five-dimensional VS method, a 100% success rate with the perturbed ranges of the initial position within 25 mm for translation and 20° for rotation, and a 48% efficiency improvement. Moreover, a planetary gear component assembly process case study, where the robot aims to automatically put the gears on the gear shafts, is conducted to demonstrate the applicability of the proposed method in practice.

Fully automated thyroid ultrasound screening utilizing multi-modality image and anatomical prior

There is a high prevalence of thyroid nodules in the general population. Early detection is essential for the treatment of malignant thyroid nodules. Ultrasound has the advantage of being non-invasive and radiation-free, and is sufficient for early diagnosis of malignant nodules. However, ultrasound image acquisition relies on manual manipulation of the probe by the sonographer, making it difficult to screen populations for malignant thyroid nodules. Robotic ultrasound systems (RUS) are expected to replace sonographers in ultrasound scanning and improve the standardization and reproducibility of ultrasound examinations. However, there is currently no RUS that can fully autonomously screen for thyroid nodules. In this paper, we propose a fully automated two-step search method that mimics a physician's thyroid scanning protocol. First, using the body surface structure observed by the RGBD camera, we use a bimodal detection network for the initial localization of the human neck. The bimodal detection network achieves an average accuracy of 0.986. Second, we designed a tracking strategy based on the anatomical prior of the human neck to navigate the probe for thyroid ultrasound acquisition. An network-based visual servoing method enables shadow prevention and target tracking to be performed uniformly. In vitro experiments demonstrated the effectiveness of the protocol.

A collision-free visual servoing method for two space manipulators capturing tumbling satellites

A novel visual servoing method is proposed to synchronously capture a malfunctional satellite’s docking ring by two space manipulators. Due to the lack of conspicuous visual marks, structured light is implemented to estimate the ring’s pose. To ensure the capturing process’s security, two aspects are investigated. Spatially, capturing target points for two manipulators are planned together by considering the chaser’s orientation. As a consequence, collision risk between two manipulators is eliminated. Temporally, by adding duration parameters in trajectory generation algorithm, capturing synchronism for two manipulators is guaranteed. Experimental results show that, by the proposed method, a tumbling target was successfully captured by two space manipulators. Compared to existing methods, the collision risk between two manipulators is thoroughly eliminated and the capturing synchronism is improved remarkably.

Robocentric Model-Based Visual Servoing for Quadrotor Flights

This article proposes a robocentric formulation for quadrotor visual servoing. This formulation presents the task-specific state dynamics of the quadrotor in its body reference frame. Compared with other visual servoing methods, our method allows tightly and integrated state estimation and control on the same robocentric model, and allows a faster system response in aggressive quadrotor flights. On the theory level, we prove the controllability and observability of the proposed robocentric model. Then, we design an on-manifold Kalman filter for the estimation and an on-manifold iterative model predictive control for motion planning and control. We verify our proposed formulation and controller in two crucial quadrotor flight tasks: 1) hovering; and 2) dynamic obstacle avoidance. Experiment results show that the quadrotor is able to resist large external disturbances and recover its position and orientation from two reference visual features. Moreover, the quadrotor is able to avoid dynamic obstacles reliably at a relative speed up to 7.4 m/s, demonstrating the effectiveness of our visual servoing methods in agile quadrotor flights.

Dynamic Target Tracking of Unmanned Aerial Vehicles Under Unpredictable Disturbances

This study proposes an image-based visual servoing (IBVS) method based on a velocity observer for an unmanned aerial vehicle (UAV) for tracking a dynamic target in Global Positioning System (GPS)-denied environments. The proposed method derives the simplified and decoupled image dynamics of underactuated UAVs using a constructed virtual camera and then considers the uncertainties caused by the unpredictable rotations and velocities of the dynamic target. A novel image depth model that extends the IBVS method to track a rotating target with arbitrary orientations is proposed. The depth model ensures image feature accuracy and image trajectory smoothness in rotating target tracking. The relative velocities of the UAV and the dynamic target are estimated using the proposed velocity observer. Thanks to the velocity observer, translational velocity measurements are not required, and the control chatter caused by noise-containing measurements is mitigated. An integral-based filter is proposed to compensate for unpredictable environmental disturbances in order to improve the anti-disturbance ability. The stability of the velocity observer and IBVS controller is analyzed using the Lyapunov method. Comparative simulations and multistage experiments are conducted to illustrate the tracking stability, anti-disturbance ability, and tracking robustness of the proposed method with a dynamic rotating target.