Visual Servo Control Research Articles

Robot Closed-Loop Grasping Based on Deep Visual Servoing Feature Network

Robot visual servoing for grasping has long been challenging to execute in complex visual environments because of issues with efficient feature extraction. This paper proposes a novel visual servoing grasping approach based on the Deep Visual Servoing Feature Network (DVSFN) to tackle this issue. The approach enables feasible to extract scale-invariant point features and target bounding boxes in real time by building an effective single-stage multi-dimensional feature extractor. The DVSFN is then integrated into a Levenberg–Marquardt–based image visual servoing (LM-IBVS) controller. The above creates a mapping link between the robot’s joint space and image features. The robot is then guided in positioning and grabbing by converting the difference between the expected and present features into the corresponding robot joint velocities. Experimental results demonstrate that the proposed method achieves a mean average precision (mAP) of 0.80 and 0.87 for detecting target bounding boxes and point features, respectively, in scenarios with significant lighting variations and occlusions. Under low-light and partial occlusion conditions, the method achieves an average grasping success rate approximately 80%.

Camera-based control system of a planar cable-driven parallel robot intended for functional rehabilitation

Abstract This paper investigates a closed-loop visual servo control scheme for controlling the position of a fully constrained cable-driven parallel robot (CDPR) designed for functional rehabilitation tasks. The control system incorporates real-time position correction using an Intel RealSense camera. Our CDPR features four cables exiting from pulleys, driven by AC servomotors, to move the moving platform (MP). The focus of this work is the development of a control scheme for a closed-loop visual servoing system utilizing depth/RGB images. The developed algorithm uses this data to determine the actual Cartesian position of the MP, which is then compared to the desired position to calculate the required Cartesian displacement. This displacement is fed into the inverse kinematic model to generate the servomotor commands. Three types of trajectories (circular, square, and triangular) are used to test the controller’s compliance with its position. Compared to the open-loop control of the robot, the new control system increases positional accuracy and effectively handles cable behavior, various perturbations, and modeling errors. The obtained results showed significant improvements in control performance, notably reduced root mean square error and maximal error in terms of position.

Visual Servoing Using Sliding-Mode Control with Dynamic Compensation for UAVs’ Tracking of Moving Targets

An Image-Based Visual Servoing Control (IBVS) structure for target tracking by Unmanned Aerial Vehicles (UAVs) is presented. The scheme contains two stages. The first one is a sliding-model controller (SMC) that allows one to track a target with a UAV; the control strategy is designed in the function of the image. The proposed SMC control strategy is commonly used in control systems that present high non-linearities and that are always exposed to external disturbances; these disturbances can be caused by environmental conditions or induced by the estimation of the position and/or velocity of the target to be tracked. In the second instance, a controller is placed to compensate the UAV dynamics; this is a controller that allows one to compensate the velocity errors that are produced by the dynamic effects of the UAV. In addition, the corresponding stability analysis of the sliding mode-based visual servo controller and the sliding mode dynamic compensation control is presented. The proposed control scheme employs the kinematics and dynamics of the robot by presenting a cascade control based on the same control strategy. In order to evaluate the proposed scheme for tracking moving targets, experimental tests are carried out in a semi-structured working environment with the hexarotor-type aerial robot. For detection and image processing, the Opencv C++ library is used; the data are published in an ROS topic at a frequency of 50 Hz. The robot controller is implemented in the mathematical software Matlab.

Robust pollination for tomato farming using deep learning and visual servoing

Abstract In this work, we propose a novel approach for tomato pollination that utilizes visual servo control. The objective is to meet the growing demand for automated robotic pollinators to overcome the decline in bee populations. Our approach focuses on addressing this challenge by leveraging visual servo control to guide the pollination process. The proposed method leverages deep learning to estimate the orientations and depth of detected flower, incorporating CAD-based synthetic images to ensure dataset diversity. By utilizing a 3D camera, the system accurately estimates flower depth information for visual servoing. The robustness of the approach is validated through experiments conducted in a laboratory environment with a 3D printed tomato flower plant. The results demonstrate a high detection rate, with a mean average precision of 91.2 %. Furthermore, the average depth error for accurately localizing the pollination target is impressively minimal, measuring only 1.1 cm. This research presents a promising solution for tomato pollination, showcasing the effectiveness of visual-guided servo control and its potential to address the challenges posed by diminishing bee populations in greenhouses.

Enhancing quadrotor robustness control using image-based visual servoing (IBVS) with fuzzy logic

Visual servoing is a commonly employed approach in robotics and unmanned aerial vehicles (UAVs) that facilitates accurate object positioning and movement control by utilizing visual feedback. As quadrotors gain popularity, more control methods have been developed. This article presents a method for enhancing quadrotor robustness control using image-based visual servoing (IBVS) with fuzzy logic. Unlike traditional visual servoing, which relies on a fixed gain and often encounters challenges with velocity convergence and maintaining the object in the field of view, this method is designed to enhance the visual servo control of quadrotors by dynamically adjusting the gain of the IBVS system through a fuzzy logic controller. This controller adaptively adjusts the servo gain in response to feature errors and the depth of the object's feature points. MATLAB simulations clearly demonstrate the superior performance of this fuzzy logic integrated method compared to classical approaches, showcasing enhanced control capabilities in challenging environments.

Image-Based Auto-Focus Microscope System with Visual Servo Control for Micro-Stereolithography.

Micro-stereolithography (μSL) is an advanced additive manufacturing technique that enables the fabrication of highly precise microstructures with fine feature resolution. One of the primary challenges in μSL is achieving and maintaining precise focus throughout the fabrication process. For the successful application of μSL, it is essential to maintain the sample surface within a focal depth of several microns. Despite the growing interest in auto-focus devices, limited attention has been directed towards auto-focus systems in image-based auto-focus microscope systems for precision μSL. To address this challenge, we propose an image-based auto-focus microscope system incorporating visual servo control. In the optical design, a transflective beam splitter is employed, allowing the laser beam to pass through for fabrication while reflecting the focused beam on the sample surface to the microscope and camera. Utilizing captured spot images and the Foucault knife-edge test, a deep learning-based laser spot image processing algorithm is developed to determine the focus position based on spot size and the number of spot pixels on both sides. Experimental results demonstrate that the proposed auto-focus system effectively determines the relative position of the focal point using the laser spot image and achieves auto-focusing through visual servo control.

Adaptive visual servoing control for uncalibrated robot manipulator with uncertain dead-zone constraint

This paper addresses the adaptive visual tracking control problem of an uncalibrated camera robot system with unknown dead-zone inputs. The uncertainties include camera extrinsic and intrinsic parameters, robot dynamic parameters, and feature depth parameters. The control achieves a better performance by establishing a smooth inverse model of the dead-zone input, which eliminates the nonlinear effect of the actuator constraint. A novel adaptive control scheme is presented which does not require a priori knowledge of the parameter intervals of dead-zone model, to update the parameter values online. Furthermore, adaptive laws are provided to estimate the uncertainties that exist both in robot and in camera models. Meanwhile, in order to avoid the singularity of the image Jacobian matrix, a projection algorithm is also introduced. Subsequently, a novel robust adaptive controller together with dead-zone compensation is established so that the tracking error image space converges to the origin. Simulations and experimental studies are conducted to verify the effectiveness of the proposed method.

Neural Network-Based Robust Guaranteed Cost Control for Image-Based Visual Servoing of Quadrotor.

In this article, a neural network (NN)-based robust guaranteed cost control design is proposed for image-based visual servoing (IBVS) control of quadrotors. According to the dynamics of three subsystems (yaw, height, and lateral subsystems) derived from the quadrotor IBVS dynamic model, the main control design is to solve the robust control problem for the time-varying lateral subsystem with angle constraints and uncertain disturbances. Considering the system dynamics, a two-loop structure is conducted. The outer loop uses the linear quadratic regulator to solve the Riccati equation for the lateral image feature system, and the inner loop adopts the optimal robust guaranteed cost control to solve the lateral velocity system. For the lateral velocity system, the optimal robust control problem is transformed to solve the modified Hamilton-Jacobi-Bellman equation of the corresponding optimal control problem utilizing adaptive dynamic programming. The implementation is accomplished with the time-varying NN and the designed estimated weight update law. In addition, the stability and effectiveness are proved by the theoretic proof and simulations.

Image-Based Distributed Predictive Visual Servo Control for Cooperative Tracking of Multiple Fixed-Wing UAVs

Image-Based Distributed Predictive Visual Servo Control for Cooperative Tracking of Multiple Fixed-Wing UAVs

Discrete-Time Visual Servoing Control with Adaptive Image Feature Prediction Based on Manipulator Dynamics.

In this paper, a practical discrete-time control method with adaptive image feature prediction for the image-based visual servoing (IBVS) scheme is presented. In the discrete-time IBVS inner-loop/outer-loop control architecture, the time delay caused by image capture and computation is noticed. Considering the dynamic characteristics of a 6-DOF manipulator velocity input system, we propose a linear dynamic model to describe the motion of a robot end effector. Furthermore, for better estimation of image features and smoothing of the robot's velocity input, we propose an adaptive image feature prediction method that employs past image feature data and real robot velocity data to adopt the prediction parameters. The experimental results on a 6-DOF robotic arm demonstrate that the proposed method can ensure system stability and accelerate system convergence.

A novel concurrent learning-based fixed-time convergent visual depth observer for weakly persistently exciting perspective dynamical systems

The problem of synthesizing a fixed-time depth observer based on monocular image feedback is considered in this study. The key challenge lies in the fact that the perspective dynamical system used to model visual motion is typically characterized as being weakly persistently exciting, which complicates observer synthesis. Furthermore, the key to achieving the task objective (safe obstacle avoidance, for instance) lies in the synthesis of a depth observer that achieves rapid convergence of the (static) obstacle depth estimate to the ground truth in a known fixed-time. To address these challenges, and in contrast with prior schemes that rely on a motion-restrictive persistency of excitation (PE) condition for ensuring exponential convergence, a novel adaptive observer framework is considered in this study that incorporates a concurrent learning (CL) term for ensuring fixed-time observer convergence. In particular, the use of concurrent learning allows for the synthesis of a relaxed finite-time excitation condition that relies on historical data recorded over a dynamic sliding window in the recent past that the proposed observer relies on to ensure fixed-time convergence. Thus, a continuous-time reduced-order observer formulation is presented that relies on camera motion data to achieve fixed-time convergence to a uniform ultimate bound for a suitably large choice of the observer gains. Experimental results are used to demonstrate the efficacy of the proposed scheme in the presence of significant measurement noise. A performance comparison study is also undertaken to demonstrate superior performance of the proposed scheme compared to leading alternative designs. Finally, the practical applicability of the proposed scheme is verified by incorporating the proposed scheme within a reactive navigation scheme to accomplish obstacle avoidance. By incorporating a suitably informative CL term within the observer framework, the proposed scheme eliminates the need to rely on a difficult-to-verify PE condition, thus rendering it more suitable for practical applications like visual target-tracking and visual servo control.

Visual Servoing and Kalman Filter Applied to Parallel Manipulator 3-RRR

This study introduces a novel methodology integrating computer vision, visual servo control, and the Kalman Filter to precisely estimate object locations for a 3-RRR planar type parallel manipulator. Through kinematic analysis and the development of a vision system using color indicators, the research enhances the ability of the manipulator to track object trajectories, especially in cases of occlusion. Employing Eye-to-Hand visual servo control, the research further refines the visual orientation of the sensor for optimal end effector and object identification. The incorporation of the Kalman Filter as a robust estimator for occluded objects underscores the predictive accuracy of the system. Results demonstrate the effectiveness of the methodology in trajectory generation and object tracking, with potential implications for improving robotic manipulators in dynamic environments. This comprehensive approach not only advances the fields of kinematic control and visual servoing but also opens new avenues for future research in complex spatial manipulations.

Nonlinear Observer-Based Visual Servoing and Vibration Control of Flexible Robotic Manipulators With a Fixed Camera.

This article studies the visual servoing and vibration suppression control for flexible manipulators when the system states are unmeasurable and only the image feedback is available. The dynamic equations of flexible manipulators are decomposed into the slow and fast subsystems based on the singular perturbation theory. The nonlinear observers based on the state transformation using the Lie derivatives are proposed to estimate the unmeasurable system states and unknown camera intrinsic parameters at the same time. Then, the image-based controllers utilizing the estimated states are, respectively, designed in the slow and fast subsystems to regulate the image positions of feature points and suppress the vibration of flexible manipulators simultaneously. In the proposed approach, only the visual feedback is required to generate the control input for flexible manipulators, which simplifies the controller implementation. The stability of the proposed control scheme is proved based on the Lyapunov theory. Finally, experimental results on a flexible single-link manipulator are provided to demonstrate the effectiveness of the proposed control approach.

Automated Piezo-Assisted Sperm Immobilization

Sperm immobilization is a crucial procedure in clinical cell surgery for infertility treatment. Current immobilization is implemented by tapping the sperm tail with a glass micropipette, but its effectiveness is restricted by sperm orientation and ineffective membrane ablation. Ineffective ablation leads to limited release of oocyte activating factors and lowers fertilization rate; and sperm swim in small angles relative to the micropipette tip cannot be tapped due to the risk of damaging the sperm’s genetic materials contained in the sperm head. This paper reports automated piezo-assisted sperm immobilization with enhanced efficacy of cell membrane ablation and sperm orientation control. The designed piezo drill consists of two orthogonal vibration modules to generate controlled micropipette vibration along axial and lateral axes. Through stiffness modeling, the flexure joints guide the motion of the central beam of each vibration module. To achieve sperm orientation control, whirl flow is induced by both axial and lateral vibration of the micropipette tip. To immobilize sperm, only micropipette’s axial vibration is generated to prevent lateral vibration from damaging sperm head. A visual servoing scheme is developed by decoupling sperm wiggling from positioning error for immobilization. Experimental results showed that sperm orientation control by the piezo drill achieved an error of 1.4 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$^{\circ}$</tex-math> </inline-formula> and a time cost of 2.5 s. Visual servoing with sperm wiggling decoupling achieved a positioning error of 1.7 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mu$</tex-math> </inline-formula> m. Furthermore, the piezo-assisted sperm immobilization technique led to effective membrane ablation. With membrane-impermeable stains, it took 5.6 s for the immobilized sperm to be stained after piezo-assisted immobilization, significantly less than 49.2 s by conventional micropipette tapping. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This work tackled the challenge of ineffective membrane ablation and orientation limit in clinical cell surgeries. Conventional manual immobilization suffers from low membrane ablation efficacy, which leads to limited release of oocyte activating factors and lowers fertilization rate. Moreover, sperm swim in small angles relative to the micropipette tip cannot be tapped due to the risk of damaging the sperm’s genetic materials contained in the sperm head. In this paper, we propose automation techniques for effective membrane ablation and orientation control of sperm. A clinically compatible piezo drill is developed to generate controllable micropipette motion along both axial and lateral directions. The whirl flow generated by micropipette vibration is employed to rotate sperm, which greatly increased the number of available sperm for immobilization. A visual servoing controller is developed to keep the sperm at the center of field of view for immobilization by decoupling sperm wiggling from positioning error. The developed methods can be generalized to the manipulation of other types of cells. The piezo drill can be used for effective membrane ablation of oocyte, embryo, yeast cell and so on. The orientation control strategy leveraging piezo-induced whirl flow is applicable to non-contact rotation of a variety of microorganism.

High-Precision Drilling by Anchor-Drilling Robot Based on Hybrid Visual Servo Control in Coal Mine

Rock bolting is a commonly used method for stabilizing the surrounding rock in coal-mine roadways. It involves installing rock bolts after drilling, which penetrate unstable rock layers, binding loose rocks together, enhancing the stability of the surrounding rock, and controlling its deformation. Although recent progress in drilling and anchoring equipment has significantly enhanced the efficiency of roof support in coal mines and improved safety measures, how to deal with drilling rigs’ misalignment with the through-hole center remains a big issue, which may potentially compromise the quality of drilling and consequently affect the effectiveness of bolt support or even result in failure. To address this challenge, this article presents a robotic teleoperation system alongside a hybrid visual servo control strategy. Addressing the demand for high precision and efficiency in aligning the drilling rigs with the center of the drilling hole, a hybrid control strategy is introduced combining position-based and image-based visual servo control. The former facilitates an effective approach to the target area, while the latter ensures high-precision alignment with the center of the drilling hole. The robot teleoperation system employs the binocular vision measurement system to accurately determine the position and orientation of the drilling-hole center, which serves as the designated target position for the drilling rig. Leveraging the displacement and angle sensor information installed on each joint of the manipulator, the system utilizes the kinematic model of the manipulator to compute the spatial position of the end-effector. It dynamically adjusts the spatial pose of the end-effector in real time, aligning it with the target position relative to its current location. Additionally, it utilizes monocular vision information to fine-tune the movement speed and direction of the end-effector, ensuring rapid and precise alignment with the target drilling-hole center. Experimental results demonstrate that this method can control the maximum alignment error within 7 mm, significantly enhancing the alignment accuracy compared to manual control. Compared with the manual control method, the average error of this method is reduced by 41.2%, and the average duration is reduced by 4.3 s. This study paves a new path for high-precision drilling and anchoring of tunnel roofs, thereby improving the quality and efficiency of roof support while mitigating the challenges associated with significant errors and compromised safety during manual control processes.

Fast Convergent Antinoise Dual Neural Network Controller With Adaptive Gain for Flexible Endoscope Robots.

Manual rigid endoscopes have defects such as a low efficiency, difficult operation, and safety risks, and the antinoise interference ability, convergence speed, and control accuracy of the neural network control technology for the existing autonomous endoscopes are often ignored. Solving these problems is important for the stable operation of endoscopes. Therefore, a new adaptive fast convergent antinoise dual neural network (AFA-DNN) controller for the visual servo control of ten-degree of freedom flexible endoscope robots (FERs) with physical constraints is proposed in this work. First, the control scheme of the FERs is formulated as a quadratic programming problem, and then, an AFA-DNN visual servo controller is designed for the FERs. The adaptive gains of the controller can accelerate the convergence, improve the antinoise ability, and increase the convergence accuracy of the controller. Then, according to the Lyapunov theory, the fast convergence of the AFA-DNN in finite time is proven for both noise-free and noisy conditions. The experimental results indicate that the FER controlled by the proposed AFA-DNN can accurately track various trajectories and that the AFA-DNN has a better antinoise interference ability, higher convergence accuracy, and faster convergence speed than conventional methods. The convergence speed of the AFA-DNN is increased by a factor of 4.22 by using the adaptive gains. Experiments also indicate that the AFA-DNN remains well functioning under various noise disturbances (such as constant, periodic, linear, and Gaussian noise).

A High-Precision Hand-Eye Coordination Localization Method under Convex Relaxation Optimization.

Traditional switching operations require on-site work, and the high voltage generated by arc discharges can pose a risk of injury to the operator. Therefore, a combination of visual servo and robot control is used to localize the switching operation and construct hand-eye calibration equations. The solution to the hand-eye calibration equations is coupled with the rotation matrix and translation vectors, and it depends on the initial value determination. This article presents a convex relaxation global optimization hand-eye calibration algorithm based on dual quaternions. Firstly, the problem model is simplified using the mathematical tools of dual quaternions, and then the linear matrix inequality convex optimization method is used to obtain a rotation matrix with higher accuracy. Afterwards, the calibration equations of the translation vectors are rewritten, and a new objective function is established to solve the coupling influence between them, maintaining positioning precision at approximately 2.9 mm. Considering the impact of noise on the calibration process, Gaussian noise is added to the solutions of the rotation matrix and translation vector to make the data more closely resemble the real scene in order to evaluate the performance of different hand-eye calibration algorithms. Eventually, an experiment comparing different hand-eye calibration methods proves that the proposed algorithm is better than other hand-eye calibration algorithms in terms of calibration accuracy, robustness to noise, and stability, satisfying the accuracy requirements of switching operations.

A robot-assisted tracheal intubation system based on a soft actuator?

Tracheal intubation is the gold standard of airway protection and constitutes a pivotal life-saving technique frequently employed in emergency medical interventions. Hence, in this paper, a system is designed to execute tracheal intubation tasks automatically, offering a safer and more efficient solution, thereby alleviating the burden on physicians. The system comprises a tracheal tube with a bendable front end, a drive system, and a tip endoscope. The soft actuator provides two degrees of freedom for precise orientation. It is fabricated with varying-hardness silicone and reinforced with fibers and spiral steel wire for flexibility and safety. The hydraulic actuation system and tube feeding mechanism enable controlled bending and delivery. Object detection of key anatomical features guides the robotic arm and soft actuator. The control strategy involves visual servo control for coordinated robotic arm and soft actuator movements, ensuring accurate and safe tracheal intubation. The kinematics of the soft actuator were established using a constant curvature model, allowing simulation of its workspace. Through experiments, the actuator is capable of 90° bending as well as 20° deflection on the left and right sides. The maximum insertion force of the tube is 2N. Autonomous tracheal intubation experiments on a training manikin were successful in all 10 trials, with an average insertion time of 45.6s. Experimental validation on the manikin demonstrated that the robot tracheal intubation system based on a soft actuator was able to perform safe, stable, and automated tracheal intubation. In summary, this paper proposed a safe and automated robot-assisted tracheal intubation system based on a soft actuator, showing considerable potential for clinical applications.

Crop Recognition Method Based On End-effector

With the increasing global population, the demand for agriculture is also on the rise. The crucial stages of agricultural production, namely fruit identification and picking, play a vital role in enhancing product quality and minimizing losses. Traditional manual processing methods, although time-tested, are not only inefficient but also challenging to maintain consistency, making them inadequate to meet the large-scale requirements of modern agricultural production. Consequently, the integration of automation technology has become a necessity. For agricultural robot the machine vision system often need to work in two typical environments, the field environment and the orchard environment. Depending on the varying objectives of operation in diverse environments, crop robots require the ability to rapidly identify fruits within images featuring significant color disparities between crops and two-dimensional field backgrounds. Consequently, a visual servo control system is being investigated. A novel camera attitude search method that employs active visual servo technology to minimize occlusions during orchard search is proposed. The recognition function of the end-effector is exceedingly crucial. The precision of the end effector's identification capabilities directly influences the success rate of automated operations. This is particularly evident in fruit picking, sorting, and other tasks, where the diverse shapes, maturity levels, and colors of fruits present significant challenges to the robotic arm's end effector. The rapid advancement of deep learning technology, however, offers a novel solution for the recognition of fruits by the robotic arm's end effector. By emulating the human visual system, deep learning models can extract the feature representation of fruits from vast amounts of data, enabling accurate identification of fruits in various conditions.

Model-free visual servoing based on active disturbance rejection control and adaptive estimator for robotic manipulation without calibration

PurposeVisual feedback control is a promising solution for robots work in unstructured environments, and this is accomplished by estimation of the time derivative relationship between the image features and the robot moving. While some of the drawbacks associated with most visual servoing (VS) approaches include the vision–motor mapping computation and the robots’ dynamic performance, the problem of designing optimal and more effective VS systems still remains challenging. Thus, the purpose of this paper is to propose and evaluate the VS method for robots in an unstructured environment.Design/methodology/approachThis paper presents a new model-free VS control of a robotic manipulator, for which an adaptive estimator aid by network learning is proposed using online estimation of the vision–motor mapping relationship in an environment without the knowledge of statistical noise. Based on the adaptive estimator, a model-free VS schema was constructed by introducing an active disturbance rejection control (ADRC). In our schema, the VS system was designed independently of the robot kinematic model.FindingsThe various simulations and experiments were conducted to verify the proposed approach by using an eye-in-hand robot manipulator without calibration and vision depth information, which can improve the autonomous maneuverability of the robot and also allow the robot to adapt its motion according to the image feature changes in real time. In the current method, the image feature trajectory was stable in the camera field range, and the robot’s end motion trajectory did not exhibit shock retreat. The results showed that the steady-state errors of image features was within 19.74 pixels, the robot positioning was stable within 1.53 mm and 0.0373 rad and the convergence rate of the control system was less than 7.21 s in real grasping tasks.Originality/valueCompared with traditional Kalman filtering for image-based VS and position-based VS methods, this paper adopts the model-free VS method based on the adaptive mapping estimator combination with the ADRC controller, which is effective for improving the dynamic performance of robot systems. The proposed model-free VS schema is suitable for robots’ grasping manipulation in unstructured environments.