Visual Servoing Scheme Research Articles

Vision-Based Localization Method for Picking Points in Tea-Harvesting Robots

To address the issue of accurately recognizing and locating picking points for tea-picking robots in unstructured environments, a visual positioning method based on RGB-D information fusion is proposed. First, an improved T-YOLOv8n model is proposed, which improves detection and segmentation performance across multi-scale scenes through network architecture and loss function optimizations. In the far-view test set, the detection accuracy of tea buds reached 80.8%; for the near-view test set, the mAP0.5 values for tea stem detection in bounding boxes and masks reached 93.6% and 93.7%, respectively, showing improvements of 9.1% and 14.1% over the baseline model. Secondly, a layered visual servoing strategy for near and far views was designed, integrating the RealSense depth sensor with robotic arm cooperation. This strategy identifies the region of interest (ROI) of the tea bud in the far view and fuses the stem mask information with depth data to calculate the three-dimensional coordinates of the picking point. The experiments show that this method achieved a picking point localization success rate of 86.4%, with a mean depth measurement error of 1.43 mm. The proposed method improves the accuracy of picking point recognition and reduces depth information fluctuations, providing technical support for the intelligent and rapid picking of premium tea.

A Data-Driven Image-Based Visual Servoing Scheme for Redundant Manipulators With Unknown Structure and Singularity Solution

A Data-Driven Image-Based Visual Servoing Scheme for Redundant Manipulators With Unknown Structure and Singularity Solution

Research on visual servo strategy for the uncalibrated non-standard manipulator

Abstract Aiming at the problem that the hand-eye calibration of the uncalibrated non-standard manipulator cannot be addressed by the general calibration method, i.e., the eye-in-hand and eye-to-hand calibration method of the visual servo system was proposed to guide the manipulator without calibration. First, a camera was installed on the manipulator to complete the eye-in-hand calibration and achieve transformation from the end coordinate system to the camera coordinate system. Second, a second camera was used to achieve eye-to-hand calibration and obtain the transformation relationship from the end coordinate system to the calibration board. The entire calibration process only requires the collection of images from two scenarios, i.e., the docking and the completion of the docking processes. The implementation results show that the rotation error between groups is less than 1.5°, and the translation error between groups is less than 8 mm. Therefore, the proposed visual servo strategy can accurately guide the uncalibrated non-standard manipulator.

Discrete-Time Visual Servoing Control with Adaptive Image Feature Prediction Based on Manipulator Dynamics.

In this paper, a practical discrete-time control method with adaptive image feature prediction for the image-based visual servoing (IBVS) scheme is presented. In the discrete-time IBVS inner-loop/outer-loop control architecture, the time delay caused by image capture and computation is noticed. Considering the dynamic characteristics of a 6-DOF manipulator velocity input system, we propose a linear dynamic model to describe the motion of a robot end effector. Furthermore, for better estimation of image features and smoothing of the robot's velocity input, we propose an adaptive image feature prediction method that employs past image feature data and real robot velocity data to adopt the prediction parameters. The experimental results on a 6-DOF robotic arm demonstrate that the proposed method can ensure system stability and accelerate system convergence.

A Physics-Informed Neural Network Approach to Augmented Dynamics Visual Servoing of Multirotors.

This article presents a visual servoing strategy that integrates the capabilities of a physics-informed neural network (PINN) to estimate system uncertainties and inaccuracies with a dynamics-centered visual servoing technique for multirotors. The proposed method effectively combines these approaches, eliminating the need for inverse Jacobian calculations to determine multirotor motion by directly relating pixel variations to the multirotor's torque and thrust inputs, while also strengthening the method's robustness through the utilization of the PINN to model and address uncertainties in camera and multirotor parameters, as well as the modeling inaccuracies inherent in the dynamics-centered visual servoing technique. In contrast to existing state-of-the-art data-driven approaches, the proposed PINN approach requires, on average, 65% less labeled data to characterize uncertainties and inaccuracies. To ensure real-time implementation of the visual servoing model, the PINN-learned model is combined with an adaptive horizon monotonically weighted nonlinear model predictive controller (NMPC), capable of processing control efforts at rates 10 times faster than existing Tube MPC and Adaptive MPC strategies. These findings are validated through real-time trajectory tracking experiments, which not only highlight the effectiveness of the proposed approach in approximating modeling inaccuracies but also its capability in handling uncertainties upto 70% in camera parameters.

Automated Piezo-Assisted Sperm Immobilization

Sperm immobilization is a crucial procedure in clinical cell surgery for infertility treatment. Current immobilization is implemented by tapping the sperm tail with a glass micropipette, but its effectiveness is restricted by sperm orientation and ineffective membrane ablation. Ineffective ablation leads to limited release of oocyte activating factors and lowers fertilization rate; and sperm swim in small angles relative to the micropipette tip cannot be tapped due to the risk of damaging the sperm’s genetic materials contained in the sperm head. This paper reports automated piezo-assisted sperm immobilization with enhanced efficacy of cell membrane ablation and sperm orientation control. The designed piezo drill consists of two orthogonal vibration modules to generate controlled micropipette vibration along axial and lateral axes. Through stiffness modeling, the flexure joints guide the motion of the central beam of each vibration module. To achieve sperm orientation control, whirl flow is induced by both axial and lateral vibration of the micropipette tip. To immobilize sperm, only micropipette’s axial vibration is generated to prevent lateral vibration from damaging sperm head. A visual servoing scheme is developed by decoupling sperm wiggling from positioning error for immobilization. Experimental results showed that sperm orientation control by the piezo drill achieved an error of 1.4 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$^{\circ}$</tex-math> </inline-formula> and a time cost of 2.5 s. Visual servoing with sperm wiggling decoupling achieved a positioning error of 1.7 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mu$</tex-math> </inline-formula> m. Furthermore, the piezo-assisted sperm immobilization technique led to effective membrane ablation. With membrane-impermeable stains, it took 5.6 s for the immobilized sperm to be stained after piezo-assisted immobilization, significantly less than 49.2 s by conventional micropipette tapping. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This work tackled the challenge of ineffective membrane ablation and orientation limit in clinical cell surgeries. Conventional manual immobilization suffers from low membrane ablation efficacy, which leads to limited release of oocyte activating factors and lowers fertilization rate. Moreover, sperm swim in small angles relative to the micropipette tip cannot be tapped due to the risk of damaging the sperm’s genetic materials contained in the sperm head. In this paper, we propose automation techniques for effective membrane ablation and orientation control of sperm. A clinically compatible piezo drill is developed to generate controllable micropipette motion along both axial and lateral directions. The whirl flow generated by micropipette vibration is employed to rotate sperm, which greatly increased the number of available sperm for immobilization. A visual servoing controller is developed to keep the sperm at the center of field of view for immobilization by decoupling sperm wiggling from positioning error. The developed methods can be generalized to the manipulation of other types of cells. The piezo drill can be used for effective membrane ablation of oocyte, embryo, yeast cell and so on. The orientation control strategy leveraging piezo-induced whirl flow is applicable to non-contact rotation of a variety of microorganism.

An Advanced IBVS-Flatness Approach for Real-Time Quadrotor Navigation: A Full Control Scheme in the Image Plane

This article presents an innovative method for planning and tracking the trajectory in the image plane for the visual control of a quadrotor. The community of researchers working on 2D control widely recognizes this challenge as complex, because a trajectory defined in image space can lead to unpredictable movements of the robot in Cartesian space. While researchers have addressed this problem for mobile robots, quadrotors continue to face significant challenges. To tackle this issue, the adopted approach involves considering the separation of altitude control from the other variables, thus reducing the workspace. Furthermore, the movements of the quadrotor (pitch, roll, and yaw) are interdependent. Consequently, the connection between the inputs and outputs cannot be reversed. The task complexity becomes significant. To address this issue, we propose the following scenario: When the quadrotor is equipped with a downward-facing camera, flying at high altitude is sensible to spot a target. However, to minimize disturbances and conserve energy, the quadrotor needs to descend in altitude. This can result in the target being lost. The solution to this problem is a new methodology based on the principle of differential flatness, allowing the separation of altitude control from the other variables. The system first detects the target at high altitude, then plots a trajectory in the image coordinate system between the acquired image and the desired image. It is crucial to emphasize that this step is performed offline, ensuring that the image processing time does not affect the control frequency. Through the proposed trajectory planning, complying with the constraints of differential flatness, the quadrotor can follow the imposed dynamics. To ensure the tracking of the target while following the generated trajectory, the proposed control law takes the form of an Image Based Visual Servoing (IBVS) scheme. We validated this method using the RVCTOOLS environment in MATLAB. The DJI Phantom 1 quadrotor served as a testbed to evaluate, under real conditions, the effectiveness of the proposed control law. We specifically designed an electronic card to transfer calculated commands to the DJI Phantom 1 control joystick via Bluetooth. This card integrates a PIC18F2520 microcontroller, a DAC8564 digital-to-analogue converter, and an RN42 Bluetooth module. The experimental results demonstrate the effectiveness of this method, ensuring the precise tracking of the target as well as the accurate tracking of the path generated in the image coordinate system.

Two-phase visual servoing for capturing tumbling non-cooperative satellites with a space manipulator

In this paper, a visual servoing approach is developed to capture the docking rings of tumbling non-cooperative satellites with a space manipulator. The primary challenge addressed is the potential for the docking ring to leave the monocular camera’s field-of-view as the manipulator approaches the target, due to the ring’s large size. To solve this issue, a two-phase visual servoing scheme combining a monocular camera and a three-line structured light vision system is proposed. In an effort to augment the success rate and safety of capture operations, several constraints are formulated, encompassing manipulator’s kinematics, monocular camera’s field-of-view, obstacle avoidance, structured light’s breakpoints and smooth capture. Subsequently, a nonlinear model predictive controller is proposed to manage these constraints in real-time and regulate the system. System models are established based on image moments and pose for each phase, selecting these features as visual feedback to simplify the formulation of servo constraints and avoid the complex circle-based pose measurement. Furthermore, to ensure unbiased predictions, the model disturbances arising from the imprecise estimation of target motion parameter are observed using an extended Kalman filter, which are then incorporated into the predictive control framework. The simulation results demonstrate the effectiveness of this scheme.

A data-driven acceleration-level scheme for image-based visual servoing of manipulators with unknown structure.

The research on acceleration-level visual servoing of manipulators is crucial yet insufficient, which restricts the potential application range of visual servoing. To address this issue, this paper proposes a quadratic programming-based acceleration-level image-based visual servoing (AIVS) scheme, which considers joint constraints. Besides, aiming to address the unknown problems in visual servoing systems, a data-driven learning algorithm is proposed to facilitate estimating structural information. Building upon this foundation, a data-driven acceleration-level image-based visual servoing (DAIVS) scheme is proposed, integrating learning and control capabilities. Subsequently, a recurrent neural network (RNN) is developed to tackle the DAIVS scheme, followed by theoretical analyses substantiating its stability. Afterwards, simulations and experiments on a Franka Emika Panda manipulator with eye-in-hand structure and comparisons among the existing methods are provided. The obtained results demonstrate the feasibility and practicality of the proposed schemes and highlight the superior learning and control ability of the proposed RNN. This method is particularly well-suited for visual servoing applications of manipulators with unknown structure.

A Study on Robust Finite-Time Visual Servoing with a Gyro-Stabilized Surveillance System

This article presents the design and validation of a novel visual servoing scheme for a surveillance system. In this system, a two-axis gimbal mechanism operates the rotation of a camera which is able to provide visual information on the tracked target for the control system. The control objective is to bring the target’s projection to the center of the image plane with the smallest steady-state error and a smooth transient response, even with the unpredictable motion of the target and the influence of external disturbances. To fulfill these tasks, the proposed control scheme is designed consisting of two parts: (1) an observer estimates simultaneously the matched and unmatched disturbances; and (2) a motion control law guarantees the finite-time stability and visual servoing performance. Finally, experiments are conducted for validation and evaluation. The proposed control system shows its consistency and ought to perform better than previous approaches.

Design and Optimal Pose‐Constrained Visual Servoing of a Novel Active Flexible Endoscope Holder System for Solo Laparoscopic Surgery

Laparoscopic surgery (LS) has become an effective and widely accepted therapy for patients. However, performing LS requires advanced training and skills, as it demands high precision and dexterity due to visual feedback depending on the collaboration between the surgeon and endoscope assistant, restricted field of view, and limited dexterity inside the body. Thus, a flexible endoscope holder system with an automatic control scheme is proposed to assist LS performance, allowing the surgeon to perform solo LS. A DNA‐inspired helix‐based structure with wide‐angle and constant curvature bending is presented and made of printable nylon with selective laser sintering. The proposed flexible mechanism reduces the cost of manufacturing and assembling. Also, it has a hollow design for embedded sensing and actuation. Then, an optimization method is proposed, which provides a reference for the endoscope installation, robot placement, and pose selection. In addition, aiming at the misorientation problem, a pose‐constrained visual servoing control scheme for the endoscope system is developed. Both simulation and physical experiments are conducted to verify the effectiveness and feasibility of the proposed control scheme. Experimental results demonstrate that the proposed visual servoing scheme can maintain the optimized pose and reduce misorientation during automatic view steering.

A general visual-servoing control strategy for active non-cooperative target tracking of space manipulators

Non-cooperative target tracking plays an important role in many space missions. Recent studies on non-cooperative target tracking are mostly concentrated in the field of passive target tracking. Passive non-cooperative target tracking can effectively be used in the initial phases of a close-range rendezvous manoeuvre. Instead, during the terminal phases of such manoeuvres, active object tracking(AOT) techniques are more convenient. This study propose a general visual servo strategy for the active non-cooperative target tracking of space manipulator. Our proposed method do not need any a priori information about the targets and can be used for space manipulators of any configuration and binocular cameras of any types. Our strategy includes a real-time and accurate position estimation algorithm for non-cooperative targets and a motion planning algorithm combined Position-based visual servo(PBVS) and Image-based visual servo(IBVS). The experiment verified that the proposed strategy well solved the active non-cooperative target tracking task and can be used as a baseline for solving such tracking problems which provides a reference for subsequent studies.

Hybrid vision/strain-based control strategy for a parallel manipulator with flexible links

Parallel manipulators may present higher speed/acceleration ratios and energy efficiency when compared with serial manipulators. In addition, reducing their components’ inertia can further improve performance. However, this design alternative might yield vibrations requiring novel joint and task space control strategies. While the former involves precise models, the latter might require adequate computation vision schemes. These requirements impose critical challenges. This manuscript proposes a hybrid control strategy and experimentally investigates using a 3RRR prototype with flexible links. This hybrid strategy comprises two control loops: position-based visual servo and strain-based feedback control schemes. The strain-based loop uses a sliding mode control and the time derivative of the signals obtained by strain gauges installed in the flexible links. Compared with the position-based visual servo scheme, the hybrid strategy can considerably attenuate the vibrations since the approach reduced the overshoot response by 36% and the maximum value of the Euclidean error on position by 38%, on average.

Automatic Assembly Technology of Dense Small Screws for Flat Panel Parts

Screw assemblies of radar are characterized by numerous densely distributed small screws of various types, rendering their manual assembly difficult and inefficient. To improve the assembly quality and efficiency, an automatic assembly system was developed. First, the position and attitude relationships among the workpiece, camera, and base coordinate systems of the manipulator were obtained using three-point location and nine-point calibration methods. Second, a visual servoing strategy is designed for guiding the manipulator such that its position can be adjusted adaptively to achieve the high-precision positioning of threaded holes. Third, a method for evaluating assembly quality is developed. The templates of angle–torque curves of various screws were produced by fitting experimental data. Furthermore, the assembly quality was determined by calculating the similarity between the angle–torque curves and templates based on the Fréchet distance. Finally, the feasibility was verified using a flat plate component with five threaded holes.

Robust homography-based visual servo control for a quadrotor UAV tracking a moving target

In this study, a new robust homography-based visual tracking control approach for the quadrotor unmanned aerial vehicle (UAV) is developed. Specifically, employing the homography matrix as feedback, a hierarchical homography-based visual servoing (HBVS) scheme with a new command attitude extraction method to account for the underactuation of UAV is proposed. On this basis, a smooth hyperbolic tangent function is fulfilled as an augmented part of the backstepping control scheme, which guarantees the non-negative total thrust and avoid singularity. Additionally, a cascaded filter-based estimator and adaptive laws with integrable functions are embedded to counteract uncertainties including external perturbations, unknown acceleration of the moving target, and unknown image depth, and to facilitate the system’s asymptotic stability simultaneously. The theoretical analysis testifies that the whole close-loop system is asymptotically stable. Simulations further verify that the proposed HBVS controller can realize the visual tracking with a superior performance.

Autonomous detection and sorting of litter using deep learning and soft robotic grippers.

Road infrastructure is one of the most vital assets of any country. Keeping the road infrastructure clean and unpolluted is important for ensuring road safety and reducing environmental risk. However, roadside litter picking is an extremely laborious, expensive, monotonous and hazardous task. Automating the process would save taxpayers money and reduce the risk for road users and the maintenance crew. This work presents LitterBot, an autonomous robotic system capable of detecting, localizing and classifying common roadside litter. We use a learning-based object detection and segmentation algorithm trained on the TACO dataset for identifying and classifying garbage. We develop a robust modular manipulation framework by using soft robotic grippers and a real-time visual-servoing strategy. This enables the manipulator to pick up objects of variable sizes and shapes even in dynamic environments. The robot achieves greater than 80% classified picking and binning success rates for all experiments; which was validated on a wide variety of test litter objects in static single and cluttered configurations and with dynamically moving test objects. Our results showcase how a deep model trained on an online dataset can be deployed in real-world applications with high accuracy by the appropriate design of a control framework around it.

IBVS based on adaptive sliding mode control for a quadrotor target tracking under perturbations

This paper presents the design of a visual control formulated on an adaptive sliding mode controller for a quadrotor executing a target tracking task subject to disturbances. An image projection of the target from a virtual camera approach, and an image-based visual servoing technique are considered to obtain a singularity-free set of image features to control the position and yaw of the rotorcraft. While, an adaptive sliding mode strategy improves the robustness against bounded external perturbations and uncertainties and provides adaptivity to the visual servoing scheme. Furthermore, an analysis based on Lyapunov theory provides sufficient conditions that guarantee the stability of the closed-loop system. A comparison of the proposed adaptive visual servoing against two recent visual servoing strategies is provided, showing superiority in simulation results. Finally, experimental tests of a Parrot AR.Drone 2.0 tracking a static and moving target further demonstrates the advantages and performance.

Hybrid Eye-in-Hand/Eye-to-Hand Image Based Visual Servoing for Soft Continuum Arms

Soft continuum arms (SCAs) that are controlled by visual servoing (VS) present trade-offs between the camera range and tracking accuracy. Cameras placed at a distance (eye-to-hand) can observe a larger workspace area and the SCA tip, while a camera at the end effector (eye-in-hand) can more accurately survey the target. In this letter, we present a hybrid eye-to-hand and eye-in-hand VS scheme to track a desired object in the SCA's worksapce. When the target is not in the field-of-view of the tip camera, hand-to-eye VS is implemented using a wide field-of-view camera on the soft robot's base, to servo the soft robot's tip to a feasible region where the target is expected to be seen by the tip camera. This region is estimated by solving an optimization problem that finds the best region to place the SCA assuming a constant curvature model for the SCA. When the target is seen by the tip camera, the system switches to a hand-in-eye controller that keeps the target in the desired image position of the tip camera. Experimental results on the popular <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$BR^{2}$</tex-math></inline-formula> SCA demonstrates the effectiveness of the hybrid VS scheme under practical settings that include external disturbances.

Logarithmic Observation of Feature Depth for Image-Based Visual Servoing

Due to the robustness to robot modeling and camera calibration errors and avoidance of complete target geometry, image-based visual servoing has always been an important topic in the fields such as robotics, computer vision and so forth. When the image information obtained by the camera is mapped to the robotic task space to design the servoing control law, the resulting interaction matrix, which links the spatial velocity of the camera to the temporal variation of the selected image features, depends on the unknown feature depths. The use of inaccurate feature depths may influence the stability and robustness of the controller, and even cause the failure of the task. In this article, based on the perspective camera model, by employing the principle of reduced order observer, a novel logarithmic observer is presented for on-line recovery of feature depth. Compared with the typical observers now available, the presented observer offers several advantages: global convergence, a faster convergence rate of error structure than exponential error structure, a less restricted observability condition and greater robustness against measurements with noise. The comparison results of numerical simulations indicate the superiority of the presented observer, and real experiments with Kinect v2 sensor further validate the effectiveness of the presented observer in practical situation. Note to Practitioners—This article was motivated by the depth problem in the image-based visual servoing scheme, but it can also be used in other situations where the image depth information is needed, such as 3D reconstruction, robot navigation, etc. The existing depth acquisition methods include TOF sensors, stereo vision, depth observers and so on. However, TOF sensors are sensitive to light conditions, and the mounting space of stereo vision is slightly large, and there is contradiction between observation performance and computational complexity in most existing observers. In this article, a novel structure of logarithmic reduced order observer is described in detail, which can be utilized to estimate the depth information of images easily. The simulations and experiments verify the good performance of the observer. The limitations of the given observer are that the estimation accuracy is not very good under weak excitation, and the camera needs to be calibrated in advance. Future work will focus on overcoming these two limitations.

A Visual Servoing Scheme for Autonomous Aquaculture Net Pens Inspection Using ROV.

Aquaculture net pens inspection and monitoring are important to ensure net stability and fish health in the fish farms. Remotely operated vehicles (ROVs) offer a low-cost and sophisticated solution for the regular inspection of the underwater fish net pens due to their ability of visual sensing and autonomy in a challenging and dynamic aquaculture environment. In this paper, we report the integration of an ROV with a visual servoing scheme for regular inspection and tracking of the net pens. We propose a vision-based positioning scheme that consists of an object detector, a pose generator, and a closed-loop controller. The system employs a modular approach that first utilizes two easily identifiable parallel ropes attached to the net for image processing through traditional computer vision methods. Second, the reference positions of the ROV relative to the net plane are extracted on the basis of a vision triangulation method. Third, a closed-loop control law is employed to instruct the vehicle to traverse from top to bottom along the net plane to inspect its status. The proposed vision-based scheme has been implemented and tested both through simulations and field experiments. The extensive experimental results have allowed the assessment of the performance of the scheme that resulted satisfactorily and can supplement the traditional aquaculture net pens inspection and tracking systems.