Position-based Visual Servoing Research Articles

Deep learning approach for detecting tomato flowers and buds in greenhouses on 3P2R gantry robot

In recent years, significant advancements have been made in the field of smart greenhouses, particularly in the application of computer vision and robotics for pollinating flowers. Robotic pollination offers several benefits, including reduced labor requirements and preservation of costly pollen through artificial tomato pollination. However, previous studies have primarily focused on the labeling and detection of tomato flowers alone. Therefore, the objective of this study was to develop a comprehensive methodology for simultaneously labeling, training, and detecting tomato flowers specifically tailored for robotic pollination. To achieve this, transfer learning techniques were employed using well-known models, namely YOLOv5 and the recently introduced YOLOv8, for tomato flower detection. The performance of both models was evaluated using the same image dataset, and a comparison was made based on their Average Precision (AP) scores to determine the superior model. The results indicated that YOLOv8 achieved a higher mean AP (mAP) of 92.6% in tomato flower and bud detection, outperforming YOLOv5 with 91.2%. Notably, YOLOv8 also demonstrated an inference speed of 0.7 ms when considering an image size of 1920×1080\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$1920 \ imes 1080$$\\end{document} pixels resized to 640×640\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$640 \ imes 640$$\\end{document} pixels during detection. The image dataset was acquired during both morning and evening periods to minimize the impact of lighting conditions on the detection model. These findings highlight the potential of YOLOv8 for real-time detection of tomato flowers and buds, enabling further estimation of flower blooming peaks and facilitating robotic pollination. In the context of robotic pollination, the study also focuses on the deployment of the proposed detection model on the 3P2R gantry robot. The study introduces a kinematic model and a modified circuit for the gantry robot. The position-based visual servoing method is employed to approach the detected flower during the pollination process. The effectiveness of the proposed visual servoing approach is validated in both un-clustered and clustered plant environments in the laboratory setting. Additionally, this study provides valuable theoretical and practical insights for specialists in the field of greenhouse systems, particularly in the design of flower detection algorithms using computer vision and its deployment in robotic systems used in greenhouses.

Model-free visual servoing based on active disturbance rejection control and adaptive estimator for robotic manipulation without calibration

PurposeVisual feedback control is a promising solution for robots work in unstructured environments, and this is accomplished by estimation of the time derivative relationship between the image features and the robot moving. While some of the drawbacks associated with most visual servoing (VS) approaches include the vision–motor mapping computation and the robots’ dynamic performance, the problem of designing optimal and more effective VS systems still remains challenging. Thus, the purpose of this paper is to propose and evaluate the VS method for robots in an unstructured environment.Design/methodology/approachThis paper presents a new model-free VS control of a robotic manipulator, for which an adaptive estimator aid by network learning is proposed using online estimation of the vision–motor mapping relationship in an environment without the knowledge of statistical noise. Based on the adaptive estimator, a model-free VS schema was constructed by introducing an active disturbance rejection control (ADRC). In our schema, the VS system was designed independently of the robot kinematic model.FindingsThe various simulations and experiments were conducted to verify the proposed approach by using an eye-in-hand robot manipulator without calibration and vision depth information, which can improve the autonomous maneuverability of the robot and also allow the robot to adapt its motion according to the image feature changes in real time. In the current method, the image feature trajectory was stable in the camera field range, and the robot’s end motion trajectory did not exhibit shock retreat. The results showed that the steady-state errors of image features was within 19.74 pixels, the robot positioning was stable within 1.53 mm and 0.0373 rad and the convergence rate of the control system was less than 7.21 s in real grasping tasks.Originality/valueCompared with traditional Kalman filtering for image-based VS and position-based VS methods, this paper adopts the model-free VS method based on the adaptive mapping estimator combination with the ADRC controller, which is effective for improving the dynamic performance of robot systems. The proposed model-free VS schema is suitable for robots’ grasping manipulation in unstructured environments.

Visual Servoing Architecture of Mobile Manipulators for Precise Industrial Operations on Moving Objects

Although the use of articulated robots and AGVs is common in many industrial sectors such as automotive or aeronautics, the use of mobile manipulators is not widespread nowadays. Even so, the majority of applications separate the navigation and manipulation tasks, avoiding simultaneous movements of the platform and arm. The capability to use mobile manipulators to perform operations on moving objects would open the door to new applications such as the riveting or screwing of parts transported by conveyor belts or AGVs. This paper presents a novel position-based visual servoing (PBVS) architecture for mobile manipulators for precise industrial operations on moving parts. The proposed architecture includes a state machine to guide the process through the different phases of the task to ensure its correct execution. The approach has been validated in an industrial environment for screw-fastening operations, obtaining promising results and metrics.

Second-Order Position-Based Visual Servoing of a Robot Manipulator

Second-Order Position-Based Visual Servoing of a Robot Manipulator

Integrated Application and Optimized Design Strategies of Automatic and Servo Control Systems for Industrial Robots

Abstract At present, industrial robots have been widely used in industrial fields. This topic focuses on optimizing the system of servo control technology for the automatic control of industrial robots. Taking the optimization of robot automatic grasping technology with visual servo as an example, image-based and position-based visual servo control schemes are designed, respectively, and adaptive gain and first-order non-singular linear equations are used for visual servo performance enhancement. Open-loop control and closed-loop control are combined to create the visual servo-based automatic robot grasping docking method. The dynamic tracking effect of visual servo technology and the grasping performance of automatic grasping technology are evaluated through test experiments. The results can be obtained that the image-based and position-based visual servo control schemes can realize better dynamic tracking of objects, and the tracking accuracy is kept within 1.1mm and 8mm in the uniform and variable speed experiments, respectively. The robot’s automatic target grasping error gradually decreases to about 0 with the increase in the number of steps, and the system exhibits high grasping accuracy (over 96.67%) and speed (within 0.80s) at an operating speed lower than 20 m/s. Dynamic tracking performance and target grasping accuracy are certain features of the automatic robot grasping method based on the visual servo in this paper.

A Leader-follower formation control of mobile robots by position-based visual servo method using fisheye camera

This paper presents a leader-follower formation control of multiple mobile robots by position-based method using a fisheye camera. A fisheye camera has a wide field of view and recognizes a wide range of objects. In this paper, the fisheye camera is first modeled on spherical coordinates and then a position estimation technique is proposed by using an AR marker based on the spherical model. This paper furthermore presents a method for estimating the velocity of a leader robot based on a disturbance observer using the obtained position information. The proposed techniques are combined with a formation control based on the virtual structure. In this paper, the formation controller and velocity estimator can be designed independently, and the stability analysis of the total system is performed by using Lyapunov theorem. The effectiveness of the proposed method is demonstrated by simulation and experiments using two real mobile robots.

A general visual-servoing control strategy for active non-cooperative target tracking of space manipulators

Non-cooperative target tracking plays an important role in many space missions. Recent studies on non-cooperative target tracking are mostly concentrated in the field of passive target tracking. Passive non-cooperative target tracking can effectively be used in the initial phases of a close-range rendezvous manoeuvre. Instead, during the terminal phases of such manoeuvres, active object tracking(AOT) techniques are more convenient. This study propose a general visual servo strategy for the active non-cooperative target tracking of space manipulator. Our proposed method do not need any a priori information about the targets and can be used for space manipulators of any configuration and binocular cameras of any types. Our strategy includes a real-time and accurate position estimation algorithm for non-cooperative targets and a motion planning algorithm combined Position-based visual servo(PBVS) and Image-based visual servo(IBVS). The experiment verified that the proposed strategy well solved the active non-cooperative target tracking task and can be used as a baseline for solving such tracking problems which provides a reference for subsequent studies.

Position-based visual servoing of a 6-RSS parallel robot using adaptive sliding mode control

The trajectory tracking control of parallel robots is challenging due to their complicated dynamics and kinematics. This paper proposes a position-based visual servoing (PBVS) approach for a 6-Revolute-Spherical-Spherical (6-RSS) parallel robot using adaptive sliding mode control in Cartesian space. A photogrammetry sensor C-Track 780 in the eye-to-hand configuration is adopted to measure the real-time pose of the robot end-effector, which can avoid the calculation of robot forward kinematics and provide more flexibility for controller design. An adaptive Kalman filter is utilized to deal with uncertain noises in visual measurements to increase the pose estimation accuracy. A sliding mode controller with strong robustness is designed to cope with system uncertainties, and a radial basis function (RBF) neural network is incorporated to realize the auto-tuning of the control gains, which make the robot effectively track different trajectories with time-varying conditions in real applications. Based on Lyapunov theorem, the stability analysis of the controller has been done. Experiments have been conducted to validate the effectiveness of the proposed strategy and illustrate the superiority of the designed controller.

Hybrid vision/strain-based control strategy for a parallel manipulator with flexible links

Parallel manipulators may present higher speed/acceleration ratios and energy efficiency when compared with serial manipulators. In addition, reducing their components’ inertia can further improve performance. However, this design alternative might yield vibrations requiring novel joint and task space control strategies. While the former involves precise models, the latter might require adequate computation vision schemes. These requirements impose critical challenges. This manuscript proposes a hybrid control strategy and experimentally investigates using a 3RRR prototype with flexible links. This hybrid strategy comprises two control loops: position-based visual servo and strain-based feedback control schemes. The strain-based loop uses a sliding mode control and the time derivative of the signals obtained by strain gauges installed in the flexible links. Compared with the position-based visual servo scheme, the hybrid strategy can considerably attenuate the vibrations since the approach reduced the overshoot response by 36% and the maximum value of the Euclidean error on position by 38%, on average.

Position-Based Visual Servo Control of Dual Robotic Arms With Unknown Kinematic Models: A Cerebellum- Inspired Approach

The inverse kinematics problem of robotic arms can be solved effectively based on the kinematic model. However, the control of robotic arms with unknown kinematic models remains challenging. This article proposes a position-based visual servo (PBVS) control system for dual robotic arms with unknown kinematic models. First, a unified PBVS control method based on dual gradient neural dynamic (DGND) models is developed to achieve the position and orientation control of a single robotic arm with unknown kinematic model. Next, a cerebellum-inspired DGND (CIDGND) control system is devised to achieve higher accuracy. More importantly, a control system for dual robotic arms is proposed based on the CIDGND method to expand the working space of the primary robotic arm, by autonomously adjusting the camera pose with another robotic arm. Finally, simulations and experiments are performed to verify the efficacy of the proposed control systems. The results validate that the DGND method can achieve the PBVS control of robotic arms in the absence of kinematic models and the CIDGND method is able to reduce the tracking error significantly. Comparative results reveal that the CIDGND method is effectively and robustly better than existing methods.

A High-Certainty Visual Servo Control Method for a Space Manipulator with Flexible Joints.

This paper introduces a novel high-certainty visual servo algorithm for a space manipulator with flexible joints, which consists of a kinematic motion planner and a Lyapunov dynamics model reference adaptive controller. To enhance kinematic certainty, a three-stage motion planner is proposed in Cartesian space to control the intermediate states and minimize the relative position error between the manipulator and the target. Moreover, a planner in joint space based on the fast gradient descent algorithm is proposed to optimize the joint's deviation from the centrality. To improve dynamic certainty, an adaptive control algorithm based on Lyapunov stability analysis is used to enhance the system's anti-disturbance capability. As to the basic PBVS (position-based visual servo methods) algorithm, the proposed method aims to increase the certainty of the intermediate states to avoid collision. A physical experiment is designed to validate the effectiveness of the algorithm. The experiment shows that the visual servo motion state in Cartesian space is basically consistent with the planned three-stage motion state, the average joint deviation index from the centrality is less than 40%, and the motion trajectory consistency exceeds 90% under different inertial load disturbances. Overall, this method reduces the risk of collision by enhancing the certainty of the basic PBVS algorithm.

Vision Guided Dynamic Synchronous Path Tracking Control of Dual Manipulator Cooperative System

Abstract Compared with a single manipulator manufacturing cell, a dual manipulator cooperative system has more advantages in reconfigurability and flexibility. However, there are calibration errors and multi-source disturbances in the collaborative process, which lead to the processing trajectory accuracy defects of large-scale associated machining features. To solve the above problems, a practical path tracking synchronous control algorithm based on position-based visual servoing (PBVS) is proposed in this paper for the dual manipulator cooperative system, the proposed dynamic path tracking cross-coupled sliding mode controller (PTCSMC) scheme can realize dynamic paths correction while executing the pre-planned paths. In addition, for the cross-coupled technology to be applied into the proposed control algorithm for dynamic path tracking based on the real-time feedback of the highly repeatable 3D visual measurement instrument (VMI), the tracking and synchronous errors of the dual manipulators converge synchronously to zero. Finally, the stability of proposed control algorithm is proven by the Lyapunov method. In the end, the real-time line and circle path tracking experimental results using two industrial manipulators demonstrate the effectiveness of the proposed algorithm.

Hybrid Visual Servo Control of a Robotic Manipulator for Cherry Tomato Harvesting

This paper aims to develop a visual servo control of a robotic manipulator for cherry tomato harvesting. In the robotic manipulator, an RGB-depth camera was mounted to the end effector to acquire the poses of the target cherry tomatoes in space. The eye-in-hand-based visual servo controller guides the end effector to implement eye–hand coordination to harvest the target cherry tomatoes, in which a hybrid visual servo control method (HVSC) with the fuzzy dynamic control parameters was proposed by combining position-based visual servo (PBVS) control and image-based visual servo (IBVS) control for the tradeoff of both performances. In addition, a novel cutting and clipping integrated mechanism was designed to pick the target cherry tomatoes. The proposed tomato-harvesting robotic manipulator with HVSC was validated and evaluated in a laboratory testbed based on harvesting implementation. The results show that the developed robotic manipulator using HVSC has an average harvesting time of 9.40 s/per and an average harvesting success rate of 96.25% in picking cherry tomatoes.

Boosting Performance of Visual Servoing Using Deep Reinforcement Learning From Multiple Demonstrations

In this study, knowledge of multiple controllers was used and combined with deep reinforcement learning (RL) to train a visual servoing (VS) technique. Deep RL algorithms were successful in solving complicated control problems, however they generally require a large amount of data before they achieve an acceptable performance. We developed a method that generates online hyper-volume action bounds from demonstrations of multiple controllers (experts) to address the issue of insufficient data in RL. The agent then continues to explore the created bounds to find more optimized solutions and gain more rewards. By doing this, we cut out pointless agent explorations, which results in a reduction in training time as well as an improvement in performance of the trained policy. During the training process, we used domain randomization and domain adaptation to make the VS approach robust in the real world. As a result, we showed a 51% decrease in training time to achieve the desired level of performance, compared to the case when RL was used solely. The findings showed that the developed method outperformed other baseline VS methods (image-based VS, position-based VS, and hybrid-decoupled VS) in terms of VS error convergence speed and maintained higher manipulability.

Space Noncooperative Object Active Tracking With Deep Reinforcement Learning

Actively tracking an arbitrary space noncooperative object relied on visual sensor remains a challenging problem. In this article, we provide an open-source benchmark for space noncooperative object visual tracking including simulated environment, evaluation toolkit, and a position-based visual servoing (PBVS) baseline algorithm, which can facilitate the research in this topic especially for those methods based on deep reinforcement learning. We also present an end-to-end active visual tracker based on deep Q-learning, named as DRLAVT, which learns approximately optimal policy merely took color or RGBD images as input. To the best of authors knowledge, it is the first intelligent agent used for active visual tracking in aerospace domain. The experiment results show that our DRLAVT achieves an excellent robustness and real-time performance compared with the PBVS baseline, benefitted from the design of complex neural network and efficient reward function. In addition, the multiple targets training adopted in this article effectively guarantees the transferability of DRLAVT by forcing the agent to learn optimal control policy with respect to motion patterns of the target.

Low-Complexity Control for Vision-Based Landing of Quadrotor UAV on Unknown Moving Platform

This article addresses the vision-based landing problem of a low-cost quadrotor on an unknown moving platform. A robust landing controller is developed, which consists of the design of the low-complexity outer-loop controller and the geometric inner-loop attitude controller. First, an error transformation based on prescribed performance is designed to guarantee the landing behaviors and deal with the intermediate control signal of backstepping approaches, resulting in a low-complexity position-based visual servoing (PBVS) design. In addition, the proposed PBVS controller exhibits strong robustness against an uncertain relative dynamic system due to no incorporation of any prior knowledge of the moving platform. Next, a modified geometric attitude controller is presented by characterizing the geometric properties of rotation matrices intrinsically. Finally, the stability analysis is presented using Lyapunov stability theory, and the effectiveness of the proposed control strategy is demonstrated through numerical simulations and experiments.

Neural network based visual servo for quick-change device alignment in context of fusion reactor remote maintenance

The remote maintenance process for fusion reactor is complex, which can be very time-consuming and labor-consuming. This paper proposes a modified neural network based visual servo method to align a quick-change device with robot camera system. A classical position-based visual servo control law is used to guide the robot to reach the desired position. The key target for the above visual servo controller is to obtain a robust pose estimator to calculate the quick-change device pose with respect to the camera. This pose estimator is trained using RepVGG model with a self built image samples. An attention mechanism is added to the neural network to enhance the stability of pose prediction for reflective metal objects. The robot joint speed is also smoothed to reduce the image motion blur effect and make the visual servo process stable. The performance of the proposed visual servo controller was verified on an UR5 robot, and the results show that the stable and rough alignment of the quick-change device can be realized.

Autonomous 6-DOF Manipulator Operation for Moving Target by a Capture and Placement Control System.

The robot control technology combined with a machine vision system provides a feasible method for the autonomous operation of moving target. However, designing an effective visual servo control system is a great challenge. For the autonomous operation of the objects moving on the pipeline, this article is dedicated to developing a capture and placement control system for the six degrees of freedom (6-DOF) manipulator equipped with an eye-in-hand camera. Firstly, a path planning strategy of online capture and offline placement is proposed for real-time capture and efficient placement. Subsequently, to achieve the fast, stable, and robust capture for a moving target, a position-based visual servo (PBVS) controller is developed by combining estimated velocity feedforward and refined PID control. Feedforward control is designed using the estimated velocity by a proposed motion estimation method for high response speed. PID control is refined by dead zone constraint to reduce the manipulator’s jitter caused by the frequent adjustment of manipulator control system. Besides, the proportional, integral, and differential coefficients of PID controller are adaptively tuned by fuzzy control to reject the noise, disturbance, and dynamic variation in the capture process. Finally, validation experiments are performed on the constructed ROS–Gazebo simulation platform, demonstrating the effectiveness of the developed control system.

Closed-form camera pose and plane parameters estimation for moments-based visual servoing of planar objects

Image moments are global descriptors of an image and can be used to achieve control-decoupling properties in visual servoing. However, only a few methods completely decouple the control. This study introduces a novel camera pose estimation method, which is a closed-form solution, based on the image moments of planar objects. Traditional position-based visual servoing estimates the pose of a camera relative to an object, but the pose estimation method directly estimates the pose of an initial camera relative to a desired camera. Because the estimation method is based on plane parameters, a plane parameters estimation method based on the 2D rotation, 2D translation, and scale invariant moments is also proposed. A completely decoupled position-based visual servoing control scheme from the two estimation methods above was adopted. The new scheme exhibited asymptotic stability when the object plane was in the camera field of view. Simulation results demonstrated the effectiveness of the two estimation methods and the advantages of the visual servo control scheme compared with the classical method.

Deep learning-based pose prediction for visual servoing of robotic manipulators using image similarity

The accuracy of pose prediction is crucial in learning-based visual servoing. Motivated by the fact that the more similar observed images are, the closer the camera poses, we propose a joint training strategy with a two-part loss function in this paper. One part is the least absolute deviation (L1) loss function, which is defined by the error between the predicted pose and the pose label. The other is the mean similarity image measurement loss function (MSIM), which is related to the image’s brightness, contrast, and structure similarity and is determined by the differences between the input image and the image corresponding to the predicted pose. Meanwhile, a data generator based on spherical projection is created to generate data uniformly for training a CNN model, and position-based visual servoing (PBVS) is designed for a robotic manipulator after pose prediction. A numeric simulation and real experiments are conducted in a virtual environment and with a UR3 manipulator. The results show that the proposed method can realize more accurate pose prediction and is robust to occlusion disturbance, and PBVS is achieved by using monocular images.