Visual Servoing Research Articles

Visual Servoing NMPC Applied to UAVs for Photovoltaic Array Inspection

Visual Servoing NMPC Applied to UAVs for Photovoltaic Array Inspection

VSG: Visual Servo Based Geolocalization for Long-Range Target in Outdoor Environment

VSG: Visual Servo Based Geolocalization for Long-Range Target in Outdoor Environment

Dynamic visual servo control methods for continuous operation of a fruit harvesting robot working throughout an orchard

Fruit-picking robots are crucial for achieving efficient orchard harvesting. To genuinely meet the commercial production needs of farmers, the new generation of fruit-picking robots must be capable of demonstrating complete and continuous observation, movement, and picking behaviors throughout complex orchards, akin to real human employees. This poses systematic challenges, as many prior researches have focused solely on a part of the continuous operation of the entire orchard, such as fruit positioning, navigation, path planning, or grasping. These isolated basic functions are important but insufficient for fulfilling operational requirements on a macro scale and continuous situation. Developing an efficient control method for each basic module and constructing their internal coordination is vital for transitioning a harvesting robot from a functional prototype to a practical machine. In this context, this study tackles the visual servo control problem for efficient locomotion, picking, and their seamless integration. A set of vision algorithms for locomotion destination estimation, real-time self-positioning, and dynamic harvesting is proposed. Additionally, a solid coordination mechanism for continuous locomotion and picking behavior is established. Each method offers distinct advantages, such as improved accuracy, adaptability to varying conditions, and enhanced picking efficiency, enabling the robot to operate autonomously and continuously. Comprehensive field experiments validated the soundness of the methods. The primary contribution of this study lies in addressing the challenge of continuous operation in an entire orchard as a systematic problem and providing new insights into control methods for the future development of highly autonomous, practical, and user-oriented fruit harvesting systems.

In-Depth Evaluation of Automated Fruit Harvesting in Unstructured Environment for Improved Robot Design

Using modern machines like robots comes with its set of challenges when encountered with unstructured scenarios like occlusion, shadows, poor illumination, and other environmental factors. Hence, it is essential to consider these factors while designing harvesting robots. Fruit harvesting robots are modern automatic machines that have the ability to improve productivity and replace labor for repetitive and laborious harvesting tasks. Therefore, the aim of this paper is to design an improved orange-harvesting robot for a real-time unstructured environment of orchards, mainly focusing on improved efficiency in occlusion and varying illumination. The article distinguishes itself with not only an efficient structural design but also the use of an enhanced convolutional neural network, methodologically designed and fine-tuned on a dataset tailored for oranges integrated with position visual servoing control system. Enhanced motion planning uses an improved rapidly exploring random tree star algorithm that ensures the optimized path for every robot activity. Moreover, the proposed machine design is rigorously tested to validate the performance of the fruit harvesting robot. The unique aspect of this paper is the in-depth evaluation of robots to test five areas of performance that include not only the accurate detection of the fruit, time of fruit picking, and success rate of fruit picking, but also the damage rate of fruit picked as well as the consistency rate of the robot picking in varying illumination and occlusion. The results are then analyzed and compared with the performance of a previous design of fruit harvesting robot. The study ensures improved results in most aspects of the design for performance in an unstructured environment.

A Study on Robust Finite-Time Visual Servoing with a Gyro-Stabilized Surveillance System

This article presents the design and validation of a novel visual servoing scheme for a surveillance system. In this system, a two-axis gimbal mechanism operates the rotation of a camera which is able to provide visual information on the tracked target for the control system. The control objective is to bring the target’s projection to the center of the image plane with the smallest steady-state error and a smooth transient response, even with the unpredictable motion of the target and the influence of external disturbances. To fulfill these tasks, the proposed control scheme is designed consisting of two parts: (1) an observer estimates simultaneously the matched and unmatched disturbances; and (2) a motion control law guarantees the finite-time stability and visual servoing performance. Finally, experiments are conducted for validation and evaluation. The proposed control system shows its consistency and ought to perform better than previous approaches.

Accuracy Enhancement of Industrial Robots Based on Visual Servoing Using Optimal Adaptive RBFNN Integral Terminal Fractional-Order Super-Twisting Algorithm

This paper proposes a novel adaptive robust control scheme for accuracy enhancement of eye-to-hand photogrammetry-based industrial robots subject to uncertainties. The proposed method uses two control loops: internal and external loops. The former is the dynamic controller designed for controlling the robot’s joints. The external loop is the kinematic controller to minimize the error of the end-effector detected by the photogrammetry sensor. An adaptive integral terminal fractional-order super-twisting algorithm (AITFOSTA) is developed and employed for both control loops. AITFOSTA is an integral sliding-mode controller (ISMC) whose nominal control law is terminal. Its switching part is replaced with a fractional-order super-twisting algorithm (FOSTA), reducing the chattering to a great extent while rejecting the uncertainties. Additionally, an adaptive uncertainty and disturbance estimator based on radial basis function neural networks (RBFNNs) is designed and employed to reduce the uncertainty bounds, contributing to further chattering reduction. The stability analysis of the proposed controller is also presented. Simulation and experimental results show the superiority of the proposed method over other well-known approaches by reaching an unprecedented tracking accuracy, i.e. 0.06 mm and 0.18[Formula: see text] for position and orientation, respectively.

A Novel Flexible Endomicroscopic Scanning Instrument with Visual Servoing Control

Optical biopsy using probe‐based confocal laser endomicroscopy (pCLE) and optical coherence tomography (OCT) technologies performs in vivo tissue scanning of organ surfaces and plays a crucial role in early cancer diagnosis. Herein, a flexible endomicroscopic scanning instrument based on a 3‐degree‐of‐freedom flexible hinge structure and visual servoing control is presented. The instrument integrates OCT and pCLE. OCT information compensates for the imaging distance of pCLE, which can combine the advantages of the two technologies. The mechanism allows for tissue surface scanning with a wire‐driven active bending part and dynamic adjustment of the probe‐tissue distance using a combination of wire and spring‐driven, which satisfy large‐area scanning while maintaining consistence tissue contacts. Local optimization‐based visual servoing control is proposed to optimize the tissue scanning. Through the scanning experiments, high‐quality mosaics of uneven surfaces are obtained with an effective area that is larger than 4 mm2. In vivo animal trial further verifies the feasibility and clinical potential of the instrument.

Stereo Visual Servoing Control of a Soft Endoscope for Upper Gastrointestinal Endoscopic Submucosal Dissection.

Quickly and accurately completing endoscopic submucosal dissection (ESD) operations within narrow lumens is currently challenging because of the environment's high flexibility, invisible collision, and natural tissue motion. This paper proposes a novel stereo visual servoing control for a dual-segment robotic endoscope (DSRE) for ESD surgery. Departing from conventional monocular-based methods, our DSRE leverages stereoscopic imaging to rapidly extract precise depth data, enabling quicker controller convergence and enhanced surgical accuracy. The system's dual-segment configuration enables agile maneuverability around lesions, while its compliant structure ensures adaptability within the surgical environment. The implemented stereo visual servo controller uses image features for real-time feedback and dynamically updates gain coefficients, facilitating rapid convergence to the target. In visual servoing experiments, the controller demonstrated strong performance across various tasks. Even when subjected to unknown external forces, the controller maintained robust performance in target tracking. The feasibility and effectiveness of the DSRE were further verified through ex vivo experiments. We posit that this novel system holds significant potential for clinical application in ESD surgeries.

Vision-based UAV adaptive tracking control for moving targets with velocity observation

An adaptive image-based visual servoing (IBVS) controller is designed for a quadrotor unmanned aerial vehicle (UAV) to achieve robust tracking for moving targets, under underactuation and tight coupling constraints of UAV kinematics. Specifically, image features are selected from perspective image moments of a planar target to obtain virtual feature dynamics regarding UAV kinematics and dynamics. By constructing an auxiliary variable, a translational velocity observer for moving target is constructed by using virtual image features. An IBVS tracking controller is designed without target geometric information by combining UAV and visual feature dynamics. Designed controller and observer make the UAV robustly reach desired height and track the moving target, despite uncertainty of target movement. The controller has asymptotical convergence performance, and the target velocity is observed according to Lyapunov stability analysis. Simulation and experimental results show that the proposed method has smoother and more accurate performance in motion tracking and target velocity prediction under system uncertainty.

Visual Servoing of Flexible Manipulators With Unknown Camera Intrinsic Parameters and Vibration States

Visual Servoing of Flexible Manipulators With Unknown Camera Intrinsic Parameters and Vibration States

4-DOF Visual Servoing of a Robotic Flexible Endoscope With a Predefined-Time Convergent and Noise-Immune Adaptive Neural Network

4-DOF Visual Servoing of a Robotic Flexible Endoscope With a Predefined-Time Convergent and Noise-Immune Adaptive Neural Network

Research on trajectory planning algorithm of manipulator based on visual servo

AbstractBackgroundWith the development and progress of society, not only substations, but also various industries are moving towards automation, intelligence, and unmanned direction. There is an increasing need for machinery to replace human activities. However, how to improve the ability of robotic arms to cope with various working environments and emergencies, and develop real‐time, high‐speed, and accurate trajectory planning algorithms has become a hot topic of discussion.PurposeIn order to develop a reasonable trajectory planning algorithm for robotic arms, this article studies the trajectory planning algorithm of robotic arms based on visual servoing, in order to improve the trajectory operation ability of robotic arms.MethodThe feasibility of the algorithm was tested through experiments.ResultsIt was found that the angle, angular velocity, and angular acceleration curves of both the second and fifth sixth joints of the robotic arm were very smooth, and there were no problems such as faults.ConclusionThis confirms the effectiveness of the algorithm's operation, which can assist the robotic arm in safe, accurate, and efficient operations on the basis of visual servoing.

Progress in artificial intelligence-based visual servoing of autonomous unmanned aerial vehicles (UAVs)

Unmanned aerial vehicles (UAVs) have attracted massive attention in many engineering and practical applications in the last years for their characteristics and operation flexibility. For the UAV system, suitable control systems are required to operate appropriately and efficiently. An emerging control technique is visual servoing utilizing the onboard camera systems for inspecting the UAV's environment and autonomously controlling the UAV's operation. Artificial intelligence (AI) techniques are widely deployed in the visual servoing of autonomous UAV applications. Despite the increasing research in the field of AI-based visual control of UAV systems, comprehensive review articles that showcase the general trends and future directions in this field of research are limited. This work comprehensively examines the application and advancements of AI-enhanced visual servoing in autonomous UAV systems, covering critical control tasks and offering insights into future research directions for enhancing performance and applicability which is limited in the current literature. The paper first reviews the application of intelligent visual servoing systems for autonomously executing various UAV control tasks, including 3D UAV positioning, aerial and ground object following, obstacle avoidance, and autonomous landing. Second, the research progresses in applying AI techniques in the visual servoing of autonomous UAV systems are discussed and analyzed. Finally, future directions and critical research gaps for further improving the performance and applicability of intelligent visual servoing systems are included.

Learning an Image-Based Visual Servoing Controller for Object Grasping

Adaptive and cooperative control of arms and fingers for natural object reaching and grasping, without explicit 3D geometric pose information, is observed in humans. In this study, an image-based visual servoing controller, inspired by human grasping behavior, is proposed for an arm-gripper system. A large-scale dataset is constructed using Pybullet simulation, comprising paired images and arm-gripper control signals mimicking expert grasping behavior. Leveraging this dataset, a network is directly trained to derive a control policy that maps images to cooperative grasp control. Subsequently, the learned synergy grasping policy from the network is directly applied to a real robot with the same configuration. Experimental results demonstrate the effectiveness of the algorithm. Videos can be found at https://www.bilibili.com/video/BV1tg4y1b7Qe/ .

Image-based visual servoing with Kalman filter and swarm intelligence optimisation algorithm

The article proposes a new Kalman depth estimation and an improved swarm intelligence optimisation algorithm for adaptive tuning of servo gain for image-based visual servo control. First, a Kalman depth estimation model is established from the principle of image-based visual servoing, and two state equations are designed for depth estimation based on the number of state quantities. Second, the improved sparrow search algorithm is proposed to tune the servo gain adaptively to improve the convergence speed and stability. To verify the effectiveness of the proposed method, the conventional image-based visual servoing and conventional Kalman estimation are reproduced and compared with the proposed method, and the simulation is completed on the Simulink simulation platform for verification. Finally, the experiments are completed in the robotic arm experimental platform. Both the simulation and experimental results show the effectiveness of the proposed method, which reduces the redundancy of the camera and shortens the convergence time.

A Study of an Agricultural Indoor Robot for Harvesting Edible Bird Nests in Vietnam

This study demonstrates robot technology for harvesting edible bird’s nests within swiftlet houses. A comprehensive manipulator’s movement analysis of harvesting operation with a separating tool is provided for precisely collecting swiftlet nests. A robotic manipulator mounted on a mobile platform with a vision system is also analyzed and evaluated in this study. The actual harvesting or separating the swiftlet nests is performed with visual servo feedback. The manipulator performs the gross motions of separating tools and removing the nests under computer control with velocity and position feedback. The separating principle between the objective nest and wooden frame has been applied to a demonstration removal of nests using a four-degrees-of-freedom manipulator to perform the gross movements of tool. The actual separations using this system are accomplished as fast as the manipulator can be controlled to perform the necessary deceleration and topping at the end of separating. This is typically 2.0 s. This efficiency underscores the system’s capability for swift and precise operation in harvesting an edible bird nest task.

Vision based crop row navigation under varying field conditions in arable fields

Accurate crop row detection is often challenged by the varying field conditions present in real-world arable fields. Traditional colour based segmentation is unable to cater for all such variations. The lack of comprehensive datasets in agricultural environments limits the researchers from developing robust segmentation models to detect crop rows. We present a dataset for crop row detection with 11 field variations from sugar beet and maize crops. We also present a novel crop row detection algorithm for visual servoing in crop row fields. The proposed method uses deep learning based crop row skeleton segmentation method followed by a crop row scanning algorithm that identifies the central crop row which the robot then follows. The unique dataset we used with skeleton representations for crop row detection enables robust crop row detection in challenging real world field conditions. Our algorithm can detect crop rows against varying field conditions such as curved crop rows, weed presence, discontinuities, growth stages, tramlines, shadows and light levels. Dense weed presence within inter-row space and discontinuities in crop rows were the most challenging field conditions for our crop row detection algorithm. An End-of Row detector algorithm was developed to detect the end of the crop row and navigate the robot towards the headland area when it reaches the end of the crop row.

Robust adaptive repetitive learning control for manipulators with visual servoing

This study introduces a novel robust adaptive repetitive learning controller (RARLC) designed for periodic tasks in vision-based robot systems. The controller eliminates the requirement for prior knowledge of the camera's intrinsic and extrinsic parameters, depth information of image features, and robot dynamic parameters. By using a depth-independent Jacobian matrix, the controller can estimate depth and unknown system parameters online. The key aspect of the approach is the fast learning ability from experience and outstanding robustness against noise and initial disturbance. This is accomplished by introducing two repetitive learning terms for the visual servo and nonlinear dynamic systems. Additionally, the method involves incorporating filtered error information and utilizes a projection mapping function to constrain estimated parameters within upper and lower bounds, thereby enhancing robustness against noise. The experimental and simulation results demonstrate that the robot's tracking errors significantly decrease after only three periods, indicating excellent performance of the controller in terms of convergence speed and servoing precision. Finally, we compared the developed controller with other controllers and examined the effectiveness of control gains in the context of tracking accuracy and robustness to image noise using simulation techniques.

Integrated Application and Optimized Design Strategies of Automatic and Servo Control Systems for Industrial Robots

Abstract At present, industrial robots have been widely used in industrial fields. This topic focuses on optimizing the system of servo control technology for the automatic control of industrial robots. Taking the optimization of robot automatic grasping technology with visual servo as an example, image-based and position-based visual servo control schemes are designed, respectively, and adaptive gain and first-order non-singular linear equations are used for visual servo performance enhancement. Open-loop control and closed-loop control are combined to create the visual servo-based automatic robot grasping docking method. The dynamic tracking effect of visual servo technology and the grasping performance of automatic grasping technology are evaluated through test experiments. The results can be obtained that the image-based and position-based visual servo control schemes can realize better dynamic tracking of objects, and the tracking accuracy is kept within 1.1mm and 8mm in the uniform and variable speed experiments, respectively. The robot’s automatic target grasping error gradually decreases to about 0 with the increase in the number of steps, and the system exhibits high grasping accuracy (over 96.67%) and speed (within 0.80s) at an operating speed lower than 20 m/s. Dynamic tracking performance and target grasping accuracy are certain features of the automatic robot grasping method based on the visual servo in this paper.

Neural Optimal Control for Constrained Visual Servoing via Learning From Demonstration

This paper proposes a novel optimal control scheme for constrained image based visual servoing of a robot manipulator. For a robot manipulator with an eye-on-hand configuration, visibility constraint is an essential requirement to avoid servo failure, while robot’s actuator limits must also be satisfied. To ensure this, the constraints are modelled implicitly via learning the task and defining safe regions using expert human demonstrations via mixture of Dynamic Movement Primitives (DMPs). The visual servoing problem is then formulated as a closed-loop optimal control problem using these constraint model where a desired target (possibly time-varying) is obtained by acting upon the feedback from the real-time visual sensors. The visual servo control loop consists of a single network adaptive critic optimal tracking control scheme whose weights are tuned using Lyapunov stability criteria. The stability and the performance of the proposed control scheme is shown theoretically via Lyapunov approach and also verified experimentally using a seven degree of freedom (DOF) Franka Emika and six DOF Universal Robot (UR) 10 manipulator. The approach is also demonstrated on a use case scenarios in mock-up convenience store and warehouse setup. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —The applications of robots in busy warehouses, healthcare sectors and convenience store, has a big societal impact. The scenarios in these real-world problems often consist of a dynamic environment. Therefore, sensor-based localization and planning, along with satisfying robot and task constraints are needed. These naturally adds up to the programming costs in addition to the robot platform and actuation. However, modern robots need intuitive and easy programming for more pervasive in a practical societal application. In our proposed method, this sensor-based planning with environmental constraints is modelled implicitly using the Programming by Demonstration framework. The user needs to catch the robot arm by his hand and teach the task at hand, and the framework captures the task and robot constraints while generalizing to new goals. This modelling, along with cost optimal controller, generates real-time constraint aware robot manipulation trajectories for the demonstrated task in dynamic environments.