Robotic Visual Servoing Research Articles

Robust Adaptive Robotic Visual Servo Grasping with Guaranteed Field of View Constraints

Robust Adaptive Robotic Visual Servo Grasping with Guaranteed Field of View Constraints

Deep neural network-based robotic visual servoing for satellite target tracking.

In response to the costly and error-prone manual satellite tracking on the International Space Station (ISS), this paper presents a deep neural network (DNN)-based robotic visual servoing solution to the automated tracking operation. This innovative approach directly addresses the critical issue of motion decoupling, which poses a significant challenge in current image moment-based visual servoing. The proposed method uses DNNs to estimate the manipulator's pose, resulting in a significant reduction of coupling effects, which enhances control performance and increases tracking precision. Real-time experimental tests are carried out using a 6-DOF Denso manipulator equipped with an RGB camera and an object, mimicking the targeting pin. The test results demonstrate a 32.04% reduction in pose error and a 21.67% improvement in velocity precision compared to conventional methods. These findings demonstrate that the method has the potential to improve efficiency and accuracy significantly in satellite target tracking and capturing.

Image-Based Composite Learning Robot Visual Servoing With an Uncalibrated Eye-to-Hand Camera

Image-Based Composite Learning Robot Visual Servoing With an Uncalibrated Eye-to-Hand Camera

IHUVS: Infinite Homography-Based Uncalibrated Methodology for Robotic Visual Servoing

The construction of task functions in robotic manipulation is of paramount importance for uncalibrated visual servoing. The existing methods generally use image information as control variables and estimate the image Jacobian matrix online, thus possessing issues relating to convergence, and image Jacobian matrix singularities. Therefore, this work proposes a novel methodology dubbed infinite homography-based uncalibrated visual servoing (IHUVS), in which the visual control of the robot end-effector pose is decomposed into its rotational and translational components. The corresponding rotational controller designs the visual servoing task function using the relationship between the infinite homography matrix and rotation matrix, and employs the Kronecker product to derive linear equations for rotational control, as well as to conduct the associated task error analysis. Meanwhile, the translational controller utilizes Kalman filtering for online estimation of the Jacobian matrix that is required by the proportional control scheme. The robot end-effector motion in Cartesian space is generated via the IHUVS method, without knowing the camera's intrinsic parameters and the robot hand-eye relationship. A simulation analysis is carried out to assess the algorithm's numerical performance, while robotic visual servoing experiments are also conducted to verify the accuracy and efficacy of the proposed IHUVS method.

Robot Visual Servoing Grasping Based on Top-Down Keypoint Detection Network

Robot Visual Servoing Grasping Based on Top-Down Keypoint Detection Network

A novel multidimensional uncalibration method applied to six-axis manipulators.

This study proposes a multidimensional uncalibrated technique for tracking and grasping dynamic targets by a robotic arm in the eye-in-hand mode. This method avoids complex and cumbersome calibration processes, enabling machine vision tasks to be adaptively applied in a variety of complex environments, which solved the problem of traditional calibration methods being unstable in complex environments. The specific method used in this study is first, in the eye-in-hand mode, the robotic arm moves along the x, y, and z axes in sequence, and images are taken before and after each movement. Thereafter, the image Jacobian matrix is calculated from the three (or more) sets of images collected. Finally, the robotic arm converts the target coordinates in the real-time captured images by the camera into coordinates in the robotic arm coordinate system through the image Jacobian matrix and performs real-time tracking. This study tests the dynamic quasi-Newton method for estimating the Jacobian matrix and optimizes the initialization coupling problem using the orthogonal moving method. This optimization scheme significantly shortens the iteration process, making the uncalibrated technology more fully applied in the field of dynamic object tracking. In addition, this study proposes a servo control algorithm with predictive compensation to mitigate or even eliminate the systematic error caused by time delay in dynamic target tracking in robot visual servo systems.

Visual-admittance-based model predictive control for nuclear collaborative robots

This paper presents a novel visual-admittance-based model predictive control scheme to cope with the problem of vision/force control and several constraints of a nuclear collaborative robotic visual servoing system. A visual-admittance model considering the desired image feature and force command in the image feature space is proposed. Moreover, a novel constraint scheme of the model predictive control (MPC) is proposed to cancel the overshoot in the interaction force control for most cases by taking the desired force command as the constraint of the proposed MPC. Via applying the robotic dynamic image-based visual servoing (IBVS) model, some other constraints, such as the actuator saturation, joint angle, and visual limits, can be satisfied simultaneously. The simulation results for the two-degrees-of-freedom (DOF) robot manipulator with an eye-to-hand camera are present to demonstrate the effectiveness of the proposed controller.

Deep Reinforcement Learning for Parameter Tuning of Robot Visual Servoing

Robot visual servoing controls the motion of a robot through real-time visual observations. Kinematics is a key approach to achieving visual servoing. One key challenge of kinematics-based visual servoing is that it requires time-varying parameter configuration throughout the entire process of one task. Parameter tuning is also necessary when applying to different tasks. The existing work on parameter tuning either lacks adaptation or cannot automate the tuning of all parameters. Meanwhile, the transferability of existing methods from one task to another is low. This work develops a Deep Reinforcement Learning (DRL) framework for robot visual servoing, which can automate all parameters tuning for one task and across tasks. In visual servoing, forward kinematics focuses on motion speed, while inverse kinematics focuses on the smoothness of motion. Therefore, we develop two separate modules in the proposed DRL framework. One tunes time-varying Forward Kinematics parameters to accelerate the motion, and the other tunes the Inverse Kinematics parameters to ensure smoothness. Moreover, we customize a knowledge transfer method to generalize the proposed DRL models to various robot tasks without reconstructing the neural network. We verify the proposed method on simulated robot tasks. The experimental results show that the proposed method outperforms the state-of-the-art methods and manual parameter configuration in terms of movement speed and smoothness in one task and across tasks.

Visual Positioning of Nasal Swab Robot Based on Hierarchical Decision.

This study focuses on a robot vision localization method for coping with the operational task of automatic nasal swab sampling. The application is important in the detection and epidemic prevention of Corona Virus Disease 2019 (COVID-19) to alleviate the large-scale negative impact of individuals suffering from pneumonia owing to COVID-19. In this method, the idea of a hierarchical decision network is used to consider the strong infectious characteristics of the COVID-19, which is followed by processing the robot behavior constraint condition. The visual navigation and positioning method using a single-arm robot for sampling is also planned, which considers the operation characteristics of medical staff. In the decision network, the risk factor for potential contact infection caused by swab sampling operations is established to avoid the spread among personnel. A robot visual servo control with artificial intelligence characteristics is developed to achieve a stable and safe nasal swab sampling operation. Experiments demonstrate that the proposed method can achieve good vision positioning for the robots and provide technical support for managing new major public health situations.

Random vector functional link network with L21 norm regularization for robot visual servo control with feature constraint

Random vector functional link network with L21 norm regularization for robot visual servo control with feature constraint

Constrained Image-Based Visual Servoing of Robot Manipulator with Third-Order Sliding-Mode Observer

A new image-based robot visual servo control strategy based on a third-order sliding-mode observer (TOSM) model predictive control is proposed in this study. This new control strategy solves the problem of robot visual servo control with system constraints and time-varying disturbances when the camera and model of the robot manipulator are uncertain and the joint velocity is unknown. In the proposed method, the joint velocity and system centralized uncertainties are estimated simultaneously based on a third-order sliding-mode observer, and the image-based visual servoing problem is transformed into a nonlinear optimization problem based on a model predictive control method considering both visibility constraints and actuator constraints, which minimizes the predicted trajectory cost function to generate the control signal for each cycle. Simulations were carried out to verify the effectiveness of the proposed control scheme.

Assembly and testing of a pan-tilt camera mechanism for robotics visual servoing applications / Montagem e teste de um mecanismo com câmera pan-tilt para aplicações de servovisão robótica

This work is about the assembling and testing of a camera on a PAN-TILT mechanism. The prototype is also composed of two direct current (DC) motors with gearboxes, fixed to the mechanism's rotation axes. The function of the motors is to provide the necessary torques to rotate the motion axes and provide a desired orientation for the camera. The ultimate goal is to obtain a fully functional mechanism to serve as a test bench for robotics visual servoing applications and for testing future control algorithms.

Visual Servoing of Unknown Objects for Family Service Robots

Aiming at satisfying the increasing demand of family service robots for housework, this paper proposes a robot visual servoing scheme based on the randomized trees to complete the visual servoing task of unknown objects in natural scenes. Here, “unknown” means that there is no prior information on object models, such as template or database of the object. Firstly, an object to be manipulated is randomly selected by user prior to the visual servoing task execution. Then, the raw image information about the object can be obtained and used to train a randomized tree classifier online. Secondly, the current image features can be computed using the well-trained classifier. Finally, the visual controller can be designed according to the error of image feature, which is defined as the difference between the desired image features and current image features. Five visual positioning of unknown objects experiments, including 2D rigid object and 3D non-rigid object, are conducted on a MOTOMAN-SV3X six degree-of-freedom (DOF) manipulator robot. Experimental results show that the proposed scheme can effectively position an unknown object in complex natural scenes, such as occlusion and illumination changes. Furthermore, the developed robot visual servoing scheme has an excellent positioning accuracy within 0.05 mm positioning error.

Design of Robot Vision Servo Control System Based on Image

Visual servo is a closed-loop control system of robot, which takes the image information obtained by visual sensor as feedback. Generally speaking, visual servo plays an important role in robot control, which is one of the main research directions in the field of robot control and plays a decisive role in the development of intelligent robots. In order to make the robot competent for more complex tasks and work more intelligently, autonomously and reliably, it is necessary not only to improve the control system of the robot, but also to obtain more and better information about the working environment of the robot. This paper introduces the principle and basic realization method of robot visual servo based on image, and expounds the problems and solutions in image feature extraction and visual servo controller design. In order to further expand the application field of robots and improve the operation performance of robots, robots must have higher intelligence level and stronger adaptability to the environment, so as to manufacture intelligent robots that can replace human labor.

Recent Advances on Robot Visual Servo Control Methods

Background: In order to enable robots to complete complex tasks, robots are required to have higher intelligence and adaptability. RVSC (Robot Visual Servo Control) is an organic combination of machine vision and servo control and is a complex system with strong coupling and nonlinearity, which improves the performance of robots and provides feasible solutions for solving practical problems in many fields. Objective: To review the latest development of RVSC methods and provide references for researchers in related fields. Methods: RVSC related patents and literature were analyzed from the aspects of the algorithm, innovation and application. In addition, the typical RVSC methods are summarized. Results: This paper analyzes the advantages and disadvantages of various methods in many patents and literature. Moreover, the typical applications, potential problems and future development directions of this field is put forward. Conclusion: RVSC has better performance than traditional control methods. In addition, it can also be combined with some algorithms or knowledge to further improve performance. But, the further improvements of RVSC are needed in the following areas: the convenience of operation, the optimization of trajectory planning, the combination of multiple sensors, etc. Furthermore, more patents of the RVSC also should be invented.

Predictive visual control network for occlusion solution in human-following robot

PurposeThe purpose of this paper is to propose a new video prediction-based methodology to solve the manufactural occlusion problem, which causes the loss of input images and uncertain controller parameters for the robot visual servo control.Design/methodology/approachThis paper has put forward a method that can simultaneously generate images and controller parameter increments. Then, this paper also introduced target segmentation and designed a new comprehensive loss. Finally, this paper combines offline training to generate images and online training to generate controller parameter increments.FindingsThe data set experiments to prove that this method is better than the other four methods, and it can better restore the occluded situation of the human body in six manufactural scenarios. The simulation experiment proves that it can simultaneously generate image and controller parameter variations to improve the position accuracy of tracking under occlusions in manufacture.Originality/valueThe proposed method can effectively solve the occlusion problem in visual servo control.

Visual Servoing of Flying Robot Based on Fuzzy Adaptive Linear Active Disturbance Rejection Control

Visual servoing is the most common control scheme for military and civil robots. For the flying robot, a nonlinear and strongly coupled system, the traditional control methods have the problems of poor adaptive ability and sensitivity to the disturbance in the complex environment. This brief deals with the disadvantages of slow response speed and weak anti-interference ability of flying robot visual servoing by using a fuzzy adaptive linear active disturbance rejection control (FLADRC) scheme. To improve the response speed and robustness of the system, the effect of the controller parameters on the performance is analyzed and quantified into fuzzy rules, and the fuzzy adaptive method is used to adjust the parameters of the linear extended state observer in real-time. Furthermore, a high-speed and high-precision visual positioning algorithm based on YOLOv4 and efficient large-scale stereo matching (ELAS) is proposed. Finally, the performances of FLADRC, LADRC, PID, and Fuzzy-PID are compared by simulation experiments, and the effectiveness of the proposed scheme is proved.

An Intelligence Image Processing Method of Visual Servo System in Complex Environment

In robot visual servoing system, effective detected the region of interest area (ROI) in target images is the first problem should be solved, but the detection is susceptible to an unstructured environment. In this paper, an instance segmentation network algorithm based on Mask Region Convolutional Neural Networks (Mask R-CNN) framework is proposed for ROI image preprocessing. As instance segmentation technology can distinguish complex environmental information, so first with this advantage filter out message such as shape or similar area to target image, then with semantic segmentation technology add special category labels to the filtered images and distinguish different individual instances in similar categories. Finally through the series steps aforementioned, robot vision system can overcome the impact of environmental factors and identify the target image. The proposed method is used to detect target image under five constraints such as occlusion and reflection, result shows the algorithm can effectively deal with challenges that brought by complex constraints, and even can predict the location data of missing information based on some image information. In addition, based on the algorithm proposed in this paper, we used one seven axis robot visual servo platform, executed visual servoing experiment under different unstructured environments, further verifies the effectiveness of our proposed method.

Deep learning the sound of light to guide surgeries

Photoacoustic imaging utilizes light and sound to make images by transmitting laser pulses that illuminate regions of interest, which subsequently absorb the light, causing thermal expansion and the generation of sound waves that are detected with conventional ultrasound transducers. The Photoacoustic and Ultrasonic Systems Engineering (PULSE) Lab is developing novel methods that use photoacoustic imaging to guide surgeries with the ultimate goal of eliminating surgical complications caused by injury to important structures like major blood vessels and nerves that are otherwise hidden from a surgeons immediate view. This invited presentation will summarize our recent work to learn from the physics of sound propagation in tissue and develop acoustic beamforming algorithms that improve image quality, using state-of-the-art deep learning methods. These deep learning methods hold promise for robotic tracking tasks, visualization and visual servoing of surgical tool tips, and assessment of relative distances between the surgical tool and nearby critical structures (e.g., major blood vessels and nerves that if injured will cause severe complications, paralysis, or patient death). Impacted surgeries and procedures include neurosurgery, spinal fusion surgery, hysterectomies, cardiac interventions, and biopsies.

Eye-in-Hand Visual Servoing Enhanced With Sparse Strain Measurement for Soft Continuum Robots

In the feature/object tracking of eye-in-hand visual servoing, 2D motion estimation relying only on image plane feedback is easily affected by vision occlusion, blurring, or poor lighting. For the commonly-used template matching method, tracking performance greatly depends on the image quality. Fiber Bragg gratings (FBGs), a type of high-frequency flexible strain sensor, can be used as an assistant device for soft robot control. We propose a method to enhance motion estimation in soft robotic visual servoing by fusing the results from template matching and FBG wavelength shifts to achieve more accurate tracking in applications such as minimally invasive surgery. Path following performance is validated in a simulated laparoscopic scene and LEGO-constructed scene, demonstrating significant improvement to feature tracking and robot motion, even under external forces.