Visual Servoing Control Scheme Research Articles

High-Precision Drilling by Anchor-Drilling Robot Based on Hybrid Visual Servo Control in Coal Mine

Rock bolting is a commonly used method for stabilizing the surrounding rock in coal-mine roadways. It involves installing rock bolts after drilling, which penetrate unstable rock layers, binding loose rocks together, enhancing the stability of the surrounding rock, and controlling its deformation. Although recent progress in drilling and anchoring equipment has significantly enhanced the efficiency of roof support in coal mines and improved safety measures, how to deal with drilling rigs’ misalignment with the through-hole center remains a big issue, which may potentially compromise the quality of drilling and consequently affect the effectiveness of bolt support or even result in failure. To address this challenge, this article presents a robotic teleoperation system alongside a hybrid visual servo control strategy. Addressing the demand for high precision and efficiency in aligning the drilling rigs with the center of the drilling hole, a hybrid control strategy is introduced combining position-based and image-based visual servo control. The former facilitates an effective approach to the target area, while the latter ensures high-precision alignment with the center of the drilling hole. The robot teleoperation system employs the binocular vision measurement system to accurately determine the position and orientation of the drilling-hole center, which serves as the designated target position for the drilling rig. Leveraging the displacement and angle sensor information installed on each joint of the manipulator, the system utilizes the kinematic model of the manipulator to compute the spatial position of the end-effector. It dynamically adjusts the spatial pose of the end-effector in real time, aligning it with the target position relative to its current location. Additionally, it utilizes monocular vision information to fine-tune the movement speed and direction of the end-effector, ensuring rapid and precise alignment with the target drilling-hole center. Experimental results demonstrate that this method can control the maximum alignment error within 7 mm, significantly enhancing the alignment accuracy compared to manual control. Compared with the manual control method, the average error of this method is reduced by 41.2%, and the average duration is reduced by 4.3 s. This study paves a new path for high-precision drilling and anchoring of tunnel roofs, thereby improving the quality and efficiency of roof support while mitigating the challenges associated with significant errors and compromised safety during manual control processes.

Target location method of intelligent deicing robot based on nonlinear auto disturbance rejection neural network

A visual navigation scheme for intermittent walking deicing robots was designed to address the issues of multiple iterations, long computation time, and poor real-time performance in the nonlinear optimization of the original SLAM system. A computer image acquisition unit, a computer image processing module, and a visual navigation parameter extraction algorithm were also designed. Implemented a visual navigation system for deicing robots based on image processing. It can meet the navigation requirements of the deicing robot. Simulation and experiments have shown that the method proposed in this paper can quickly and accurately identify abnormal point clouds. The visual servo control scheme constructed in this paper is aimed at robot task operations with unknown system calibration and target depth information. By calibrating the robot vision system, the conversion between camera pixel coordinates and robot base coordinates is achieved, with a transmission frame rate of 53.65 per second; The maximum error in positioning accuracy in space is 10.6 mm. The feature trajectory of visual space is smooth and stable within the camera's field of view, and the end movement of Cartesian space robots is stable without rebound, resulting in high grasping and positioning accuracy.

Uncalibrated and Unmodeled Image-Based Visual Servoing of Robot Manipulators Using Zeroing Neural Networks.

Neural networks have been widely investigated for the control of robot manipulators and recurrent neural network (RNN) is accepted as a powerful tool for visual servoing. Different from existing control schemes for robot-camera systems, this article proposes a novel image-based visual servoing (IBVS) control scheme for both the regulation and tracking control of robot manipulators in the framework of a special class of RNN, termed zeroing neural network (ZNN), which does not require prior knowledge about camera configuration and kinematic model parameters. The proposed control scheme is composed of a data-driven mapping estimator and a controller, both of which are designed based on ZNN. To facilitate the deployment of the proposed IBVS control scheme, a discrete-time version of the proposed control scheme is developed. Theoretical analysis for the proposed method is presented in terms of convergence, stability, and robustness. In addition, simulations and experiments are carried out based on different types of robot-camera systems to verify the efficacy and portability of the proposed control scheme for solving regulation and trajectory IBVS problems. Moreover, comparative studies are performed to reveal the merits of the proposed control scheme.

Design and Optimal Pose‐Constrained Visual Servoing of a Novel Active Flexible Endoscope Holder System for Solo Laparoscopic Surgery

Laparoscopic surgery (LS) has become an effective and widely accepted therapy for patients. However, performing LS requires advanced training and skills, as it demands high precision and dexterity due to visual feedback depending on the collaboration between the surgeon and endoscope assistant, restricted field of view, and limited dexterity inside the body. Thus, a flexible endoscope holder system with an automatic control scheme is proposed to assist LS performance, allowing the surgeon to perform solo LS. A DNA‐inspired helix‐based structure with wide‐angle and constant curvature bending is presented and made of printable nylon with selective laser sintering. The proposed flexible mechanism reduces the cost of manufacturing and assembling. Also, it has a hollow design for embedded sensing and actuation. Then, an optimization method is proposed, which provides a reference for the endoscope installation, robot placement, and pose selection. In addition, aiming at the misorientation problem, a pose‐constrained visual servoing control scheme for the endoscope system is developed. Both simulation and physical experiments are conducted to verify the effectiveness and feasibility of the proposed control scheme. Experimental results demonstrate that the proposed visual servoing scheme can maintain the optimized pose and reduce misorientation during automatic view steering.

A general visual-servoing control strategy for active non-cooperative target tracking of space manipulators

Non-cooperative target tracking plays an important role in many space missions. Recent studies on non-cooperative target tracking are mostly concentrated in the field of passive target tracking. Passive non-cooperative target tracking can effectively be used in the initial phases of a close-range rendezvous manoeuvre. Instead, during the terminal phases of such manoeuvres, active object tracking(AOT) techniques are more convenient. This study propose a general visual servo strategy for the active non-cooperative target tracking of space manipulator. Our proposed method do not need any a priori information about the targets and can be used for space manipulators of any configuration and binocular cameras of any types. Our strategy includes a real-time and accurate position estimation algorithm for non-cooperative targets and a motion planning algorithm combined Position-based visual servo(PBVS) and Image-based visual servo(IBVS). The experiment verified that the proposed strategy well solved the active non-cooperative target tracking task and can be used as a baseline for solving such tracking problems which provides a reference for subsequent studies.

Image-Based Visual Servoing of Unmanned Aerial Manipulators for Tracking and Grasping a Moving Target

In this paper, an image-based visual servoing (IBVS) control strategy is proposed for the unmanned aerial manipulator (UAM) system to track and grasp a moving target. Specifically, a robust-adaptive velocity observer is designed to estimate the relative velocity between the tracked target and the UAM platform. Based on the velocity observer, an IBVS controller using onboard camera of the UAM platform is proposed for moving target tracking without velocity measurement. Then, the barrier Lyapunov function is introduced into the UAM platform IBVS controller to ensure the safety of target tracking. Besides, another virtual camera is constructed on manipulator end-effector to compensate for the tracking error of the UAM platform. As a benefit, the eye-to-hand onboard camera ensures the global view of the UAM, and the eye-in-hand virtual camera of the manipulator ensures the accuracy of the grasping task. Finally, the stability of the proposed IBVS control strategy is analyzed through Lyapunov theory. The comparative simulations are provided to illustrate the target tracking performance of the proposed method. The experimental results demonstrate that the proposed method can be applied to the UAM with a low-cost sensor suite to realize the tasks of tracking and grasping a moving target.

Finite-Time Tracking of Moving Platform With Single Camera for Quadrotor Autonomous Landing

This paper presents a finite-time visual servoing control strategy for the autonomous landing of a quadrotor onto a tilting and moving landing vehicle. The proposed method, called Finite-Time Dynamic Visual Servo (FTDVS) control, utilizes a computer vision technique called Virtual Reticle Image Plane (VRIP) to track four observable features on the landing plane called landmarks. To represent the sea state, target-plane tilting motion is modelled using monochromatic sinusoidal waves. VRIP exploits the sinusoidal tilting pose of the landing plane and calculates the time period of the tilting motion. Based on this time period, the proposed FTDVS control strategy enables the quadrotor to search for suitable landing windows on a tilting target plane. The proposed finite-time controller converges the tracking errors to zero in finite-time during a landing maneuver. First, a rigorous stability analysis of the FTDVS control law is presented, followed by simulations demonstrating acceptable performance in terms of tracking accuracy when compared to existing controllers for autonomous landing on moving targets. The proposed method has been experimentally validated in an indoor environment.

Image-based prescribed performance visual servoing control of a QUAV with hysteresis quantised input

ABSTRACT In this paper, a novel image-based prescribed performance visual servoing control scheme with hysteresis quantised input is proposed for quadrotor unmanned aerial vehicles (QUAVs, for short). First, based on the perspective projection principle, effective image features are defined on a image plane called virtual image plane, and the decoupled image feature dynamics are achieved with respect to target points. Then, the image feature dynamics and QUAV dynamics are combined to derive a nonlinear dynamic system, and the nonlinear system is decoupled into two subsystems based on the commonly used inner–outer loop control framework. Furthermore, the prescribed performance control-based visual servoing control scheme with hysteresis quantizer is designed, and the necessary stability proof is presented. Finally, the numerical simulations and experiments are conducted to validate the effectiveness of the proposed control scheme.

ROS-Industrial based robotic cell for Industry 4.0: Eye-in-hand stereo camera and visual servoing for flexible, fast, and accurate picking and hooking in the production line

The fourth industrial revolution envisages the use of modern smart technologies to automate traditional manufacturing and industrial practices. However, industrial robots execute mostly pre-programmed jobs and are not able to face challenging tasks in unstructured environments. Industry 4.0 pushes for flexibility on target changes and autonomy. In line with the new principles of Industry 4.0, the proposed work describes an autonomous industrial cell that employs several smart technologies for loading jewelry pieces from a conveyor belt to a hooking frame built on purpose. The cell involves an industrial robot, a custom gripper, pneumatic and electric actuators with the aim of moving and opening the frame hooks, and a custom vision pipeline for detecting the feature of interest during the picking and hooking phases. The implemented pipeline makes use of a stereo camera pair mounted under the robot gripper and two fixed monocular cameras. The method employs HOG feature descriptors and machine learning algorithms for the detection. The software architecture is a component-based designed architecture that uses ROS as the underlying framework and ROS-Industrial packages to control the robot. The robot is controlled with position-based commands to reach intermediate positions in the workspace and with velocity command to implement a visual servoing control scheme that runs at 30 Hz and adjusts the robot position with the feedback of the vision during picking and hooking. The proposed visual servoing approach, thanks to the design of the stereo camera and choice of the optics, is able to perceive the features until the final movement phase, differently from most of the visual servoing employed in the literature that, due to the use of RGB-D camera or other vision apparatus, use an open control loop at a standoff distance. The presented work reaches an accuracy of 95% with a cycle time under 8 s.

Uncalibrated Adaptive Visual Servoing of Robotic Manipulators with Uncertainties in Kinematics and Dynamics

In the study, we propose a novel adaptive visual servoing control scheme for robotic manipulators with kinematic and dynamic uncertainties, where the camera used is uncalibrated, which implies that its intrinsic and extrinsic parameters are unavailable for measurement. For our scheme, a depth-independent composite Jacobian matrix is constructed to make visual parameters and robotic physical parameters appear linearly in a parametrized uniform form so that an adaptive algorithm can be developed to estimate their values. With the raised adaptive algorithm, the potential singularity of the Jacobian matrix can be well circumvented by updating estimated parameters in an appropriate tiny range of actual values. With our scheme, the asymptotic convergence of the image tracking error to zero is established successfully, in addition to the signal boundedness of the closed-loop system. The effectiveness of the proposed scheme is confirmed by simulation results based on a 6-DOF PUMA manipulator.

Visual path following with obstacle avoidance for quadcopters in indoor environments

In this paper, we propose a hierarchical visual servo control scheme for following visual paths with quadcopters in indoor environments. The control scheme is able to deal with three tasks simultaneously, considering as order of hierarchy: first, collision avoidance task, second visibility task (keep point features in the camera’s field of view), and finally, visual servoing task. The strategy allows a commitment of active tasks (also the smooth deactivation of tasks) for avoiding conflicts between them. The proposed scheme uses a monocular RGB front-facing camera as single sensor, i.e., only 2D information is required for the control law computation. That information is used for online reactive visual path following, i.e., no 3D reconstruction of the environment is required, and no path or motion planning for obstacle avoidance is performed while the robot follows the visual path. The effectiveness of the proposed control scheme is supported by a stability analysis and experiments with a low-cost commercial quadcopter in scenarios with static obstacles.

High precision PCB Soldering With Pin Springback Compensation by Robotic Micromanipulation

Through-hole components (THCs) soldering is widely reserved in electronic industry. Although recent progress in soldering robot has greatly improved the productivity, how to deal with metal pins' misalignment with the through-hole center remains a big issue, which may affect the soldering quality, reduce the product reliability, or even result in failure. To address this challenge, this article presents a robotic micromanipulation system and a hybrid visual servoing control strategy with springback compensation. First, we analyze the deformation and recovery phenomenal of the metal pin during alignment and then establish its elastoplastic model and springback model accordingly. In view of high precision and efficiency in electronics assembly, we propose a hybrid control strategy by combining the position-based and image-based visual servoing control, where the former is for approaching the target region efficiently and the latter is for high precision pin alignment. With the developed model and control strategy, the robotic micromanipulation system can automatically detect the pin's position and posture by a microscope camera (magnification 15 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">${\times }$</tex-math></inline-formula> ) and control the end-effector to align the pin to the through-hole center based on a predicted overshoot. The experimental results verify that our approach can confine the maximum alignment error in three pixels ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\sim$</tex-math></inline-formula> 37 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">${\mu }$</tex-math></inline-formula> m) with 100% success rate, which substantially improves the soldering accuracy of current techniques. This article paves a new path for high precision THCs soldering, which will not only directly improve the quality and reliability of PCBs in industry, but also inspire the application of robotics in micromanufacturing and microfabrication.

Eye-in-Hand Visual Servoing Control of Robot Manipulators Based on an Input Mapping Method

In image-based visual servoing (IBVS), parametric uncertainties tend to cause the model inaccuracy and limit the control performance. Considering these uncertainties can be embodied by the output–input data from the visual servoing system, this brief proposes an eye-in-hand visual servoing control (VSC) scheme based on the input mapping method, which directly utilizes the past output–input data to enhance the original feedback control law rather than identifying the model. The system with the input mapping method is proven to not only maintain the stability of the original VSC but also accelerate the convergent rate. The results of the experiments on a manipulator with an eye-in-hand camera demonstrate the superiority of our proposed method.

Visual servoing control of omnidirectional mobile robots based on disturbance observer

This study studies a critical issue that linear velocity and steering angular velocity of the wheels of omni-directional mobile robot are easily affected by the unknown external disturbances such as wind direction and road flatness during the visual servoing stabilization task. To overcome such challenging problems, this study proposes a disturbance observer based visual servoing control strategy. In configuration, the controller is divided into two parts: the quasi-min-max model predictive controller is designed to accommodate the state constraint and input constraint of the system, and complete the visual stabilization task without disturbance. A nonlinear disturbance observer is designed to estimate the external disturbance and feedback compensate the effect of the actual disturbance on the wheel speed. The simulation results and real experiments show that the proposed control strategy can stably accomplish the visual servoing stabilization task even with a large external disturbance.

A Kinematic Modeling and Control Scheme for Different Robotic Endoscopes: A Rudimentary Research Prototype

In image-guided robotic surgery, there exist different endoscopes coupled either with specialized surgical robots (SSRs) or general industrial robots (GIRs). In general, SSRs mechanically respect the remote-center-of-motion (RCM) constraints with directly and explicitly controllable degrees-of-freedom (DoFs), whereas GIRs meet the constraints at the algorithmic level in an implicit manner, with a loss of direct control of some RCM DoFs. Conventionally, different robotic endoscopes are treated as different monolithic systems. Then, kinematic models and control schemes are separately established to automate different robotic endoscopes. This means that a similar analysis must be followed for each system, which is tedious and time-consuming. This letter introduces a modular method to analyze the individual kinematics of the robotic holder, the endoscope shaft and the surgical endoscope with RCM constraints explicitly handled. For achieving automation, the integrated kinematics of a generic robotic endoscope is determined by combining the kinematics of the modular elements. Considering that cameras are intrinsically embedded sensors, a visual servo control scheme applicable to different robotic endoscopes is formulated to incorporate four explicit (virtual) DoFs characterizing the RCM constraints in the control law. Simulations and experiments performed using three different robotic endoscopes validate the effectiveness and practicality of the kinematic modeling and control scheme for automatic field-of-view adjustment.

Virtual camera-based visual servoing for rotorcraft using monocular camera and gyroscopic feedback

In this paper, an image-based visual servoing (IBVS) control strategy with a virtual camera frame is proposed for multirotor vehicles. Compared to previous works, the proposed IBVS controller requires minimal sensors in the position loop: a monocular camera and gyroscope. To achieve this, the visual feature projected onto the virtual image plane is associated with the plane’s normal instead of relying on an additional attitude estimator and prior knowledge of the plane’s inclination. Furthermore, we show that the ratio velocity, when scaled by the image moment, exhibits a similar dynamics to the linear velocity. The finding allows the quantity to be recovered and used for control with a monocular camera without other metric cues. To provide feedback for the controller, an Extended Kalman filter for estimating the ratio velocity, target plane’s inclination, and relative rotation between the current and reference camera frame is developed using only monocular vision and gyroscopic measurements. To validate the proposed controller and estimation strategy, both simulation and real-world flight experiments were carried out. The quadrotor smoothly and robustly tracked both dynamic horizontal and static inclined targets, without prior knowledge of the target’s inclination. Overall, the proposed regime offers a lightweight and robust alternative IBVS solution for rotorcraft.

Constrained Image-Based Visual Servoing of Robot Manipulator with Third-Order Sliding-Mode Observer

A new image-based robot visual servo control strategy based on a third-order sliding-mode observer (TOSM) model predictive control is proposed in this study. This new control strategy solves the problem of robot visual servo control with system constraints and time-varying disturbances when the camera and model of the robot manipulator are uncertain and the joint velocity is unknown. In the proposed method, the joint velocity and system centralized uncertainties are estimated simultaneously based on a third-order sliding-mode observer, and the image-based visual servoing problem is transformed into a nonlinear optimization problem based on a model predictive control method considering both visibility constraints and actuator constraints, which minimizes the predicted trajectory cost function to generate the control signal for each cycle. Simulations were carried out to verify the effectiveness of the proposed control scheme.

Closed-form camera pose and plane parameters estimation for moments-based visual servoing of planar objects

Image moments are global descriptors of an image and can be used to achieve control-decoupling properties in visual servoing. However, only a few methods completely decouple the control. This study introduces a novel camera pose estimation method, which is a closed-form solution, based on the image moments of planar objects. Traditional position-based visual servoing estimates the pose of a camera relative to an object, but the pose estimation method directly estimates the pose of an initial camera relative to a desired camera. Because the estimation method is based on plane parameters, a plane parameters estimation method based on the 2D rotation, 2D translation, and scale invariant moments is also proposed. A completely decoupled position-based visual servoing control scheme from the two estimation methods above was adopted. The new scheme exhibited asymptotic stability when the object plane was in the camera field of view. Simulation results demonstrated the effectiveness of the two estimation methods and the advantages of the visual servo control scheme compared with the classical method.

Robotic manipulation based on 3-D visual servoing and deep neural networks

The critical challenge, for robot–object-interaction, is to estimate visually the pose of the target object in a 3D space and combine it into a vision-based control scheme in manipulation applications. This paper proposes a novel reliable framework for deep ConvNet combined with visual servoing using a single RGB camera. We introduce an extensive system called Deep-Visual-Servoing (DVS) that addresses an integration of: (I) training of deep-CNNs using synthetic dataset only and operates successfully in real-world scenario, (II) continuous 3 D object pose estimation as the sensing feedback in a 3D visual servoing control scheme, and (III) design, integration and experimentation of visual servoing approach based on Lyapunov’s theory. The proposed deep based learning approach, the kinematic modeling and controller design are experimentally verified and discussed using the 6 DOF UR5 manipulator.

An onboard-eye-to-hand visual servo and task coordination control for aerial manipulator based on a spherical model

Compared with the unmanned aerial vehicle (UAV), the aerial manipulator extends the ability to interact with the environment by using the onboard manipulator. The complex dynamic characteristics of the combined system increase the challenges of accurately control the aerial manipulator to approximate the target object. In this paper, an onboard-eye-to-hand visual servo and task coordination control strategy is proposed for the aerial manipulator to enhance the accurate manipulation and flight stability. By establishing a new spherical coordinate error equation, the flying platform (quadrotor) and onboard manipulator can be controlled simultaneously. Meanwhile, the multi-task coordinated control scheme is adopted to achieve precise positioning and grasping of the target object, including estimating the object pose, dynamically compensating the change of the center of gravity of the aerial manipulator, and avoiding the onboard manipulator joint limitations. The stability of the closed-loop system is analyzed by the Lyapunov method. Finally, the feasibility of the proposed visual servo method is verified in simulations and real-world outdoor experiments, respectively. The strengths of the proposed method are demonstrated by comparing it with the conventional visual servo method.