Visual Servoing Control Law Research Articles

Visual Servoing of Rigid-Link Flexible-Joint Manipulators in the Presence of Unknown Camera Parameters and Boundary Output

Exact position control of the flexible-joint manipulator (FJM) is a challenging task due to the manipulator’s nonlinearity and underactuated characteristic. For the three-dimensional (3-D) rigid-link FJM, the image-based visual servoing (IBVS) approach with link-position output constraint and unknown camera parameters is investigated in this article. The controller is designed to deal with three problems. First, the visual servoing control law is divided into two domains based on the singular perturbation method, where the control input for the fast subsystem is developed to damp out the flexible joint’s vibration. Second, the adaptive updating law for estimating the unknown camera parameters is presented in the slow subsystem. Third, to guarantee that the rigid-link position keep in set constraints, the controller for the slow subsystem is designed by introducing a barrier Lyapunov function. According to the Lyapunov theorem of stability, the proposed controller for the FJM is theoretically proved to be asymptotically stable. Several numerical simulation experiments are provided to illustrate that the presented control scheme is effective.

Neural network based visual servo for quick-change device alignment in context of fusion reactor remote maintenance

The remote maintenance process for fusion reactor is complex, which can be very time-consuming and labor-consuming. This paper proposes a modified neural network based visual servo method to align a quick-change device with robot camera system. A classical position-based visual servo control law is used to guide the robot to reach the desired position. The key target for the above visual servo controller is to obtain a robust pose estimator to calculate the quick-change device pose with respect to the camera. This pose estimator is trained using RepVGG model with a self built image samples. An attention mechanism is added to the neural network to enhance the stability of pose prediction for reflective metal objects. The robot joint speed is also smoothed to reduce the image motion blur effect and make the visual servo process stable. The performance of the proposed visual servo controller was verified on an UR5 robot, and the results show that the stable and rough alignment of the quick-change device can be realized.

Adaptive visual servo control law for finite-time tracking to land quadrotor on moving platform using virtual reticle algorithm

This paper proposed an image-based visual servoing (IBVS) control law for a quadrotor that is equipped with a single monocular camera attached to its bottom. For control purposes virtual reticle plane (VRP) algorithm is used to track the relative 3D position of the quadrotor to the tilting and moving target landing platform within the range of the camera’s field of view (FOV). In this article, the landing platform’s tilting motion is considered sinusoidal type oscillatory motion for the overhead camera. For control purposes, an adaptive finite-time control (AFTC) is proposed, based on the finite-time control (FTC) and adaptive approximation of uncertainties. A constructive combination of FTC and adaptive approximation inherits benefits of both to overcome each other’s limitations. The task of the controller is to regulate the position error to zero in time calculated by VRP. The experimental results confirmed the effectiveness of the VRP algorithm to track the desired parameters of the moving target. Finally, simulations are performed to illustrate the effectiveness and improved performance of the proposed AFTC in response time, robustness, and tracking accuracy.

Direct Visual Servoing in the Frequency Domain

In this letter, we propose an original approach to extend direct visual servoing to the frequency domain. Whereas most of visual servoing approaches relied on the geometric features, recent works have highlighted the importance of taking into account the photometric information of the entire images. This leads to direct visual servoing (DVS) approaches. In this letter we propose no longer to consider the image itself in the spatial domain but its transformation in the frequency domain. The idea is to consider the Discrete Cosine Transform (DCT) which allows to represent the image in the frequency domain in terms of a sum of cosine functions that oscillate at various frequencies. This leads to a new set of coordinates in a new precomputed orthogonal basis, the coefficients of the DCT. We propose to use these coefficients as the visual features that are then considered in a visual servoing control law. We then exhibit the analytical formulation of the interaction matrix related to these coefficients. Experimental results validate our approach.

Direct Visual Servoing Using Wavelet Coefficients

This paper focuses on the design of a 6 degrees of freedom visual servoing control law. Instead of using geometric visual features in standard vision-based approaches, the proposed controller makes use of wavelet coefficients as control signal inputs. It uses the multiple resolution coefficients representing the wavelet transform of an image in the spatial domain. The main contributions are the definition of the multiple resolution wavelet interaction model that links the time-variation of wavelet coefficients to the robot spatial velocity and the associated task function controller. The proposed control law was tested and validated experimentally using a commercial micromanipulator in an eye-to-hand configuration. To be able to judge the efficiency of the control law, several validation tests were carried out under different conditions of use i.e., large illumination variations, noisy images, partial occlusions, and using unknown three-dimensional scenes. It is also demonstrated experimentally that the proposed approach outperforms, the well-known photometry visual servoing as well as a feature-based visual servoing, namely in unfavorable conditions of use.

Structured Light-Based Visual Servoing for Robotic Pipe Welding Pose Optimization

In order to optimize the pose of welding torch preplanned by offline programming, a structured light-based visual servoing method is proposed. First of all, a series of phase shifting patterns are projected to acquire the so-called phase map. Afterwards, unlike usual feature extraction methods, which were based on 3-D cloud, a cylinder axis is extracted directly from the phase map to represent the connecting pipes’ cylindrical surface. Then, a visual servoing control law based on the axis combined with the seam center in phase map is proposed to optimize the pose of the welding torch. Moreover, global asymptotic stability of this method is proved. Finally, simulations and real experiments are performed to demonstrate the effectiveness and robustness of this method. Results show this method can improve the mean error of deviated distance and angle of offline programming by 73.5% and 82.5%, respectively.

A Soft, Steerable Continuum Robot That Grows via Tip Extension.

Soft continuum robots exhibit access and manipulation capabilities in constrained and cluttered environments not achievable by traditional robots. However, navigation of these robots can be difficult due to the kinematics of these devices. Here we describe the design, modeling, and control of a soft continuum robot with a tip extension degree of freedom. This design enables extremely simple navigation of the robot through decoupled steering and forward movement. To navigate to a destination, the robot is steered to point at the destination and the extension degree of freedom is used to reach it. Movement of the tip is always in the direction tangent to the end of the robot's backbone, independent of the shape of the rest of the backbone. Steering occurs by inflating multiple series pneumatic artificial muscles arranged radially around the backbone and extending along the robot's whole length, while extension is implemented using pneumatically driven tip eversion. We present models and experimentally verify the growing robot kinematics. Control of the growing robot is demonstrated using an eye-in-hand visual servo control law that enables growth and steering of the robot to designated locations.

Model-Based Coarse-Fine Virtual Calibration and Visual Servo for Augmented Reality-Assisted Peg-in-Hole Microassembly

A three-dimensional (3D)-virtual calibration and visual servo are implemented for augmented reality (AR)-assisted peg-in-hole microassembly operations. By employing 3D model and ray casting, the 3D coordinates on virtual mating rod correspondent to the two-dimensional (2D) virtual image points are extracted. The detecting and tracking of image feature points for calibration is carried out by the proposed algorithm of regional template matching (TM) and scanning with edge fitting (RTM-SEF). For achieving subpixel error between the feature points in real and virtual images, a coarse-fine virtual calibration method is proposed. In regard to the image viewed by the real and virtual cameras, a calibrated virtual camera is utilized to track the mating rod. A visual servo control law including coarse and fine tuning is proposed to ensure sub-pixel error between the most important feature point in the real and virtual images. The AR technology is mainly employed in the alignment between micropeg and mating hole for inserting a micropeg of diameter 80 μm with length 1–1.4 mm into a mating rod with 100 μm hole.

Series Pneumatic Artificial Muscles (sPAMs) and Application to a Soft Continuum Robot.

We describe a new series pneumatic artificial muscle (sPAM) and its application as an actuator for a soft continuum robot. The robot consists of three sPAMs arranged radially round a tubular pneumatic backbone. Analogous to tendons, the sPAMs exert a tension force on the robot's pneumatic backbone, causing bending that is approximately constant curvature. Unlike a traditional tendon driven continuum robot, the robot is entirely soft and contains no hard components, making it safer for human interaction. Models of both the sPAM and soft continuum robot kinematics are presented and experimentally verified. We found a mean position accuracy of 5.5 cm for predicting the end-effector position of a 42 cm long robot with the kinematic model. Finally, closed-loop control is demonstrated using an eye-in-hand visual servo control law which provides a simple interface for operation by a human. The soft continuum robot with closed-loop control was found to have a step-response rise time and settling time of less than two seconds.

Image-Based Visual Servoing of a Quadrotor Using Virtual Camera Approach

In this paper, an image-based visual servoing control law is proposed for a quadrotor unmanned aerial vehicle using an on-board monocular camera and an inertial measurement unit sensor. Based on the perspective projection model, suitable image features are defined on a rotated image plane called virtual image plane, thus a decoupled image feature dynamics is achieved. Then, a translational velocity observer is presented using these image features. The image feature dynamics and quadrotor dynamics are combined to derive a nonlinear controller. The controller is based on backstepping technique to account for the underactuation of the quadrotor. The image-based visual servoing controller only needs three point features, which make it useable in general environment. The closed-loop system is proved globally asymptotic stable by means of Lyapunov analysis. Computer simulations that regulate a quadrotor to a desired position with respect to (w.r.t.) four points lying on a horizontal plane and three points lying on a full rotated slope are conducted separately. Smooth and efficient trajectories are obtained both in virtual image plane and Cartesian space. Finally, experimental tests including pushing and pulling the visual target are conducted to verify the validity and robustness of the proposed controller. The proposed control law regulates the quadrotor to a desired position, defined by desired image, from an unknown initial position, which can be used in monitoring, landing, and other applications.

Adaptive-Repetitive Visual-Servo Control of Low-Flying Aerial Robots via Uncalibrated High-Flying Cameras

This paper presents the design and implementation of an adaptive-repetitive visual-servo control system for a moving high-flying vehicle (HFV) with an uncalibrated camera to monitor, track, and precisely control the movements of a low-flying vehicle (LFV) or mobile ground robot. Applications of this control strategy include the use of high-flying unmanned aerial vehicles (UAVs) with computer vision for monitoring, controlling, and coordinating the movements of lower altitude agents in areas, for example, where GPS signals may be unreliable or nonexistent. When deployed, a remote operator of the HFV defines the desired trajectory for the LFV in the HFV’s camera frame. Due to the circular motion of the HFV, the resulting motion trajectory of the LFV in the image frame can be periodic in time, thus an adaptive-repetitive control system is exploited for regulation and/or trajectory tracking. The adaptive control law is able to handle uncertainties in the camera’s intrinsic and extrinsic parameters. The design and stability analysis of the closed-loop control system is presented, where Lyapunov stability is shown. Simulation and experimental results are presented to demonstrate the effectiveness of the method for controlling the movement of a low-flying quadcopter, demonstrating the capabilities of the visual-servo control system for localization (i.e.,, motion capturing) and trajectory tracking control. In fact, results show that the LFV can be commanded to hover in place as well as track a user-defined flower-shaped closed trajectory, while the HFV and camera system circulates above with constant angular velocity. On average, the proposed adaptive-repetitive visual-servo control system reduces the average RMS tracking error by over 77% in the image plane and over 71% in the world frame compared to using just the adaptive visual-servo control law.

Dynamic Visual Servoing for a Quadrotor Using a Virtual Camera

This paper presents a dynamic image-based visual servoing (IBVS) control law for a quadrotor unmanned aerial vehicle (UAV) equipped with a single fixed on-board camera. The motion control problem is to regulate the relative position and yaw of the vehicle to a moving planar target located within the camera’s field of view. The control law is termed dynamic as it’s based on the dynamics of the vehicle. To simplify the kinematics and dynamics, the control law relies on the notion of a virtual camera and image moments as visual features. The convergence of the closed-loop is proven to be globally asymptotically stable for a horizontal target. In the case of nonhorizontal targets, we modify the control using a homography decomposition. Experimental and simulation results demonstrate the control law’s performance.

Incremental visual servo control of robotic manipulator for autonomous capture of non-cooperative target

This paper develops a new autonomous incremental visual servo control law for the robotic manipulator to capture a non-cooperative target, where the control input is the incremental joint angle to avoid the multiple solutions in the existing inverse kinematics. The position and motion of the non-cooperative target are estimated by an eye-to-hand vision system in real time by integrated photogrammetry and extended Kalman filter. The estimated position and motion of the target are fed into the newly developed position-based visual servo control law to drive the manipulator incrementally towards the dynamically predicted interception point between trajectories of the end effector and the target. To validate the proposed approach, a hardware-in-the-loop simulation has been conducted where the position and motion of the target is estimated by a real eye-to-hand camera and fed into the simulation of the robotic manipulator. The simulation results show the proposed incremental visual servo control law is stable and able to avoid the multiple solutions in the total inverse kinematics.

Alignment of a flexible sheet object with position-based and image-based visual servoing

This paper describes visual servoing methods to align an end edge of a flexible sheet object such as paper to a specified line segment in 3D space. In order to manipulate the end edge pose which may be distant from a grasping point by a robot hand on the flexible sheet, we incorporate online estimation of the relative pose between the end edge and the robot hand into visual servoing control laws. Two types of feature values to be regulated, position-based and image-based ones, are defined. The position-based one consists of 3D pose parameters of the end edge, which are estimated by fitting a polynomial surface model to stereo-measured edge points around the sheet. The image-based one consists of image pixel values within narrow regions around the end edge. Through numerical simulations, we confirm that the position-based features are effective in achieving quick response while the image-based features are robust against measurement noise. In order to combine these contrasting behaviors, we introduce a weighted combination of the position-based and image-based features. Experimental results show both of the position-based method and the weighted combination method work well and suggest the possibility that the weighted combination method can reduce final alignment errors.

Adaptive Visual Servo Control of Robotic Harvesting Systems

Harvesting efficiency is a critical factor in determining whether robotic harvesting can be economically superior to hand picking. One practical challenge in robotic harvesting is to grasp a moving fruit, where unsuccessful pick cycles may result in lower harvesting efficiency. The fruit motion can be due to exogenous disturbances such as wind gust, canopy unloading, and particularly, fruit detachment forces. In this paper, an adaptive visual servo control law is presented to estimate and subsequently compensate the unknown time-varying fruit motion. The fruit motion in the image plane and along the optical axis is modeled as a second-order spring-mass system. The unknown parameters of fruit motion are identified using an adaptive parameter update law, and a robust feedback term is included in the control law to account for modeling uncertainties. Lyapunov-based stability analysis guarantees uniformly ultimately bounded regulation of the robot to the fruit position. Simulation results are provided to verify the feasibility of the developed controller.

Laser steering using virtual trifocal visual servoing

This paper focuses on the development of a weakly calibrated three-view-based visual servoing control law applied to the laser steering process. It proposes to revisit the conventional trifocal constraints governing a three-view geometry for a more suitable use in the design of an efficient trifocal vision-based control. Thereby, an explicit control law is derived, without any matrix inversion, which allows to simply prove the global exponential stability of the control. Moreover, only ‘’ are necessary to design a fast trifocal control system. Thanks to the simplicity of the implementation, our control law is fast, accurate, robust to errors on the weak calibration and it exhibits good behavior in terms of convergence and decoupling. This was demonstrated by different experimental validations performed on a test-bench for the steering of laser spot on a 2D and 3D surface using a two degrees-of-freedom commercial piezoelectric mirror, as well as in preliminary cadaver trials using an endoluminal micromirror prototype.

Nonlinear dynamic image-based visual servoing of a quadrotor

This paper proposes a dynamic image-based visual servoing (IBVS) control law for a quadrotor unmanned aerial vehicle (UAV) equipped with a single fixed on-board camera facing downward. The motion control problem is to regulate the relative lateral position of the vehicle to a stationary target located on the ground. The control law is termed dynamic as it is based on the dynamics and kinematics of the vehicle. The proposed design uses a nonlinear input-dependent change of state coordinates and its error dynamics are proven to be locally exponentially stable with an estimate provided for the region of attraction. Experimental and simulation results demonstrate the method's ease of on-board implementation, performance, and robustness. The simulation and experimental results include a comparison with an established dynamic IBVS method. This comparison shows the proposed method can provide similar performance with the benefit of reduced complexity.

A New Virtual Visual Servo Algorithm

In this paper, on the basis of the robot visual servo system, a virtual visual servo algorithm for pose estimation is put forward. By reducing the image error between the current image features of real object and the features of virtual models, the rotation and translation relation of the virtual model is changed by visual servo control law, until the pose of virtual model overlaps the one of real object, and then object pose estimation is obtained. In addition, RANSAC algorithm is also introduced to improve the robustness of the proposed algorithm. Finally the experiment about camera external parameters estimation shows that our algorithm gets better results than the available parameter estimation algorithms.

21105 超広角レンズの広角特性を維持した特徴ベーストビジュアルサーボ(OS10-4【ロボティクス・メカトロニクス(4)】)

Abstract: Feature-based visual servo determines camera movement by comparing a current feature image and the goal feature image. This method does not require a geometric model of the object because it does not need a 3D information of the object. Therefore, it is robust to various disturbances. Various methods have been proposed. For example using a wide-angle lens enables the camera to capture the object in a wide range However, using a wide-angle lens in the conventional control law, the camera would conduct waste motion because of the peripheral distortion of the wide-angle lens. In addition, if the distortion is corrected, the wide-angle properties will be lost. This paper proposes a new Feature-based visual servoing control law by preserving wide-angle properties of super wide-angle lens.

마커인식과 혼합 비주얼 서보잉 기법을 통한 이동로봇의 자세 안정화 제어

This paper proposes a posture stabilization control algorithm for a wheeled mobile robot using hybrid visual servo control method with a position based and an image based visual servoing (PBVS and IBVS). To overcome chattering phenomena which were shown in the previous researches using a simple switching function based on a threshold, the proposed hybrid visual servo control law introduces the fusion function based on a blending function. Then, the chattering problem and rapid motion of the mobile robot can be eliminated. Also, we consider the nonlinearity of the wheeled mobile robot unlike the previous visual servo control laws using linear control methods to improve the performances of the visual servo control law. The proposed posture stabilization control law using hybrid visual servoing is verified by a theoretical analysis and simulation and experimental results.