Image-based Visual Servoing Method Research Articles

An Innovative Collision-Free Image-Based Visual Servoing Method for Mobile Robot Navigation Based on the Path Planning in the Image Plan.

In this article, we present an innovative approach to 2D visual servoing (IBVS), aiming to guide an object to its destination while avoiding collisions with obstacles and keeping the target within the camera's field of view. A single monocular sensor's sole visual data serves as the basis for our method. The fundamental idea is to manage and control the dynamics associated with any trajectory generated in the image plane. We show that the differential flatness of the system's dynamics can be used to limit arbitrary paths based on the number of points on the object that need to be reached in the image plane. This creates a link between the current configuration and the desired configuration. The number of required points depends on the number of control inputs of the robot used and determines the dimension of the flat output of the system. For a two-wheeled mobile robot, for instance, the coordinates of a single point on the object in the image plane are sufficient, whereas, for a quadcopter with four rotating motors, the trajectory needs to be defined by the coordinates of two points in the image plane. By guaranteeing precise tracking of the chosen trajectory in the image plane, we ensure that problems of collision with obstacles and leaving the camera's field of view are avoided. Our approach is based on the principle of the inverse problem, meaning that when any point on the object is selected in the image plane, it will not be occluded by obstacles or leave the camera's field of view during movement. It is true that proposing any trajectory in the image plane can lead to non-intuitive movements (back and forth) in the Cartesian plane. In the case of backward motion, the robot may collide with obstacles as it navigates without direct vision. Therefore, it is essential to perform optimal trajectory planning that avoids backward movements. To assess the effectiveness of our method, our study focuses exclusively on the challenge of implementing the generated trajectory in the image plane within the specific context of a two-wheeled mobile robot. We use numerical simulations to illustrate the performance of the control strategy we have developed.

Dynamic Target Tracking of Unmanned Aerial Vehicles Under Unpredictable Disturbances

This study proposes an image-based visual servoing (IBVS) method based on a velocity observer for an unmanned aerial vehicle (UAV) for tracking a dynamic target in Global Positioning System (GPS)-denied environments. The proposed method derives the simplified and decoupled image dynamics of underactuated UAVs using a constructed virtual camera and then considers the uncertainties caused by the unpredictable rotations and velocities of the dynamic target. A novel image depth model that extends the IBVS method to track a rotating target with arbitrary orientations is proposed. The depth model ensures image feature accuracy and image trajectory smoothness in rotating target tracking. The relative velocities of the UAV and the dynamic target are estimated using the proposed velocity observer. Thanks to the velocity observer, translational velocity measurements are not required, and the control chatter caused by noise-containing measurements is mitigated. An integral-based filter is proposed to compensate for unpredictable environmental disturbances in order to improve the anti-disturbance ability. The stability of the velocity observer and IBVS controller is analyzed using the Lyapunov method. Comparative simulations and multistage experiments are conducted to illustrate the tracking stability, anti-disturbance ability, and tracking robustness of the proposed method with a dynamic rotating target.

Robust Image-Based Landing Control of a Quadrotor on an Unpredictable Moving Vehicle Using Circle Features

This paper addresses the landing problem of a quadrotor on an unpredictable moving vehicle, using a robust image-based visual servoing (IBVS) method. The circle-based image moments are defined to construct image dynamics, and the passivity-like property of the circle features is preserved by reprojecting to the virtual image plane. The landing control system is decoupled into translation and rotation modules due to the rotation invariance of the proposed circle features. First, by exploiting the error transformation in the image space, a robust IBVS controller can overcome the lack of both the desired depth information of target features and the velocity feedback of the target. Next, an adaptive geometric attitude controller is developed directly using rotation matrices to avoid the singularities of Euler-angles and the ambiguity of quaternions. One benefit of the proposed scheme is that it can potentially improve the camera visibility, guarantee the transient and steady-state behaviors in image space, and be efficiently implemented on the low-cost quadrotor. Finally, The stability analysis is presented using Lyapunov stability theory on cascaded systems, and the effectiveness of the proposed control strategy is demonstrated through simulations and experiments. Note to Practitioners—The motivation of this paper is to investigate a practical control strategy for the image-based landing control of underactuated quadrotors on an unpredictable moving vehicle. In most of the existing image-based landing control schemes for underactuated quadrotors, having the prior predictive model of the moving landing vehicle to provide a feed-forward compensation during the landing maneuver. However, due to the fact that the landing environment and vehicle are primarily stochastic, resulting in no predictive models are valid in practice. Therefore, this paper suggests a robust image-based landing control strategy without the model or state of the moving landing vehicle. In particular, a novel virtual circle feature, possessing the characteristic of rotation invariance, is designed for the landing of underactuated quadrotors, which decouples the landing system and simplifies the control design. Moreover, the image feature errors are directly retained within prescribed performance funnels in the image space. As a result, the transient and steady-state landing behaviors can be implicitly guaranteed in Cartesian space. The stability and convergence of the system are analyzed mathematically and the experiment using quadrotors provides promising results. In ongoing research, we are addressing the issues of collision avoidances and unknown disturbances to provide a more realistic setup for the autonomous deployment and recovery of underactuated quadrotors in GPS-denied environments.

Towards realizing a visual UAV flying environment: A novel approach based aerial imagery to construct a dataset for visual servoing

Realizing a visual flying environment in various scenarios is an instrumental key to establishing visual hardware-in-the-loop (HIL) simulation for flying UAVs. To simulate a desired UAV flying mission for a specific building landing in arbitrary geodetic coordinates, a new approach is presented. The approach fulfills the UAV landing HIL via three key elements: Building, Training, and Servoing. Building a flying dynamic scene which is not limited to a certain scenario, but can be utilized for generating different aerial remote sensing scenes according to desired geodetic coordinates (location-free), flying mission, perspective angles, and approaching velocities. The dynamic scene is constructed via RHINO graphics software based on georeferenced images taken at different altitudes. Training a deep detection framework on the extracted dataset from each constructed scene to localize a specific building as a landing platform. Servoing a pan–tilt camera on an embedded system by a pure image-based visual servoing (IBVS) method to coincide the detected building center with the acquired frame center during the flight. Tiny-YOLOv4 is picked as a light detector that runs on Nvidia nano with a servo kit and achieves a satisfactory result even with two wind disturbances models in a real-time condition. Thus, verifying a successful UAV landing on a specific building approach is accomplished.

Adaptive neural network control for image‐based visual servoing of robot manipulators

This paper presents a novel adaptive neural network control strategy for image-based visual servoing (IBVS) of robotic manipulators with both eye-in-hand and eye-to-hand camera configurations in the presence of unknown dynamics and external disturbances. The IBVS method is combined with the adaptive neural network to construct the proposed adaptive neural network controller to solve the visual servoing control problem of robots. The adaptive neural network based IBVS controller is designed based on the depth-independent interaction matrix, which can be trained on-line to identify the visual servoing robotic system modeling errors. Moreover, the proposed method can approach the unknown nonlinear dynamics for both eye-in-hand and eye-to-hand camera configurations without requiring the robot dynamics to be linearly parameterizable, and the exact knowledge of the robot structure is not needed. On the basis of the nonlinear robot dynamics, the Lyapunov stability analysis is given to prove the asymptotical convergence of the image position and velocity errors. Simulation results for both camera configurations are provided to demonstrate the performance of the proposed adaptive neural network based approach.

A Switching Image-Based Visual Servoing Method for Cooperative Continuum Robots

A Switching Image-Based Visual Servoing Method for Cooperative Continuum Robots

Optimized hybrid decoupled visual servoing with supervised learning

This study proposes an optimized hybrid visual servoing approach to overcome the imperfections of classical two-dimensional, three-dimensional and hybrid visual servoing methods. These imperfections are mostly convergence issues, non-optimized trajectories, expensive calculations and singularities. The proposed method provides more efficient optimized trajectories with shorter camera path for the robot than image-based and classical hybrid visual servoing methods. Moreover, it is less likely to lose the object from the camera field of view, and it is more robust to camera calibration than the classical position-based and hybrid visual servoing methods. The drawbacks in two-dimensional visual servoing are mostly related to the camera retreat and rotational motions. To tackle these drawbacks, rotations and translations in Z-axis have been separately controlled from three-dimensional estimation of the visual features. The pseudo-inverse of the proposed interaction matrix is approximated by a neuro-fuzzy neural network called local linear model tree. Using local linear model tree, the controller avoids the singularities and ill-conditioning of the proposed interaction matrix and makes it robust to image noises and camera parameters. The proposed method has been compared with classical image-based, position-based and hybrid visual servoing methods, both in simulation and in the real world using a 7-degree-of-freedom arm robot.

Coarse-to-Fine Visual Object Catching Strategy Applied in Autonomous Airport Baggage Trolley Collection

In this study, a systematic solution for mobile manipulation with the application of autonomous trolley collection in the airport is presented. The developed system features a novel integrated hardware and software system, incorporating a specially designed manipulator capable of tightly catching the trolley. Particularly, we innovatively present a supervisory control guided coarse-to-fine visual trolley catching strategy in large-scale airport environments. At the coarsest level, an image-based visual servoing method is utilized to approach the trolley from far to near, where a learning-based object detection method is adopted to identify trolleys in the environment in real-time. At the finest level, we present an accurate point cloud registration based object pose estimation method to promote the object manipulation with a position-based visual servoing controller. The tightly integrated system and the subsystems are evaluated through conducting experiments in the trolley collection task and comparing the key module with the state of the art. Extensive experiments demonstrate that the proposed framework can find and fetch the trolley robustly. Notably, the proposed object detection and pose estimation methods exhibit higher accuracy, providing credible perceptual approaches for mobile manipulation. Moreover, the proposed framework can be easily extended to other mobile manipulation tasks for object search and retrieval.

Dynamic Image-Based Visual Servoing of Monocular Camera Mounted Omnidirectional Mobile Robots Considering Actuators and Target Motion via Fuzzy Integral Sliding Mode Control

In this article, an image-based visual servoing (IBVS) method for monocular camera mounted three-wheeled omnidirectional mobile robots (OMRs) is proposed using a novel fuzzy integral sliding mode control (FISMC) method. Most of the existing studies for IBVS of wheeled mobile robots (WMRs) have focused only on kinematics, even though no consideration for the OMR dynamics, actuators, and target motion can lead to much degraded performance. Accordingly, a camera mounted OMR is first modeled considering the OMR dynamics, actuator dynamics, and disturbance effects of the target object motion. On the basis of the OMR model, a dynamic IBVS method (i.e., IBVS based on kinematics, dynamics, and actuators) is then proposed to replace the kinematic IBVS method (i.e., IBVS based only on kinematics) such that the effects of the OMR dynamics, actuators, and target motion can be accommodated by using the FISMC method. In particular, the proposed FISMC method for the IBVS of OMRs can eliminate the reaching phase to make the states remain on the sliding surface all the time and improve the transient performance significantly when compared with the existing methods. The robust stability of the overall system using the proposed system is demonstrated by Lyapunov stability analysis. Simulation results are also provided to show that where the kinematic IBVS achieves much degraded performance owing to the OMR dynamics and target motion, the proposed FISMC-based dynamic IBVS method compensates for these effects and achieve the satisfactory feature point tracking performance, verifying the effectiveness of the proposed method.

Adaptive visual servoing with an uncalibrated camera using extreme learning machine and Q-leaning

In this paper, a novel image-based visual servoing (IBVS) method using Extreme Learning Machine (ELM) and Q-learning is proposed to solve the problems of complex modeling and selection of the servo gain. First, the pseudoinverse of the interaction matrix is approached by ELM which avoids the singularity of the interaction matrix effectively and is robust to interferences such as feature noises and camera calibration errors. Second, a reinforcement learning method, Q-learning, is adopted to adaptively adjust the servo gain in order to improve the convergence speed and stability. Compared with other methods, ELM has better generalization performance, faster operation speed and a unique optimal solution. Also, Q-learning has self-learning ability without experience in advance. The effectiveness of the proposed method is validated by simulations and experiment on a 6-DOF robot with eye-in-hand configuration.

A new image-based visual servoing method with velocity direction control

In image-based visual servoing (IBVS), for more satisfying trajectories, the target is always expected to move in straight lines to the desired positions. In this paper, a new IBVS controller for trajectory adjustment is obtained by optimizing a multi-objective function about adjusting a direction in the image plane. By applying the control law based on the optimized parameters, less redundant motion can be achieved in contrast to the traditional methods. Meanwhile, it will be beneficial for reducing the risk of the target leaving the field of view (FOV) and the robustness to the uncertainties of depth and camera internal parameters remains. The experimental results of the 6 degrees of freedom (DOF) robot with eye-in-hand configuration demonstrate the effectiveness and practicability of the proposed method.

Adaptive Switch Image-based Visual Servoing for Industrial Robots

In this paper, an adaptive switch image-based visual servoing (IBVS) controller for industrial robots is presented. The proposed control algorithm decouples the rotational and translational camera motions and decomposes the IBVS control into three separate stages with different gains. This method can increase the system response speed and improve the tracking performance of IBVS while the proposed adaptive law deals with the uncertainties of the monocular camera in eye-in-hand configuration. The stability of the designed controller is proved using Lyapunov method. Experimental results on a 6 degree of freedom (DOF) robot show the significant enhancement of the control performance over other IBVS methods, in terms of the response time and tracking performance. Also the designed visual servoing controller demonstrates its capability to overcome some of the inherent drawbacks of IBVS, such its inability to perform a 180° camera rotation around its center.

A learning approach to image-based visual servoing with a bagging method of velocity calculations

Visual servoing allows accurate control of positioning and motion relative to a stationary or moving target using vision and is the subject of many studies. Most servoing gains for image-based visual servoing methods are selected heuristically or empirically so accuracy is affected. This study uses reinforcement learning to adaptively tune the proportional servoing gain over sequences of image based visual servoing, instead of using a constant gain. The system is used to control hovering and tracking for a stationary or slowly moving target. An image Jacobian matrix with four dimensions is constructed because a quad-rotor drone features under-actuated dynamics. The Moore-Penrose pseudo inverse method is usually used to calculate the inverse image Jacobian matrix, but this study uses a bagging approach to calculate the inverse kinematics. The desired velocity is obtained from time-varying image errors, which gives greater robustness. An adaptive method to calculate the servoing gain method is proposed, whereby the selection of an appropriate servo gain over time is regarded as a reinforcement learning problem. The proposed visual servoing control system is implemented and tested experimentally using a quad-rotor drone system. The experimental results demonstrate that the proposed method is more robust and converges faster than other methods.

Decoupled Visual Servoing With Fuzzy Q-Learning

The objective of visual servoing aims to control an object's motion with visual feedbacks and becomes popular recently. Problems of complex modeling and instability always exist in visual servoing methods. Moreover, there are few research works on selection of the servoing gain in image-based visual servoing (IBVS) methods. This paper proposes an IBVS method with Q -Learning, where the learning rate is adjusted by a fuzzy system. Meanwhile, a synthetic preprocess is introduced to perform feature extraction. The extraction method is actually a combination of a color-based recognition algorithm and an improved contour-based recognition algorithm. For dealing with underactuated dynamics of the unmanned aerial vehicles (UAVs), a decoupled controller is designed, where the velocity and attitude are decoupled through attenuating the effects of underactuation in roll and pitch and two independent servoing gains, for linear and angular motion servoing, respectively, are designed in place of single servoing gain in traditional methods. For further improvement in convergence and stability, a reinforcement learning method, Q -Learning, is taken for adaptive servoing gain adjustment. The Q -Learning is composed of two independent learning agents for adjusting two serving gains, respectively. In order to improve the performance of the Q -Learning, a fuzzy-based method is proposed for tuning the learning rate. The results of simulations and experiments on control of UAVs demonstrate that the proposed method has better properties in stability and convergence than the competing methods.

Partially Decoupled Image-Based Visual Servoing Using Different Sensitive Features

A new image-based visual servoing method based on sensitive features is presented to separately realize the position control and orientation control. Line features are used for the orientation control because of their sensitivities to rotational motions. Point features and area size features are employed to realize the position control since area size features are very sensitive to the objects’ depths. The translations resulting from rotational motions are introduced into the position control as the compensation in order to eliminate the influence of the camera’s motions on the point features. The depths for all active features are estimated via interaction matrices, features variations, and the executed camera motions. The proposed method can keep the tracked objects in the camera’s field of view in the visual servoing process. In addition, the determination methods of the interaction matrices for point, line, and area size features are proposed. Comparing to the traditional method, the proposed determination method of the interaction matrix for line is independent from the parameters of the plane containing the line. Experimental results verify the effectiveness of the proposed methods.

Multi-cameras visual servoing for dual-arm coordinated manipulation

SUMMARYAlthough image-based visual servoing (IBVS) provides good performance in many dual-arm manipulation applications, it reveals some fatal limitations when dealing with a large position and orientation uncertainty. The object features may leave the camera's field of view, and the dual-arm robot may not converge to their goal configurations. In this paper, a novel vision-based control strategy is presented to resolve these limitations. A visual path planning method for dual-arm end-effector features is proposed to regulate the large initial poses to the pre-alignment poses. Then, the visual constraints between the position and orientation of two objects are established, and the sequenced subtasks are performed to attain the pose alignment of two objects by using a multi-tasks IBVS method. The proposed strategy has been implemented on a MOTOMAN robot to perform the alignment tasks of plug–socket and cup–lid, and results indicate that the plug and socket with the large initial pose errors 145.4 mm, 43.8○ (the average errors of three axes) are successfully aligned with the allowed pose alignment errors 3.1 mm, 1.1○, and the cup and lid with the large initial pose errors 131.7 mm, 20.4○ are aligned with the allowed pose alignment errors −2.7 mm, −0.8○.

Imaged-Based Visual Servo Control for a VTOL Aircraft

This paper presents a novel control strategy to force a vertical take-off and landing (VTOL) aircraft to accomplish the pinpoint landing task. The control development is based on the image-based visual servoing method and the back-stepping technique; its design differs from the existing methods because the controller maps the image errors onto the actuator space via a visual model which does not contain the depth information of the feature point. The novelty of the proposed method is to extend the image-based visual servoing technique to the VTOL aircraft control. In addition, the Lyapunov theory is used to prove the asymptotic stability of the VTOL aircraft visual servoing system, while the image error can converge to zero. Furthermore, simulations have been also conducted to demonstrate the performances of the proposed method.

Visual-based quadrotor control by means of fuzzy cognitive maps

By applying an image-based visual servoing (IBVS) method, the intelligent image-based controlling of a quadrotor type unmanned aerial vehicle (UAV) tracking a moving target is studied in this paper. A fuzzy cognitive map (FCM) is a soft computing method which is classified as a fuzzy neural system and exploits the main aspects of fuzzy logic and neural network systems; so it seems to be a suitable choice for implementing a vision-based intelligent technique. An FCM has been employed in implementing an IBVS scheme on a quadrotor UAV, so that the UAV can track a moving target on the ground. For this purpose, by properly combining the perspective image moments, some features with the desired characteristics for controlling the translational and yaw motions of a UAV have been presented. In designing a vision-based control method for a UAV quadrotor, there are some challenges, including the target mobility and not knowing the height of UAV above the target. Also, no sensor has been installed on the moving object and the changes of its yaw angle are not available. Despite all the stated challenges, the proposed method, which uses an FCM in controlling the translational motion and the yaw rotation of a UAV, adequately enables the quadrotor to follow the moving target. The simulation results for different paths show the satisfactory performance of the designed controller.

AN ADAPTIVE SCHEME FOR IMAGE-BASED VISUAL SERVOING OF AN UNDERACTUATED UAV

An image-based visual servoing (IBVS) method is proposed for controlling the 3D translational motion and the yaw rotation of a quadrotor. The dynamic model of this unmanned aerial vehicle (UAV) is considered at the design stage to account for its underactuation. In contrast with previous IBVS methods for underactuated UAVs, which used spherical image moments as visual features, the proposed controller makes use of appropriately defined perspective moments. As a consequence, we gain a clear improvement in performance, as satisfactory trajectories are obtained in both image and Cartesian space. In addition, an adaptation mechanism is included in the controller to achieve robust performance in spite of uncertainties related to the depth of the image features and to the dynamics of the robot. Simulation results in both nominal and perturbed conditions are presented to validate the proposed method.

자율주행 모바일 역진자의 비주얼서보잉에 대한 연구

This paper proposes an optimal three-dimensional coordinate implementation of the vision sensor using two CCD cameras. The PBVS (Position based visual servoing) is implemented using the positional information obtained from images. Stereo vision by PBVS method that has enhanced every frame using calibration parameters is effective in the distance calculation. The IBVS (Image based visual servoing) is also implemented using the difference between reference and obtained images. Stereo vision by IBVS method calculates the distance using rotation angle of motors that correspond eyes and neck without enhanced images. The PBVS method is compared with the IBVS method in terms of advantages, disadvantages, computing time, and performances. Finally, the IBVS method is applied for the dual arm manipulator on the mobile inverted pendulum. The autonomous mobile inverted pendulum is successfully demonstrated using the center of the manipulator's mass.