3D Visual Servoing Research Articles

An Innovative Collision-Free Image-Based Visual Servoing Method for Mobile Robot Navigation Based on the Path Planning in the Image Plan.

In this article, we present an innovative approach to 2D visual servoing (IBVS), aiming to guide an object to its destination while avoiding collisions with obstacles and keeping the target within the camera's field of view. A single monocular sensor's sole visual data serves as the basis for our method. The fundamental idea is to manage and control the dynamics associated with any trajectory generated in the image plane. We show that the differential flatness of the system's dynamics can be used to limit arbitrary paths based on the number of points on the object that need to be reached in the image plane. This creates a link between the current configuration and the desired configuration. The number of required points depends on the number of control inputs of the robot used and determines the dimension of the flat output of the system. For a two-wheeled mobile robot, for instance, the coordinates of a single point on the object in the image plane are sufficient, whereas, for a quadcopter with four rotating motors, the trajectory needs to be defined by the coordinates of two points in the image plane. By guaranteeing precise tracking of the chosen trajectory in the image plane, we ensure that problems of collision with obstacles and leaving the camera's field of view are avoided. Our approach is based on the principle of the inverse problem, meaning that when any point on the object is selected in the image plane, it will not be occluded by obstacles or leave the camera's field of view during movement. It is true that proposing any trajectory in the image plane can lead to non-intuitive movements (back and forth) in the Cartesian plane. In the case of backward motion, the robot may collide with obstacles as it navigates without direct vision. Therefore, it is essential to perform optimal trajectory planning that avoids backward movements. To assess the effectiveness of our method, our study focuses exclusively on the challenge of implementing the generated trajectory in the image plane within the specific context of a two-wheeled mobile robot. We use numerical simulations to illustrate the performance of the control strategy we have developed.

Quadrotor UAV Dynamic Visual Servoing Based on Differential Flatness Theory

In this paper, we propose 2D dynamic visual servoing (Dynamic IBVS), where a quadrotor UAV tries to track a moving target using a single facing-down perspective camera. As an application, we propose the tracking of a car-type vehicle. In this case, data related to the altitude and the lateral angles have no importance for the visual system. Indeed, to perform the tracking, we only need to know the longitudinal displacements (along the x and y axes) and the orientation along the z-axis. However, those data are necessary for the quadrotor’s guidance problem. Thanks to the concept of differential flatness, we demonstrate that if we manage to extract the displacements according to the three axes and the orientation according to the yaw angle (the vertical axis) of the quadrotor, we can control all the other variables of the system. For this, we consider a camera equipped with a vertical stabilizer that keeps it in a vertical position during its movement (a gimbaled camera). Other specialized sensors measure information regarding altitude and lateral angles. In the case of classic 2D visual servoing, the elaboration of the kinematic torsor of the quadrotor in no way guarantees the physical realization of instructions, given that the quadrotor is an under-actuated system. Indeed, the setpoint has a dimension equal to six, while the quadrotor is controlled only by four inputs. In addition, the dynamics of a quadrotor are generally very fast, which requires a high-frequency control law. Furthermore, the complexity of the image processing stage can cause delays in motion control, which can lead to target loss. A new dynamic 2D visual servoing method (Dynamic IBVS) is proposed. This method makes it possible to generate in real time the necessary movements for the quadrotor in order to carry out the tracking of the target (vehicle) using a single point of this target as visual information. This point can represent the center of gravity of the target or any other part of it. A control by flatness has been proposed, which guarantees the controllability of the system and ensures the asymptotic convergence of the generated trajectory in the image plane. Numerical simulations are presented to show the effectiveness of the proposed control strategy.

Mobile Robot Control Based on 3D Visual Servoing: A New Approach Combining Pose Estimation by Neural Network and Differential Flatness

This paper deals with 3D visual servoing applied to mobile robots in the presence of measurement disturbances, caused in particular by target occlusion. We propose a new approach based on the flatness concept. In 3D visual servoing, the task is performed out of image coordinate space and targets may leave the camera field of view during navigation (servoing). Forced to navigate blindly during one or more periods of time, the robot will use our new open-loop control algorithm inspired by the flatness concept. The 3D visual servoing method is performed using robot pose estimation. This estimation generally contains some errors. The exact position of the robot is therefore not guaranteed, and robust feedback control is necessary to reject these errors in the input. To solve this problem, we propose a new pose estimation method that uses neural networks. We reduce the complexity of the architecture of the neural networks used (the number of variables to estimate) by proving that the location and the orientation of the robot can be ensured by using a single point in the image coordinate space for mobile robots with two degrees of freedom. To show the efficiency of the proposed algorithm, we use the RVCTOOLS MATLAB toolbox.

DeepNet-Based 3D Visual Servoing Robotic Manipulation

The fourth industrial revolution (industry 4.0) demands high-autonomy and intelligence robotic manipulators. The goal is to accomplish autonomous manipulation tasks without human interventions. However, visual pose estimation of target object in 3D space is one of the critical challenges for robot-object interaction. Incorporating the estimated pose into an autonomous manipulation control scheme is another challenge. In this paper, a deep-ConvNet algorithm is developed for object pose estimation. Then, it is integrated into a 3D visual servoing to achieve a long-range mobile manipulation task using a single camera setup. The proposed system integrates (1) deep-ConvNet training using only synthetic single images, (2) 6DOF object pose estimation as sensing feedback, and (3) autonomous long-range mobile manipulation control. The developed system consists of two main steps. First, a perception network trains on synthetic datasets and then efficiently generalizes to real-life environment without postrefinements. Second, the execution step takes the estimated pose to generate continuous translational and orientational joint velocities. The proposed system has been experimentally verified and discussed using the Husky mobile base and 6DOF UR5 manipulator. Experimental findings from simulations and real-world settings showed the efficiency of using synthetic datasets in mobile manipulation task.

Robotic manipulation based on 3-D visual servoing and deep neural networks

The critical challenge, for robot–object-interaction, is to estimate visually the pose of the target object in a 3D space and combine it into a vision-based control scheme in manipulation applications. This paper proposes a novel reliable framework for deep ConvNet combined with visual servoing using a single RGB camera. We introduce an extensive system called Deep-Visual-Servoing (DVS) that addresses an integration of: (I) training of deep-CNNs using synthetic dataset only and operates successfully in real-world scenario, (II) continuous 3 D object pose estimation as the sensing feedback in a 3D visual servoing control scheme, and (III) design, integration and experimentation of visual servoing approach based on Lyapunov’s theory. The proposed deep based learning approach, the kinematic modeling and controller design are experimentally verified and discussed using the 6 DOF UR5 manipulator.

Automated real-time 3D visual servoing control of single cell surgery with application to microinjection processes

Automated real-time 3D visual servoing control of single cell surgery with application to microinjection processes

Automated real-time 3D visual servoing control of single cell surgery with application to microinjection processes

Automated real-time 3D visual servoing control of single cell surgery with application to microinjection processes

Improving the Manipulability of a Redundant Arm Using Decoupled Hybrid Visual Servoing

This study proposes a hybrid visual servoing technique that is optimised to tackle the shortcomings of classical 2D, 3D and hybrid visual servoing approaches. These shortcomings are mostly the convergence issues, image and robot singularities, and unreachable trajectories for the robot. To address these deficiencies, 3D estimation of the visual features was used to control the translations in Z-axis as well as all rotations. To speed up the visual servoing (VS) operation, adaptive gains were used. Damped Least Square (DLS) approach was used to reduce the robot singularities and smooth out the discontinuities. Finally, manipulability was established as a secondary task, and the redundancy of the robot was resolved using the classical projection operator. The proposed approach is compared with the classical 2D, 3D and hybrid visual servoing methods in both simulation and real-world. The approach offers more efficient trajectories for the robot, with shorter camera paths than 2D image-based and classical hybrid VS methods. In comparison with the traditional position-based approach, the proposed method is less likely to lose the object from the camera scene, and it is more robust to the camera calibrations. Moreover, the proposed approach offers greater robot controllability (higher manipulability) than other approaches.

Adaptive 3D Visual Servoing of a Scara Robot Manipulator with Unknown Dynamic and Vision System Parameters

In the present work, we develop an adaptive dynamic controller based on monocular vision for the tracking of objects with a three-degrees of freedom (DOF) Scara robot manipulator. The main characteristic of the proposed control scheme is that it considers the robot dynamics, the depth of the moving object, and the mounting of the fixed camera to be unknown. The design of the control algorithm is based on an adaptive kinematic visual servo controller whose objective is the tracking of moving objects even with uncertainties in the parameters of the camera and its mounting. The design also includes a dynamic controller in cascade with the former one whose objective is to compensate the dynamics of the manipulator by generating the final control actions to the robot even with uncertainties in the parameters of its dynamic model. Using Lyapunov’s theory, we analyze the two proposed adaptive controllers for stability properties, and, through simulations, the performance of the complete control scheme is shown.

Mobile Robot Control Based on 2D Visual Servoing: A New Approach Combining Neural Network With Variable Structure and Flatness Theory

This paper focuses on the 2D visual servo-control of a mobile robot using a neural network (NN) with variable structure. The interaction matrix relating camera movement and changes in visual characteristics requires an estimation phase to determine its parameters as well as a camera calibration phase. It is common in applications related to mobile robotics that the robot model contains uncertainties generated by the sliding phenomenon. We suggest online identification, using NN to avoid this problem. The RBF NN is used to estimate the block formed by the interaction matrix and the reverse robot. Since the number of variables to be estimated is large, this can lead to the use of an excessive number of RBFs. We propose to use a single point of the scene which is sufficient to solve the problem. This problem reduction is possible thanks to flatness theory which allows to reduce the number of NN inputs from 8 inputs (4 image points) -generally used in the literature- to 2 (one image point) only. In order to further reduce the complexity of the proposed algorithm, the number of neurons for each layer and for each iteration is optimized. We use a neural network with variable structure to reach this objective. The very encouraging results obtained validate the proposed approach.

Improved eye-vergence visual servoing system in longitudinal direction with RM-GA

Visual servoing is a control method to manipulate the motion of the robot using visual information, which aims to realize “working while watching.” However, the visual servoing towards moving target with hand–eye cameras fixed at hand is inevitably affected by hand dynamical oscillation. To overcome this defect of the hand–eye fixed camera system, an eye-vergence system has been put forward, where the pose of the cameras could be rotated to observe the target object. The visual servoing controllers of hand and eye-vergence are installed independently, so that it can observe the target object at the center of camera images through eye-vergence function. In this research, genetic algorithm (GA) is used as a pose tracking method, which is called “Real-Time Multi-step GA(RM-GA),” solves on-line optimization problems for 3D visual servoing. The performances of real-time object tracking using eye-vergence system and “RM-GA” method have been examined, and also the pose tracking accuracy has been verified.

楕円体近似モデルを用いた複眼立体ビジュアルサーボによる対象物追従

楕円体近似モデルを用いた複眼立体ビジュアルサーボによる対象物追従

Passivity‐based adaptive 3D visual servoing without depth and image velocity measurements for uncertain robot manipulators

SummaryIn this work, we consider the 3D visual tracking problem for a robot manipulator with uncertainties in the kinematic and dynamic models. The visual feedback is provided by a fixed and uncalibrated camera located above the robot workspace. The Cartesian motion of the robot end effector can be separated into a 1D motion parallel to the optical axis of the camera and a 2D motion constrained on a plane orthogonal to this axis. Thus, the control design can be simplified, and the overall visual servoing system can be partitioned in two almost‐independent subsystems. Adaptive visual servoing schemes, based on a kinematic approach, are developed for image‐based look‐and‐move systems allowing for both depth and planar tracking of a reference trajectory, without using image velocity and depth measurements. In order to include the uncertain robot kinematics and dynamics in the presented solution, we develop a cascade control strategy based on an indirect/direct adaptive method. The stability and convergence properties are analyzed in terms of Lyapunov‐like functions and the passivity‐based formalism. Numerical simulations including hardware‐in‐the‐loop results, obtained with a robot manipulator and a web camera, are presented to illustrate the performance and feasibility of the proposed control scheme. Copyright © 2016 John Wiley & Sons, Ltd.

Characterization of the Positioning Accuracy and Precision of MEMS Die Servoing Using Model-Based Visual Tracking

This work deals with an automated micro-assembly of MEMS die holders using a 3D visual servoing approach. Visual positioning which was used is based on eye-to-hand tracking principle. The 3D poses of the MEMS are supplied in a real-time by a 3D (CAD) model-based tracking algorithm. Having executed the visual servoing the accuracy and repeatability of this method was examined. The micro-assembly process was performed and the accuracy calculated according to ISO 9283. The achieved accuracy was 0.345 μm and repeatability was 4.3 μm. The rotation accuracy was less than 0.13°.

On intensity-based 3D visual servoing

This article investigates the problem of pose-based visual servoing whose equilibrium state is defined via a reference image. Differently from most solutions, this work directly exploits the pixel intensities without any feature extraction or matching. Intensity-based methods provide for higher accuracy and versatility. Another central idea of this work concerns the exploitation of the observability issue associated to monocular systems, which always occurs around the equilibrium. This overall framework allows for developing a family of new 3D visual servoing techniques with varying degrees of computational complexity and of prior knowledge. Importantly, they are arranged hierarchically from the simplest to the optimal ones, all in a unified scheme. Three new methods are then presented, and their closed-loop performances are experimentally assessed using a six degree-of-freedom robotic arm. As an additional contribution, these results refute the common belief that correct camera calibration and pose recovery are crucial to the accuracy of 3D visual servoing techniques.

A Generalized Visual Aid System for Teleoperation Applied to Satellite Servicing

This paper presents the latest results of a newly developed visual aid system for direct teleoperation. This method is extended to visual control to make an efficient teleoperation system by combining direct teleoperation and automatic control. On the one hand, an operator can conduct direct teleoperation with 3D graphic prediction simulation established by the VR technique. In order to remove inconsistencies between the virtual and real environments, a practical model-matching method is investigated. On the other hand, to realize real-time visual servoing control, a particular object recognition and pose estimation algorithm based on polygonal approximation is investigated to ensure a low computational cost for image processing. To avoid undesired forces involved in contact operation, 3D visual servoing incorporating a compliant control based on impedance control is developed. Finally, in a representative laboratory environment, a typical satellite servicing experiment is carried out based on this combined system. Experimental results demonstrate the feasibility and the effectiveness of the proposed method.

Photometric visual servoing for omnidirectional cameras

2D visual servoing consists in using data provided by a vision sensor for controlling the motions of a dynamic system. Most of visual servoing approaches has relied on the geometric features that have to be tracked and matched in the image acquired by the camera. Recent works have highlighted the interest of taking into account the photometric information of the entire image. This approach was tackled with images of perspective cameras. We propose, in this paper, to extend this technique to central cameras. This generalization allows to apply this kind of method to catadioptric cameras and wide field of view cameras. Several experiments have been successfully done with a fisheye camera in order to control a 6 degrees of freedom robot and with a catadioptric camera for a mobile robot navigation task.

Fuzzy vs Nonfuzzy in 2D Visual Servoing for Robot Manipulators

In this paper we consider the tracking control problem of planar robots manipulators via visual servoing in the presence of parametric uncertainties associated with the robot dynamics and the camera parameters. An image-based visual servo control is developed using a Takagi-Sugeno (T-S) fuzzy model. The design includes an image coordinate velocity observer. To the best knowledge of the authors such a proposal remains hitherto unpublished. The new scheme is compared experimentally with two different non-fuzzy algorithms showing the good performance and advantages of the complete system.

A Real-time 3D Pose Based Visual Servoing Implementation for an Autonomous Mobile Robot Manipulator

Today, the manipulation of objects by mobile robots is still a challenging task. This task is commonly decomposed on three stages: a) approaching to the objects, b) path planning and trajectory execution of the manipulator arm and finally c) fine tuning and grasping. In this work is presented an implementation of a 3D pose visual servoing for an autonomous mobile manipulator dealing with the last stage of the manipulation task (fine tuning and grasping). The methodology proposed consists of three steps: a) beginning with a fast monocular image segmentation, followed by b) 3D model reconstruction and finally c) pose estimation to feedback fine tuning manipulation control loop. Objects and end-effector are colored in different colors and their models are supposed to be known. Our mobile manipulator prototype consists of a stereo camera under a binocular stand-alone configuration and an anthropomorphic 7DoF arm with a parallel end-effector (gripper). Our methodology runs in real time and is suitable to perform continuous visual servoing. Experimental results are reported.

Template Tracking and Visual Servoing for Alignment Tasks with Autonomous Underwater Vehicles

Alignment of underwater vehicles with respect to underwater structures is of utmost importance for many tasks performed in marine environments, specially for those requiring manipulation and, therefore, a specific and stable vehicle pose. These skills are crucial in the case of autonomous vehicles, where there is not a human in the loop in charge of tele-operating the vehicle to a suitable position and attitude with respect to the target. In this paper we apply 2D visual servoing techniques based on template tracking to the problem of autonomously reaching and keeping a vehicle pose relative to a target in a typical underwater intervention scenario. Assuming that the object of interest has been already detected, we develop a method that autonomously aligns the vehicle relative to it, and keeps the pose during the task. The method is validated, first in simulation, and then in a water tank with the Girona 500 vehicle