Visual Servoing Algorithm Research Articles

Visual servoing using an optimized trajectory planning technique for a 4 DOFs robotic manipulator

Although visual servoing has been considered as a solution to increase dexterity and intelligence of the robotic systems specially in unstructured environments, some prominent deficiencies are preventing it from practical employment. Trajectory planning is a solution to overcome the shortcomings of visual servoing and makes it practical for industrial applications. In this paper, a new trajectory planning technique is developed to perform image-based visual servoing (IBVS) tasks for a 4 DOFs robotic manipulator system. In this method, the camera’s velocity screw is parameterized using time-based profiles. The parameters of the velocity profile are then determined such that the velocity profile takes the robot to its desired position. This is done by minimizing the errors between the initial and desired features. A depth estimation technique is proposed to provide the trajectory planning algorithm with an accurate initial depth. This algorithm is tested and validated via the experiment on a 4 DOFs Denso robot in an eye-in-hand configuration. Experimental results demonstrate that the proposed method provides with a reliable visual servoing algorithm by overcoming the IBVS drawbacks such as surpassing the system limits and causing instability of the system in fulfilling the tasks which require a 180° rotation of the camera about its center.

An Unmanned Ground Vehicles Experimental Setup for Image-Based Object Tracking

This paper presents a practical design of a visual servoing algorithm for mobile robots. The algorithm, implemented in mobile robots equipped with a camera, facilitates to detect and follow a specific object autonomously. The proposed scheme infers information directly from the images obtained by the camera, and then digital signal processing techniques are implemented in order to detect the object of interest and to generate the necessary movements of the robot in order to perform the tracking task. The algorithm provides effective performance when using images of objects that have well known characteristics, e.g., color and the size. Experimental results obtained with omnidirectional mobile robot performing indoors show that the proposed methodology allows locating and following a specific object by using the proposed image-based visual servoing control.

혼합 비주얼 서보잉을 통한 모바일 로봇의 물체 추종

This paper proposes a hybrid visual servoing algorithm for the object tracking by a mobile robot with the stereo camera. The mobile robot with the stereo camera performs an object recognition and object tracking using the SIFT and CAMSHIFT algorithms for the hybrid visual servoing. The CAMSHIFT algorithm using stereo camera images has been used to obtain the three-dimensional position and orientation of the mobile robot. With the hybrid visual servoing, a stable balance control has been realized by a control system which calculates a desired angle of the center of gravity whose location depends on variations of link rotation angles of the manipulator. A PID controller algorithm has adopted in this research for the control of the manipulator since the algorithm is simple to design and it does not require unnecessary complex dynamics. To demonstrate the control performance of the hybrid visual servoing, real experiments are performed using the mobile manipulator system developed for this research.

A New Virtual Visual Servo Algorithm

In this paper, on the basis of the robot visual servo system, a virtual visual servo algorithm for pose estimation is put forward. By reducing the image error between the current image features of real object and the features of virtual models, the rotation and translation relation of the virtual model is changed by visual servo control law, until the pose of virtual model overlaps the one of real object, and then object pose estimation is obtained. In addition, RANSAC algorithm is also introduced to improve the robustness of the proposed algorithm. Finally the experiment about camera external parameters estimation shows that our algorithm gets better results than the available parameter estimation algorithms.

Calibración de cámaras de tiempo de vuelo: Ajuste adaptativo del tiempo de integración y análisis de la frecuencia de modulación

The depth perception is essential in many manipulation tasks, visual inspection and robot navigation. The cameras of Time of Flight (TOF) generate range images which provide depth measurements in real time. However, the distance parameter computed from these cameras is strongly dependent on the integration time set for the sensor and the frequency of modulation used by the integrated lighting system. In this paper, a methodology for automatic setting of integration time and an experimental analysis of ToF camera behavior adjusting its modulation frequency is presented. This method has been successfully tested on visual servoing algorithms with architecture ‘eye-in-hand’ in which the sensory system consists of a ToF camera, in addition this methodology can be applied to other workspaces and scenarios.

Particle Filtering for Industrial 6DOF Visual Servoing

Visual servoing allows the introduction of robotic manipulation in dynamic and uncontrolled environments. This paper presents a position-based visual servoing algorithm using particle filtering. The objective is the grasping of objects using the 6 degrees of freedom of the robot manipulator in non-automated industrial environments using monocular vision. A particle filter has been added to the position-based visual servoing algorithm to deal with the different noise sources of those industrial environments. Experiments performed in the real industrial scenario of ROBOFOOT ( http://www.robofoot.eu/ ) project showed accurate grasping and high level of stability in the visual servoing process.

Mechatronics Systems Supported by Vision Techniques

In the paper an application of vision methods and algorithms in various domains that contribute to mechatronics is presented. Regarding mechatronics devices and machines as robots, a vision system employed for a testing station simulating an industrial assembly line is discussed. Some numerical aspects concerning image pre-processing, analysis and geometrical transformations commonly used in robotics were introduced. To accomplish an effective investigation, the developed methodology and algorithms were implemented and verified on an experimental setup composed of two industrial robots and automation devices cooperating with two various vision systems. In the case of underwater robots for tank inspection, image pre-processing and analysis algorithms for the robot's position estimation, an image scale calculation and wall crack detection were investigated. An active vibration control system is treated as a mechatronic device which contains mechanical parts, electronics and software. In this example, a visual servoing architecture based on image features for controlling an active vibration control system was examined. For an effective investigation and synthesis of visual servoing algorithms, a MATLAB/Simulink/dSPACE hardware–software environment was employed. A vision system was used to analyze vibration amplitude of the vibro-isolation mass of the active suspension system and to provide a feedback control signal. The connection of 3D vision techniques with modal analysis was shown. Within the confines of the project a methodology for amplitude of vibration measurement and a software tool for modal analysis realization based on visual data were developed. The 3D measurements and structure of the construction were obtained by application and development of passive 3-D vision techniques. From this area, ‘structure from motion’ techniques were developed. In the experimental research, a mechatronic test stand was designed and manufactured enabling automatic two-axis control of a camera. A frame structure was built, in which a guiding-rail system was mounted enabling straight-line motion of a camera. Additionally, a system realizing rotational motion of a camera was built in. To control the experiment stand, software was created enabling the combination of the hardware-software part of the stand with the software part of a vision system. A tool was developed for the purpose of modal analysis and estimation of the quantities characterizing dynamic properties of the structure based on vision signals. As a conclusion, the presented, implemented and tested vision methods in various hardware-software programming platforms are discussed

Adaptive and robust algorithms and tests for visual-based navigation of a space robotic manipulator

Optical navigation for guidance and control of robotic systems is a well-established technique from both theoretic and practical points of view. According to the positioning of the camera, the problem can be approached in two ways: the first one, “hand-in-eye”, deals with a fixed camera, external to the robot, which allows to determine the position of the target object to be reached. The second one, “eye-in-hand”, consists in a camera accommodated on the end-effector of the manipulator. Here, the target object position is not determined in an absolute reference frame, but with respect to the image plane of the mobile camera. In this paper, the algorithms and the test campaign applied to the case of the planar multibody manipulator developed in the Guidance and Navigation Lab at the University of Rome La Sapienza are reported with respect to the eye-in-hand case. In fact, being the space environment the target application for this research activity, it is quite difficult to imagine a fixed, non-floating camera in the case of an orbital grasping maneuver. The classic approach of Image Base Visual Servoing considers the evaluation of the control actions directly on the basis of the error between the current image of a feature and the image of the same feature in a final desired configuration. Both simulation and experimental tests show that such a classic approach can fail when navigation errors and actuation delays are included. Moreover, changing light conditions or the presence of unexpected obstacles can lead to a camera failure in target acquisition. In order to overcome these two problems, a Modified Image Based Visual Servoing algorithm and an Extended Kalman Filtering for feature position estimation are developed and applied. In particular, the filtering shows a quite good performance if target's depth information is supplied. A simple procedure for estimating initial target depth is therefore developed and tested. As a result of the application of all the novel approaches proposed, the experimental test campaign shows a remarkable increase in the robustness of the guidance, navigation and control systems.

A Universal State-Space Approach to Uncalibrated Model-Free Visual Servoing

In this paper, a universal state-space approach to uncalibrated model-free visual servoing is proposed and analyzed in detail. After an updated review of the problem, the current research development is transformed and categorized into three important schemes: Broyden–Gauss–Newton method in state space, Broyden recursive least squares method in state space, and Kalman–Bucy filter method, which are compared and investigated comprehensively. A Broyden population partition method in state space, or BP-SS-20-P, is proposed and experimentally verified as the best one at tracking fast and complicated target maneuver among those enlisted in this comparative study. Besides, a robustness theorem of uncalibrated model-free visual servoing algorithms is proposed. The state-space approach of uncalibrated model-free visual servoing provides a unified perspective, a standardized platform and much more flexibility for future work and improvement.

Image-based visual servoing through micropart reflection for the microassembly process

This paper presents an image-based visual servoing algorithm to perform the microassembly process with an uncalibrated manipulator. The proposed algorithm requires only the use of the visual information from a single-vision camera to evaluate the unknown Jacobian matrix. Two methodologies were examined to estimate the Jacobian matrix on-line. Through monitoring the selected feature and the image reflection from the surface, the 3D position between the slot and the micropart can be evaluated successfully for the assembly process. Experimental results confirmed that the Jacobian matrix computed from both methods can evaluate the position with an accuracy of 3.6 µm initially. By using a proportional gain control, the position accuracy can be improved to within 1 µm. Measurement noise during the image acquisition is determined to be one of the root causes of the evaluation accuracy.

A visual-inertial servoing method for tracking object with two landmarks and an inertial measurement unit

This paper introduces a robot visual-inertial tracking algorithm for a robot manipulator intended to track an object using inertial sensors incorporated into the object. To create this algorithm, the inertial Jacobian is first newly defined in order to show the relationship between an angle set velocity vector of the object and the angular velocity vector of the robot tip. Then, the inertial Jacobian is combined with the conventional image Jacobian. Therefore, the proposed algorithm requires only two landmarks with the help of an inertial measurement unit to track a moving object with six degrees of freedom, while at least three landmarks are required in the conventional stereo visual servoing algorithm. Further, the possession of a multi-rate controller allows the integration system to out-perform conventional systems in the tracking of an object’s attitude change. A suggested application of the proposed method is tracking and selection of a container from a shipping vessel that is being affected by large waves. Simulations and experiments were conducted to verify the feasibility of the proposed methodology.

頸部血流計測ロボットシステムの開発―画像情報を用いた超音波プローブの自動位置決め・追従―

One of the important tasks required for sonographers is an accurate positioning of ultrasound probe. Especially, the measurement of the diameter of the blood vessel is affected by the positioning error of the probe. For this purpose, we have been developing a robot system for the medical ultrasound diagnosis, which composed of a 6-DOFs probe manipulator and supporting arm. In this paper, we present an implementation of fully automated positioning algorithm of the ultrasound probe on the carotid artery, by means of image feedback. A probe holding robot firstly detects the patient skin, then scans and detects the longitudinal section of the carotid artery, guide the probe towards the position where the tissue layers of the artery walls are most clearly observed. Furthermore, a new method of the visual servoing for tracking the out-of-plane motion for the medical ultrasound is presented. Automated positioning algorithm is tested in eleven volunteers and the results show that the system successfully completed the task in 91% of the trials. Visual servoing algorithm is also tested and the experimental results show its effectiveness.

A Vision-Based Automatic Landing Method for Fixed-Wing UAVs

In this paper, a vision-based landing system for small-size fixed-wing unmanned aerial vehicles (UAVs) is presented. Since a single GPS without a differential correction typically provide position accuracy of at most a few meters, an airplane equipped with a single GPS only is not guaranteed to land at a designated location with a sufficient accuracy. Therefore, a visual servoing algorithm is proposed to improve the accuracy of landing. In this scheme, the airplane is controlled to fly into the visual marker by directly feeding back the pitch and yaw deviation angles sensed by the forward-looking camera during the terminal landing phase. The visual marker is a monotone hemispherical airbag, which serves as the arresting device while providing a strong and yet passive visual cue for the vision system. The airbag is detected by using color- and moment-based target detection methods. The proposed idea was tested in a series of experiments using a blended wing-body airplane and proven to be viable for landing of small fixed-wing UAVs.

An improved self-calibration approach based on adaptive genetic algorithm for position-based visual servo

An improved self-calibrating algorithm for visual servo based on adaptive genetic algorithm is proposed in this paper. Our approach introduces an extension of Mendonca-Cipolla and G. Chesi’s self-calibration for the position-based visual servo technique which exploits the singular value property of the essential matrix. Specifically, a suitable dynamic online cost function is generated according to the property of the three singular values. The visual servo process is carried out simultaneous to the dynamic self-calibration, and then the cost function is minimized using the adaptive genetic algorithm instead of the gradient descent method in G. Chesi’s approach. Moreover, this method overcomes the limitation that the initial parameters must be selected close to the true value, which is not constant in many cases. It is not necessary to know exactly the camera intrinsic parameters when using our approach, instead, coarse coding bounds of the five parameters are enough for the algorithm, which can be done once and for all off-line. Besides, this algorithm does not require knowledge of the 3D model of the object. Simulation experiments are carried out and the results demonstrate that the proposed approach provides a fast convergence speed and robustness against unpredictable perturbations of camera parameters, and it is an effective and efficient visual servo algorithm.

Model-Free Uncalibrated Visual Servoing Using Recursive Least Squares

In this paper, a model free uncalibrated visual servoing algorithm based on recursive least squares is proposed and discussed in depth. No robot kinetics or dynamics, camera calibration or target model to be tracked would be ever needed. Based on the brief retrospection of RLS filter as well as its theoretical analysis, i.e. Wiener-Hopf condition, the core uncalibrated visual servoing algorithm based on recursive least squares is explored in detail. The experimental results of both static and moving targets applied on a Puma-560 6DOFs industrial robot simulation verifies its performances.

Asymptotic Stability of Uncalibrated Eye-in-Hand Visual Servoing: An Affine Invariance Perspective

In this paper, asymptotic stability of uncalibrated eye-in-hand visual servoing is proved in an affine invariance perspective. After a brief retrospection on the uncalibrated eye-in-hand visual servoing, the affine invariance framework is introduced with discussion in depth. Then the visual servoing algorithm is reconstructed in an affine invariance framework, or more precisely as an affine contravariance algorithm, with its complete asymptotic stability proved, proposed as a convergence theorem. The affine invariance perspective enroots the series algorithm on a more solid and fruitful mathematical background and finally, the paper would discuss several potential research realms of the topic of visual servoing.

Position-Based Visual Servoing in Industrial Multirobot Cells Using a Hybrid Camera Configuration

This paper deals with the problem of position-based visual servoing in a multiarm robotic cell equipped with a hybrid eye-in-hand/eye-to-hand multicamera system. The proposed approach is based on the real-time estimation of the pose of a target object by using the extended Kalman filter. The data provided by all the cameras are selected by a suitable algorithm on the basis of the prediction of the object self-occlusions, as well as of the mutual occlusions caused by the robot links and tools. Only an optimal subset of image features is considered for feature extraction, thus ensuring high estimation accuracy with a computational cost independent of the number of cameras. A salient feature of the paper is the implementation of the proposed approach to the case of a robotic cell composed of two industrial robot manipulators. Two different case studies are presented to test the effectiveness of the hybrid camera configuration and the robustness of the visual servoing algorithm with respect to the occurrence of occlusions

A Dual Imaging System for Flip-Chip Alignment Using Visual Servoing

Electronic components must be precisely aligned on printed circuit (PC) boards in chip packaging, especially in flip-chip assembly. We propose a vision system to provide relative positioning information between the flip-chip and substrate to ensure accurate alignment. The dual imaging system (DIS) we introduce to ensure alignment consists of a zoom lens, beam splitter, mirror, charge-coupled device (CCD) and light-emitting diode (LED) illumination, and simultaneously observes solder bumps on flip-chips and pad patterns on substrates using one camera. Features are extracted from the preprocessed image frame containing the flip-chip and substrate to obtain positioning and orientation errors between the chip and substrate. Based on measured error, visual servoing determines the instantaneous velocity input of flip-chips at each servoing iteration time and controls relative positioning and orientation precisely on-line. Fine servoing on a 3-axis stages uses dual imaging for visual alignment. Experiments under different parameter conditions to evaluate dual imaging system and visual servoing algorithm performance confirmed the feasibility of our proposal.

Remote programming over multiple heterogeneous robots: a case study on distributed multirobot architecture

PurposeThe authors of this paper aim to describe the design of distributed architectures for the remote control of multirobot systems. A very good example of remote robot programming in order to validate these architectures is in fact the remote visual servoing control. It uses sequences of camera inputs in order to bring the robots to the desired position, in an iterative way. In fact, in this paper, we enabled the students and scientists in our university to experiment with their remote visual servoing algorithms through a remote real environment instead of using simulation tools.Design/methodology/approachSince 2001, the authors have been using the UJI‐TeleLab as a tool to allow students and scientists to program remotely several vision‐based network robots. During this period it has been learnt that multithread remote programming combined with a distributed multirobot architecture, as well as advanced multimedia user interfaces, are very convenient, flexible and profitable for the design of a Tele‐Laboratory. The distributed system architecture permits any external algorithm to have access to almost every feature of several network robots.FindingsPresents the multirobot system architecture and its performance by programming two closed loop experiments using the Internet as communication media between the user algorithm and the remote robots (i.e. remote visual servoing). They show which conditions of Internet latencies and bandwidth are appropriate for the visual servoing loop. We must take into account that the real images are taken from the remote robot scenario and the experiment algorithm is executed from the client side at the user place. Moreover, the distributed multirobot architecture is validated by performing a multirobot programming example using two manipulators and a mobile robot.Research limitations/implicationsFuture work will pursue the development of more sophisticated visual servoing loops using external cameras, pan/tilt and also stereo cameras. Indeed, the stereo cameras control introduces an interesting difficulty related to their synchronization during the loop, which introduces the need to implement Real Time Streaming Protocol (RTSP) based camera monitoring. By using camera servers that support RTSP (e.g. Helix Producer, etc.) it means sending the differences between the frames instead of sending the whole frame information for every iteration.Practical implicationsThe distributed multirobot architecture has been validated since 2003 within the education and training scenario. Students and researchers are able to use the system as a tool to rapidly implement complex algorithms in a simple manner. The distributed multirobot architecture is being applied as well within the industrial robotics area in order to program remotely two synchonized robots.Originality/valueThis paper is an original contribution to the network robots field, since it presents a generic architecture to program remotelly a set of heterogeneous robots. The concept of network robot recently came up at the Workshop “network robots” within the IEEE ICRA 2005 World Congress.

Epipole-based visual servoing for mobile robots

This paper proposes a visual servoing algorithm for mobile robot navigation based on the epipolar geometry retrieved by object profiles. The main motivation for this approach is that for unstructured scenes the task of solving correspondences is definitely a harder task than contour detection. An unstructured three-dimensional (3-D) scene consists mainly of objects whose most important 2-D features are their apparent contours in the image plane. Apparent contours are used to estimate the positions of the epipoles and some special symmetry conditions whereby the visual servoing is able to steer the mobile robot to the desired position.