Visual Servo Control System Research Articles

High-Accuracy Positioning of an Industrial Robot Using Image/PSD-Based Hybrid Servo Control

This article presents a new and effective multi-sensor-based control strategy for high-accuracy/precision and high-efficiency automatic robot positioning in the field of industrial robot calibration. The strategy combines both coarsely visual servo and fine position-sensitive detector (PSD) servo control methods. In a large field of view, an image-based visual servo control system is employed to roughly guide the laser spot, which is from a single laser pointer attached to the end-effector of a robot to project to the segmented PSDs with a resolution of better than 0.1 µm. Once the laser spot is projected onto the active area of the PSD surface, the control will automatically be switched to the PSD feedback and servoing for fine positioning. An image/PSD-based servoing system has been presented. Based on the system, the controller design and switch logic of the hybrid servoing are given. The experimental results conducted on an ABB industrial robot IRB1600 verified the effectiveness of the developed visual/PSD hybrid servo controllers, as well as demonstrated that the high accuracy of 30 µm of robot positioning can be approached. The development of the hybrid control method and system has played a major role in achieving high-performance automatic robot calibration.

A real-time fuzzy reasoning based control system for catching a moving goldfish

Designing a real-time visual tracking system to catch a goldfish is a complex task because of the large amount of streaming video data that must be transmitted and processed immediately when tracking the goldfish. Usually, building such visual servoing systems requires the application of high-cost specialized hardware and the development of complicated visual control software. In this paper, a novel low-cost, real-time visual servo control system is presented. The system uses stereo vision consisting of two calibrated cameras to acquire images of the goldfish, and applies the continuously adaptive mean shift (CAMSHIFT) vision-tracking algorithm to provide feedback of a fish’s real-time position at a high frame rate. It then employs a 5-axis robot manipulator controlled by a fuzzy reasoning system to catch the fish. This visual tracking and servoing system is less sensitive to lighting influences and thus performs more efficiently. Experiments with the proposed method yielded very good results, as the system’s real-time 3D vision successfully tracked two fish and guided the manipulator, which has a net attached to its end effector, to catch one of them.

Quaternion‐based visual servo control in the presence of camera calibration error

AbstractVisual servo control systems use information from images along with knowledge of the optic parameters (i.e. camera calibration) to position the camera relative to some viewed object. If there are inaccuracies in the camera calibration, then performance degradation and potentially unpredictable response from the visual servo control system may occur. Motivated by the desire to incorporate robustness to the camera calibration, different control methods have been developed. Previous adaptive/robust controllers (especially for six degree‐of‐freedom camera motion) rely heavily on properties of the rotation parameterization to formulate state estimates and a measurable closed‐loop error system. All of these results are based on the singular axis–angle parameterization. Motivated by the desire to express the rotation by a non‐singular parameterization, efforts in this paper address the question: Can state estimates and a measurable closed‐loop error system be crafted in terms of the quaternion parameterization when the camera calibration parameters are unknown? To answer this question, a contribution of this paper is the development of a robust controller and closed‐loop error system based on a new quaternion‐based estimate of the rotation error. A Lyapunov‐based analysis is provided which indicates that the controller yields asymptotic regulation of the rotation and translation error signals given a sufficient approximate of the camera calibration parameters. Simulation results are provided that illustrate the performance of the controller for a range of calibration uncertainty. Copyright © 2009 John Wiley & Sons, Ltd.

Visual Servo Control Achieving Nanometer Resolution in $X$–$Y$–$Z$

This paper presents a three-axis vision motion sensor and its applications to visual servo control. The vision sensor is integrated with a three-axis piezo stage to form a visual servo control system that achieves nanometer resolution in all three <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">x</i> - <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">y</i> - <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">z</i> motion axes. Motion measurement is achieved using a single interferometer-equipped optical microscope. A real-time image-processing algorithm that processes interference fringe patterns and that achieves nanometer out-of-plane resolution is presented. Furthermore, a feedback-control scheme is introduced to control the sensor plane using an Objective-Z-Positioner to enable automatic tracking of moving objects. It expands the out-of-plane measurement range of the vision sensor beyond its inherent depth of field of several micrometers to 100 mum and beyond. An integrated visual servo system is implemented and experimental results are shown.

Volumetric Calibration of Stereo Camera in Visual Servo Based Robot Control

The primary objective of the paper is to propose a calibration method for a stereo camera used in a visual servo control for a robot manipulator. Specifically, projection matrix between the stereo camera and world coordinates is established using few calibration points and solved using the single value decomposition technique. Then calibration accuracy is compared for a randomized and designed set of points, and economical number of calibration points is recommended. Additionally, the non-linear lens distortion is modeled and corrected to improve the accuracy. In addition, this research focuses on the development and implementation of a fully automated visual servo control system using a stereo camera that is calibrated by proposed method.

Vision-Based Robotic Motion Control for Non-autonomous Environment

A visual servo control system with SOPC structure is implemented on a retrofitted Mitsubishi RV-M2 robotic system. The hardware circuit has the functions of quadrature encoder decoding, limit switch detecting, pulse width modulation (PWM) generating and CMOS image signal capturing. The software embedded in Nios II micro processor has the functions of using UART to communicate with PC, robotic inverse kinematics calculation, robotic motion control schemes, digital image processing algorithm and gobang game AI algorithm. A CMOS color image sensor is used to catch the environment time-varying change for robotic vision-based servo control. A gobang game is planned to investigate the implementation problem of the visual servo robotic motion control in non-autonomous environment. A model-free intelligent self-organizing fuzzy control (SOFC) strategy is employed to design the robotic joint controller. A vision based trajectory planning algorithm is designed to calculate the desired angular positions or trajectory of each robotic joint. The experimental results show that this visual servo control robot has reliable control actions.

A Real-time Visual Servo Architecture for a Golden Fish Catching System

Designing a real-time visual tracking system to catch golden fish is a complex task due to difficulties involved in the precise determination of a fish's relative 3D positions in the water tank. In this paper we present a new visual servo control system using stereo vision to acquire the 3D positions of the golden fish. The continuously adaptive mean shift (CAMSHIFT) algorithm and optical flow vision tracking methods are applied to provide, at a high frame-rate, the real-time feedback of the position and motion trajectory of the fish being tracked. A visual servo architecture for controlling two low-cost, well calibrated cameras for tracking the golden fish and a 5-axis robot manipulator for catching a specified golden fish are constructed using standard off-the-shelf hardware and open source software. The designed visual tracking and servo system is less sensitive to lighting influences and thus performs more efficiently. Experiments of the proposed method yielded very good results as the real-time 3D vision was able to successfully track two fish and guide the manipulator to catch one of them with a net attached to the manipulator's end-effector.

Image Moment Based Visual Servo Control and Its Application

The conventional image based visual servo control would lead to image singularities that might cause control instabilities, and this problem can be solved by introducing the image moments as features in visual servo control.Where the Jacobian matrix is full rank and upper triangular,thus it has the maximal decoupled structure and simplifies the controller design.Then,PD based visual controller is presented to improve the performance of the visual servo control system instead of the commonly used P controller. Genetic algorithm based PD parameters tuning method is applied to get the optimal parameters.The method proposed is used to align the micro peg and hole,and the experiment results show that the object can reach its desired position faster and more smoothly with PD visual controller.

Image-based fuzzy control system

A novel image-based fuzzy control (IBFC) scheme is developed to imitate the way humans use visual information to control objects. A CCD camera gathers images of the controlled plant, and a simple algorithm analyses the images. The proposed image analysis algorithm utilises image information more intuitively than visual servo control systems. The difference between a reference image and the current image is numerically expressed and directly used by a fuzzy control system using a human-like control law. To investigate the effectiveness of the proposed IBFC scheme, it is applied to control an inverted pendulum system. Simulation results show that the IBFC system can achieve favourable tracking performance without prior knowledge of the controlled plant.

A Research of Image-Based Visual Control in Microassembly under a Microscope

A new visual servo control system under microscope was introduced based on the research of image-based computer visual control system, including the installation and calibration of camera, deduction of the image Jacobian and the algorithm of controlling and so on. The experiments of motion controlling and positioning showed that one pixel about 2.97μm can be reached easily in single axes under 5× eyepiece and under the same condition a 400μm micro shaft was assembled into a micro-gear with 415μm inner bore. Knowing from the control algorithm, the accuracy of the system is independent on errors in the robot kinematics or the camera calibration. The research results indicate that this method is a full closed-loop visual servo control system with a high reliability and simple control algorithm and it can be used in microassembly system.

An ultraprecision six-axis visual servo-control system

This paper presents the development of a novel active visual measurement technique, laterally sampled white-light interferometry (L-SWLI), which is capable of real-time visual tracking of six-degree-of-freedom (6-DOF) rigid body motion with near-nanometer precision. The visual tracking system is integrated with a 6-DOF motion stage to realize an ultraprecision six-axis visual servo-control system. Contrary to conventional interferometric techniques, L-SWLI obtains the complete pose of the target object from a single image frame, therefore allowing real-time tracking. Six-DOF motions are obtained from measuring the fringe pattern on multiple surfaces of the object or from a single surface with additional information gained from conventional image-processing techniques. The feasibility of the visually servoed motion scheme was demonstrated on a micro cantilever. The cantilever was maneuvered in a three-dimensional space with near-nanometer motion resolution in all three translational axes.

Performance Tests for Visual Servo Control Systems, with Application to Partitioned Approaches to Visual Servo Control

Visual servoing has been a viable method of robot manipulator control for more than a decade. Initial developments involved position-based visual servoing (PBVS), in which the control signal exists in Cartesian space. The younger method, image-based visual servoing (IBVS), has seen considerable development in recent years. PBVS and IBVS offer tradeoffs in performance, and neither can solve all tasks that may confront a robot. In response to these issues, several methods have been devised that partition the control scheme, allowing some motions to be performed in the manner of a PBVS system, while the remaining motions are performed using an IBVS approach. To date, there has been little research that explores the relative strengths and weaknesses of these methods. In this paper we present such an evaluation. We have chosen three recent visual servo approaches for evaluation in addition to the traditional PBVS and IBVS approaches. We posit a set of performance metrics that mea sure quantitatively the performance of a visual servo controller for a specific task. We then evaluate each of the candidate visual servo methods for four canonical tasks with simulations and with experiments in a robotic work cell.

Camera modelling for visual servo control applications

When designing a visual servo system, it is important to have a complete and accurate model of the imaging process. Unmodelled imaging dynamics may play an important role in the stability and performance of such systems. In this paper, we present a detailed camera model which can be used in the design and analysis of visual servo systems. Using the free-standing acrobot as a testbed, we analyze the effects of unmodelled imaging dynamics on visual servo control systems. We show that certain camera parameters strongly influence the performance of this system, and that accurate modeling is necessary for proper selection of imaging hardware.

Global performance evaluation of image features for visual servo control

The performance of a visual servo control system depends on the set of image features used in the control loop. Although some local performance measures have been used for evaluating the image features, their usage in the process of feature selection requires on-line computation that is difficult to realize in real-time, especially with a large number of candidate features. In this article, we introduce a global measure for evaluating the performance of image features for visual servo tasks. This measure can be computed off-line and it takes into account several desirable properties of the image features, including minimization of singularities in the image Jacobian, linearity of feature variation, and maximization of feature resolution. For a given kinematic and imaging model of a robot/camera setup, the measure can be used for a variety of visual servo tasks. A numerical approximation scheme is presented along with several computed examples to illustrate the utility of this measure.

A tutorial on visual servo control

This article provides a tutorial introduction to visual servo control of robotic manipulators. Since the topic spans many disciplines our goal is limited to providing a basic conceptual framework. We begin by reviewing the prerequisite topics from robotics and computer vision, including a brief review of coordinate transformations, velocity representation, and a description of the geometric aspects of the image formation process. We then present a taxonomy of visual servo control systems. The two major classes of systems, position-based and image-based systems, are then discussed in detail. Since any visual servo system must be capable of tracking image features in a sequence of images, we also include an overview of feature-based and correlation-based methods for tracking. We conclude the tutorial with a number of observations on the current directions of the research field of visual servo control.

Adaptive Control of Robots Using Visual Feedback

Vision based robot control systems provide capabilities for modifying positions to accommodate uncertainty in the task environment. Vision based robot control research has focused on vision processing issues, while control system design has been limited to ad hoc strategies In this paper, we formalize an analytical approach to the design of dynamic robot visual servo control systems An image-based structure represents a new approach to visual servo control which uses local image features as feedback control signals image-based control presents formidable engineering problems for controller design, including complex dynamics, kinematics, and feature transformations, as well as unknown parameters, and measurement noise and delays A model reference adaptive Controller (MRAC) is designed to satisfy these requirements Simulation studies are described which indicate the feasibility and expected performance of this approach for systems with 1, 2. and 3 degrees of freedom.