Visual Servoing Architecture Research Articles

Visual Servoing Architecture of Mobile Manipulators for Precise Industrial Operations on Moving Objects

Although the use of articulated robots and AGVs is common in many industrial sectors such as automotive or aeronautics, the use of mobile manipulators is not widespread nowadays. Even so, the majority of applications separate the navigation and manipulation tasks, avoiding simultaneous movements of the platform and arm. The capability to use mobile manipulators to perform operations on moving objects would open the door to new applications such as the riveting or screwing of parts transported by conveyor belts or AGVs. This paper presents a novel position-based visual servoing (PBVS) architecture for mobile manipulators for precise industrial operations on moving parts. The proposed architecture includes a state machine to guide the process through the different phases of the task to ensure its correct execution. The approach has been validated in an industrial environment for screw-fastening operations, obtaining promising results and metrics.

A Short Review of Deep Learning Methods in Visual Servoing Systems

Abstract This survey explores the evolution and applications of Visual Servoing Systems in robotics, emphasizing the transition from traditional image processing techniques to the incorporation of neural networks for feature extraction and control. Robotic systems, integral to manufacturing, surveillance, and healthcare, increasingly rely on Visual Servoing for enhanced interaction within various work environments. Initially focused on auxiliary sensors like visual sensors for robustness and accuracy, recent advances have seen a shift towards integrating deep learning methods for direct control and feature extraction. The survey covers the differences that emerge from classical Visual Servoing architectures to novel methods involving Deep Learning, highlighting their respective advantages and limitations regarding stability, precision, and real-time applicability. Innovative approaches, such as Direct Visual Servoing and the use of siamese networks for camera position estimation, demonstrate significant progress in overcoming the challenges of traditional Visual Servoing. Through detailed examination of leading research, the survey highlights the potential of neural networks to revolutionize this domain by enhancing feature extraction, reducing reliance on precise calibration, and improving control laws for complex robotic tasks.

Dynamic visual servoing with Kalman filter-based depth and velocity estimator

Camera calibration error, vision latency, nonlinear dynamics, and so on present a major challenge for designing the control scheme for a visual servoing system. Although many approaches on visual servoing have been proposed, surprisingly, only a few of them have taken into account system dynamics in the control design of a visual servoing system. In addition, the depth information of feature points is essential in the image-based visual servoing architecture. As a result, to cope with the aforementioned problems, this article proposes a Kalman filter-based depth and velocity estimator and a modified image-based dynamic visual servoing architecture that takes into consideration the system dynamics in its control design. In particular, the Kalman filter is exploited to deal with the problems caused by vision latency and image noise so as to facilitate the estimation of the joint velocity of the robot using image information only. Moreover, in the modified image-based dynamic visual servoing architecture, the computed torque control scheme is used to compensate for system dynamics and the Kalman filter is used to provide accurate depth information of the feature points. Results of visual servoing experiments conducted on a two-degree of freedom planar robot verify the effectiveness of the proposed approach.

A review on vision-based control of flexible manipulators

In this review, recent developments in the field of flexible robot arm control using visual servoing are reviewed. In comparison to rigid robots, the end-effector position of flexible links cannot be obtained precisely enough with respect to position control using kinematic information and joint variables. To solve the task here the use of a vision sensor (camera) system, visual servoing is proposed to realize the task of control flexible manipulators with improved quality requirements. The paper is organized as follows: the visual servoing architectures will be reviewed for rigid robots first. The advantages, disadvantages, and comparisons between different approaches of visual servoing are carried out. The using of visual servoing to control flexible robot is addressed next. Open problems such as state variables estimation as well as the combination of different sensor properties as well as some application-oriented points related to flexible robot are discussed in detail.

Mechatronics Systems Supported by Vision Techniques

In the paper an application of vision methods and algorithms in various domains that contribute to mechatronics is presented. Regarding mechatronics devices and machines as robots, a vision system employed for a testing station simulating an industrial assembly line is discussed. Some numerical aspects concerning image pre-processing, analysis and geometrical transformations commonly used in robotics were introduced. To accomplish an effective investigation, the developed methodology and algorithms were implemented and verified on an experimental setup composed of two industrial robots and automation devices cooperating with two various vision systems. In the case of underwater robots for tank inspection, image pre-processing and analysis algorithms for the robot's position estimation, an image scale calculation and wall crack detection were investigated. An active vibration control system is treated as a mechatronic device which contains mechanical parts, electronics and software. In this example, a visual servoing architecture based on image features for controlling an active vibration control system was examined. For an effective investigation and synthesis of visual servoing algorithms, a MATLAB/Simulink/dSPACE hardware–software environment was employed. A vision system was used to analyze vibration amplitude of the vibro-isolation mass of the active suspension system and to provide a feedback control signal. The connection of 3D vision techniques with modal analysis was shown. Within the confines of the project a methodology for amplitude of vibration measurement and a software tool for modal analysis realization based on visual data were developed. The 3D measurements and structure of the construction were obtained by application and development of passive 3-D vision techniques. From this area, ‘structure from motion’ techniques were developed. In the experimental research, a mechatronic test stand was designed and manufactured enabling automatic two-axis control of a camera. A frame structure was built, in which a guiding-rail system was mounted enabling straight-line motion of a camera. Additionally, a system realizing rotational motion of a camera was built in. To control the experiment stand, software was created enabling the combination of the hardware-software part of the stand with the software part of a vision system. A tool was developed for the purpose of modal analysis and estimation of the quantities characterizing dynamic properties of the structure based on vision signals. As a conclusion, the presented, implemented and tested vision methods in various hardware-software programming platforms are discussed

Predictive control of nonlinear visual servoing systems using image moments

The visual servoing architectures with an eye-in-hand configuration, composed of a 6-d.o.f. manipulator robot and a video camera, are categorised as high-coupled and nonlinear systems. For this reason, in order to have a stable and convergent behaviour, advanced techniques like predictive control are needed. These types of techniques can solve the potential problems that can appear in classical (image-based visual servoing) while different type of constraints for visual servoing system can be fulfilled easier. Three different visual predictive control architectures based on image moments are proposed. The model for the image moments predictor used in the first architecture is an extension of a local model point features predictor. The second predictive approach uses an image moment-based predictor to describe the future behaviour of the system over a certain camera velocity scenario directly in the image moments space. In order to have a faster convergence and in the same time a desired behaviour of the servoing system, a reference trajectory is employed. This represents the main contribution of the last predictive control technique proposed in this study. A Matlab simulator was developed and all the visual servoing architectures described in the study were implemented. The simulation results showed that all the proposed visual predictive control approaches are stable and convergent when dealing with visual servoing tasks, but the best performances were assigned to the reference trajectory-based image moments architecture.

Predictive control architecture for real-time image moments based servoing of robot manipulators

Abstract This paper presents a real-time architecture for visual servoing of robot manipulators using nonlinear based predictive control. A 6 d.o.f robot manipulator with an eye-in-configuration is used to design the visual servoing architecture and a real-time implementation using Matlab is done. The results of different experiments conducted with two types of image moments based controllers (proportional and predictive with reference trajectory) are presented and discussed.

Developing robust vision modules for microsystems applications

In this work, several robust vision modules are developed and implemented for fully automated micromanipulation. These are autofocusing, object and end-effector detection, real-time tracking and optical system calibration modules. An image based visual servoing architecture and a path planning algorithm are also proposed based on the developed vision modules. Experimental results are provided to assess the performance of the proposed visual servoing approach in positioning and trajectory tracking tasks. Proposed path planning algorithm in conjunction with visual servoing imply successful micromanipulation tasks.

Application of the visual servoing for an active vibration control system

In this paper a visual servoing architecture based on image features for controlling an active vibration control system is presented. The developed vision system was used to calculate amplitude of the vibration of the vibroisolated mass and to provide a visual feedback signal in control loop. To accomplish an effective investigation and synthesis of visual servoing algorithms. hardware-software environment MATLAB/SimulinkldSPACE was proposed. This environment was used for visual servoing architecture implementation and testing based on PID and Smith/Marshall controllers. The method of design is based on rapid control prototyping, where the key is automatic code generation and real time implementation of control algorithms directly from Simulink block.

3D and 2D Visual servoing architectures for a PUMA560 robot

The aim of this work is to present a control system for a six degrees freedom Puma robot using 3D and 2D Visual Servoing. A new Robot puma model is presented. In this system the camera is placed on the robot's end-effector and the goal is to control the robot pose to follow a target or a trajectory. A control law based on features extracted from camera images in 2D or based in the target model in 3D is used. Simulation results show that the strategy works well and that 3D visual servoing is more efficient to large displacements but more sensible to the noise