Real-time Visual Servoing Research Articles

Visual Servo Kinematic Control of Delta Robot using YOLOv5 Algorithm

Industrial delta robots with motion-and vision-base controllers, providing high-precision and fast/flexible motion, normally come with high cost and complex software. Low-cost delta robots with vision solution could be beneficial to vast industries to increase productivity and to reduce labor cost. A delta robot using a low-cost motion controller and an open-source vision system is developed to accomplish real-time visual servoing with high motion accuracy. In the low-cost motion controller, three parallel links’ upper-arm angles, computed from inverse kinematics for a given desired target position by a high-level computer, are used as reference position commands for three AC-motor drives. A low-level Arduino microcontroller is employed to convert these links’ angles to high-frequency pulses and on-off signals for synchronously controlling three motor angles and direction. Experimental results of a point-to-point motion tracking exhibit high-precision repeatability. Synchronous pulse generation from Arduino microcontroller and structural misalignments of parallel links are major challenges for achieving high motion accuracy. For the vision-based system, the YOLOv5 algorithm is implemented along with a Python GUI Application. Then, the visual-servo performance is evaluated on localization accuracy and recognition rate of 3-color objects. However, a partial object occlusion can reduce the visual classification rate. A sorting task of 4-category medicine boxes demonstrate a high-speed pick-and-place operation using the low-cost visual-servo system of this delta robot. Therefore, integration of low-cost visual servoing with this delta robot can revolutionize various industries, like automobile, pharmaceutical, and food sectors, in separating, sorting and packing applications.

The Prediction of Badminton Flight Trajectory Based on an Intelligent Compensator

A real-time visual servo badminton trajectory tracking system (VSBTTS) is proposed in this study. VSBTTS is composed of two parts: the image unit and the tracking operation unit. The primary function of the image unit is to segment the moving shuttlecock from a continuous image taken by the camera and detect the moving object. The tracking operation unit is used to track parts of the shuttlecock found in the image unit. Then, the flight trajectory of the shuttlecock is estimated; however, it is difficult for tracking operation units to avoid estimation errors, preventing them from achieving full accuracy. In this study, a fuzzy compensation controller is used to compensate for the system estimation errors of the flight trajectory. Using the fuzzy compensator, the trajectory of the shuttlecock is accurately estimated. Experimental results demonstrate the usefulness of the proposed shuttlecock trajectory tracking system.

A Cascaded and Adaptive Visual Predictive Control Approach for Real-Time Dynamic Visual Servoing

In the past two decades, Unmanned Aerial Vehicles (UAVs) have gained attention in applications such as industrial inspection, search and rescue, mapping, and environment monitoring. However, the autonomous navigation capability of UAVs is aggravated in GPS-deprived areas such as indoors. As a result, vision-based control and guidance methods are sought. In this paper, a vision-based target-tracking problem is formulated in the form of a cascaded adaptive nonlinear Model Predictive Control (MPC) strategy. The proposed algorithm takes the kinematics/dynamics of the system, as well as physical and image constraints into consideration. An Extended Kalman Filter (EKF) is designed to estimate uncertain and/or time-varying parameters of the model. The control space is first divided into low and high levels, and then, they are parameterised via orthonormal basis network functions, which makes the optimisation- based control scheme computationally less expensive, therefore suitable for real-time implementation. A 2-DoF model helicopter, with a coupled nonlinear pitch/yaw dynamics, equipped with a front-looking monocular camera, was utilised for hypothesis testing and evaluation via experiments. Simulated and experimental results show that the proposed method allows the model helicopter to servo toward the target efficiently in real-time while taking kinematic and dynamic constraints into account. The simulation and experimental results are in good agreement and promising.

Automated real-time 3D visual servoing control of single cell surgery with application to microinjection processes

Automated real-time 3D visual servoing control of single cell surgery with application to microinjection processes

Automated real-time 3D visual servoing control of single cell surgery with application to microinjection processes

Automated real-time 3D visual servoing control of single cell surgery with application to microinjection processes

Application of a high-throughput microcrystal delivery system to serial femtosecond crystallography.

Microcrystal delivery methods are pivotal in the use of serial femtosecond crystallography (SFX) to resolve the macromolecular structures of proteins. Here, the development of a novel technique and instruments for efficiently delivering microcrystals for SFX are presented. The new method, which relies on a one-dimensional fixed-target system that includes a microcrystal container, consumes an extremely low amount of sample compared with conventional two-dimensional fixed-target techniques at ambient temperature. This novel system can deliver soluble microcrystals without highly viscous carrier media and, moreover, can be used as a microcrystal growth device for SFX. Diffraction data collection utilizing this advanced technique along with a real-time visual servo scan system has been successfully demonstrated for the structure determination of proteinase K microcrystals at 1.85 Å resolution.

Real-time Visual Servoing Control of an UAV Having Two-Cameras

Real-time Visual Servoing Control of an UAV Having Two-Cameras

Skeleton-Based Visual Servoing in Unknown Environments

The generalized Voronoi diagram (GVD) is a powerful environment representation that defines a set of paths at maximal distance from the obstacles. Many works implicitly use this property to define safe navigation strategies for a mobile robot, but, in practice, only a few explicitly extracts in real time the GVD from the robot perception. In this paper, a real-time skeleton-based visual servoing approach is proposed for safe autonomous navigation of mobile robots in unknown structured environments. The control is based on an approximation of the local GVD using the delta medial axis, a fast and robust skeletonization algorithm. The latter produces a filtered skeleton of the freespace surrounding the robot using a pruning parameter that takes into account the robot size. A unified virtual perception frame is also introduced to project the navigable freespace obtained by possibly multiple sensors and observations. After freespace skeletonization and GVD approximation, a visual servoing control is designed and studied to ensure autonomous robot navigation along the extracted GVD branches. The approach is first tested using a simulated environment. Then, an experimental validation is carried out on a real differential wheelchair allowing the system to navigate safely and autonomously without any human intervention. Static and dynamic environment scenarios are studied. The method is immediately generalizable to any differential drive wheeled robot.

Robotic assembly of smartphone back shells with eye-in-hand visual servoing

Abstract Research concerning smartphone assembly has attracted increasing attention from the manufacturing industry. This paper presents an automatic back shell assembly system using real-time visual servoing in an eye-in-hand configuration. Look-then-move and look-and-move control approaches are sequentially proposed to handle distal and proximal assembly tasks, respectively. During distal assembly, binocular vision is employed to assist a robotic manipulator to locate and reconstruct the positions and orientations of a cellular phone and its back shell in the work space. After lifting the back shell up and then reaching the top of the cellular phone, there is no binocular field of view. A seemingly novel way of 3-D reconstruction by two uniocular fields of view of the two cameras is proposed. With the pair of monocular vision sensors, two monocular visual servo control laws are employed sequentially to perform the proximal assembly task autonomously. Based on the experimental results, the robotic manipulator can be driven to hold the back shell by a vacuum absorption device and position it precisely onto the smartphone to accomplish the automatic assembly task even if the vision system is only roughly calibrated. The proposed control approach for distal and proximal assembly appears to be efficient, flexible, and reliable for potential applications in industrial manufacturing.

High-Resolution Real-Time Visual Inspection System for Micro Assembly

For precise assembly of miniature parts, the precise inspection for parts’ posture and real-time servo control for assembly greatly depend on the performance of visual inspection system. This paper proposed a high-resolution real-time visual inspection system of micro assembly. The CMOS image sensor and high-speed digital signal processing chip were chosen to design the image acquisition module, image processing module and image display module. High-accuracy display on the common display device was implemented with the video encoding chip and FPGA. The test results showed that the processing speed with preprocessing could reach 3.5 frames per second with 5 mega-pixel resolution, and the display accuracy after threshold processing had little loss. Micro parts assembly experiment and high accuracy Peg-in-Hole assembly experiment are done to test the performance of the proposed visual inspection system. This visual inspection system can be used for high-resolution real-time micro assembly and other real-time visual servo control.

Development of a calibrating algorithm for Delta Robot’s visual positioning based on artificial neural network

Delta robot with vision system can automatically control the end-actuator to accurately grasp moving objects on the conveyor belt. Establishment of the mapping relationship between the image feature space and the robot working space form a closed-loop chain for transformational link between the robot coordinate, camera coordinate and conveyor belt coordinate. The vision system calibration is a basic problem of robot vision research and implementation. The artificial neural networks (ANN) which has learning ability, adaptive ability and nonlinear function approximation ability can establish the nonlinear relationship between space points and pixel points to complete accurate calibration of the vision system. The convergence speed of calibration algorithm affects the real-time visual servo system. The calibration precision, generalization ability and calibration space of algorithm influence the robot grasping accuracy. Therefore, a new calibration technique for delta robot’s vision system was presented in this paper. The algorithm combines ANN with Faugeras vision system calibration technology. The setting of the initial value, network structure and the choice of the activation function is based on the model of Faugeras vision system calibration algorithm, which makes the actual output of the network closer to the target output. Experiments proved that this algorithm has higher calibration accuracy and generalization ability compared with the conventional calibration algorithm, as well as faster convergence speed compared with the conventional artificial neural network structure in the case of high calibration accuracy.

A TSR Visual Servoing System Based on a Novel Dynamic Template Matching Method.

The so-called Tethered Space Robot (TSR) is a novel active space debris removal system. To solve its problem of non-cooperative target recognition during short-distance rendezvous events, this paper presents a framework for a real-time visual servoing system using non-calibrated monocular-CMOS (Complementary Metal Oxide Semiconductor). When a small template is used for matching with a large scene, it always leads to mismatches, so a novel template matching algorithm to solve the problem is presented. Firstly, the novel matching algorithm uses a hollow annulus structure according to a FAST (Features from Accelerated Segment) algorithm and makes the method be rotation-invariant. Furthermore, the accumulative deviation can be decreased by the hollow structure. The matching function is composed of grey and gradient differences between template and object image, which help it reduce the effects of illumination and noises. Then, a dynamic template update strategy is designed to avoid tracking failures brought about by wrong matching or occlusion. Finally, the system synthesizes the least square integrated predictor, realizing tracking online in complex circumstances. The results of ground experiments show that the proposed algorithm can decrease the need for sophisticated computation and improves matching accuracy.

A Generalized Visual Aid System for Teleoperation Applied to Satellite Servicing

This paper presents the latest results of a newly developed visual aid system for direct teleoperation. This method is extended to visual control to make an efficient teleoperation system by combining direct teleoperation and automatic control. On the one hand, an operator can conduct direct teleoperation with 3D graphic prediction simulation established by the VR technique. In order to remove inconsistencies between the virtual and real environments, a practical model-matching method is investigated. On the other hand, to realize real-time visual servoing control, a particular object recognition and pose estimation algorithm based on polygonal approximation is investigated to ensure a low computational cost for image processing. To avoid undesired forces involved in contact operation, 3D visual servoing incorporating a compliant control based on impedance control is developed. Finally, in a representative laboratory environment, a typical satellite servicing experiment is carried out based on this combined system. Experimental results demonstrate the feasibility and the effectiveness of the proposed method.

Distributed Computation Particle Filters on GPU Architectures for Real-Time Control Applications

We present the design, analysis, and real-time implementation of a distributed computation particle filter on a graphic processing unit (GPU) architecture that is especially suited for fast real-time control applications. The proposed filter architecture is composed of a number of local subfilters that can share limited information among each other via an arbitrarily chosen abstract connected communication topology. We develop a detailed implementation procedure for GPU architectures focusing on distributed resampling as a crucial step in our approach, and describe alternative methods in the literature. We analyze the role of the most important parameters such as the number of exchanged particles and the effect of the particle exchange topology. The significant speedup and increase in performance obtained with our framework with respect to both available GPU solutions and standard sequential CPU methods enable particle filter implementations in fast real-time feedback control systems. This is illustrated via experimental and simulation results using a real-time visual servoing problem of a robotic arm capable of running in closed loop with an update rate of 100 Hz, while performing particle filter calculations that involve over one million particles.

A fractional order control strategy for visual servoing systems

In this paper, a control strategy based on fractional calculus for visual servoing systems is proposed. The image-based control strategy is designed using a point features based fractional-order PI controller. A real-time visual servoing system, composed of a manipulator robot with 6 degrees of freedom (d.o.f.) with an eye-in-hand camera, is used for performance evaluation of the proposed control strategy. The image acquisition and processing, together with the computing of the image-based control law are implemented in MATLAB. Using planar static objects, real-time experiments are conducted and the results reveal that the image-based fractional-order PI controller outperforms the conventional image-based integer-order PI controller.

Visual Servoing Considering Sensing Dynamics and Robot Dynamics

For many desirable applications of vision guided industrial robots, real-time visual servoing is necessary but also challenging. Difficulty comes from the limited sampling rate and response time of typical machine vision systems equipped on industrial robots. These factors are addressed as the dynamics of visual sensing. In addition, robot dynamics should also be fully considered when designing the control law. Considering these aspects, this paper presents a control scheme of visual servoing. A dual-rate adaptive tracking filter is presented to compensate the visual sensing dynamics. Based on the compensated vision feedback, the techniques of multi-surface sliding control and dynamic surface control are used to formulate a two-layer control law for target tracking. System kinematics and dynamics are decoupled and dealt with by the two layers of the control law respectively. The proposed method is validated through experiments on a SCARA robot.

Visual Servoing Tracking Control of a Ball and Plate System: Design, Implementation and Experimental Validation

This paper presents the design, implementation and validation of real-time visual servoing tracking control for a ball and plate system. The position of the ball is measured with a machine vision system. The image processing algorithms of the machine vision system are pipelined and implemented on a field programmable gate array (FPGA) device to meet real-time constraints. A detailed dynamic model of the system is derived for the simulation study. By neglecting the high-order coupling terms, the ball and plate system model is simplified into two decoupled ball and beam systems, and an approximate input-output feedback linearization approach is then used to design the controller for trajectory tracking. The designed control law is implemented on a digital signal processor (DSP). The validity of the performance of the developed control system is investigated through simulation and experimental studies. Experimental results show that the designed system functions well with reasonable agreement with simulations.

Design and implementation of robust visual servoing control of an inverted pendulum with an FPGA-based image co-processor

This paper presents the design and implementation of robust real-time visual servoing control with an FPGA-based image co-processor for a rotary inverted pendulum. The position of the pendulum is measured with a machine vision system. The pendulum used in the proposed system is much shorter than those used in published vision-based pendulum control system studies, which makes the system more difficult to control. The image processing algorithms of the machine vision system are pipelined and implemented on a field programmable gate array (FPGA) device to meet real-time constraints. To enhance robustness to model uncertainty and to attenuate disturbance and sensor noise, the design of the stabilizing controller is formulated as a problem of the mixed H 2/ H ∞ control, which is then solved using the linear matrix inequality (LMI) approach. The designed control law is implemented on a digital signal processor (DSP). The effectiveness of the controller and the FPGA-based image co-processor is verified through simulation and experimental studies. The experimental results show that the designed system can robustly control an inverted pendulum in real-time.

A real-time fuzzy reasoning based control system for catching a moving goldfish

Designing a real-time visual tracking system to catch a goldfish is a complex task because of the large amount of streaming video data that must be transmitted and processed immediately when tracking the goldfish. Usually, building such visual servoing systems requires the application of high-cost specialized hardware and the development of complicated visual control software. In this paper, a novel low-cost, real-time visual servo control system is presented. The system uses stereo vision consisting of two calibrated cameras to acquire images of the goldfish, and applies the continuously adaptive mean shift (CAMSHIFT) vision-tracking algorithm to provide feedback of a fish’s real-time position at a high frame rate. It then employs a 5-axis robot manipulator controlled by a fuzzy reasoning system to catch the fish. This visual tracking and servoing system is less sensitive to lighting influences and thus performs more efficiently. Experiments with the proposed method yielded very good results, as the system’s real-time 3D vision successfully tracked two fish and guided the manipulator, which has a net attached to its end effector, to catch one of them.

A new fuzzy visual servoing with application to robot manipulator

Many stereo vision algorithms require that the images which are very similar in appearance, and the distance between the two cameras should be small enough. Normal visual servoing of robot manipulator needs both image information and joint velocities. In this paper we make two modifications to overcome these problems: 1) a new simple stereo vision model is proposed; 2) fuzzy controller based on only visual information is implemented to a robot manipulator. We successfully apply our new algorithms to real-time visual servoing of an industrial robot. The stereo model proposed in this paper can also be extended to other applications such as, mobile robots, 3D reconstruction, etc.