Position-based Visual Servo Control Research Articles

Hybrid vision/strain-based control strategy for a parallel manipulator with flexible links

Parallel manipulators may present higher speed/acceleration ratios and energy efficiency when compared with serial manipulators. In addition, reducing their components’ inertia can further improve performance. However, this design alternative might yield vibrations requiring novel joint and task space control strategies. While the former involves precise models, the latter might require adequate computation vision schemes. These requirements impose critical challenges. This manuscript proposes a hybrid control strategy and experimentally investigates using a 3RRR prototype with flexible links. This hybrid strategy comprises two control loops: position-based visual servo and strain-based feedback control schemes. The strain-based loop uses a sliding mode control and the time derivative of the signals obtained by strain gauges installed in the flexible links. Compared with the position-based visual servo scheme, the hybrid strategy can considerably attenuate the vibrations since the approach reduced the overshoot response by 36% and the maximum value of the Euclidean error on position by 38%, on average.

Position-Based Visual Servo Control of Dual Robotic Arms With Unknown Kinematic Models: A Cerebellum- Inspired Approach

The inverse kinematics problem of robotic arms can be solved effectively based on the kinematic model. However, the control of robotic arms with unknown kinematic models remains challenging. This article proposes a position-based visual servo (PBVS) control system for dual robotic arms with unknown kinematic models. First, a unified PBVS control method based on dual gradient neural dynamic (DGND) models is developed to achieve the position and orientation control of a single robotic arm with unknown kinematic model. Next, a cerebellum-inspired DGND (CIDGND) control system is devised to achieve higher accuracy. More importantly, a control system for dual robotic arms is proposed based on the CIDGND method to expand the working space of the primary robotic arm, by autonomously adjusting the camera pose with another robotic arm. Finally, simulations and experiments are performed to verify the efficacy of the proposed control systems. The results validate that the DGND method can achieve the PBVS control of robotic arms in the absence of kinematic models and the CIDGND method is able to reduce the tracking error significantly. Comparative results reveal that the CIDGND method is effectively and robustly better than existing methods.

Hybrid Visual Servo Control of a Robotic Manipulator for Cherry Tomato Harvesting

This paper aims to develop a visual servo control of a robotic manipulator for cherry tomato harvesting. In the robotic manipulator, an RGB-depth camera was mounted to the end effector to acquire the poses of the target cherry tomatoes in space. The eye-in-hand-based visual servo controller guides the end effector to implement eye–hand coordination to harvest the target cherry tomatoes, in which a hybrid visual servo control method (HVSC) with the fuzzy dynamic control parameters was proposed by combining position-based visual servo (PBVS) control and image-based visual servo (IBVS) control for the tradeoff of both performances. In addition, a novel cutting and clipping integrated mechanism was designed to pick the target cherry tomatoes. The proposed tomato-harvesting robotic manipulator with HVSC was validated and evaluated in a laboratory testbed based on harvesting implementation. The results show that the developed robotic manipulator using HVSC has an average harvesting time of 9.40 s/per and an average harvesting success rate of 96.25% in picking cherry tomatoes.

Low-Complexity Control for Vision-Based Landing of Quadrotor UAV on Unknown Moving Platform

This article addresses the vision-based landing problem of a low-cost quadrotor on an unknown moving platform. A robust landing controller is developed, which consists of the design of the low-complexity outer-loop controller and the geometric inner-loop attitude controller. First, an error transformation based on prescribed performance is designed to guarantee the landing behaviors and deal with the intermediate control signal of backstepping approaches, resulting in a low-complexity position-based visual servoing (PBVS) design. In addition, the proposed PBVS controller exhibits strong robustness against an uncertain relative dynamic system due to no incorporation of any prior knowledge of the moving platform. Next, a modified geometric attitude controller is presented by characterizing the geometric properties of rotation matrices intrinsically. Finally, the stability analysis is presented using Lyapunov stability theory, and the effectiveness of the proposed control strategy is demonstrated through numerical simulations and experiments.

Neural network based visual servo for quick-change device alignment in context of fusion reactor remote maintenance

The remote maintenance process for fusion reactor is complex, which can be very time-consuming and labor-consuming. This paper proposes a modified neural network based visual servo method to align a quick-change device with robot camera system. A classical position-based visual servo control law is used to guide the robot to reach the desired position. The key target for the above visual servo controller is to obtain a robust pose estimator to calculate the quick-change device pose with respect to the camera. This pose estimator is trained using RepVGG model with a self built image samples. An attention mechanism is added to the neural network to enhance the stability of pose prediction for reflective metal objects. The robot joint speed is also smoothed to reduce the image motion blur effect and make the visual servo process stable. The performance of the proposed visual servo controller was verified on an UR5 robot, and the results show that the stable and rough alignment of the quick-change device can be realized.

Closed-form camera pose and plane parameters estimation for moments-based visual servoing of planar objects

Image moments are global descriptors of an image and can be used to achieve control-decoupling properties in visual servoing. However, only a few methods completely decouple the control. This study introduces a novel camera pose estimation method, which is a closed-form solution, based on the image moments of planar objects. Traditional position-based visual servoing estimates the pose of a camera relative to an object, but the pose estimation method directly estimates the pose of an initial camera relative to a desired camera. Because the estimation method is based on plane parameters, a plane parameters estimation method based on the 2D rotation, 2D translation, and scale invariant moments is also proposed. A completely decoupled position-based visual servoing control scheme from the two estimation methods above was adopted. The new scheme exhibited asymptotic stability when the object plane was in the camera field of view. Simulation results demonstrated the effectiveness of the two estimation methods and the advantages of the visual servo control scheme compared with the classical method.

Visual Object Tracking and Servoing Control of a Nano-Scale Quadrotor: System, Algorithms, and Experiments

There are two main trends in the development of unmanned aerial vehicle (UAV) technologies: miniaturization and intellectualization, in which realizing object tracking capabilities for a nano-scale UAV is one of the most challenging problems. In this paper, we present a visual object tracking and servoing control system utilizing a tailor-made 38 g nano-scale quadrotor. A lightweight visual module is integrated to enable object tracking capabilities, and a micro positioning deck is mounted to provide accurate pose estimation. In order to be robust against object appearance variations, a novel object tracking algorithm, denoted by RMCTer, is proposed, which integrates a powerful short-term tracking module and an efficient long-term processing module. In particular, the long-term processing module can provide additional object information and modify the short-term tracking model in a timely manner. Furthermore, a position-based visual servoing control method is proposed for the quadrotor, where an adaptive tracking controller is designed by leveraging backstepping and adaptive techniques. Stable and accurate object tracking is achieved even under disturbances. Experimental results are presented to demonstrate the high accuracy and stability of the whole tracking system.

Coarse-to-Fine Visual Object Catching Strategy Applied in Autonomous Airport Baggage Trolley Collection

In this study, a systematic solution for mobile manipulation with the application of autonomous trolley collection in the airport is presented. The developed system features a novel integrated hardware and software system, incorporating a specially designed manipulator capable of tightly catching the trolley. Particularly, we innovatively present a supervisory control guided coarse-to-fine visual trolley catching strategy in large-scale airport environments. At the coarsest level, an image-based visual servoing method is utilized to approach the trolley from far to near, where a learning-based object detection method is adopted to identify trolleys in the environment in real-time. At the finest level, we present an accurate point cloud registration based object pose estimation method to promote the object manipulation with a position-based visual servoing controller. The tightly integrated system and the subsystems are evaluated through conducting experiments in the trolley collection task and comparing the key module with the state of the art. Extensive experiments demonstrate that the proposed framework can find and fetch the trolley robustly. Notably, the proposed object detection and pose estimation methods exhibit higher accuracy, providing credible perceptual approaches for mobile manipulation. Moreover, the proposed framework can be easily extended to other mobile manipulation tasks for object search and retrieval.

A Compact Surgical Robot System for Craniomaxillofacial Surgery and its Preliminary Study.

Craniomaxillofacial surgery has the characteristics of complex anatomical structure, narrow surgical field, and easy damage to nerves, blood vessels, and other structures. Compared with the traditional bare-hand operation, robot-assisted craniofacial surgery is expected to achieve a more stable and accurate surgical operation. So we have developed a robot-assisted craniofacial surgery system. A compact mechanism design was adopted for the robot system, integrates with visual and force perception modules. The motion analysis and working space analysis are carried out on the mechanical structure. The binocular vision module is integrated and the robot hand-eye calibration process was completed. The target tracking method based on staple is used to achieve tracking and monitoring of the target area. A distributed robot control system based on CAN bus technology is designed, and a position-based visual servo control method is adopted. Then the precision test of the robot system prototype and the drilling experiment of the 3D printed mandible model were carried out. The average pixel error of the vision module is 0.15 pixels. Based on the staple tracking method, the average center error rate of the image is 0.3175 mm, and the overlap rate is 88.76%. The drilling experiment of the mandible model showed that the average entrance position error is 1.76 ± 0.36 mm, the average target position error is 1.62 ± 0.27 mm, and the angle error is 5.36 ± 0.31 degrees. The designed craniofacial robot system can better assist surgeons to complete the mandibular osteotomy.

Collision Avoidance for Redundant Robots in Position-Based Visual Servoing

To tackle the problem on trajectory planning or the design of control law, this paper introduces a visual servoing system for a manipulator with redundant joints that the trajectory of the manipulator approaching the target is determined spontaneously by the visual control law. The proposed method resolves joint solution for visual servoing and obstacle avoidance. The work comprises of two procedures, feature extraction for position-based visual servoing (PBVS) and collision avoidance within the working envelope. In the PBVS control, the target pose must be reconstructed with respect to the robot and this results in a Cartesian motion-planning problem. Once the geometric relationship between the target and the end effector is determined, a secure inverse kinematics method incorporating trajectory planning is used to solve the solution of the redundant manipulator by the virtual repulsive torque method. Therefore, the links of the manipulator can always maintain a safe distance from obstacles while approaching the target smoothly. The proposed method is verified with its applicability in experiments using an eye-in-hand manipulator with seven joints. For reusability and extensibility, the system has been coded and constructed in the framework of the Robot Operating System so as that the developed algorithms can be disseminated to different platforms.

Robust Visual Servoing Control for Ground Target Tracking of Quadrotors

In this brief, the robust position-based visual servoing control problem is addressed for a quadrotor with a down-facing monocular camera to perform ground target tracking tasks in outdoor environments. The object is to regulate the position of the quadrotor relative to the ground target by using a minimum sensor set, including an inertial measurement unit, an ultrasonic sensor, and a vision sensor. The robust compensators are designed in the visual servoing control system to enhance the robustness against nonlinearities, couplings, and uncertainties, including parameter uncertainties and external disturbances in the translational and rotational quadrotor dynamics. Theoretical analysis and experimental results demonstrate the effectiveness of the proposed robust visual servoing control method.

Modeling and vision-based control of large-dimension cable-driven parallel robots using a multiple-camera setup

This paper deals with the modeling and vision-based control of large-dimension cable-driven parallel robots. Inverse kinematics and instantaneous inverse kinematics models are derived from the elastic catenary cable modeling. These models turn out to be dependent on the pose of the mobile platform (end-effector), on the cable tangent directions and on the cable tensions. In order to control the motion of the robot, a position-based visual servo control is used, where the mobile platform pose is measured by vision and used for regulation. A multi-camera setup and load cells provide the aforementioned desired measurements, i.e., the mobile platform pose, the directions of the tangents to the cables, and the cable tensions. The proposed approach was validated in experiments on the large-dimension cable-driven parallel robot prototype CoGiRo of global dimensions 15 m × 11 m × 6 m (L × l × h). A maximum error of less than 1 cm in position and 0.5∘ in orientation was achieved. Moreover, in the case of cable-driven parallel robots larger than the prototype CoGiRo, simulations were conducted in order to assess the influence on the vision-based control of four instantaneous inverse kinematics models.

Incremental visual servo control of robotic manipulator for autonomous capture of non-cooperative target

This paper develops a new autonomous incremental visual servo control law for the robotic manipulator to capture a non-cooperative target, where the control input is the incremental joint angle to avoid the multiple solutions in the existing inverse kinematics. The position and motion of the non-cooperative target are estimated by an eye-to-hand vision system in real time by integrated photogrammetry and extended Kalman filter. The estimated position and motion of the target are fed into the newly developed position-based visual servo control law to drive the manipulator incrementally towards the dynamically predicted interception point between trajectories of the end effector and the target. To validate the proposed approach, a hardware-in-the-loop simulation has been conducted where the position and motion of the target is estimated by a real eye-to-hand camera and fed into the simulation of the robotic manipulator. The simulation results show the proposed incremental visual servo control law is stable and able to avoid the multiple solutions in the total inverse kinematics.

Position-based visual servo control of autonomous robotic manipulators

This paper concerns the position-based visual servo control of autonomous robotic manipulators in space. It focuses on the development of a real-time vision-based pose and motion estimation algorithm of a non-cooperative target by photogrammetry and extended Kalman filter for robotic manipulators to perform autonomous capture. Optical flow algorithm is adopted to track the target features in order to improve the image processing efficiency. Then, a close-loop position-based visual servo control strategy is devised to determine the desired pose of the end-effector at the rendezvous point based on the estimated pose and motion of the target. The corresponding desired joint angles of the robotic manipulator in the joint space are derived by the inverse kinematics of the robotic manipulator. The developed algorithm and position-based visual servo control strategy are validated experimentally by a custom built robotic manipulator with an eye-in-hand configuration. The experimental results demonstrate the proposed estimation algorithm and control scheme are feasible and effective.

혼합 비주얼 서보 제어 기법을 이용한 이동로봇의 목표물 추종

This paper proposes a target tracking algorithm for wheeled mobile robots using in various fields. For the stable tracking, we apply a vision system to a mobile robot which can extract targets through image processing algorithms. Furthermore, this paper presents an algorithm to position the mobile robot at the desired location from the target by estimating its relative position and attitude. We show the problem in the tracking method using the Position-Based Visual Servo(PBVS) control, and propose a tracking method, which can achieve the stable tracking performance by combining the PBVS control with Image-Based Visual Servo(IBVS) control. When the target is located around the outskirt of the camera image, the target can disappear from the field of view. Thus the proposed algorithm combines the control inputs with of the hyperbolic form the switching function to solve this problem. Through both simulations and experiments for the mobile robot we have confirmed that the proposed visual servo control method is able to enhance the stability compared to of the method using only either PBVS or IBVS control method.

Simulation and experiments for a vision-based control of a 6R robot

In this paper, we present both the simulation and experimental results for position-based visual servoing control of a 6R robot using two stationary cameras . This method can deal with real-time changes in the relative position of the target with respect to the robot, while providing greater accuracy. In addition, the servo control structure is independent of the target pose coordinates. Direct and inverse kinematics equations, in addition to dynamic equations of the 6R robot, have been derived and simulated. Then we prescribe simulation of image processing, object recognition and pose estimation for the end effector and target-object in 3D Cartesian space and visual control of the robot. Afterwards, performance tests of 6R robot using two cameras have been implemented. Finally, experimental results obtained from actual implementation of visual control and tests of 6R robot in lab are presented. Analysis of error and test data has been carried out according to ISO9283, ANSI-RIA R15.05–2 standards and the MATLAB statistical toolbox. An efficient algorithm is designed for collision detection and contact determination between the 6R robot links and its environment during their rotation in controlling and performance tests. In this technique, bounding spheres on different parts of 6R robot are used to detect collisions among them.

Robotic execution of everyday tasks by means of external vision/force control

In this article, we present an integrated manipulation framework for a service robot, that allows to interact with articulated objects at home environments through the coupling of vision and force modalities. We consider a robot which is observing simultaneously his hand and the object to manipulate, by using an external camera (i.e. robot head). Task-oriented grasping algorithms (Proc of IEEE Int Conf on robotics and automation, pp 1794–1799, 2007) are used in order to plan a suitable grasp on the object according to the task to perform. A new vision/force coupling approach (Int Conf on advanced robotics, 2007), based on external control, is used in order to, first, guide the robot hand towards the grasp position and, second, perform the task taking into account external forces. The coupling between these two complementary sensor modalities provides the robot with robustness against uncertainties in models and positioning. A position-based visual servoing control law has been designed in order to continuously align the robot hand with respect to the object that is being manipulated, independently of camera position. This allows to freely move the camera while the task is being executed and makes this approach amenable to be integrated in current humanoid robots without the need of hand-eye calibration. Experimental results on a real robot interacting with different kind of doors are presented.

Visual servo control. I. Basic approaches

This paper is the first of a two-part series on the topic of visual servo control using computer vision data in the servo loop to control the motion of a robot. In this paper, we describe the basic techniques that are by now well established in the field. We first give a general overview of the formulation of the visual servo control problem. We then describe the two archetypal visual servo control schemes: image-based and position-based visual servo control. Finally, we discuss performance and stability issues that pertain to these two schemes, motivating the second article in the series, in which we consider advanced techniques

A modified resolved acceleration controller for position-based visual servoing

This paper presents modifications to the resolved acceleration controller design to develop a Cartesian controller suitable for use with position-based visual servoing. The development of any Cartesian control for relative robot end-point position control with respect to an object, requires the solution to the problem of relative orientation measurement and control. This problem is due to the nonlinear nature of orientation and the complex transformations used for its representation. The solution proposed in this paper, is to use the orientation representation of roll-pitch-yaw angles in a vector form, and calculate the difference between the current vector and the desired vector to represent the orientation error. The Jacobian matrices which relate the roll-pitch-yaw rates to the joint velocities are developed. The resulting relationships are then used in the development of a modified resolved acceleration controller that is applicable to position-based visual servo control.