Hand-eye Camera Research Articles

Visual Servoing of a 3R Robot by Metaheuristic Algorithms

In planar robot control, the singularity problem is frequently encountered when determining the geometric or kinematic robot model. To overcome this problem, a new method is presented in this paper. Indeed, this method is used to compute the joint positions of 3R robots with high accuracy. The main idea of the proposed method is to use a metaheuristic algorithm for retrieving the coordinates of four reference points in the object image that is captured by a hand-eye camera of the 3R robot. Then, the joint positions are estimated by using a metaheuristic algorithm. The resulting positions are then used for controlling the 3R robot for moving to the desired position. The simulation experiments are conducted by using several metaheuristic algorithms with the same population size (N = 400). The obtained results show the high accuracy of the proposed method in terms of determining the exact positions of the 3R robot joints, which leads to finding the optimal robot trajectory.

Design and experiment of on-orbit assembly ground simulation robot

This paper proposes a space telescope assembly scheme and a ROS-based on-orbit assembly robot simulation system. First, the system fuses the hand-eye camera and the global camera ArUco marked image information to complete the identification and position and pose estimation of the structural parts to be assembled. Then, the pose information is transmitted to the ROS motion control node through Modbus-TCP communication, and the dual-arm robot system is controlled to complete motion planning and obstacle avoidance, arriving at the designated position for precise pick and place. After many simulation experiments of the designed space telescope model assembly scheme, it has been proved that the target object pose estimation algorithm is accurate, and the dual-arm collaborative robot has high position accuracy, which can realize the ground simulation autonomous assembly of large-scale space facilities. The system has certain practical application value with high integration, strong expansibility and good portability.

Robust monocular 3D object pose tracking for large visual range variation in robotic manipulation via scale-adaptive region-based method

Many robot manipulation processes involve large visual range variation between the hand-eye camera and the object, which in turn causes object scale change of a large span in the image sequence captured by the camera. In order to accurately guide the manipulator, the relative 6 degree of freedom (6D) pose between the object and manipulator is continuously required in the process. The large-span scale change of the object in the image sequence often leads to the 6D pose tracking failure of the object for existing pose tracking methods. To tackle this problem, this article proposes a novel scale-adaptive region-based monocular pose tracking method. Firstly, the impact of the object scale on the convergence performance of the local region-based pose tracker is meticulously tested and analyzed. Then, a universal region radius calculation model based on object scale is built based on the statical analysis result. Finally, we develop a novel scale-adaptive localized region-based pose tracking model by merging the scale-adaptive radius selection mechanism into the local region-based method. The proposed method adjusts local region size according to the scale of the object projection and achieves robust pose tracking. Experiment results on synthetic and real image sequences indicate that the proposed method achieves better performance over the traditional localized region-based method in manipulator operation scenarios which involve large visual range variation.

Grasping of arbitrary shape objects by a two-finger hand with an RGB-D hand-eye camera

Grasping of arbitrary shape objects by a two-finger hand with an RGB-D hand-eye camera

A Multi-Scale Grasp Detector Based on Fully Matching Model

Robotic grasping is an essential problem at both the household and industrial levels, and unstructured objects have always been difficult for grippers. Parallel-plate grippers and algorithms, focusing on partial information of objects, are one of the widely used approaches. However, most works predict single-size grasp rectangles for fixed cameras and gripper sizes. In this paper, a multi-scale grasp detector is proposed to predict grasp rectangles with different sizes on RGB-D or RGB images in real-time for hand-eye cameras and various parallel-plate grippers. The detector extracts feature maps of multiple scales and conducts predictions on each scale independently. To guarantee independence between scales and efficiency, fully matching model and background classifier are applied in the network. Based on analysis of the Cornell Grasp Dataset, the fully matching model can match all labeled grasp rectangles. Furthermore, background classification, along with angle classification and box regression, functions as hard negative mining and background predictor. The detector is trained and tested on the augmented dataset, which includes images of 320 × 320 pixels and grasp rectangles ranging from 20 to more than 320 pixels. It performs up to 98.87% accuracy on image-wise dataset and 97.83% on object-wise split dataset at a speed of more than 22 frames per second. In addition, the detector, which is trained on a single-object dataset, can predict grasps on multiple objects.

Robotic hand-arm system for on-orbit servicing missions in Tiangong-2 Space Laboratory

PurposeThe purpose of this study is to develop a robotic hand–arm system for on-orbit servicing missions at the Tiangong-2 (TG-2) Space Laboratory.Design/methodology/approachThe hand–arm system is mainly composed of a lightweight arm, a dexterous hand, an electrical cabinet, a global camera, a hand–eye camera and some human–machine interfaces. The 6-DOF lightweight arm and the 15-DOF dexterous hand adopt the modular design philosophy that greatly reduces the design cycle and cost. To reduce the computational burden on the central controller and simplify system maintenance, an electrical system which has a hierarchical structure is introduced.FindingsThe prototypical operating experiments completed in TG-2 space laboratory demonstrate the performance of the hand–arm system and lay foundations for the future applications of space manipulators.Originality/valueThe main contributions of this paper are as follows a robotic hand–arm system which can perform on-orbit servicing missions such as grasping the electric drill, screwing the bolt, unscrewing J599 electrical connector has been developed; a variable time step motion plan method is proposed to adjust the trajectories of the lightweight arm to reduce or eliminate the collision force; and a dexterous hand uses the coordinated grasp control based on the object Cartesian stiffness to realize stable grasp. To solve the kinematic mapping from the cyber glove commands to the dexterous hand, a fingertip-position-based method is proposed to acquire precise solutions.

Modeling and planning of a space robot for capturing tumbling target by approaching the Dynamic Closest Point

Space debris or uncontrolled satellites bring new and critical challenges for space robots to capture and remove them, especially when the targets are tumbling. In the face of these challenges, most existing research only focused on capturing a fixed point on the target. Distinguished from previous research, this paper attempts to investigate an autonomous tumbling target capturing method by approaching the “Dynamic Closest Point” (DCP), that is, the most suitable point on the target to be grasped and always closest to the end effector. First, an uncontrolled space object and a space robotic system are modeled, and the existence of the DCP is proved. For a practical malfunction spacecraft, the motion characteristics of the DCP and a fixed point are theoretically analyzed and compared as regards different motion types and rotation vectors. Furthermore, the desired trajectory is generated for the end effector of the space robot to approach the DCP autonomously, based on the position and velocity of the DCP estimated in real-time by hand-eye cameras. The joint velocities and angles are then resolved by combining the differential kinematics and movement constraints on the end effector. Finally, an Adams-Simulink co-simulation system is developed. Based on this, the proposed method is verified by single-arm and dual-arm capturing cases. Simulation results demonstrate that the capture time and movement ranges of the end effector are both greatly reduced with the DCP method. In contrast with traditional capturing methods, the efficiency of this proposed method is largely improved.

Improved eye-vergence visual servoing system in longitudinal direction with RM-GA

Visual servoing is a control method to manipulate the motion of the robot using visual information, which aims to realize “working while watching.” However, the visual servoing towards moving target with hand–eye cameras fixed at hand is inevitably affected by hand dynamical oscillation. To overcome this defect of the hand–eye fixed camera system, an eye-vergence system has been put forward, where the pose of the cameras could be rotated to observe the target object. The visual servoing controllers of hand and eye-vergence are installed independently, so that it can observe the target object at the center of camera images through eye-vergence function. In this research, genetic algorithm (GA) is used as a pose tracking method, which is called “Real-Time Multi-step GA(RM-GA),” solves on-line optimization problems for 3D visual servoing. The performances of real-time object tracking using eye-vergence system and “RM-GA” method have been examined, and also the pose tracking accuracy has been verified.

Robust, fast and accurate vision-based localization of a cooperative target used for space robotic arm

When a space robotic arm deploys a payload, usually the pose between the cooperative target fixed on the payload and the hand-eye camera installed on the arm is calculated in real-time. A high-precision robust visual cooperative target localization method is proposed. Combing a circle, a line and dots as markers, a target that guarantees high detection rates is designed. Given an image, single-pixel-width smooth edges are drawn by a novel linking method. Circles are then quickly extracted using isophotes curvature. Around each circle, a square boundary in a pre-calculated proportion to the circle radius is set. In the boundary, the target is identified if certain numbers of lines exist. Based on the circle, the lines, and the target foreground and background intensities, markers are localized. Finally, the target pose is calculated by the Point-3-Perspective algorithm. The algorithm processes 8 frames per second with the target distance ranging from 0.3m to 1.5m. It generated high-precision poses of above 97.5% on over 100,000 images regardless of camera background, target pose, illumination and motion blur. At 0.3m, the rotation and translation errors were less than 0.015° and 0.2mm. The proposed algorithm is very suitable for real-time visual measurement that requires high precision in aerospace.

Frequency response experiments of eye-vergence visual servoing in lateral motion with 3D evolutionary pose tracking

Visual servoing towards moving target with hand-eye cameras fixed at hand is inevitably affected by hand dynamical oscillations. Therefore, it is difficult to make target position keep always at the center of camera's view, as nonlinear dynamical effects of whole manipulator stand against tracking ability. To overcome this defect of the hand-eye fixed camera system, an eye-vergence system has been put forward, where the cameras could rotate to observe the target object. The visual servoing controllers of hand and eye-vergence are installed independently, so that it can observe the target object at the center of camera images through eye-vergence function. The dynamical superiorities of eye-vergence system are verified through frequency response experiments, comparing with hand tracking performances and the proposed eye-vergence tracking performances. This paper analyzes the performance of 3D-object position and orientation tracking, in which orientation representation method is based on quaternion, and the orientation tracking results are shown with more comprehensive analysis of system performance.

配電作業用ロボットにおける工具箱上のボルトの把持

We have been developing a power distribution line maintenance robot system. This system will autonomously carry out basic tasks in the maintenance work. For examples, “Grasping a bolt”, “Inserting a bolt”, “Tightening a nut” and so on. We perform a task performance test “Grasp the bolt on the tool box”. The system grasps the bolt on tool box under visual feedback from the hand-eye camera. Applied visual feedback control does not require camera calibration and arm calibration because this visual feedback adjusts the bolt and the gripper in one image plane using the hand-eye camera.

Attitude disturbance correction for binocular vision cameras in rendezvous and docking

Vision-based relative position and attitude measurement methods are widely used in rendezvous and docking (RVD). For example, a structured light vision-based sensor (the advanced video guidance sensor) was included as one of the primary proximity navigation sensors in the Orbital Express mission [1]. A monocular vision-based sensor (hand–eye camera mounted on the end effector and monitor camera mounted on the first joint of the robot arm) is used in Engineering Test Satellite No. 7 to locate cooperative targets [2]. In the Front-end Robotics Enabling Near-term Demonstration, a binocular vision-based sensor (camera head unit LIDAR) is used to locate noncooperative targets [3].

Analyses about Trackability of Hand-Eye-Vergence Visual Servoing in Lateral Direction

Visual Servoing to Moving Target with Fixed Hand-Eye Cameras Mounted at Hand of Robot is Inevitably Be Affected by Hand Dynamical Oscillations, then it is Hard to Keep Target at the Centre of Camera’s Image, since Nonlinear Dynamical Effects of Whole Manipulator Stand against Tracking Ability. in Order to Solve this Problem, an Eye-Vergence System, where the Visual Servoing Controller of Hand and Eye-Vergence is Controlled Independently, so that the Cameras can Observe the Target Object at the Center of the Camera Images through Eye-Vergence Functions. the Eyes with Light Mass Make the Cameras’ Eye-Sight Direction Rotate Quickly, so the Track Ability of the Eye-Vergence Motion is Superior to the One of Fixed Hand-Eye Configuration. in this Report Merits of Eye-Vengence Visual Servoing for Pose Tracking Have been Confirmed through Frequency Response Experiments.

Fusion of Multi-Vision of Industrial Robot in MAS-Based Smart Space

The paper presents a fusion method of muti-vision of industrial robot in a smart space based on multi-agent system(MAS), the robotic multi-vision consists of top-view, side-view, front-view and hand-eye cameras, the moving hand-eye provide vision guidance and give the estimation of robot position, other three cameras are used for target recognition and positioning. Each camera is connected to an agent based on an image-processing computer that aims at analyzing image rapidly and satisfying the real-time requirement of data processing. As a learning strategy of robotic vision, a back-propagation neural network(BPNN) with 3-layer-architecture is first constructed for each agent and is independently trained as a classifier of target recognition using batch gradient descent method based on the region features extracted from the images of target samples(typical mechanical parts), and then the outputs of trained BPNNs in MAS-based smart space are fused with Dempster-Shafer evidence theory to form a final recognition decision, the experimental results of typical mechanical parts show that fusion of multi-vision can improve the robotic vision accuracy and MAS-based smart space will contribute to the parallel processing of immense image data in robotic multi-vision system.

A Ground-Based Validation System of Teleoperation for a Space Robot

Teleoperation of space robots is very important for future on-orbit service. In order to assure the task is accomplished successfully, ground experiments are required to verify the function and validity of the teleoperation system before a space robot is launched. In this paper, a ground-based validation subsystem is developed as a part of a teleoperation system. The subsystem is mainly composed of four parts: the input verification module, the onboard verification module, the dynamic and image workstation, and the communication simulator. The input verification module, consisting of hardware and software of the master, is used to verify the input ability. The onboard verification module, consisting of the same hardware and software as the onboard processor, is used to verify the processor's computing ability and execution schedule. In addition, the dynamic and image workstation calculates the dynamic response of the space robot and target, and generates emulated camera images, including the hand-eye cameras, global-vision camera and rendezvous camera. The communication simulator provides fidelity communication conditions, i.e., time delays and communication bandwidth. Lastly, we integrated a teleoperation system and conducted many experiments on the system. Experiment results show that the ground system is very useful for verified teleoperation technology.

Hand-Eye Motion-Invariant Pose Estimation with Online 1-Step GA -3D Pose Tracking Accuracy Evaluation in Dynamic Hand-Eye Oscillation-

This paper presents online pose measurement for a 3-dimensional (3-D) object detected by stereo hand-eye cameras. Our proposal improves 3-D pose tracking accuracy by compensating for the fictional motion of the target in camera images stemming from the ego motion of the hand-eye camera caused by dynamic manipulator oscillation. This motion feed-forward (MFF) is combined into the evolutionary search of a genetic algorithm (GA) and fitness evaluation based on stereo model matching whose pose is expressed using a unit quaternion. The proposal’s effectiveness was confirmed in simulation tracking an object’s 3-D pose adversely affected by hand-eye camera oscillations induced by dynamic effects of robot motion.

Autonomous target capturing of free-floating space robot: Theory and experiments

SUMMARYSpace robotic systems are expected to play an increasingly important role in the future. Unlike on the earth, space operations require the ability to work in the unstructured environment. Some autonomous behaviors are necessary to perform complex and difficult tasks in space. This level of autonomy relies not only on vision, force, torque, and tactile sensors, but also the advanced planning and decision capabilities. In this paper, the authors study the autonomous target capturing from the issues of theory and experiments. Firstly, we deduce the kinematic and dynamic equations of space robotic system. Secondly, the visual measurement model of hand–eye camera is created, and the image processing algorithms to extract the target features are introduced. Thirdly, we propose an autonomous trajectory planning method, directly using the 2D image features. The method predicts the target motion, plans the end-effector's velocities and solves the inverse kinematic equations using practical approach to avoid the dynamic singularities. At last, numeric simulation and experiment results are given. The ground experiment system is set up based on the concept of dynamic simulation and kinematic equivalence. With the system, the experiments of autonomous capturing a target by a free-floating space robot, composed of a 6-DOF manipulator and a satellite as its base, are conducted, and the results validate the proposed algorithm.

Autonomous Path Planning and Experiment Study of Free-floating Space Robot for Target Capturing

Space robotic systems are expected to play an increasingly important role in the future. The robotic on-orbital service, whose key is the capturing technology, becomes research hot in recent years. In this paper, the authors propose an autonomous path planning method for target capturing. The task is described in Cartesian space and it can drive the manipulator to approach the target along the closest path. Firstly, the target feature is extracted based on the measured information via the hand-eye camera, and the target pose (position and orientation) and velocities (linear velocity and angular velocity) are estimated using Kalman filtering technology. Then, a numerically feasible approach is presented to plan the manipulator motion and avoid the dynamic singularities, which are transformed into real-time kinematic singularities avoiding problem. Thirdly, the potential disturbance on the base due to the manipulator's motion is estimated, and the joint rates are autonomously adjusted to reduce the disturbance if it is beyond the allowed bound. At last, a ground experiment system is set up based on the concept of dynamic emulation and kinematic equivalence. With the experiment system, the autonomous target capturing experiments are conducted. The experiment results validate the proposed algorithm.

Visual Servoing to Catch Fish Using Global/Local GA Search

This paper presents a vision-related technique for a real-time visual servoing. The method utilizes the global search feature of a genetic algorithm (GA) and a local search technique of the GA. In the GA process, the computation of the fitness function is based on the configuration of an object model designated as the surface-strips model. To evaluate the effectiveness of the proposed recognition method, experiments to track a fish by a hand-eye camera and catch the fish with a net attached to the hand of the manipulator have been performed.

Evolutionary Pose Measurement by Stereo Model Matching

This paper presents a pose measurement method of a 3-D object. The proposed method utilizes an evolutionary search technique of the genetic algorithm (GA) and a fitness evaluation based on a matching stereo model, named as surface-strips model here. The unprocessed gray-scale image, called a raw image, is used in order to perform recognition of a target using known target object shape. Here, the problem to recognize the position/orientation of the target object is converted to an optimization problem of a fitness function that consists in the computation of the brightness difference between an internal surface and a contour-strips. In order to evaluate the proposed 3-D recognition method, experiments to detect position/orientation of a rectangular solid block have been conducted to show its effectiveness of recognizing objects in static image. Furthermore, experiments to recognize a ball on a turning table by a robot manipulator equipped with two hand-eye cameras have also been conducted to show the effectiveness of this method for Real-Time visual servoing.