A Human–Robot Collaboration Method Using a Pose Estimation Network for Robot Learning of Assembly Manipulation Trajectories From Demonstration Videos

Abstract

The wide application of industrial robots has greatly improved assembly efficiency and reliability. However, determining how to efficiently teach a robot to perform assembly manipulation trajectories from demonstration videos is a challenging issue. This paper proposes a method integrating deep learning, image processing, and an iteration model to predict the real assembly manipulation trajectory of a human hand from a video without specific depth information. First, a pose estimation network, Keypoint-RCNN, is used to accurately estimate hand pose in the two-dimensional (2-D) image of each frame in a video. Second, image processing is applied to map the 2-D hand pose estimated by the neural network with the real 3-D assembly space. An iteration model based on the trust region algorithm is proposed to solve for the quaternions and translation vectors of two frames. All the quaternions and translation vectors form the predicted assembly manipulation trajectories. Finally, a UR3 robot is used to imitate the assembly operation based on the predicted manipulation trajectories. The results show that the robot could successfully imitate various operations based on the predicted manipulation trajectories.