Hierarchical deep reinforcement learning to drag heavy objects by adult-sized humanoid robot

Abstract

Most research on robot manipulation focuses on objects that are light enough for the robot to pick them up. However, in our daily life, some objects are too big or too heavy to be picked up or carried, so that dragging them is necessary. Although bipedal humanoid robots have nowadays good mobility on level ground, dragging unfamiliar objects including large and heavy objects on various surfaces is an interesting research area with many applications, which will provide insights into human manipulation and will encourage the development of novel algorithms for robot motion planning and control. This is a challenging problem, not only because of the unknown and potentially variable friction of the foot, but also because the feet of the robot may slip during unbalanced poses. In this paper, we propose a novel hierarchical deep learning algorithm that learns how to drag heavy objects with an adult-sized humanoid robot for the first time. First, we present a Three-layered Convolution Volumetric Network (TCVN) for 3D object classification with point clouds volumetric occupancy grid integration. Second, we propose a lightweight real-time instance segmentation method named Tiny-YOLACT for the detection and classification of the floor surface. Third, we propose a deep Q-learning algorithm to learn the policy control of the Center of Mass of the robot (DQL-COM). The DQL-COM algorithm learning is bootstrapped using the ROS Gazebo simulator. After initial training, we complete training on the THORMANG-Wolf, a 1.4 m tall adult-sized humanoid robot with 27 degrees of freedom and weighing 48 kg, on three distinct types of surfaces. We evaluate the performance of our approach by dragging eight different types of objects (e.g., a small suitcase, a large suitcase, a chair). The extensive experiments (480 times on the real robot) included dragging a heavy object with a mass of 84.6 kg (two times greater than the robot’s weight) and showed remarkable success rates of 92.92% when combined with the force–torque sensors, and 83.75% without force–torque sensors.