SURRL: Structural Unsupervised Representations for Robot Learning

Abstract

Revolutionary advances have occurred in robot learning research, where a resurgence in reinforcement learning (RL) algorithms has fueled breakthroughs in acquiring complicated robotic skills without human intervention. Unfortunately, one limitation that has hampered reinforcement learning methods is that RL may be quite inefficient, that is, it may cost unrealistic learning time and prohibitively large numbers of trajectories to provide implausible models for achieving multi-task learning. In a broader perspective, the realization of robotics control by RL relies heavily on the availability of compact and expressive representations of the state spaces. In this paper, we propose to learn vivid general structural representations by utilizing structural prior knowledge of robots. Particularly, a novel framework called Structural Unsupervised Representations for Robot Learning (SURRL) is presented to enable multi-task learning. The task-agnostic sample trajectories are leveraged to learn the structural representations which are constructed by Graph Auto-Encoder in an unsupervised fashion. When learning a new task, the learned structural representations can be directly adopted for subsequently policy learning without training from scratch. Extensive experiments on continuous robotic environments, including dexterous manipulation tasks, characterize that our method’s effectiveness to learn optimal policies for handling multiple tasks learning, demonstrating significant improvements over other competitive methods.