Quadruped Reinforcement Learning without Explicit State Estimation

Abstract

Reinforcement learning is a promising approach to developing legged robot locomotion controllers. The gen-eral process of development is: large-scale training in the virtual simulation environment to obtain reliable control policy network, and then the policy network is deployed to real legged robot. In the training procedure, a complete robot state increases the speed of training and the stability of policy. The robot's states like body velocities are easily available in simulation training, but they are difficult to obtain in the real robot, hence specifically designed robot state estimators are needed. However, the development of state estimators requires expert knowledge related to control theory and robotics, limiting the direct application of reinforcement learning to robots. To take advantage of the end-to-end mapping of artificial neural networks, we simplified the existing reinforcement learning process for quadruped robots and propose a training method based on curriculum learning in this work. The proposed method can produce a reliable policy that does not require robot state estimator and only take raw sensors data. The feasibility of the proposed method is verified in simulations and real quadrupedal robot. Video of the quadrupedal robot is available at www.youtube.com/watch?v=-iho4KIlEPw.