Curriculum Adversarial Training for Robust Reinforcement Learning

Abstract

Reinforcement learning with adversarial training is currently a key method for improving the robustness of DRL. However, in adversarial training, especially for unstable or disturbance-sensitive systems, the adversary always learns the policy significantly faster than the DRL agent and thus easily generates powerful perturbations. The agent cannot effectively adapt to the overly powerful adversary, which leads to unstable training and even failure to learn the robust policy. In this work, we propose a novel adversarial training method, called Curriculum Adversarial Training, inspired by the idea of curriculum learning. The method dynamically adjusts the strength of the adversary through natural curriculum learning for progressive adversarial training. Thus, the DRL system considers to reasonable learning rules, and the agent faces a suitable learning process. Furthermore, we adopt an advanced action space perturbation method with a most attractive ability as the adversary during training. The proposed method is compared with popular baseline methods through MuJoCo tasks. Experimental results show that our method can improve the robustness of the policy significantly and adapt to uncertain environment effectively.