Parallel Curriculum Experience Replay in Distributed Reinforcement Learning

Abstract

Distributed training architectures have been shown to be effective to improve the performance of reinforcement learning algorithms. However, their performances are still poor for problems with sparse rewards, e.g., the scoring task with or without goalkeeper for robots in RoboCup soccer. It is challenging to solve these tasks in reinforcement learning, especially for those that require combining high-level actions with flexible control. To address these challenges, we introduce a distributed training framework with parallel curriculum experience replay that can collect different experiences in parallel and then automatically identify the difficulty of these subtasks. Experiments on the domain of simulated RoboCup soccer show that, the approach is effective and outperforms existing reinforcement learning methods.