Distributed Reinforcement Learning with States Feature Encoding and States Stacking in Continuous Action Space

Abstract

The practical application of reinforcement learning agents is often bottlenecked by the duration of training time. To accelerate training, practitioners often turn to distributed reinforcement learning architectures to parallelize and accelerate the training process. This work, we utilize the distributed reinforcement learning architecture to deal with continuous control tasks. The Importance Weighted Actor-Learner Architectures (IMPALA) decouples the acting and learning process to reduce queuing time. IMPALA attains higher scores on the new DMLab-30 set and the Atari-57 set because of its high performance, good scalability, and high efficiency. We extend IMPALA on the continuous control tasks with three changes. We encoder states into low dimensional data to establish an action distribution function that the agents have the ability to exploit and explore. A queue buffer is used to store a mini-batch data and discard them after training. In order to make the agent take appropriate action in the continuous control environment, we stack the past three steps states that attempt to make the robot moves smoothly. Finally, experiments are carried out on Mujoco tasks. The results show that our work is better than other distributed reinforcement learning algorithms.