PMA-DRL: A parallel model-augmented framework for deep reinforcement learning algorithms

Abstract

In recent years, aided by deep neural networks (DNNs), reinforcement learning (RL) algorithms have been achieving great success in more and more tasks. In general, model-free RL algorithms are widely applicable, but sometimes suffer from low sample efficiency. Although the environment dynamics can be incorporated into model-free RL algorithms to enhance sample efficiency, which can be transformed into the joint algorithm of model-free and model-based RL algorithms, the model bias of dynamics model may still hurt the performance. Another attractive study direction for RL algorithms, is using parallel strategy. Meanwhile, parallel RL algorithms can achieve outstanding results under less time cost, but are still with low sample efficiency, because most of such algorithms are vanilla model-free ones. Therefore, aiming to enhance the model performance for RL algorithms with DNNs, in this paper, we propose a novel parallel model-augmented framework, called PMA-DRL, to combine the dynamics model and parallel RL algorithms together. First, we introduce a local dynamics model (LDM) for each local agent model (LAM) in parallel RL algorithms. Next, we propose to build a better LDM by using an ensemble of LDMs, and such ensemble can help to reduce model bias, because the experienced observations under different LAMs are always diverse. Then, the observation diversity can be further increased by employing LDMs to explore. Finally, we implement experiments on classic control tasks (CCTs) and Atari games, which demonstrates the efficiency of our proposed framework.