Preferential Experience Collection with Frequency based Intrinsic Reward for Deep Reinforcement Learning

Abstract

Experience replay plays a key role in the success of deep reinforcement learning (DRL), where the experience collection is defined as the process of placing experience tuples into a replay buffer. Although this deterministic experience collection works well for the posterior replay, there is still a potential to improve the efficiency by using a preferential way. In this paper, we propose a preferential experience collection (PEC) method for experience replay in model-free DRL, where the reward is composed of extrinsic and intrinsic components. Since designing a good extrinsic reward is difficult especially in a complicated environment, this paper considers to establish a novel self-supervised intrinsic reward (IR) for model-free DRL. By looking into the frequency domain, the intrinsic reward is designed such that the agent can understand the behavior itself to some extent in addition to completing the task. Extensive experimental results demonstrate that our method can improve the efficiency and stability of the model-free DRL.