Joint Action Representation and Prioritized Experience Replay for Reinforcement Learning in Large Discrete Action Spaces

Abstract

In dealing with the large discrete action spaces, a joint action representation and prioritized experience replay method is proposed in this paper, which consists of three modules. In the first module, we use the k-nearest neighbor method to reduce the dimensionality of the original action space, generating a compact action space, and then the critic network is introduced to further evaluate and filter this compact space to obtain the optimal action. Note that the optimal action may have inconsistency with the actual desired action. Then in the second module, we introduce a multi-step update technique to reduce the training variance when storing data in the replay buffer. In the third module, considering the existence of correlation between samples when sampling data, we assign the corresponding weight to the sample experience by calculating the absolute value of temporal difference error and use such a non-uniform sampling method to prioritize the samples for sampling. Experimental results on four benchmark environments demonstrate the effectiveness and efficiency of the proposed method in dealing with the large discrete action spaces.