Multiple Frequency Bands Temporal State Representation for Deep Reinforcement Learning

Abstract

Deep reinforcement learning has achieved significant success in solving sequential decision-making tasks. Excellent models usually require the input of valid state signals during training, which is challenging to encode temporal state features for the deep reinforcement learning model. To address this issue, recent methods attempt to encode multi-step sequential state signals so as to obtain more comprehensive observational information. However, these methods usually have a lower performance on complex continuous control tasks because mapping the state sequence into a low-dimensional embedding causes blurring of the immediate state features. In this paper, we propose a multiple frequency bands temporal state representation learning framework. The temporal state signals are decomposed into discrete state signals of various frequency bands by Discrete Fourier Transform (DFT). Then, feature signals filtered out different high-frequency bands are generated. Meanwhile, the mask generator evaluates the weights of signals of various frequency bands and encodes high-quality representations for agent training. Our intuition is that temporal state representations considering multiple frequency bands have high fidelity and stability. We conduct experiments tasks and verify that our method has obvious advantages over the baseline in complex continuous control tasks such as Walker and Crawler.