Improving Deep Reinforcement Learning With Mirror Loss

Abstract

Recent years have witnessed the great breakthrough of deep reinforcement learning (DRL) in various AI applications, but the training process needs a very large amount of samples and huge computational costs. To alleviate the low sample efficiency issue, one feasible solution is to improve the state representation learning. We uncover that the agents, trained by the original DRL algorithm, face severe performance degradation in mirrored game environments. As mirror symmetry is an important property of the environment, the poor performance in the mirrored situation indicates that the agents are not fully aware of the essence of the environment. In order to handle this problem and make use of the property to attain better state representation, we propose a novel mirror loss, which serves as an auxiliary module to bring mirror symmetry representation to the DRL agent. It is model-agnostic and prompts the DRL agent to make logically consistent mirrored actions in the mirrored environment. We conduct experiments on OpenAI Gym Atari environments and a more complex RL task, Mahjong AI, and the results demonstrate the efficiency and versatility of our method.