Image Augmentation Based Momentum Memory Intrinsic Reward for Sparse Reward Visual Scenes

Abstract

Many real-life tasks can be abstracted as sparse reward visual scenes, which can make it difficult for an agent to accomplish tasks accepting only images and sparse reward. To address this problem, we split it into two parts: visual representation and sparse reward, and propose our novel framework, called Image Augmentation based Momentum Memory Intrinsic Reward ( <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">IAMMIR</b> ), which combines self-supervised representation learning with intrinsic motivation. For visual representation, we acquire a representation driven by a combination of image-augmented forward dynamics and reward. To handle sparse reward, we design a new type of intrinsic reward called Momentum Memory Intrinsic Reward ( <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MMIR</b> ), which uses the difference between the outputs from the current model (online network) and the historical model (target network) to indicate the agent's state familiarity. We evaluate our method on a visual navigation task with sparse reward in Vizdoom and demonstrate that it achieves state-of-the-art performance in terms of sample efficiency. Our method is at least 2 times faster than existing methods and reaches a 100% success rate.