Recurrent attention unit: A new gated recurrent unit for long-term memory of important parts in sequential data

Abstract

Gated recurrent unit (GRU) is a variant of the recurrent neural network (RNN). It has been widely used in many applications, such as handwriting recognition and natural language processing. However, GRU can only memorize the sequential information, but lacks the capability of adaptively paying attention to important parts in the sequences. In this paper, we propose a novel RNN model, called recurrent attention unit (RAU), which can seamlessly integrate the attention mechanism into the interior of the GRU cell by adding an attention gate. The attention gate enhances the ability of RAU to remember long-term information and pay attention to important parts in the sequential data. Extensive experiments on adding problem, image classification, sentiment classification and language modeling show that RAU consistently outperforms GRU and other related models.