Attention augmentation with multi-residual in bidirectional LSTM

Abstract

Recurrent neural networks (RNNs) have been proven to be efficient in processing sequential data. However, the traditional RNNs have suffered from the gradient diminishing problem until the advent of Long Short-Term Memory (LSTM). However, LSTM is weak in capturing long-time dependency in sequential data due to the inadequacy of memory capacity in LSTM cells. To address this challenge, we propose an Attention-augmentation Bidirectional Multi-residual Recurrent Neural Network (ABMRNN) to overcome the deficiency. We propose an algorithm which integrates both past and future information at every time step with omniscient attention model. The multi-residual mechanism has also been leveraged in the proposed model targeting the pattern of the relationship between current time step and further distant time steps instead of only one previous time step. The results of experiments show that our model outperforms the traditional statistical classifiers and other existing RNN architectures.