Variations of Training Process in Vanilla Recurrent Neural Network Framework

Abstract

Recurrent neural networks (RNNs) are capable of learning features and long term dependencies from sequential and time series data and show outstanding performance in sequential modeling tasks. However, training process in RNNs is troubled by issues in learning processes such as slow inference, vanishing gradients and difficulties in capturing long term dependencies. In this paper, we introduce a new learning technique to update the weight set as we change the input sequence which is shifted by certain amount of time in training process, instead of using a traditional way to calculate one set of the weights and bias in training time series with sequences shifted by certain amount of time series. We also consider an algorithm for an evaluation process. In the traditional way, the evaluation process is executed by using final weights and biases calculated in the training process. Instead, during the testing process, the weights and biases are iteratively updated in each sequence as done in the training process. Several numerical experiments demonstrate the efficiency of the proposed techniques.