Convergence of Gradient Descent Algorithm for Diagonal Recurrent Neural Networks

Abstract

Recurrent neural networks have been used for analysis and prediction of time series. This paper is concerned with the convergence of the gradient descent algorithm for training the diagonal recurrent neural networks. The existing convergence results consider the online gradient training algorithm based on the assumption that a very large number of (or infinitely many in theory) training samples of the time series are available, and accordingly the stochastic process theory is used to establish some convergence results of probability nature. In this paper, we consider the case that only a small number of training samples of the time series are available such that the stochastic treatment of the problem is no longer appropriate. Instead, we use the offline gradient descent algorithm for training the diagonal recurrent neural network, and we accordingly prove some convergence results of deterministic nature. The monotonicity of the error function in the iteration is also guaranteed.