A recalling-enhanced recurrent neural network: Conjugate gradient learning algorithm and its convergence analysis

Abstract

Elman network is a classical recurrent neural network with an internal delay feedback. In this paper, we propose a recalling-enhanced recurrent neural network (RERNN) which has a selective memory property. In addition, an improved conjugate algorithm with generalized Armijo search technique that speeds up the convergence rate is used to train the RERNN model. Further enhancement performance is achieved with adaptive learning coefficients. Finally, we prove weak and strong convergence of the presented algorithm. In other words, as the number of training steps increases, the following has been established for RERNN: (1) the gradient norm of the error function with respect to the weight vectors converges to zero, (2) the weight sequence approaches a fixed optimal point. We have carried out a number of simulations to illustrate and verify the theoretical results that demonstrate the efficiency of the proposed algorithm.