A modified conjugate gradient-based Elman neural network

Abstract

Elman recurrent network is a representative model with feedback mechanism. Although gradient descent method has been widely used to train Elman network, it frequently leads to slow convergence. According to optimization theory, conjugate gradient method is an alternative strategy in searching the descent direction during training. In this paper, an efficient conjugate gradient method has been presented to reach the optimal solution in two ways: (1) constructing a more effective conjugate coefficient, (2) determining adaptive learning rates in terms of the generalized Armijo search method. Experiments show that the performance of the new algorithm is superior to traditional algorithms, such as gradient descent method and conjugate gradient method. In particular, the new algorithm has better performance than the evolutionary algorithm. Finally, we prove the weak and strong convergence of the presented algorithm, i.e., the gradient norm of the error function with respect to the weight vectors converges to zero and the weight sequence approaches a fixed optimal point.