A normalised real time recurrent learning algorithm

Abstract

A real time recurrent learning (RTRL) algorithm with an adaptive-learning rate for nonlinear adaptive filters realised as fully connected recurrent neural networks (RNNs) is derived. The algorithm is obtained by minimising the instantaneous squared error at the output neuron for every time instant while the network is running. The algorithm normalises the learning rate with the L 2 norm of the external input vector and a measure of the gradients at the neurons within the network, and is hence referred to as the normalised RTRL (NRTRL) algorithm. Indeed, the algorithm degenerates into the normalised least mean square (NLMS) algorithm for a linear-single-neuron network. For a neuron with a contractive nonlinear activation function, the algorithm is shown to impose additional stability and faster convergence to the RTRL, without significant demands on additional computational complexity. The bounds imposed on the learning rate which preserve convergence of the algorithm are also provided.