Time‐varying learning rate for recurrent neural networks to solve linear equations

Abstract

Consider there exist many time‐varying factors during the recurrent neural networks training in piratical application engineering. In this paper, different from the time‐invariant case, a time‐varying learning rate is presented to accelerate the convergence speed for the recurrent neural networks (RNNs) when used to solve the linear simultaneous equations. Theoretical analysis shows that the neural networks with the time‐varying learning rate can globally converge to the theoretical solution; moreover, if the linear activation function is used, the model will be exponential convergent to the theoretical solution, and if the sign‐bi‐power activation function is used, the state solution by the neural networks will be convergent in finite time. Computer simulation results further substantiate that the time‐varying learning rate can be used for accelerating the convergence rate of the RNN models.