Acceleration of Levenberg-Marquardt training of neural networks with variable decay rate

Abstract

In the application of the standard Levenberg-Marquardt training process of neural networks, error oscillations are frequently observed and they usually aggravate on approaching the required accuracy. In this paper, a modified Levenberg-Marquardt method based on variable decay rate in each iteration is proposed in order to reduce such error oscillations. Through a certain variation of the decay rate, the time required for training of neural networks is cut down to less than half of that required in the standard Levenberg-Marquardt method. Several numerical examples are given to show the effectiveness of the proposed method.