Study on fast speed fractional order gradient descent method and its application in neural networks

Abstract

This article introduces a novel fractional order gradient descent method for the quadratic loss function. Based on Riemann-Liouville definition, a more practical fractional order gradient descent method with variable initial value is proposed to ensure convergence to the actual extremum. On this basis, the random weight particle swarm optimization algorithm is introduced to select the appropriate initial value, which not only accelerates the convergence speed, but also enhances the global convergence ability of the algorithm. To avoid complicated problems of the chain rule in fractional calculus, the parameters of output layers is trained by the new designed method, while the parameters of hidden layers still use the conventional method. By selecting proper hyper-parameters, the proposed method shows faster convergence speed than others. Finally, numerical examples are given to verify that the proposed algorithm has fast convergence speed and high accuracy under a adequate large number of independent runs.