Applications of fractional gradient descent method with adaptive momentum in BP neural networks

Abstract

A novel fractional gradient descent method with adaptive momentum is presented in this paper to improve the convergence speed and stability for BP neural network training. The fractional Grünwald-Letnikov derivative is used for the fractional gradient. The coefficient of the momentum term is set as an adaptive variable, depending on the fractional gradient of the current step and the weight change of the previous step. We give a detailed convergence proof of the proposed method. Experiments on MNIST data sets and XOR problem demonstrate that the fractional gradient descent method with adaptive momentum term can effectively improve convergence speed, maintain stability of BP neural network training, help escape from local minimum points, and enlarge the selection range of the learning rate.