Convergence of Online Gradient Method with Momentum for BP Neural Network

Abstract

The BP neural network, which uses the steepest descent method (gradient descent method) as the basic idea for learning, is often used to deal with approximation problems because of its strong nonlinear mapping ability. The gradient method with added momentum can improve the learning speed of BP neural network. We study the convergence of the online gradient method with momentum for two-layer BP neural network when the training samples are randomly arranged in each iteration. Choosing appropriate learning rates, and selecting momentum coefficients in an adaptive manner, we prove the weak and strong convergence theorems of the algorithm.