Escape from the slowdown area on training of back-propagation algorithm by using a jumping method

Abstract

The typical error back-propagation learning algorithm (BP) of neural networks with layered structure is shown to be useful in a wide range of practical applications. The slowdown of the change in the evaluation error function that occurs in the course of learning, however, has a large effect on the training period and on convergence. Slowdown seems to arise in the area where the partial derivative of the evaluation error function with respect to the network coefficient is very small. To avoid slowdown of training in the BP algorithm, this paper proposes to speed up training via jumping correction of the network coefficient. More precisely, the change of the network coefficient as a function of the number of training cycles is locally approximated by a second-order function and the coefficient is updated by jumping of that function. It is verified that slowdown in the BP algorithm can be avoided by applying the proposed method and that training converges at a high speed. The proposed method is compared with the existing prediction method, which is a speed-up technique for training, and the proposed method is shown to reduce training time to between one-third and one-half.