Acceleration of the back propagation through dynamic adaptation of the learning rate

Abstract

Standard backpropagation, as with many gradient based optimization methods converges slowly as neural networks training problems become larger and more complex. In this paper, we present a new algorithm, dynamic adaptation of the learning rate to accelerate steepest descent. The underlying idea is to partition the iteration number domain into n intervals and a suitable value for the learning rate is assigned for each respective iteration interval. We present a derivation of the new algorithm and test the algorithm on several classification problems. As compared to standard backpropagation, the convergence rate can be improved immensely with only a minimal increase in the complexity of each iteration.