Convergence of learning algorithms with constant learning rates

Abstract

The behavior of neural network learning algorithms with a small, constant learning rate, epsilon, in stationary, random input environments is investigated. It is rigorously established that the sequence of weight estimates can be approximated by a certain ordinary differential equation, in the sense of weak convergence of random processes as epsilon tends to zero. As applications, backpropagation in feedforward architectures and some feature extraction algorithms are studied in more detail.