Stochastic weight update in the backpropagation algorithm on feed-forward neural networks

Abstract

We will examine stochastic weight update in the backpropagation algorithm on feed-forward neural networks. It was introduced by Salvetti and Wilamowski in 1994 in order to improve probability of convergence and speed of convergence. However, this update method has also one another quality, its implementation is simple for arbitrary network topology. In stochastic weight update scenario, constant number of weights is randomly selected and updated. This is in contrast to classical ordered update, where always all weights are updated. We will describe exact implementation, and present example results on toy-task data with feed-forward neural network topology. Stochastic weight update is suitable to replace classical ordered update without any penalty on implementation complexity and with good chance without penalty on quality of convergence.