A Nearly Optimal Back-Propagation Learning Algorithm on a Bus-Based Architecture

Abstract

The back-propagation model is applied mainly to multiple layers of neurons with feed-forward connections. It applies the generalized delta learning rule to derive appropriate weights on connections from a set of training instances. In the past, a parallel back-propagation learning algorithm on a bus-based architecture with an infinite number of processors was proposed to speed up the learning of weights. In this paper, the parallel back-propagation learning algorithm is modified, so that it is applicable to any number of processors. The given training instances are equally partitioned and put on these processors. When learning starts, each processor searches its own training subset to update the weight matrix and to broadcast the new result. Both analytical and experimental results show that the average speed-up can reach nearly O(r) by r processors if r is much less than the number of training instances.