Training fully recurrent neural networks on a ring transputer array

Abstract

We describe the parallel implementation of fully recurrent neural networks (RRN) on a transputer-based multiprocessor system. To train the RNN, the real-time recurrent learning (RTRL) algorithm was used. The computationally intensive sequential RTRL algorithm has been transformed to an equivalent parallel algorithm, realized in a ring topology that can be matched to a variety of target architectures, ranging from application specific VLSI arrays to general purpose multiprocessor systems. A ring array of up to 19 T800 transputers was programmed to efficiently perform the parallel RTRL algorithm. The speedup of the transputer implementation was estimated both analytically and through simulations, and the effect of the communication overhead is discussed. It is shown that as more neuron units are allocated to the same processor the efficiency is increased.