Parallel implementation of neural networks

Abstract

This paper presents systematic methods, based on graph theoretic approach, for mapping of neural networks onto mesh connected SIMD arrays. The methods are applicable to a large class of multilayer network models, which can be represented in terms of sparse matrix vector operations. The class of computers, that the mappings are suitable for, encompasses most of the experimental and commercial mesh-connected SIMD arrays of processors. There are three methods described in the paper, one for the case of a processor array, which is larger or equal to the network size and two for the partitioned case, i.e. array smaller than the input data size. The methods are illustrated on an example of a multilayer perceptron with back-propagation learning, which consists ofn nuerons ande synaptic connections. For the first method, the processor array is assumed to be of sizeN×N, whereN2?n+e, and the required local memory of processors is limited to only a few registers. The implementation of a single iteration of a recall phase according to the method requires 24(N-1) shifts. For this method we have developed a software tool, which generates a sequence of pseudo instructions, such as elemental data shift and arithmetic operations, that implement a given neural network on a given size processor array. For the two partitioned methods, the processor array is of sizeP×P, whereP2≤n+e, and the local memory in the processors is of sizeO(K). The faster of the two methods requiresO(N3/P3K) time for an iteration of the recall or learning phase.