Scalable Massively Parallel Artificial Neural Networks

Abstract

Artificial Neural Networks (ANN) can be very effective for pattern recognition, function approximation, scientific classification, control, and the analysis of time series data; however they can require very large training times for large networks. Once the network is trained for a particular problem, however, it can produce results in a very short time. Traditional ANNs using back-propagation algorithm do not scale well as each neuron in one level is fully connected to each neuron in the previous level. In the present work only the neurons at the edges of the domains were involved in communication, in order to reduce the communication costs and maintain scalability. Ghost neurons were created at these processor boundaries for information communication. An object-oriented, massively-parallel ANN software package SPANN (Scalable Parallel Artificial Neural Network) has been developed and is described here. MPI was used to parallelize the C++ code. The back-propagation algorithm was used to train the network. In preliminary tests, the software was used to identify character sets consisting of 48 characters and with increasing resolutions. The code correctly identified all the characters when adequate training was used in the network. The training of a problem sizewith2billionneuronweightsonanIBMBlueGene/Lcomputerusing1000dualPowerPC 440 processors required less than 30 minutes.Various comparisons in training time, forward propagation time, and error reduction were also made.