Parallel Back-propagation Algorithm Research Articles

Fuzzy Method and Neural Network Model Parallel Implementation of Multi-Layer Neural Network Based on Cloud Computing for Real Time Data Transmission in Large Offshore Platform

Abstract With the rapid development of electronic technology, network technology and cloud computing technology, the current data is increasing in the way of mass, has entered the era of big data. Based on cloud computing clusters, this paper proposes a novel method of parallel implementation of multilayered neural networks based on Map-Reduce. Namely in order to meet the requirements of big data processing, this paper presents an efficient mapping scheme for a fully connected multi-layered neural network, which is trained by using error back propagation (BP) algorithm based on Map-Reduce on cloud computing clusters (MRBP). The batch-training (or epoch-training) regimes are used by effective segmentation of samples on the clusters, and are adopted in the separated training method, weight summary to achieve convergence by iterating. For a parallel BP algorithm on the clusters and a serial BP algorithm on uniprocessor, the required time for implementing the algorithms is derived. The performance parameters, such as speed-up, optimal number and minimum of data nodes are evaluated for the parallel BP algorithm on the clusters. Experiment results demonstrate that the proposed parallel BP algorithm in this paper has better speed-up, faster convergence rate, less iterations than that of the existed algorithms.

Parallel implementation of multilayered neural networks based on Map-Reduce on cloud computing clusters

To meet the requirements of big data processing, this paper presents an efficient mapping scheme for a fully connected multilayered neural network, which is trained by using back-propagation (BP) algorithm based on Map-Reduce of cloud computing clusters. The batch-training (or epoch-training) regimes are used by effective segmentation of samples on the clusters, and are adopted in the separated training method, weight summary to achieve convergence by iterating. For a parallel BP algorithm on the clusters and a serial BP algorithm on an uniprocessor, the required time for implementing the algorithms is derived. The performance parameters, such as speedup, optimal number and minimum of data nodes are evaluated for the parallel BP algorithm on the clusters. Experiment results demonstrate that the proposed parallel BP algorithm in this paper has better speedup, faster convergence rate, less iterations than that of the existed algorithms.

Parallel implementation of back-propagation algorithm in networks of workstations

This work presents an efficient mapping scheme for the multilayer perceptron (MLP) network trained using back-propagation (BP) algorithm on network of workstations (NOWs). Hybrid partitioning (HP) scheme is used to partition the network and each partition is mapped on to processors in NOWs. We derive the processing time and memory space required to implement the parallel BP algorithm in NOWs. The performance parameters like speed-up and space reduction factor are evaluated for the HP scheme and it is compared with earlier work involving vertical partitioning (VP) scheme for mapping the MLP on NOWs. The performance of the HP scheme is evaluated by solving optical character recognition (OCR) problem in a network of ALPHA machines. The analytical and experimental performance shows that the proposed parallel algorithm has better speed-up, less communication time, and better space reduction factor than the earlier algorithm. This work also presents a simple and efficient static mapping scheme on heterogeneous system. Using divisible load scheduling theory, a closed-form expression for number of neurons assigned to each processor in the NOW is obtained. Analytical and experimental results for static mapping problem on NOWs are also presented.

Mapping artificial neural networks to a tree shape neurocomputer

We present a parallel algorithm which trains artificial neural networks on the TUTNC (Tampere University of Technology NeuroComputer) parallel tree shape neurocomputer. The neurocomputer is designed for parallel computation and it is suitable for several artificial neural network models. Detailed mathematical notations are used to describe the parallel back-propagation algorithm. Performance analyses show that we can effectively utilize both broadcasting and global adding between processing units in the presented parallel implementation. By using the given algorithm it is possible to add more processing units to the system without any change in the number of communication transactions required.

Convergence properties of backpropagation for neural nets via theory of stochastic gradient methods. Part 1

We study here convergence properties of serial and parallel backpropagation algorithm for training of neural nets, as well as its modification with momentum term. It is shown that these algorithms can be put into the general framework of the stochastic gradient methods. This permits to consider from the same positions both stochastic and deterministic rules for the selection of components (training examples) of the error function to minimize at each iteration. We obtained weaker conditions on the stepsize for deterministic case and provide quite general synchronization rule for parallel version.