Spike Communication Research Articles

Enhancing the Robustness of Spiking Neural Networks with Stochastic Gating Mechanisms

Spiking neural networks (SNNs) exploit neural spikes to provide solutions for low-power intelligent applications on neuromorphic hardware. Although SNNs have high computational efficiency due to spiking communication, they still lack resistance to adversarial attacks and noise perturbations. In the brain, neuronal responses generally possess stochasticity induced by ion channels and synapses, while the role of stochasticity in computing tasks is poorly understood. Inspired by this, we elaborate a stochastic gating spiking neural model for layer-by-layer spike communication, introducing stochasticity to SNNs. Through theoretical analysis, our gating model can be viewed as a regularizer that prevents error amplification under attacks. Meanwhile, our work can explain the robustness of Poisson coding. Experimental results prove that our method can be used alone or with existing robust enhancement algorithms to improve SNN robustness and reduce SNN energy consumption. We hope our work will shed new light on the role of stochasticity in the computation of SNNs. Our code is available at https://github.com/DingJianhao/StoG-meets-SNN/.

MorphBungee: A 65-nm 7.2-mm2 27-μJ/image Digital Edge Neuromorphic Chip with On-Chip 802-frame/s Multi-Layer Spiking Neural Network Learning.

This paper presents a digital edge neuromorphic spiking neural network (SNN) processor chip for a variety of edge intelligent cognitive applications. This processor allows high-speed, high-accuracy and fully on-chip spike-timing-based multi-layer SNN learning. It is characteristic of hierarchical multi-core architecture, event-driven processing paradigm, meta-crossbar for efficient spike communication, and hybrid and reconfigurable parallelism. A prototype chip occupying an active silicon area of 7.2 mm2 was fabricated using a 65-nm 1P9M CMOS process. when running a 256-256-256-256-200 4-layer fully-connected SNN on downscaled 16 × 16 MNIST images. it typically achieved a high-speed throughput of 802 and 2270 frames/s for on-chip learning and inference, respectively, with a relatively low power dissipation of around 61 mW at a 100 MHz clock rate under a 1.0V core power supply, Our on-chip learning results in comparably high visual recognition accuracies of 96.06%, 83.38%, 84.53%, 99.22% and 100% on the MNIST, Fashion-MNIST, ETH-80, Yale-10 and ORL-10 datasets, respectively. In addition, we have successfully applied our neuromorphic chip to demonstrate high-resolution satellite cloud image segmentation and non-visual tasks including olfactory classification and textural news categorization. These results indicate that our neuromorphic chip is suitable for various intelligent edge systems under restricted cost, energy and latency budgets while requiring in-situ self-adaptative learning capability.

Boost event-driven tactile learning with location spiking neurons.

Tactile sensing is essential for a variety of daily tasks. Inspired by the event-driven nature and sparse spiking communication of the biological systems, recent advances in event-driven tactile sensors and Spiking Neural Networks (SNNs) spur the research in related fields. However, SNN-enabled event-driven tactile learning is still in its infancy due to the limited representation abilities of existing spiking neurons and high spatio-temporal complexity in the event-driven tactile data. In this paper, to improve the representation capability of existing spiking neurons, we propose a novel neuron model called "location spiking neuron," which enables us to extract features of event-based data in a novel way. Specifically, based on the classical Time Spike Response Model (TSRM), we develop the Location Spike Response Model (LSRM). In addition, based on the most commonly-used Time Leaky Integrate-and-Fire (TLIF) model, we develop the Location Leaky Integrate-and-Fire (LLIF) model. Moreover, to demonstrate the representation effectiveness of our proposed neurons and capture the complex spatio-temporal dependencies in the event-driven tactile data, we exploit the location spiking neurons to propose two hybrid models for event-driven tactile learning. Specifically, the first hybrid model combines a fully-connected SNN with TSRM neurons and a fully-connected SNN with LSRM neurons. And the second hybrid model fuses the spatial spiking graph neural network with TLIF neurons and the temporal spiking graph neural network with LLIF neurons. Extensive experiments demonstrate the significant improvements of our models over the state-of-the-art methods on event-driven tactile learning, including event-driven tactile object recognition and event-driven slip detection. Moreover, compared to the counterpart artificial neural networks (ANNs), our SNN models are 10× to 100× energy-efficient, which shows the superior energy efficiency of our models and may bring new opportunities to the spike-based learning community and neuromorphic engineering. Finally, we thoroughly examine the advantages and limitations of various spiking neurons and discuss the broad applicability and potential impact of this work on other spike-based learning applications.

Automated and parallelized spike collision tests to identify spike signal projections

SummaryThe spike collision test is a highly reliable technique to identify the axonal projection of a neuron recorded electrophysiologically for investigating functional spike information among brain areas. It is potentially applicable to more neuronal projections by combining multi-channel recording with optogenetic stimulation. Yet, it remains inefficient and laborious because an experimenter must visually select spikes in every channel and manually repeat spike collision tests for each neuron serially. Here, we automated spike collision tests for all channels in parallel (Multi-Linc analysis) in a multi-channel real-time processing system. The rat cortical neurons identified with this technique displayed physiological spike features consistent with excitatory projection neurons. Their antidromic spikes were similar in shape but slightly larger in amplitude compared with spontaneous spikes. In addition, we demonstrated simultaneous identification of reciprocal or bifurcating projections among cortical areas. Thus, our Multi-Linc analysis will be a powerful approach to elucidate interareal spike communication.

Fast Simulation of a Multi-Area Spiking Network Model of Macaque Cortex on an MPI-GPU Cluster.

Spiking neural network models are increasingly establishing themselves as an effective tool for simulating the dynamics of neuronal populations and for understanding the relationship between these dynamics and brain function. Furthermore, the continuous development of parallel computing technologies and the growing availability of computational resources are leading to an era of large-scale simulations capable of describing regions of the brain of ever larger dimensions at increasing detail. Recently, the possibility to use MPI-based parallel codes on GPU-equipped clusters to run such complex simulations has emerged, opening up novel paths to further speed-ups. NEST GPU is a GPU library written in CUDA-C/C++ for large-scale simulations of spiking neural networks, which was recently extended with a novel algorithm for remote spike communication through MPI on a GPU cluster. In this work we evaluate its performance on the simulation of a multi-area model of macaque vision-related cortex, made up of about 4 million neurons and 24 billion synapses and representing 32 mm2 surface area of the macaque cortex. The outcome of the simulations is compared against that obtained using the well-known CPU-based spiking neural network simulator NEST on a high-performance computing cluster. The results show not only an optimal match with the NEST statistical measures of the neural activity in terms of three informative distributions, but also remarkable achievements in terms of simulation time per second of biological activity. Indeed, NEST GPU was able to simulate a second of biological time of the full-scale macaque cortex model in its metastable state 3.1× faster than NEST using 32 compute nodes equipped with an NVIDIA V100 GPU each. Using the same configuration, the ground state of the full-scale macaque cortex model was simulated 2.4× faster than NEST.

A Network Simulator for the Estimation of Bandwidth Load and Latency Created by Heterogeneous Spiking Neural Networks on Neuromorphic Computing Communication Networks

Accelerated simulations of biological neural networks are in demand to discover the principals of biological learning. Novel many-core simulation platforms, e.g., SpiNNaker, BrainScaleS and Neurogrid, allow one to study neuron behavior in the brain at an accelerated rate, with a high level of detail. However, they do not come anywhere near simulating the human brain. The massive amount of spike communication has turned out to be a bottleneck. We specifically developed a network simulator to analyze in high detail the network loads and latencies caused by different network topologies and communication protocols in neuromorphic computing communication networks. This simulator allows simulating the impacts of heterogeneous neural networks and evaluating neuron mapping algorithms, which is a unique feature among state-of-the-art network models and simulators. The simulator was cross-checked by comparing the results of a homogeneous neural network-based run with corresponding bandwidth load results from comparable works. Additionally, the increased level of detail achieved by the new simulator is presented. Then, we show the impact heterogeneous connectivity can have on the network load, first for a small-scale test case, and later for a large-scale test case, and how different neuron mapping algorithms can influence this effect. Finally, we look at the latency estimations performed by the simulator for different mapping algorithms, and the impact of the node size.

On the Design of a Fault-Tolerant Scalable Three Dimensional NoC-Based Digital Neuromorphic System With On-Chip Learning

Neuromorphic systems have shown improvements over the years, leveraging Spiking neural networks (SNN) event-driven nature to demonstrate low power consumption. As neuromorphic systems require high integration to form a functional silicon brain-like, moving to 3D integrated circuits (3D-ICs) with three-dimensional network on chip (3D-NoC) interconnect is a suitable approach that allows scalable design, shorter connections, and lower power consumption. However, highly dense neuromorphic systems also encounter the reliability issue where a single point of failure can affect the systems’operation. Because neuromorphic systems rely heavily on spike communication, an interruption or violation in the timing of spike communication can adversely affect the performance and accuracy of a neuromorphic system. This paper presents NASH, a a fault-tolerant 3D-NoC based neuromorphic system that incorporates as processing elements, lightweight spiking neuron processing cores (SNPCs) with spike-timing-dependent-plasticity (STDP) on-chip learning. Each SNPC houses 256 leaky integrate-and-fire (LIF) neurons and 65k synapses. Evaluation results on MNIST classification, using the fault-tolerant shortest-path K-means-based multicast routing algorithm (FTSP-KMCR), show that the NASH system can maintain high accuracy for up to 30% permanent fault in the interconnect with an acceptable area and power overheads when compared to other existing systems. a This project is supported by the University of Aizu, Competitive Research Funding (CRF), Ref. UoA-P6-2020

Run-time Mapping of Spiking Neural Networks to Neuromorphic Hardware

Neuromorphic architectures implement biological neurons and synapses to execute machine learning algorithms with spiking neurons and bio-inspired learning algorithms. These architectures are energy efficient and therefore, suitable for cognitive information processing on resource and power-constrained environments, ones where sensor and edge nodes of internet-of-things (IoT) operate. To map a spiking neural network (SNN) to a neuromorphic architecture, prior works have proposed design-time based solutions, where the SNN is first analyzed offline using representative data and then mapped to the hardware to optimize some objective functions such as minimizing spike communication or maximizing resource utilization. In many emerging applications, machine learning models may change based on the input using some online learning rules. In online learning, new connections may form or existing connections may disappear at run-time based on input excitation. Therefore, an already mapped SNN may need to be re-mapped to the neuromorphic hardware to ensure optimal performance. Unfortunately, due to the high computation time, design-time based approaches are not suitable for remapping a machine learning model at run-time after every learning epoch. In this paper, we propose a design methodology to partition and map the neurons and synapses of online learning SNN-based applications to neuromorphic architectures at run-time. Our design methodology operates in two steps – step 1 is a layer-wise greedy approach to partition SNNs into clusters of neurons and synapses incorporating the constraints of the neuromorphic architecture, and step 2 is a hill-climbing optimization algorithm that minimizes the total spikes communicated between clusters, improving energy consumption on the shared interconnect of the architecture. We conduct experiments to evaluate the feasibility of our algorithm using synthetic and realistic SNN-based applications. We demonstrate that our algorithm reduces SNN mapping time by an average 780x compared to a state-of-the-art design-time based SNN partitioning approach with only 6.25% lower solution quality.

A 4096-Neuron 1M-Synapse 3.8-pJ/SOP Spiking Neural Network With On-Chip STDP Learning and Sparse Weights in 10-nm FinFET CMOS

A reconfigurable 4096-neuron, 1M-synapse chip in 10-nm FinFET CMOS is developed to accelerate inference and learning for many classes of spiking neural networks (SNNs). The SNN features digital circuits for leaky integrate and fire neuron models, on-chip spike-timing-dependent plasticity (STDP) learning, and high-fan-out multicast spike communication. Structured fine-grained weight sparsity reduces synapse memory by up to 16 $\times $ with less than 2% overhead for storing connections. Approximate computing co-optimizes the dropping flow control and benefits from algorithmic noise to process spatiotemporal spike patterns with up to 9.4 $\times $ lower energy. The SNN achieves a peak throughput of 25.2 GSOP/s at 0.9 V, peak energy efficiency of 3.8 pJ/SOP at 525 mV, and 2.3- $\mu \text{W}$ /neuron operation at 450 mV. On-chip unsupervised STDP trains a spiking restricted Boltzmann machine to de-noise Modified National Institute of Standards and Technology (MNIST) digits and to reconstruct natural scene images with RMSE of 0.036. Near-threshold operation, in conjunction with temporal and spatial sparsity, reduces energy by $17.4\times $ to 1.0- $\mu \text{J}$ /classification in a $236 \times 20$ feed-forward network that is trained to classify MNIST digits using supervised STDP. A binary-activation multilayer perceptron with 50% sparse weights is trained offline with error backpropagation to classify MNIST digits with 97.9% accuracy at 1.7- $\mu \text{J}$ /classification.

A Review of Spiking Neuromorphic Hardware Communication Systems

Multiple neuromorphic systems use spiking neural networks (SNNs) to perform computation in a way that is inspired by concepts learned about the human brain. SNNs are artificial networks made up of neurons that fire a pulse, or spike, once the accumulated value of the inputs to the neuron exceeds a threshold. One of the most challenging parts of designing neuromorphic hardware is handling the vast degree of connectivity that neurons have with each other in the form of synaptic connections. This paper analyzes the neuromorphic systems Neurogrid, Braindrop, SpiNNaker, BrainScaleS, TrueNorth, Loihi, Darwin, and Dynap-SEL; and discusses the design of large scale spiking communication networks used in such systems. In particular, this paper looks at how each of these systems solved the challenges of forming packets with spiking information and how these packets are routed within the system. The routing of packets is analyzed at two scales: How the packets should be routed when traveling a short distance, and how the packets should be routed over longer global connections. Additional topics, such as the use of asynchronous circuits, robustness in communication, connection with a host machine, and network synchronization are also covered.

Unsupervised speech recognition through spike-timing-dependent plasticity in a convolutional spiking neural network.

Speech recognition (SR) has been improved significantly by artificial neural networks (ANNs), but ANNs have the drawbacks of biologically implausibility and excessive power consumption because of the nonlocal transfer of real-valued errors and weights. While spiking neural networks (SNNs) have the potential to solve these drawbacks of ANNs due to their efficient spike communication and their natural way to utilize kinds of synaptic plasticity rules found in brain for weight modification. However, existing SNN models for SR either had bad performance, or were trained in biologically implausible ways. In this paper, we present a biologically inspired convolutional SNN model for SR. The network adopts the time-to-first-spike coding scheme for fast and efficient information processing. A biological learning rule, spike-timing-dependent plasticity (STDP), is used to adjust the synaptic weights of convolutional neurons to form receptive fields in an unsupervised way. In the convolutional structure, the strategy of local weight sharing is introduced and could lead to better feature extraction of speech signals than global weight sharing. We first evaluated the SNN model with a linear support vector machine (SVM) on the TIDIGITS dataset and it got the performance of 97.5%, comparable to the best results of ANNs. Deep analysis on network outputs showed that, not only are the output data more linearly separable, but they also have fewer dimensions and become sparse. To further confirm the validity of our model, we trained it on a more difficult recognition task based on the TIMIT dataset, and it got a high performance of 93.8%. Moreover, a linear spike-based classifier—tempotron—can also achieve high accuracies very close to that of SVM on both the two tasks. These demonstrate that an STDP-based convolutional SNN model equipped with local weight sharing and temporal coding is capable of solving the SR task accurately and efficiently.

Controlled Propagation of Spiking Dynamics in Vertical-Cavity Surface-Emitting Lasers: Towards Neuromorphic Photonic Networks

We report experimentally and in theory on the controllable propagation of spiking regimes between two interlinked vertical-cavity surface-emitting lasers (VCSELs). We show that spiking patterns generated in a first transmitter VCSEL (T-VCSEL) are communicated to a second receiver VCSEL (R-VCSEL), which responds by firing the same spiking response. Importantly, the spiking regimes from both devices had analogous temporal and amplitude characteristics, including equal number of spikes fired, same spike and interspike temporal durations, and similar spike intensity properties. These responses are analogous to the spiking communication patterns of biological neurons yet at subnanosecond speeds, this is several (up to 8) orders of magnitude faster than the timescales of biological neurons. We have also carried out numerical simulations reproducing with high degree of agreement the experimental findings. These results obtained with inexpensive, commercially available VCSELs operating at important telecom wavelengths (1300 nm) offer great prospects for the scaling of emerging VCSEL-based photonic neuronal models into network configurations for use in novel neuromorphic photonic systems. This offers high potentials for nontraditional computing paradigms beyond digital systems and able to operate at ultrafast speeds.

Supervised Learning Based on Temporal Coding in Spiking Neural Networks.

Gradient descent training techniques are remarkably successful in training analog-valued artificial neural networks (ANNs). Such training techniques, however, do not transfer easily to spiking networks due to the spike generation hard nonlinearity and the discrete nature of spike communication. We show that in a feedforward spiking network that uses a temporal coding scheme where information is encoded in spike times instead of spike rates, the network input-output relation is differentiable almost everywhere. Moreover, this relation is piecewise linear after a transformation of variables. Methods for training ANNs thus carry directly to the training of such spiking networks as we show when training on the permutation invariant MNIST task. In contrast to rate-based spiking networks that are often used to approximate the behavior of ANNs, the networks we present spike much more sparsely and their behavior cannot be directly approximated by conventional ANNs. Our results highlight a new approach for controlling the behavior of spiking networks with realistic temporal dynamics, opening up the potential for using these networks to process spike patterns with complex temporal information.

Fixed latency on-chip interconnect for hardware spiking neural network architectures

Information in a Spiking Neural Network (SNN) is encoded as the relative timing between spikes. Distortion in spike timings can impact the accuracy of SNN operation by modifying the precise firing time of neurons within the SNN. Maintaining the integrity of spike timings is crucial for reliable operation of SNN applications. A packet switched Network on Chip (NoC) infrastructure offers scalable connectivity for spike communication in hardware SNN architectures. However, shared resources in NoC architectures can result in unwanted variation in spike packet transfer latency. This packet latency jitter distorts the timing information conveyed on the synaptic connections in the SNN, resulting in unreliable application behaviour. This paper presents a SystemC simulation based analysis of the synaptic information distortion in NoC based hardware SNNs. The paper proposes a fixed spike transfer latency ring topology interconnect for spike communication between neural tiles, using a novel timestamped spike broadcast flow control scheme. The proposed architectural technique is evaluated using spike rates employed in previously reported mesh topology NoC based hardware SNN applications, which exhibited spike latency jitter over NoC paths. Results indicate that the proposed interconnect offers fixed spike transfer latency and eliminates the associated information distortion. The paper presents the micro-architecture of the proposed ring router. The FPGA validated ring interconnect architecture has been synthesised using 65nm low-power CMOS technology. Silicon area comparisons for various ring sizes are presented. Scalability of the proposed architecture has been addressed by employing a hierarchical NoC architecture.

A Learning-Enabled Neuron Array IC Based Upon Transistor Channel Models of Biological Phenomena

We present a single-chip array of 100 biologically-based electronic neuron models interconnected to each other and the outside environment through 30,000 synapses. The chip was fabricated in a standard 350 nm CMOS IC process. Our approach used dense circuit models of synaptic behavior, including biological computation and learning, as well as transistor channel models. We use Address-Event Representation (AER) spike communication for inputs and outputs to this IC. We present the IC architecture and infrastructure, including IC chip, configuration tools, and testing platform. We present measurement of small network of neurons, measurement of STDP neuron dynamics, and measurement from a compiled spiking neuron WTA topology, all compiled into this IC.

Dynamical System Guided Mapping of Quantitative Neuronal Models Onto Neuromorphic Hardware

We present an approach to map neuronal models onto neuromorphic hardware using mathematical insights from dynamical system theory. Quantitatively accurate mappings are important for neuromorphic systems to both leverage and extend existing theoretical and numerical cortical modeling results. In the present study, we first calibrate the on-chip bias generators on our custom hardware. Then, taking advantage of the hardware's high-throughput spike communication, we rapidly estimate key mapping parameters with a set of linear relationships for static inputs derived from dynamical system theory. We apply this mapping procedure to three different chips, and show close matching to the neuronal model and between chips-the Jenson-Shannon divergence was reduced to at least one tenth that of the shuffled control. We confirm that our mapping procedure generalizes to dynamic inputs: Silicon neurons match spike timings of a simulated neuron with a standard deviation of 3.4% of the average inter-spike interval.

Bistability induces episodic spike communication by inhibitory neurons in neuronal networks

Bistability is one of the important features of nonlinear dynamical systems. In neurodynamics, bistability has been found in basic Hodgkin-Huxley equations describing the cell membrane dynamics. When the neuron is clamped near its threshold, the stable rest potential may coexist with the stable limit cycle describing periodic spiking. However, this effect is often neglected in network computations where the neurons are typically reduced to threshold firing units (e.g., integrate-and-fire models). We found that the bistability may induce spike communication by inhibitory coupled neurons in the spiking network. The communication is realized in the form of episodic discharges with synchronous (correlated) spikes during the episodes. A spiking phase map is constructed to describe the synchronization and to estimate basic spike phase locking modes.

Efficient spike communication in the MUSIC multi-simulation framework

MUSIC is a standard API and software library allowing large-scale neuronal network simulators, or other applications, to exchange data within a parallel computer during runtime [1]. It promotes interoperability between models written for different simulators and allows models to be re-used to build a larger model system, a multi-simulation. In addition, it allows for independent development of pre- and post-processing tools, for example for scientific visualization. A prototype implementation of the API in the form of a C++ library was released under the GPL license in early 2009. The simulators NEST, Neuron and MOOSE are adapted for the MUSIC library. Work in progress includes extension of MUSIC to support multi-scale multi-simulations [2] and connecting MUSIC-enabled applications to robotic hardware [3]. In the MUSIC prototype implementation, spikes are communicated using pair-wise MPI_Send/MPI_Recv. While this scales well for one-to-one-like connectivity, there are cases where such a communication scheme might not use machine resources optimally, e.g. for all-to-all-like connectivity in a simulation running on a large number of MPI processes. Here we report on recent developments of the library providing an alternative communication algorithm based on collective MPI communication. We present benchmark results comparing the scaling of the two communication algorithms for different topologies of connectivity. In order to put maximal stress on the MUSIC implementation, we used artificial neuronal populations lacking their own dynamics (no computation) so that spike communication could be studied in isolation. We ran a multi-simulation where a simulation A communicated spikes to a simulation B. The simulations A and B consisted of 100000 neurons each. Every neuron in B received spikes from 10000 randomly selected neurons in A, each spiking at a frequency of 10Hz. Buffered spikes were communicated every 5 ms. Figure ​Figure11 shows wall-clock time for 100 simulated seconds for the two algorithms on two computer architectures. For reference, the wall-clock time for 128 processors on the BG/L (165 s) is 1.4 % of the wall-clock time for simulation of 100 s of the symmetric multi-simulation benchmark in [1] which includes computation of the neuronal dynamics. The scaling of the pair-wise algorithm on the Cray system might be possible to improve by allocating nodes within short network distance. Figure 1 Scaling of multi-simulation with two connected populations. We describe two communication algorithms used in MUSIC and give a quantitative evaluation of their scaling performance. Preliminary results indicate that the new allgather-based algorithm scales better than the pair-wise algorithm for all-to-all connectivity. It also performs better than the pair-wise algorithm on the Cray XE6 with more than 100 processors.

A 32 GBit/s communication SoC for a waferscale neuromorphic system

State-of-the-art large-scale neuromorphic systems require a sophisticated, high-bandwidth communication infrastructure for the exchange of spike events between units of the neural network. These communication infrastructures are usually built around custom-designed FPGA systems. However, the overall bandwidth requirements and the integration density of very large neuromorphic systems necessitate a significantly more targeted approach, i.e. the development of dedicated integrated circuits. We present a VLSI realization of a neuromorphic communication system-on-chip (SoC) with a cumulative throughput of 32 GBit/s in 0.18 μ m CMOS, employing state-of-the-art circuit blocks. Several of these circuits exhibit improved performance compared to current literature, e.g. a priority queue with a speed of 31 Mkeys/s at 1.3 mW, or a 1 GHz PLL at 5 mW. The SoC contains additional neuromorphic functionality, such as configurable event delays and event ordering. The complete configuration of the neuromorphic system is also handled by the spike communication channels, in contrast to the separate channels required in the majority of current systems. At 865 Mevent/s, the SoC delivers at least a factor of eight more bandwidth than other current neuromorphic communication infrastructures.

Comparison of neuronal spike exchange methods on a Blue Gene/P supercomputer

For neural network simulations on parallel machines, interprocessor spike communication can be a significant portion of the total simulation time. The performance of several spike exchange methods using a Blue Gene/P (BG/P) supercomputer has been tested with 8–128 K cores using randomly connected networks of up to 32 M cells with 1 k connections per cell and 4 M cells with 10 k connections per cell, i.e., on the order of 4·1010 connections (K is 1024, M is 10242, and k is 1000). The spike exchange methods used are the standard Message Passing Interface (MPI) collective, MPI_Allgather, and several variants of the non-blocking Multisend method either implemented via non-blocking MPI_Isend, or exploiting the possibility of very low overhead direct memory access (DMA) communication available on the BG/P. In all cases, the worst performing method was that using MPI_Isend due to the high overhead of initiating a spike communication. The two best performing methods—the persistent Multisend method using the Record-Replay feature of the Deep Computing Messaging Framework DCMF_Multicast; and a two-phase multisend in which a DCMF_Multicast is used to first send to a subset of phase one destination cores, which then pass it on to their subset of phase two destination cores—had similar performance with very low overhead for the initiation of spike communication. Departure from ideal scaling for the Multisend methods is almost completely due to load imbalance caused by the large variation in number of cells that fire on each processor in the interval between synchronization. Spike exchange time itself is negligible since transmission overlaps with computation and is handled by a DMA controller. We conclude that ideal performance scaling will be ultimately limited by imbalance between incoming processor spikes between synchronization intervals. Thus, counterintuitively, maximization of load balance requires that the distribution of cells on processors should not reflect neural net architecture but be randomly distributed so that sets of cells which are burst firing together should be on different processors with their targets on as large a set of processors as possible.