Simulation Of Spiking Neural Networks Research Articles

SWsnn: A Novel Simulator for Spiking Neural Networks.

Spiking neural network (SNN) simulators play an important role in neural system modeling and brain function research. They can help scientists reproduce and explore neuronal activities in brain regions, neuroscience, brain-like computing, and other fields and can also be applied to artificial intelligence, machine learning, and other fields. At present, many simulators using central processing unit (CPU) or graphics processing unit (GPU) have been developed. However, due to the randomness of connections between neurons and spiking events in SNN simulation, this causes a lot of memory access time. To alleviate this problem, we developed an SNN simulator SWsnn based on the new Sunway SW26010pro processor. The SW26010pro processor consists of six core groups, each with 16 MB of local data memory (LDM). LDM has the characteristics of high-speed read and write, which is suitable for performing simulation tasks similar to SNNs. Experimental results show that SWsnn runs faster than other mainstream GPU-based simulators when simulating a certain scale of neural network, showing a strong performance advantage. To conduct larger scale simulations, SWsnn designed a simulation computation based on a large shared model of Sunway processor and developed a multiprocessor version of SWsnn based on this mode, achieving larger scale SNN simulations.

48‐3: In‐Ga‐Zn‐O Synaptic Transistor with 1 µm Channel Length for Neuromorphic Computing

In this paper, indium‐gallium‐zinc‐oxide synaptic transistors with different channel lengths ranging from 2 μm to 1 μm were demonstrated. Channel scaling effect on synaptic behaviors such as long‐term depression and long‐term potentiation was investigated. A spiking neural network simulation was conducted to verify our synaptic transistor with a channel length of 1 μm, achieving high classification accuracy of 98.05%.

Synaptic Transistor Based on In‐Ga‐Zn‐O Channel and Trap Layers with Highly Linear Conductance Modulation for Neuromorphic Computing

AbstractBrain‐inspired neuromorphic computing has drawn significant attraction as a promising technology beyond von Neumann architecture by using the parallel structure of synapses and neurons. Various artificial synapse configurations and materials have been proposed to emulate synaptic behaviors for human brain functions such as memorizing, learning, and visual processing. Especially, the memory type indium‐gallium‐zinc‐oxide (IGZO) synaptic transistor adopting a charge trapping layer (CTL) has the advantages of high stability and a low leakage current of the IGZO channel. However, the CTL material should be carefully selected and optimized to overcome the low de‐trapping efficiency, resulting from difficulty in inducing holes in the IGZO channel. In this paper, IGZO is adopted as a CTL and found out that making it degenerated is crucial to improving de‐trapping efficiency. The degenerate CTL, where electrons remain as free electrons, induces Fowler‐Nordheim tunneling by increasing the electric field across the tunneling layer. As a result, the synaptic transistor represents a high linearity of potentiation (αp: −0.03) and depression (αd: −0.47) with 64 conductance levels, which enables the spiking neural network simulation to achieve high accuracy of 98.08%. These experimental results indicate that the synapse transistor can be one of the promising candidates for neuromorphic applications.

A Novel Structured Single Device Neuron for Low Standby Power and Compact System Application

We propose a single device neuron that utilizes a ferroelectric layer, a split gate, and a truncated floating gate structure. The proposed neuron device, named Single-Device Leaky-FeFET (SD L-FeFET), successfully emulates the neuronal dynamics for both excitatory and inhibitory connections while reducing the standby power by eliminating tail current. A spiking neural network (SNN) using the newly developed SD L-FeFET neuron shows MNIST handwriting digit pattern recognition of 92.5% and face recognition of 91.7%, which is comparable performance to the state-of-art SNN simulation results using conventional complex cell designs.

SPAYK: An environment for spiking neural network simulation

In research areas such as mobile robotics and computer vision, energy and computational efficiency have become critical. This has greatly increased interest in high-efficiency neuromorphic hardware and spiking neural networks. Because neuromorphic hardware is not yet widely available, spiking neural network studies are conducted by simulations. There are numerous simulators available today, each designed for a specific purpose. In this paper, a novel and open source package (SPAYK) for simulating spiking neural networks is presented. SPAYK has been proposed to speed up spiking neural network research. In the majority of simulators, networks are expressed with differential equations and require advanced neuroscience knowledge since such simulators are generally designed for brain and neuroscience research. SPAYK, on the other hand, is specifically designed as a framework to easily design spiking neural networks for practical problems. SPAYK is an easy-to-use Python package. There are three fundamental classes in the core: the model class for creating neuron groups, the organization class for simulating tissues, and the learning class for synaptic plasticity. While developing and testing the SPAYK environment, various experiments were carried out. This study includes three of these experiments. In the first experiment, we investigated the behavior of a group of Izhikevich neurons for visual stimuli. Also, a single Izhikevich neuron has been trained to respond to a particular label in a supervised manner with synaptic plasticity. In the second experiment, a well-known experiment was repeated to validate SPAYK. In this experiment, a neuron trained by synaptic plasticity can recognize repetitive patterns in a spike train. In the third experiment, a similar neuron was simulated with stimuli with multiple labels adapted from the MNIST dataset. It has been shown that the neuron can classify a particular label by synaptic plasticity. All these experiments and the SPAYK environment are presented as open-source tools.

Efficient parameter calibration and real-time simulation of large-scale spiking neural networks with GeNN and NEST.

Spiking neural networks (SNNs) represent the state-of-the-art approach to the biologically realistic modeling of nervous system function. The systematic calibration for multiple free model parameters is necessary to achieve robust network function and demands high computing power and large memory resources. Special requirements arise from closed-loop model simulation in virtual environments and from real-time simulation in robotic application. Here, we compare two complementary approaches to efficient large-scale and real-time SNN simulation. The widely used NEural Simulation Tool (NEST) parallelizes simulation across multiple CPU cores. The GPU-enhanced Neural Network (GeNN) simulator uses the highly parallel GPU-based architecture to gain simulation speed. We quantify fixed and variable simulation costs on single machines with different hardware configurations. As a benchmark model, we use a spiking cortical attractor network with a topology of densely connected excitatory and inhibitory neuron clusters with homogeneous or distributed synaptic time constants and in comparison to the random balanced network. We show that simulation time scales linearly with the simulated biological model time and, for large networks, approximately linearly with the model size as dominated by the number of synaptic connections. Additional fixed costs with GeNN are almost independent of model size, while fixed costs with NEST increase linearly with model size. We demonstrate how GeNN can be used for simulating networks with up to 3.5 · 106 neurons (> 3 · 1012synapses) on a high-end GPU, and up to 250, 000 neurons (25 · 109 synapses) on a low-cost GPU. Real-time simulation was achieved for networks with 100, 000 neurons. Network calibration and parameter grid search can be efficiently achieved using batch processing. We discuss the advantages and disadvantages of both approaches for different use cases.

Brian2CUDA: Flexible and Efficient Simulation of Spiking Neural Network Models on GPUs.

Graphics processing units (GPUs) are widely available and have been used with great success to accelerate scientific computing in the last decade. These advances, however, are often not available to researchers interested in simulating spiking neural networks, but lacking the technical knowledge to write the necessary low-level code. Writing low-level code is not necessary when using the popular Brian simulator, which provides a framework to generate efficient CPU code from high-level model definitions in Python. Here, we present Brian2CUDA, an open-source software that extends the Brian simulator with a GPU backend. Our implementation generates efficient code for the numerical integration of neuronal states and for the propagation of synaptic events on GPUs, making use of their massively parallel arithmetic capabilities. We benchmark the performance improvements of our software for several model types and find that it can accelerate simulations by up to three orders of magnitude compared to Brian's CPU backend. Currently, Brian2CUDA is the only package that supports Brian's full feature set on GPUs, including arbitrary neuron and synapse models, plasticity rules, and heterogeneous delays. When comparing its performance with Brian2GeNN, another GPU-based backend for the Brian simulator with fewer features, we find that Brian2CUDA gives comparable speedups, while being typically slower for small and faster for large networks. By combining the flexibility of the Brian simulator with the simulation speed of GPUs, Brian2CUDA enables researchers to efficiently simulate spiking neural networks with minimal effort and thereby makes the advancements of GPU computing available to a larger audience of neuroscientists.

EvtSNN: Event-driven SNN simulator optimized by population and pre-filtering.

Recently, spiking neural networks (SNNs) have been widely studied by researchers due to their biological interpretability and potential application of low power consumption. However, the traditional clock-driven simulators have the problem that the accuracy is limited by the time-step and the lateral inhibition failure. To address this issue, we introduce EvtSNN (Event SNN), a faster SNN event-driven simulator inspired by EDHA (Event-Driven High Accuracy). Two innovations are proposed to accelerate the calculation of event-driven neurons. Firstly, the intermediate results can be reused in population computing without repeated calculations. Secondly, unnecessary peak calculations will be skipped according to a condition. In the MNIST classification task, EvtSNN took 56 s to complete one epoch of unsupervised training and achieved 89.56% accuracy, while EDHA takes 642 s. In the benchmark experiments, the simulation speed of EvtSNN is 2.9–14.0 times that of EDHA under different network scales.

Toward the Optimal Design and FPGA Implementation of Spiking Neural Networks.

The performance of a biologically plausible spiking neural network (SNN) largely depends on the model parameters and neural dynamics. This article proposes a parameter optimization scheme for improving the performance of a biologically plausible SNN and a parallel on-field-programmable gate array (FPGA) online learning neuromorphic platform for the digital implementation based on two numerical methods, namely, the Euler and third-order Runge-Kutta (RK3) methods. The optimization scheme explores the impact of biological time constants on information transmission in the SNN and improves the convergence rate of the SNN on digit recognition with a suitable choice of the time constants. The parallel digital implementation leads to a significant speedup over software simulation on a general-purpose CPU. The parallel implementation with the Euler method enables around 180× ( 20× ) training (inference) speedup over a Pytorch-based SNN simulation on CPU. Moreover, compared with previous work, our parallel implementation shows more than 300× ( 240× ) improvement on speed and 180× ( 250× ) reduction in energy consumption for training (inference). In addition, due to the high-order accuracy, the RK3 method is demonstrated to gain 2× training speedup over the Euler method, which makes it suitable for online training in real-time applications.

Beyond LIF Neurons on Neuromorphic Hardware.

Neuromorphic systems aim to provide accelerated low-power simulation of Spiking Neural Networks (SNNs), typically featuring simple and efficient neuron models such as the Leaky Integrate-and-Fire (LIF) model. Biologically plausible neuron models developed by neuroscientists are largely ignored in neuromorphic computing due to their increased computational costs. This work bridges this gap through implementation and evaluation of a single compartment Hodgkin-Huxley (HH) neuron and a multi-compartment neuron incorporating dendritic computation on the SpiNNaker, and SpiNNaker2 prototype neuromorphic systems. Numerical accuracy of the model implementations is benchmarked against reference models in the NEURON simulation environment, with excellent agreement achieved by both the fixed- and floating-point SpiNNaker implementations. The computational cost is evaluated in terms of timing measurements profiling neural state updates. While the additional model complexity understandably increases computation times relative to LIF models, it was found a wallclock time increase of only 8× was observed for the HH neuron (11× for the mutlicompartment model), demonstrating the potential of hardware accelerators in the next-generation neuromorphic system to optimize implementation of complex neuron models. The benefits of models directly corresponding to biophysiological data are demonstrated: HH neurons are able to express a range of output behaviors not captured by LIF neurons; and the dendritic compartment provides the first implementation of a spiking multi-compartment neuron model with XOR-solving capabilities on neuromorphic hardware. The work paves the way for inclusion of more biologically representative neuron models in neuromorphic systems, and showcases the benefits of hardware accelerators included in the next-generation SpiNNaker2 architecture.

Fast Simulation of a Multi-Area Spiking Network Model of Macaque Cortex on an MPI-GPU Cluster.

Spiking neural network models are increasingly establishing themselves as an effective tool for simulating the dynamics of neuronal populations and for understanding the relationship between these dynamics and brain function. Furthermore, the continuous development of parallel computing technologies and the growing availability of computational resources are leading to an era of large-scale simulations capable of describing regions of the brain of ever larger dimensions at increasing detail. Recently, the possibility to use MPI-based parallel codes on GPU-equipped clusters to run such complex simulations has emerged, opening up novel paths to further speed-ups. NEST GPU is a GPU library written in CUDA-C/C++ for large-scale simulations of spiking neural networks, which was recently extended with a novel algorithm for remote spike communication through MPI on a GPU cluster. In this work we evaluate its performance on the simulation of a multi-area model of macaque vision-related cortex, made up of about 4 million neurons and 24 billion synapses and representing 32 mm2 surface area of the macaque cortex. The outcome of the simulations is compared against that obtained using the well-known CPU-based spiking neural network simulator NEST on a high-performance computing cluster. The results show not only an optimal match with the NEST statistical measures of the neural activity in terms of three informative distributions, but also remarkable achievements in terms of simulation time per second of biological activity. Indeed, NEST GPU was able to simulate a second of biological time of the full-scale macaque cortex model in its metastable state 3.1× faster than NEST using 32 compute nodes equipped with an NVIDIA V100 GPU each. Using the same configuration, the ground state of the full-scale macaque cortex model was simulated 2.4× faster than NEST.

Deploying and Optimizing Embodied Simulations of Large-Scale Spiking Neural Networks on HPC Infrastructure.

Simulating the brain-body-environment trinity in closed loop is an attractive proposal to investigate how perception, motor activity and interactions with the environment shape brain activity, and vice versa. The relevance of this embodied approach, however, hinges entirely on the modeled complexity of the various simulated phenomena. In this article, we introduce a software framework that is capable of simulating large-scale, biologically realistic networks of spiking neurons embodied in a biomechanically accurate musculoskeletal system that interacts with a physically realistic virtual environment. We deploy this framework on the high performance computing resources of the EBRAINS research infrastructure and we investigate the scaling performance by distributing computation across an increasing number of interconnected compute nodes. Our architecture is based on requested compute nodes as well as persistent virtual machines; this provides a high-performance simulation environment that is accessible to multi-domain users without expert knowledge, with a view to enable users to instantiate and control simulations at custom scale via a web-based graphical user interface. Our simulation environment, entirely open source, is based on the Neurorobotics Platform developed in the context of the Human Brain Project, and the NEST simulator. We characterize the capabilities of our parallelized architecture for large-scale embodied brain simulations through two benchmark experiments, by investigating the effects of scaling compute resources on performance defined in terms of experiment runtime, brain instantiation and simulation time. The first benchmark is based on a large-scale balanced network, while the second one is a multi-region embodied brain simulation consisting of more than a million neurons and a billion synapses. Both benchmarks clearly show how scaling compute resources improves the aforementioned performance metrics in a near-linear fashion. The second benchmark in particular is indicative of both the potential and limitations of a highly distributed simulation in terms of a trade-off between computation speed and resource cost. Our simulation architecture is being prepared to be accessible for everyone as an EBRAINS service, thereby offering a community-wide tool with a unique workflow that should provide momentum to the investigation of closed-loop embodiment within the computational neuroscience community.

SNN Simulation Performance Prediction: A Nonempirical Method

As a third generation artificial neural network, spiking neuron network is expected to expand the artificial intelligence world. However, as a more detailed simulation of brain, a single run of spiking neural networks (SNNs) simulation can take hours to days. To get a better prediction of SNN simulation performance, existing work requires gathering result of actual runs to conduct accurate modeling. In this paper, we propose a nonempirical SNN simulation performance prediction method, prototyped in a hybrid CPU-FPGA cluster. Experiments show that our method, without actual simulation run, can get comparable accuracy with orders of magnitude less runtime cost.

The bioinspired traffic sign classifier

Abstract Objectives In this paper the research on developing convolutional spiking neural networks for traffic signs classification is presented. Unlike classical ones, spiking networks reflect the behaviour of biological neurons much more closely, by taking into account the time dimension and event-based operation. Spiking networks running on dedicated neuromorphic platforms, such as Intel Loihi, can operate with greater energy efficiency, hence they are an interesting approach for embedded solutions. Methods For convolutional spiking neural networks' design and simulation, Nengo and NengoDL libraries for Python language were used. Numerous experiments using the Leaky-Integrate-and-Fire (LIF) neuron model were conducted. The training results, with different augmentation methods and number of time steps for input image presentation were compared. Results Finally, an accuracy of up to 97% on the test set was achieved, depending on the number of time steps the input was presented to the SNN. Conclusions The proposed experiments show that using simple convolutional spiking neural network, one can achieve accuracy comparable to the classical network with the same architecture and trained on the same dataset. At the same time, running on dedicated neuromorphic hardware, such solution should be characterized by low latency and low energy consumption.

Homeostatic control of synaptic rewiring in recurrent networks induces the formation of stable memory engrams.

Brain networks store new memories using functional and structural synaptic plasticity. Memory formation is generally attributed to Hebbian plasticity, while homeostatic plasticity is thought to have an ancillary role in stabilizing network dynamics. Here we report that homeostatic plasticity alone can also lead to the formation of stable memories. We analyze this phenomenon using a new theory of network remodeling, combined with numerical simulations of recurrent spiking neural networks that exhibit structural plasticity based on firing rate homeostasis. These networks are able to store repeatedly presented patterns and recall them upon the presentation of incomplete cues. Storage is fast, governed by the homeostatic drift. In contrast, forgetting is slow, driven by a diffusion process. Joint stimulation of neurons induces the growth of associative connections between them, leading to the formation of memory engrams. These memories are stored in a distributed fashion throughout connectivity matrix, and individual synaptic connections have only a small influence. Although memory-specific connections are increased in number, the total number of inputs and outputs of neurons undergo only small changes during stimulation. We find that homeostatic structural plasticity induces a specific type of “silent memories”, different from conventional attractor states.

EDHA: Event-Driven High Accurate Simulator for Spike Neural Networks

In recent years, spiking neural networks (SNNs) have attracted increasingly more researchers to study by virtue of its bio-interpretability and low-power computing. The SNN simulator is an essential tool to accomplish image classification, recognition, speech recognition, and other tasks using SNN. However, most of the existing simulators for spike neural networks are clock-driven, which has two main problems. First, the calculation result is affected by time slice, which obviously shows that when the calculation accuracy is low, the calculation speed is fast, but when the calculation accuracy is high, the calculation speed is unacceptable. The other is the failure of lateral inhibition, which severely affects SNN learning. In order to solve these problems, an event-driven high accurate simulator named EDHA (Event-Driven High Accuracy) for spike neural networks is proposed in this paper. EDHA takes full advantage of the event-driven characteristics of SNN and only calculates when a spike is generated, which is independent of the time slice. Compared with previous SNN simulators, EDHA is completely event-driven, which reduces a large amount of calculations and achieves higher computational accuracy. The calculation speed of EDHA in the MNIST classification task is more than 10 times faster than that of mainstream clock-driven simulators. By optimizing the spike encoding method, the former can even achieve more than 100 times faster than the latter. Due to the cross-platform characteristics of Java, EDHA can run on x86, amd64, ARM, and other platforms that support Java.

Robust neuromorphic coupled oscillators for adaptive pacemakers

Neural coupled oscillators are a useful building block in numerous models and applications. They were analyzed extensively in theoretical studies and more recently in biologically realistic simulations of spiking neural networks. The advent of mixed-signal analog/digital neuromorphic electronic circuits provides new means for implementing neural coupled oscillators on compact, low-power, spiking neural network hardware platforms. However, their implementation on this noisy, low-precision and inhomogeneous computing substrate raises new challenges with regards to stability and controllability. In this work, we present a robust, spiking neural network model of neural coupled oscillators and validate it with an implementation on a mixed-signal neuromorphic processor. We demonstrate its robustness showing how to reliably control and modulate the oscillator’s frequency and phase shift, despite the variability of the silicon synapse and neuron properties. We show how this ultra-low power neural processing system can be used to build an adaptive cardiac pacemaker modulating the heart rate with respect to the respiration phases and compare it with surface ECG and respiratory signal recordings from dogs at rest. The implementation of our model in neuromorphic electronic hardware shows its robustness on a highly variable substrate and extends the toolbox for applications requiring rhythmic outputs such as pacemakers.

A spiking central pattern generator for the control of a simulated lamprey robot running on SpiNNaker and Loihi neuromorphic boards

Central pattern generator (CPG) models have long been used to investigate both the neural mechanisms that underlie animal locomotion, as well as for robotic research. In this work we propose a spiking central pattern generator (SCPG) neural network and its implementation on neuromorphic hardware as a means to control a simulated lamprey model. To construct our SCPG model, we employ the naturally emerging dynamical systems that arise through the use of recurrent neural populations in the neural engineering framework (NEF). We define the mathematical formulation behind our model, which consists of a system of coupled abstract oscillators modulated by high-level signals, capable of producing a variety of output gaits. We show that with this mathematical formulation of the CPG model, the model can be turned into a spiking neural network (SNN) that can be easily simulated with Nengo, an SNN simulator. The SCPG model is then used to produce the swimming gaits of a simulated lamprey robot model in various scenarios. We show that by modifying the input to the network, which can be provided by sensory information, the robot can be controlled dynamically in direction and pace. The proposed methodology can be generalized to other types of CPGs suitable for both engineering applications and scientific research. We test our system on two neuromorphic platforms, SpiNNaker and Loihi. Finally, we show that this category of spiking algorithms displays a promising potential to exploit the theoretical advantages of neuromorphic hardware in terms of energy efficiency and computational speed.

Spiking neural networks and hippocampal function: A web-accessible survey of simulations, modeling methods, and underlying theories

Computational modeling has contributed to hippocampal research in a wide variety of ways and through a large diversity of approaches, reflecting the many advanced cognitive roles of this brain region. The intensively studied neuron type circuitry of the hippocampus is a particularly conducive substrate for spiking neural models. Here we present an online knowledge base of spiking neural network simulations of hippocampal functions. First, we overview theories involving the hippocampal formation in subjects such as spatial representation, learning, and memory. Then we describe an original literature mining process to organize published reports in various key aspects, including: (i) subject area (e.g., navigation, pattern completion, epilepsy); (ii) level of modeling detail (Hodgkin-Huxley, integrate-and-fire, etc.); and (iii) theoretical framework (attractor dynamics, oscillatory interference, self-organizing maps, and others). Moreover, every peer-reviewed publication is also annotated to indicate the specific neuron types represented in the network simulation, establishing a direct link with the Hippocampome.org portal. The web interface of the knowledge base enables dynamic content browsing and advanced searches, and consistently presents evidence supporting every annotation. Moreover, users are given access to several types of statistical reports about the collection, a selection of which is summarized in this paper. This open access resource thus provides an interactive platform to survey spiking neural network models of hippocampal functions, compare available computational methods, and foster ideas for suitable new directions of research.

Reconfigurable MoS2 Memtransistors for Continuous Learning in Spiking Neural Networks

Artificial intelligence and machine learning are growing computing paradigms, but current algorithms incur undesirable energy costs on conventional hardware platforms, thus motivating the exploration of more efficient neuromorphic architectures. Toward this end, we introduce here a memtransistor with gate-tunable dynamic learning behavior. By fabricating memtransistors from monolayer MoS2 grown on sapphire, the relative importance of the vertical field effect from the gate is enhanced, thereby heightening reconfigurability of the device response. Inspired by biological systems, gate pulses are used to modulate potentiation and depression, resulting in diverse learning curves and simplified spike-timing-dependent plasticity that facilitate unsupervised learning in simulated spiking neural networks. This capability also enables continuous learning, which is a previously underexplored cognitive concept in neuromorphic computing. Overall, this work demonstrates that the reconfigurability of memtransistors provides unique hardware accelerator opportunities for energy efficient artificial intelligence and machine learning.