Networks Of Spiking Neurons Research Articles

Fast learning without synaptic plasticity in spiking neural networks

Spiking neural networks are of high current interest, both from the perspective of modelling neural networks of the brain and for porting their fast learning capability and energy efficiency into neuromorphic hardware. But so far we have not been able to reproduce fast learning capabilities of the brain in spiking neural networks. Biological data suggest that a synergy of synaptic plasticity on a slow time scale with network dynamics on a faster time scale is responsible for fast learning capabilities of the brain. We show here that a suitable orchestration of this synergy between synaptic plasticity and network dynamics does in fact reproduce fast learning capabilities of generic recurrent networks of spiking neurons. This points to the important role of recurrent connections in spiking networks, since these are necessary for enabling salient network dynamics. We show more specifically that the proposed synergy enables synaptic weights to encode more general information such as priors and task structures, since moment-to-moment processing of new information can be delegated to the network dynamics.

A robust balancing mechanism for spiking neural networks.

Dynamical balance of excitation and inhibition is usually invoked to explain the irregular low firing activity observed in the cortex. We propose a robust nonlinear balancing mechanism for a random network of spiking neurons, which works also in the absence of strong external currents. Biologically, the mechanism exploits the plasticity of excitatory-excitatory synapses induced by short-term depression. Mathematically, the nonlinear response of the synaptic activity is the key ingredient responsible for the emergence of a stable balanced regime. Our claim is supported by a simple self-consistent analysis accompanied by extensive simulations performed for increasing network sizes. The observed regime is essentially fluctuation driven and characterized by highly irregular spiking dynamics of all neurons.

A mean-field model of gamma-frequency oscillations in networks of excitatory and inhibitory neurons.

Gamma oscillations are widely seen in the cerebral cortex in different states of the wake-sleep cycle and are thought to play a role in sensory processing and cognition. Here, we study the emergence of gamma oscillations at two levels, in networks of spiking neurons, and a mean-field model. At the network level, we consider two different mechanisms to generate gamma oscillations and show that they are best seen if one takes into account the synaptic delay between neurons. At the mean-field level, we show that, by introducing delays, the mean-field can also produce gamma oscillations. The mean-field matches the mean activity of excitatory and inhibitory populations of the spiking network, as well as their oscillation frequencies, for both mechanisms. This mean-field model of gamma oscillations should be a useful tool to investigate large-scale interactions through gamma oscillations in the brain.

Efficient Estimation of Directed Connectivity in Nonlinear and Nonstationary Spiking Neuron Networks.

Studying directed connectivity within spiking neuron networks can help understand neural mechanisms. Existing methods assume linear time-invariant neural dynamics with a fixed time lag in information transmission, while spiking networks usually involve complex dynamics that are nonlinear and nonstationary, and have varying time lags. We develop a Gated Recurrent Unit (GRU)-Point Process (PP) method to estimate directed connectivity within spiking networks. We use a GRU to describe the dependency of the target neuron's current firing rate on the source neurons' past spiking events and a PP to relate the target neuron's firing rate to its current 0-1 spiking event. The GRU model uses recurrent states and gate/activation functions to deal with varying time lags, nonlinearity, and nonstationarity in a parameter-efficient manner. We estimate the model using maximum likelihood and compute directed information as our measure of directed connectivity. We conduct simulations using artificial spiking networks and a biophysical model of Parkinson's disease to show that GRU-PP systematically addresses varying time lags, nonlinearity, and nonstationarity, and estimates directed connectivity with high accuracy and data efficiency. We also use a non-human-primate dataset to show that GRU-PP correctly identifies the biophysically-plausible stronger PMd-to-M1 connectivity than M1-to-PMd connectivity during reaching. In all experiments, the GRU-PP consistently outperforms state-of-the-art methods. The GRU-PP method efficiently estimates directed connectivity in varying time lag, nonlinear, and nonstationary spiking neuron networks. The proposed method can serve as a directed connectivity analysis tool for investigating complex spiking neuron network dynamics.

Astrocyte control bursting mode of spiking neuron network with memristor-implemented plasticity

A mathematical model of a spiking neuron network accompanied by astrocytes is considered. The network is composed of excitatory and inhibitory neurons with synaptic connections supplied by a memristor-based model of plasticity. Another mechanism for changing the synaptic connections involves astrocytic regulations using the concept of tripartite synapses. In the absence of memristor-based plasticity, the connections between these neurons drive the network dynamics into a burst mode, as observed in many experimental neurobiological studies when investigating living networks in neuronal cultures. The memristive plasticity implementing synaptic plasticity in inhibitory synapses results in a shift in network dynamics toward an asynchronous mode. Next,it is found that accounting for astrocytic regulation in glutamatergic excitatory synapses enable the restoration of ‘normal’ burst dynamics. The conditions and parameters of such astrocytic regulation’s impact on burst dynamics established.

Research on handwritten digits recognition system based on spiking neuron network

In the 21st century, deep learning has revolutionized the fields of machine learning and computer science, attaining high accuracy in tasks such as image recognition. More layers and more parameters are stuffed into the network to achieve higher performance, making the network extremely large. A new, radically different approach was proposed to complete the tasks, such as image recognition, using a spiking neural network(SNN). The spiking neural network is event-driven rather than data-driven, which makes it more physiologically realistic and uses a lot less power. This study reviews the development of spiking neural networks and their differences from non-spiking neural networks, as well as the different encoding methods, neuronal models and update rules that have an impact on the performance of the network. It can be concluded that though SNNs can hardly achieve the same accuracy as artificial neural networks(ANN), the gap is narrowing. More strategies from ANN such as back-propagation and convolutional layers have been applied to SNNs, making it more accurate, stable and comprehensive.

Topological data analysis of the firings of a network of stochastic spiking neurons.

Topological data analysis is becoming more and more popular in recent years. It has found various applications in many different fields, for its convenience in analyzing and understanding the structure and dynamic of complex systems. We used topological data analysis to analyze the firings of a network of stochastic spiking neurons, which can be in a sub-critical, critical, or super-critical state depending on the value of the control parameter. We calculated several topological features regarding Betti curves and then analyzed the behaviors of these features, using them as inputs for machine learning to discriminate the three states of the network.

How synaptic function controls critical transitions in spiking neuron networks: insight from a Kuramoto model reduction.

The dynamics of synaptic interactions within spiking neuron networks play a fundamental role in shaping emergent collective behavior. This paper studies a finite-size network of quadratic integrate-and-fire neurons interconnected via a general synaptic function that accounts for synaptic dynamics and time delays. Through asymptotic analysis, we transform this integrate-and-fire network into the Kuramoto-Sakaguchi model, whose parameters are explicitly expressed via synaptic function characteristics. This reduction yields analytical conditions on synaptic activation rates and time delays determining whether the synaptic coupling is attractive or repulsive. Our analysis reveals alternating stability regions for synchronous and partially synchronous firing, dependent on slow synaptic activation and time delay. We also demonstrate that the reduced microscopic model predicts the emergence of synchronization, weakly stable cyclops states, and non-stationary regimes remarkably well in the original integrate-and-fire network and its theta neuron counterpart. Our reduction approach promises to open the door to rigorous analysis of rhythmogenesis in networks with synaptic adaptation and plasticity.

Bursting gamma oscillations in neural mass models.

Gamma oscillations (30-120 Hz) in the brain are not periodic cycles, but they typically appear in short-time windows, often called oscillatory bursts. While the origin of this bursting phenomenon is still unclear, some recent studies hypothesize its origin in the external or endogenous noise of neural networks. We demonstrate that an exact neural mass model of excitatory and inhibitory quadratic-integrate and fire-spiking neurons theoretically predicts the emergence of a different regime of intrinsic bursting gamma (IBG) oscillations without any noise source, a phenomenon due to collective chaos. This regime is indeed observed in the direct simulation of spiking neurons, characterized by highly irregular spiking activity. IBG oscillations are distinguished by higher phase-amplitude coupling to slower theta oscillations concerning noise-induced bursting oscillations, thus indicating an increased capacity for information transfer between brain regions. We demonstrate that this phenomenon is present in both globally coupled and sparse networks of spiking neurons. These results propose a new mechanism for gamma oscillatory activity, suggesting deterministic collective chaos as a good candidate for the origin of gamma bursts.

Short-term neuronal and synaptic plasticity act in synergy for deviance detection in spiking networks.

Sensory areas of cortex respond more strongly to infrequent stimuli when these violate previously established regularities, a phenomenon known as deviance detection (DD). Previous modeling work has mainly attempted to explain DD on the basis of synaptic plasticity. However, a large fraction of cortical neurons also exhibit firing rate adaptation, an underexplored potential mechanism. Here, we investigate DD in a spiking neuronal network model with two types of short-term plasticity, fast synaptic short-term depression (STD) and slower threshold adaptation (TA). We probe the model with an oddball stimulation paradigm and assess DD by evaluating the network responses. We find that TA is sufficient to elicit DD. It achieves this by habituating neurons near the stimulation site that respond earliest to the frequently presented standard stimulus (local fatigue), which diminishes the response and promotes the recovery (global fatigue) of the wider network. Further, we find a synergy effect between STD and TA, where they interact with each other to achieve greater DD than the sum of their individual effects. We show that this synergy is caused by the local fatigue added by STD, which inhibits the global response to the frequently presented stimulus, allowing greater recovery of TA-mediated global fatigue and making the network more responsive to the deviant stimulus. Finally, we show that the magnitude of DD strongly depends on the timescale of stimulation. We conclude that highly predictable information can be encoded in strong local fatigue, which allows greater global recovery and subsequent heightened sensitivity for DD.

Steps toward a homogenization procedure for spiking neural P systems

A spiking neural P system (SN P system) is a model of computation inspired by the mechanism of a network of spiking neurons. A homogeneous SN P system is an SN P system whose neurons have identical rule set. In this work, we introduced an algorithm that, under certain condition, transforms a non-homogeneous SN P system Π to a homogeneous SN P system Π′ that computes the same set as the system Π. Two transformation operations called neuron translation and neuron scaling are introduced which are then used by the homogenization algorithm.

Exploiting noise as a resource for computation and learning in spiking neural networks

Networks of spiking neurons underpin the extraordinary information-processing capabilities of the brain and have become pillar models in neuromorphic artificial intelligence. Despite extensive research on spiking neural networks (SNNs), most studies are established on deterministic models, overlooking the inherent non-deterministic, noisy nature of neural computations. This study introduces the noisy SNN (NSNN) and the noise-driven learning (NDL) rule by incorporating noisy neuronal dynamics to exploit the computational advantages of noisy neural processing. The NSNN provides a theoretical framework that yields scalable, flexible, and reliable computation and learning. We demonstrate that this framework leads to spiking neural models with competitive performance, improved robustness against challenging perturbations compared with deterministic SNNs, and better reproducing probabilistic computation in neural coding. Generally, this study offers a powerful and easy-to-use tool for machine learning, neuromorphic intelligence practitioners, and computational neuroscience researchers.

How network structure affects the dynamics of a network of stochastic spiking neurons.

Up to now, it still remains an open question about the relation between the structure of brain networks and their functions. The effects of structure on the dynamics of neural networks are usually investigated via extensive numerical simulations, while analytical analysis is always very difficult and thus rare. In this work, we explored the effects of a random regular graph on the dynamics of a neural network of stochastic spiking neurons, which has a bistable region when fully connected. We showed by numerical simulations that as the number of each neuron's neighbors decreases, the bistable region shrinks and eventually seems to disappear, and a critical-like transition appears. In the meantime, we made analytical analysis that explains numerical results. We hope this would give some insights into how structure affects the dynamics of neural networks from a theoretical perspective, rather than merely by numerical simulations.

Sequence learning in a spiking neuronal network with memristive synapses

Brain-inspired computing proposes a set of algorithmic principles that hold promise for advancing artificial intelligence. They endow systems with self learning capabilities, efficient energy usage, and high storage capacity. A core concept that lies at the heart of brain computation is sequence learning and prediction. This form of computation is essential for almost all our daily tasks such as movement generation, perception, and language. Understanding how the brain performs such a computation is not only important to advance neuroscience, but also to pave the way to new technological brain-inspired applications. A previously developed spiking neural network implementation of sequence prediction and recall learns complex, high-order sequences in an unsupervised manner by local, biologically inspired plasticity rules. An emerging type of hardware that may efficiently run this type of algorithm is neuromorphic hardware. It emulates the way the brain processes information and maps neurons and synapses directly into a physical substrate. Memristive devices have been identified as potential synaptic elements in neuromorphic hardware. In particular, redox-induced resistive random access memories (ReRAM) devices stand out at many aspects. They permit scalability, are energy efficient and fast, and can implement biological plasticity rules. In this work, we study the feasibility of using ReRAM devices as a replacement of the biological synapses in the sequence learning model. We implement and simulate the model including the ReRAM plasticity using the neural network simulator NEST. We investigate two types of ReRAM memristive devices: (i) a gradual, analog switching device, and (ii) an abrupt, binary switching device. We study the effect of different device properties on the performance characteristics of the sequence learning model, and demonstrate that, in contrast to many other artificial neural networks, this architecture is resilient with respect to changes in the on-off ratio and the conductance resolution, device variability, and device failure.

From clean room to machine room: commissioning of the first-generation BrainScaleS wafer-scale neuromorphic system

The first-generation of BrainScaleS, also referred to as BrainScaleS-1, is a neuromorphic system for emulating large-scale networks of spiking neurons. Following a ‘physical modeling’ principle, its VLSI circuits are designed to emulate the dynamics of biological examples: analog circuits implement neurons and synapses with time constants that arise from their electronic components’ intrinsic properties. It operates in continuous time, with dynamics typically matching an acceleration factor of 10 000 compared to the biological regime. A fault-tolerant design allows it to achieve wafer-scale integration despite unavoidable analog variability and component failures. In this paper, we present the commissioning process of a BrainScaleS-1 wafer module, providing a short description of the system’s physical components, illustrating the steps taken during its assembly and the measures taken to operate it. Furthermore, we reflect on the system’s development process and the lessons learned to conclude with a demonstration of its functionality by emulating a wafer-scale synchronous firing chain, the largest spiking network emulation ran with analog components and individual synapses to date.

Runtime Construction of Large-Scale Spiking Neuronal Network Models on GPU Devices

Simulation speed matters for neuroscientific research: this includes not only how quickly the simulated model time of a large-scale spiking neuronal network progresses but also how long it takes to instantiate the network model in computer memory. On the hardware side, acceleration via highly parallel GPUs is being increasingly utilized. On the software side, code generation approaches ensure highly optimized code at the expense of repeated code regeneration and recompilation after modifications to the network model. Aiming for a greater flexibility with respect to iterative model changes, here we propose a new method for creating network connections interactively, dynamically, and directly in GPU memory through a set of commonly used high-level connection rules. We validate the simulation performance with both consumer and data center GPUs on two neuroscientifically relevant models: a cortical microcircuit of about 77,000 leaky-integrate-and-fire neuron models and 300 million static synapses, and a two-population network recurrently connected using a variety of connection rules. With our proposed ad hoc network instantiation, both network construction and simulation times are comparable or shorter than those obtained with other state-of-the-art simulation technologies while still meeting the flexibility demands of explorative network modeling.

A proposal for leaky integrate-and-fire neurons by domain walls in antiferromagnetic insulators

Brain-inspired neuromorphic computing is a promising path towards next generation analogue computers that are fundamentally different compared to the conventional von Neumann architecture. One model for neuromorphic computing that can mimic the human brain behavior are spiking neural networks (SNNs), of which one of the most successful is the leaky integrate-and-fire (LIF) model. Since conventional complementary metal-oxide-semiconductor (CMOS) devices are not meant for modelling neural networks and are energy inefficient in network applications, recently the focus shifted towards spintronic-based neural networks. In this work, using the advantage of antiferromagnetic insulators, we propose a non-volatile magnonic neuron that could be the building block of a LIF spiking neuronal network. In our proposal, an antiferromagnetic domain wall in the presence of a magnetic anisotropy gradient mimics a biological neuron with leaky, integrating, and firing properties. This single neuron is controlled by polarized antiferromagnetic magnons, activated by either a magnetic field pulse or a spin transfer torque mechanism, and has properties similar to biological neurons, namely latency, refraction, bursting and inhibition. We argue that this proposed single neuron, based on antiferromagnetic domain walls, is faster and has more functionalities compared to previously proposed neurons based on ferromagnetic systems.

Geometry of population activity in spiking networks with low-rank structure.

Recurrent network models are instrumental in investigating how behaviorally-relevant computations emerge from collective neural dynamics. A recently developed class of models based on low-rank connectivity provides an analytically tractable framework for understanding of how connectivity structure determines the geometry of low-dimensional dynamics and the ensuing computations. Such models however lack some fundamental biological constraints, and in particular represent individual neurons in terms of abstract units that communicate through continuous firing rates rather than discrete action potentials. Here we examine how far the theoretical insights obtained from low-rank rate networks transfer to more biologically plausible networks of spiking neurons. Adding a low-rank structure on top of random excitatory-inhibitory connectivity, we systematically compare the geometry of activity in networks of integrate-and-fire neurons to rate networks with statistically equivalent low-rank connectivity. We show that the mean-field predictions of rate networks allow us to identify low-dimensional dynamics at constant population-average activity in spiking networks, as well as novel non-linear regimes of activity such as out-of-phase oscillations and slow manifolds. We finally exploit these results to directly build spiking networks that perform nonlinear computations.

Spiking Deep Residual Networks.

Spiking neural networks (SNNs) have received significant attention for their biological plausibility. SNNs theoretically have at least the same computational power as traditional artificial neural networks (ANNs). They possess the potential of achieving energy-efficient machine intelligence while keeping comparable performance to ANNs. However, it is still a big challenge to train a very deep SNN. In this brief, we propose an efficient approach to build deep SNNs. Residual network (ResNet) is considered a state-of-the-art and fundamental model among convolutional neural networks (CNNs). We employ the idea of converting a trained ResNet to a network of spiking neurons named spiking ResNet (S-ResNet). We propose a residual conversion model that appropriately scales continuous-valued activations in ANNs to match the firing rates in SNNs and a compensation mechanism to reduce the error caused by discretization. Experimental results demonstrate that our proposed method achieves state-of-the-art performance on CIFAR-10, CIFAR-100, and ImageNet 2012 with low latency. This work is the first time to build an asynchronous SNN deeper than 100 layers, with comparable performance to its original ANN.

Designing spiking neural networks for robust and reconfigurable computation

Networks of spiking neurons constitute analog systems capable of effective and resilient computing. Recent work has shown that networks of symmetrically connected inhibitory neurons may implement basic computations such that they are resilient to system disruption. For instance, if the functionality of one neuron is lost (e.g., the neuron, along with its connections, is removed), the system may be robustly reconfigured by adapting only one global system parameter. How to effectively adapt network parameters to robustly perform a given computation is still unclear. Here, we present an analytical approach to derive such parameters. Specifically, we analyze k-winners-takes-all (k-WTA) computations, basic computational tasks of identifying the k largest signals from a total of N input signals from which one can construct any computation. We identify and characterize different dynamical regimes and provide analytical expressions for the transitions between different numbers k of winners as a function of both input and network parameters. Our results thereby provide analytical insights about the dynamics underlying k-winner-takes-all functionality as well as an effective way of designing spiking neural network computing systems implementing disruption-resilient dynamics.