Networks Of Spiking Neurons Research Articles

Local dendritic balance enables learning of efficient representations in networks of spiking neurons

How can neural networks learn to efficiently represent complex and high-dimensional inputs via local plasticity mechanisms? Classical models of representation learning assume that feedforward weights are learned via pairwise Hebbian-like plasticity. Here, we show that pairwise Hebbian-like plasticity works only under unrealistic requirements on neural dynamics and input statistics. To overcome these limitations, we derive from first principles a learning scheme based on voltage-dependent synaptic plasticity rules. Here, recurrent connections learn to locally balance feedforward input in individual dendritic compartments and thereby can modulate synaptic plasticity to learn efficient representations. We demonstrate in simulations that this learning scheme works robustly even for complex high-dimensional inputs and with inhibitory transmission delays, where Hebbian-like plasticity fails. Our results draw a direct connection between dendritic excitatory-inhibitory balance and voltage-dependent synaptic plasticity as observed in vivo and suggest that both are crucial for representation learning.

Reconstruction of a Fully Paralleled Auditory Spiking Neural Network and FPGA Implementation.

This paper presents a field-programmable gate array (FPGA) implementation of an auditory system, which is biologically inspired and has the advantages of robustness and anti-noise ability. We propose an FPGA implementation of an eleven-channel hierarchical spiking neuron network (SNN) model, which has a sparsely connected architecture with low power consumption. According to the mechanism of the auditory pathway in human brain, spiking trains generated by the cochlea are analyzed in the hierarchical SNN, and the specific word can be identified by a Bayesian classifier. Modified leaky integrate-and-fire (LIF) model is used to realize the hierarchical SNN, which achieves both high efficiency and low hardware consumption. The hierarchical SNN implemented on FPGA enables the auditory system to be operated at high speed and can be interfaced and applied with external machines and sensors. A set of speech from different speakers mixed with noise are used as input to test the performance our system, and the experimental results show that the system can classify words in a biologically plausible way with the presence of noise. The method of our system is flexible and the system can be modified into desirable scale. These confirm that the proposed biologically plausible auditory system provides a better method for on-chip speech recognition. Compare to the state-of-the-art, our auditory system achieves a higher speed with a maximum frequency of 65.03 MHz and a lower energy consumption of 276.83 μJ for a single operation. It can be applied in the field of brain-computer interface and intelligent robots.

Nonlinear computations in spiking neural networks through multiplicative synapses

The brain efficiently performs nonlinear computations through its intricate networks of spiking neurons, but how this is done remains elusive. While nonlinear computations can be implemented successfully in spiking neural networks, this requires supervised training and the resulting connectivity can be hard to interpret. In contrast, the required connectivity for any computation in the form of a linear dynamical system can be directly derived and understood with the spike coding network (SCN) framework. These networks also have biologically realistic activity patterns and are highly robust to cell death. Here we extend the SCN framework to directly implement any polynomial dynamical system, without the need for training. This results in networks requiring a mix of synapse types (fast, slow, and multiplicative), which we term multiplicative spike coding networks (mSCNs). Using mSCNs, we demonstrate how to directly derive the required connectivity for several nonlinear dynamical systems. We also show how to carry out higher-order polynomials with coupled networks that use only pair-wise multiplicative synapses, and provide expected numbers of connections for each synapse type. Overall, our work demonstrates a novel method for implementing nonlinear computations in spiking neural networks, while keeping the attractive features of standard SCNs (robustness, realistic activity patterns, and interpretable connectivity). Finally, we discuss the biological plausibility of our approach, and how the high accuracy and robustness of the approach may be of interest for neuromorphic computing.

Memories in a network with excitatory and inhibitory plasticity are encoded in the spiking irregularity.

Cell assemblies are thought to be the substrate of memory in the brain. Theoretical studies have previously shown that assemblies can be formed in networks with multiple types of plasticity. But how exactly they are formed and how they encode information is yet to be fully understood. One possibility is that memories are stored in silent assemblies. Here we used a computational model to study the formation of silent assemblies in a network of spiking neurons with excitatory and inhibitory plasticity. We found that even though the formed assemblies were silent in terms of mean firing rate, they had an increased coefficient of variation of inter-spike intervals. We also found that this spiking irregularity could be read out with support of short-term plasticity, and that it could contribute to the longevity of memories.

Feedforward and feedback influences through distinct frequency bands between two spiking-neuron networks

Several studies on brain signals suggested that bottom-up and top-down influences are exerted through distinct frequency bands among visual cortical areas. It was recently shown that theta and gamma rhythms subserve feedforward, whereas the feedback influence is dominated by the alpha-beta rhythm in primates. A few theoretical models for reproducing these effects have been proposed so far. Here we show that a simple but biophysically plausible two-network motif composed of spiking-neuron models and chemical synapses can exhibit feedforward and feedback influences through distinct frequency bands. Different from previous studies, this kind of model allows us to study directed influences not only at the population level, by using a proxy for the local field potential, but also at the cellular level, by using the neuronal spiking series.

A `Rosetta stone' for the population dynamics of spiking neuron networks

A `Rosetta stone' for the population dynamics of spiking neuron networks

Analysis of cellular and synaptic mechanisms behind spontaneous cortical activity in vitro: Insights from optimization of spiking neuronal network models

Spontaneous network bursts, the intervals of intense network-wide activity interleaved with longer periods of sparse activity, are a hallmark phenomenon observed in cortical networks at postnatal developmental stages. Generation, propagation and termination of network bursts depend on a combination of synaptic, cellular and network mechanisms; however, the interplay between these mechanisms is not fully understood. We study this interplay in silico, using a new data-driven framework for generating spiking neuronal networks fitted to the microelectrode array recordings. We recorded the network bursting activity from rat postnatal cortical networks under several pharmacological conditions. In each condition, the function of specific excitatory and inhibitory synaptic receptors was reduced in order to examine their impact on global network dynamics. The obtained data was used to develop two complementary model fitting protocols for automatic model generation. These protocols allowed us to disentangle systematically the modeled cellular and synaptic mechanisms that affect the observed network bursts. We confirmed that the change in excitatory and inhibitory synaptic transmission in silico, consistent with pharmacological conditions, can account for the changes in network bursts relative to the control data. Reproducing the exact recorded network bursts statistics required adapting both the synaptic transmission and the cellular excitability separately for each pharmacological condition. Our results bring new understanding of the complex interplay between cellular, synaptic and network mechanisms supporting the burst dynamics. While here we focused on analysis of in vitro data, our approach can be applied ex vivo and in vivo given that the appropriate experimental data is available.

Microscopic theory of intrinsic timescales in spiking neural networks

A complex interplay of single-neuron properties and the recurrent network structure shapes the activity of cortical neurons. The single-neuron activity statistics differ in general from the respective population statistics, including spectra and, correspondingly, autocorrelation times. We develop a theory for self-consistent second-order single-neuron statistics in block-structured sparse random networks of spiking neurons. In particular, the theory predicts the neuron-level autocorrelation times, also known as intrinsic timescales, of the neuronal activity. The theory is based on an extension of dynamic mean-field theory from rate networks to spiking networks, which is validated via simulations. It accounts for both static variability, e.g. due to a distributed number of incoming synapses per neuron, and temporal fluctuations of the input. We apply the theory to balanced random networks of generalized linear model neurons, balanced random networks of leaky integrate-and-fire neurons, and a biologically constrained network of leaky integrate-and-fire neurons. For the generalized linear model network with an error function nonlinearity, a novel analytical solution of the colored noise problem allows us to obtain self-consistent firing rate distributions, single-neuron power spectra, and intrinsic timescales. For the leaky integrate-and-fire networks, we derive an approximate analytical solution of the colored noise problem, based on the Stratonovich approximation of the Wiener-Rice series and a novel analytical solution for the free upcrossing statistics. Again closing the system self-consistently, in the fluctuation-driven regime this approximation yields reliable estimates of the mean firing rate and its variance across neurons, the inter-spike interval distribution, the single-neuron power spectra, and intrinsic timescales.

Mean-field approximations of networks of spiking neurons with short-term synaptic plasticity.

Low-dimensional descriptions of spiking neural network dynamics are an effective tool for bridging different scales of organization of brain structure and function. Recent advances in deriving mean-field descriptions for networks of coupled oscillators have sparked the development of a new generation of neural mass models. Of notable interest are mean-field descriptions of all-to-all coupled quadratic integrate-and-fire (QIF) neurons, which have already seen numerous extensions and applications. These extensions include different forms of short-term adaptation considered to play an important role in generating and sustaining dynamic regimes of interest in the brain. It is an open question, however, whether the incorporation of presynaptic forms of synaptic plasticity driven by single neuron activity would still permit the derivation of mean-field equations using the same method. Here we discuss this problem using an established model of short-term synaptic plasticity at the single neuron level, for which we present two different approaches for the derivation of the mean-field equations. We compare these models with a recently proposed mean-field approximation that assumes stochastic spike timings. In general, the latter fails to accurately reproduce the macroscopic activity in networks of deterministic QIF neurons with distributed parameters. We show that the mean-field models we propose provide a more accurate description of the network dynamics, although they are mathematically more involved. Using bifurcation analysis, we find that QIF networks with presynaptic short-term plasticity can express regimes of periodic bursting activity as well as bistable regimes. Together, we provide novel insight into the macroscopic effects of short-term synaptic plasticity in spiking neural networks, as well as two different mean-field descriptions for future investigations of such networks.

A CMOS-based neuromorphic device for seizure detection from LFP signals

Recent research provides examples of neuromorphic systems applied to process biological signals or to interface biological tissues. Usually, in such contexts, the neuromorphic system is used for automatic anomaly detection. The automation of the long-term monitoring of biological signals holds promise for lightening the burden placed on clinicians. At the same time, the adoption of such devices potentially allows processing to be performed locally, without the need to transfer data to an external processor. In turn, on-site signal analysis makes closed-loop intervention feasible, to correct the source of the anomalies. So far, the common approach has been to deploy the network of spiking neurons either on multi-core neuromorphic platforms or on programmable units (field programmable gate array). However, if the aim is to develop wearable or even chronically implantable devices, it is imperative to move in the direction of embedded solutions, tailored and optimized for the specific application. To this end, the present study proposes a neuromorphic device implemented in CMOS technology for the detection of epileptic seizures (ictal events) from local field potential (LFP) signals. The LFP data have been acquired by a multi-electrode array in a slice of mouse hippocampus-cortex. The system includes an analog-to-event converter (AEC) encoding the recorded signals into trains of spikes, and a small spiking neural network (SNN) of 2 × 1 neurons with online biological-plausible learning. The AEC yields two spike-trains: UP spikes that account for the positive slope signal and DOWN spikes that correspond to the negative slope signal. The synapses among the input and output layers are plastic and follow the spike-timing-dependent plasticity rule. Early results show that the SNN module is able to detect ictal events with a delay of 64.98 ± 30.92 ms, consuming 50 pW. The layout of the entire system occupies 528 µm × 278 µm.

Active intrinsic conductances in recurrent networks allow for long-lasting transients and sustained activity with realistic firing rates as well as robust plasticity.

Recurrent neural networks of spiking neurons can exhibit long lasting and even persistent activity. Such networks are often not robust and exhibit spike and firing rate statistics that are inconsistent with experimental observations. In order to overcome this problem most previous models had to assume that recurrent connections are dominated by slower NMDA type excitatory receptors. Usually, the single neurons within these networks are very simple leaky integrate and fire neurons or other low dimensional model neurons. However real neurons are much more complex, and exhibit a plethora of active conductances which are recruited both at the sub and supra threshold regimes. Here we show that by including a small number of additional active conductances we can produce recurrent networks that are both more robust and exhibit firing-rate statistics that are more consistent with experimental results. We show that this holds both for bi-stable recurrent networks, which are thought to underlie working memory and for slowly decaying networks which might underlie the estimation of interval timing. We also show that by including these conductances, such networks can be trained to using a simple learning rule to predict temporal intervals that are an order of magnitude larger than those that can be trained in networks of leaky integrate and fire neurons.

Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks

Inspired by detailed modelling of biological neurons, spiking neural networks (SNNs) are investigated as biologically plausible and high-performance models of neural computation. The sparse and binary communication between spiking neurons potentially enables powerful and energy-efficient neural networks. The performance of SNNs, however, has remained lacking compared with artificial neural networks. Here we demonstrate how an activity-regularizing surrogate gradient combined with recurrent networks of tunable and adaptive spiking neurons yields the state of the art for SNNs on challenging benchmarks in the time domain, such as speech and gesture recognition. This also exceeds the performance of standard classical recurrent neural networks and approaches that of the best modern artificial neural networks. As these SNNs exhibit sparse spiking, we show that they are theoretically one to three orders of magnitude more computationally efficient compared to recurrent neural networks with similar performance. Together, this positions SNNs as an attractive solution for AI hardware implementations. The use of sparse signals in spiking neural networks, modelled on biological neurons, offers in principle a highly efficient approach for artificial neural networks when implemented on neuromorphic hardware, but new training approaches are needed to improve performance. Using a new type of activity-regularizing surrogate gradient for backpropagation combined with recurrent networks of tunable and adaptive spiking neurons, state-of-the-art performance for spiking neural networks is demonstrated on benchmarks in the time domain.

Integration, coincidence detection and resonance in networks of spiking neurons expressing Gamma oscillations and asynchronous states.

Gamma oscillations are widely seen in the awake and sleeping cerebral cortex, but the exact role of these oscillations is still debated. Here, we used biophysical models to examine how Gamma oscillations may participate to the processing of afferent stimuli. We constructed conductance-based network models of Gamma oscillations, based on different cell types found in cerebral cortex. The models were adjusted to extracellular unit recordings in humans, where Gamma oscillations always coexist with the asynchronous firing mode. We considered three different mechanisms to generate Gamma, first a mechanism based on the interaction between pyramidal neurons and interneurons (PING), second a mechanism in which Gamma is generated by interneuron networks (ING) and third, a mechanism which relies on Gamma oscillations generated by pacemaker chattering neurons (CHING). We find that all three mechanisms generate features consistent with human recordings, but that the ING mechanism is most consistent with the firing rate change inside Gamma bursts seen in the human data. We next evaluated the responsiveness and resonant properties of these networks, contrasting Gamma oscillations with the asynchronous mode. We find that for both slowly-varying stimuli and precisely-timed stimuli, the responsiveness is generally lower during Gamma compared to asynchronous states, while resonant properties are similar around the Gamma band. We could not find conditions where Gamma oscillations were more responsive. We therefore predict that asynchronous states provide the highest responsiveness to external stimuli, while Gamma oscillations tend to overall diminish responsiveness.

A robust model of Stimulus-Specific Adaptation validated on neuromorphic hardware

Stimulus-Specific Adaptation (SSA) to repetitive stimulation is a phenomenon that has been observed across many different species and in several brain sensory areas. It has been proposed as a computational mechanism, responsible for separating behaviorally relevant information from the continuous stream of sensory information. Although SSA can be induced and measured reliably in a wide variety of conditions, the network details and intracellular mechanisms giving rise to SSA still remain unclear. Recent computational studies proposed that SSA could be associated with a fast and synchronous neuronal firing phenomenon called Population Spikes (PS). Here, we test this hypothesis using a mean-field rate model and corroborate it using a neuromorphic hardware. As the neuromorphic circuits used in this study operate in real-time with biologically realistic time constants, they can reproduce the same dynamics observed in biological systems, together with the exploration of different connectivity schemes, with complete control of the system parameter settings. Besides, the hardware permits the iteration of multiple experiments over many trials, for extended amounts of time and without losing the networks and individual neural processes being studied. Following this “neuromorphic engineering” approach, we therefore study the PS hypothesis in a biophysically inspired recurrent networks of spiking neurons and evaluate the role of different linear and non-linear dynamic computational primitives such as spike-frequency adaptation or short-term depression (STD). We compare both the theoretical mean-field model of SSA and PS to previously obtained experimental results in the area of novelty detection and observe its behavior on its neuromorphic physical equivalent model. We show how the approach proposed can be extended to other computational neuroscience modelling efforts for understanding high-level phenomena in mechanistic models.

Few-Shot Learning in Spiking Neural Networks by Multi-Timescale Optimization.

Learning new concepts rapidly from a few examples is an open issue in spike-based machine learning. This few-shot learning imposes substantial challenges to the current learning methodologies of spiking neuron networks (SNNs) due to the lack of task-related priori knowledge. The recent learning-to-learn (L2L) approach allows SNNs to acquire priori knowledge through example-level learning and task-level optimization. However, existing L2L-based frameworks do not target the neural dynamics (i.e., neuronal and synaptic parameter changes) on different timescales. This diversity of temporal dynamics is an important attribute in spike-based learning, which facilitates the networks to rapidly acquire knowledge from very few examples and gradually integrate this knowledge. In this work, we consider the neural dynamics on various timescales and provide a multi-timescale optimization (MTSO) framework for SNNs. This framework introduces an adaptive-gated LSTM to accommodate two different timescales of neural dynamics: short-term learning and long-term evolution. Short-term learning is a fast knowledge acquisition process achieved by a novel surrogate gradient online learning (SGOL) algorithm, where the LSTM guides gradient updating of SNN on a short timescale through an adaptive learning rate and weight decay gating. The long-term evolution aims to slowly integrate acquired knowledge and form a priori, which can be achieved by optimizing the LSTM guidance process to tune SNN parameters on a long timescale. Experimental results demonstrate that the collaborative optimization of multi-timescale neural dynamics can make SNNs achieve promising performance for the few-shot learning tasks.

Mean-Field Models for EEG/MEG: From Oscillations to Waves

Neural mass models have been used since the 1970s to model the coarse-grained activity of large populations of neurons. They have proven especially fruitful for understanding brain rhythms. However, although motivated by neurobiological considerations they are phenomenological in nature, and cannot hope to recreate some of the rich repertoire of responses seen in real neuronal tissue. Here we consider a simple spiking neuron network model that has recently been shown to admit an exact mean-field description for both synaptic and gap-junction interactions. The mean-field model takes a similar form to a standard neural mass model, with an additional dynamical equation to describe the evolution of within-population synchrony. As well as reviewing the origins of this next generation mass model we discuss its extension to describe an idealised spatially extended planar cortex. To emphasise the usefulness of this model for EEG/MEG modelling we show how it can be used to uncover the role of local gap-junction coupling in shaping large scale synaptic waves.

Radar Emitter Identification Based on Fully Connected Spiking Neural Network

In the face of the increasing complex electromagnetic environment and new radar system, it is difficult to extract radar emitter characteristics based on manual mode to meet requirements of modern cognitive electronic warfare. In order to improve the intelligence level of radar emitter identification, a new method based on Spiking Neuron Network (SNN) for radar emitter identification is proposed in this paper. Firstly, five kinds of common radar signals are converted into two-dimensional gray scale images by using time-frequency analysis method. Then, the images are converted into spikes by Poisson coder, which are put into a fully connected spiking neural network for training and emitter identification. Finally, the simulation results prove the validity of this method by comparing with the traditional neural network.

Efficient and robust coding in heterogeneous recurrent networks.

Cortical networks show a large heterogeneity of neuronal properties. However, traditional coding models have focused on homogeneous populations of excitatory and inhibitory neurons. Here, we analytically derive a class of recurrent networks of spiking neurons that close to optimally track a continuously varying input online, based on two assumptions: 1) every spike is decoded linearly and 2) the network aims to reduce the mean-squared error between the input and the estimate. From this we derive a class of predictive coding networks, that unifies encoding and decoding and in which we can investigate the difference between homogeneous networks and heterogeneous networks, in which each neurons represents different features and has different spike-generating properties. We find that in this framework, 'type 1' and 'type 2' neurons arise naturally and networks consisting of a heterogeneous population of different neuron types are both more efficient and more robust against correlated noise. We make two experimental predictions: 1) we predict that integrators show strong correlations with other integrators and resonators are correlated with resonators, whereas the correlations are much weaker between neurons with different coding properties and 2) that 'type 2' neurons are more coherent with the overall network activity than 'type 1' neurons.

Modeling Working Memory in a Spiking Neuron Network Accompanied by Astrocytes.

We propose a novel biologically plausible computational model of working memory (WM) implemented by a spiking neuron network (SNN) interacting with a network of astrocytes. The SNN is modeled by synaptically coupled Izhikevich neurons with a non-specific architecture connection topology. Astrocytes generating calcium signals are connected by local gap junction diffusive couplings and interact with neurons via chemicals diffused in the extracellular space. Calcium elevations occur in response to the increased concentration of the neurotransmitter released by spiking neurons when a group of them fire coherently. In turn, gliotransmitters are released by activated astrocytes modulating the strength of the synaptic connections in the corresponding neuronal group. Input information is encoded as two-dimensional patterns of short applied current pulses stimulating neurons. The output is taken from frequencies of transient discharges of corresponding neurons. We show how a set of information patterns with quite significant overlapping areas can be uploaded into the neuron-astrocyte network and stored for several seconds. Information retrieval is organized by the application of a cue pattern representing one from the memory set distorted by noise. We found that successful retrieval with the level of the correlation between the recalled pattern and ideal pattern exceeding 90% is possible for the multi-item WM task. Having analyzed the dynamical mechanism of WM formation, we discovered that astrocytes operating at a time scale of a dozen of seconds can successfully store traces of neuronal activations corresponding to information patterns. In the retrieval stage, the astrocytic network selectively modulates synaptic connections in the SNN leading to successful recall. Information and dynamical characteristics of the proposed WM model agrees with classical concepts and other WM models.

Recharged Catalyst with Memristive Nitrogen Reduction Activity through Learning Networks of Spiking Neurons.

Electrocatalysis from N2 to NH3 has been increasingly studied because it provides an environmentally friendly avenue to take the place of the current Haber-Bosch method. Unfortunately, the conversion of N2 to NH3 is far below the necessary level for implementation at a large scale. Inspired by signal memory in a spiking neural network, we developed rechargeable catalyst technology to activate and remember the optimal catalytic activity using manageable electrical stimulation. Herein, we designed double-faced FeReS3 Janus layers that mimic a multiple-neuron network consisting of resistive switching synapses, enabling a series of intriguing multiphase transitions to activate undiscovered catalytic activity; the activation energy barrier is clearly reduced via an active site conversion between two nonequivalent surfaces. Electrical field-stimulated FeReS3 demonstrates a Faradaic efficiency of 43% and the highest rate of 203 μg h-1 mg-1 toward NH3 synthesis. Moreover, this rechargeable catalyst displays unprecedented catalytic performance that persists for up to 216 h and can be repeatedly activated through a simple charging operation.