Feedforward Spiking Neural Network Research Articles

Adjusted SpikeProp algorithm for recurrent spiking neural networks with LIF neurons

A problem related to the development of a supervised learning method for recurrent spiking neural networks is addressed in the paper. The widely used Leaky-Integrate-and-Fire model has been adopted as a spike neuron model. The proposed method is based on a known SpikeProp algorithm. In detail, the developed method enables gradient descent learning of recurrent or multi-layer feedforward spiking neural networks. The research included an extended verification study for the classical XOR classification problem. In addition, the developed learning method has been used to provide a spiking neural black-box model of fast processes occurring in a pressurised water nuclear reactor. The obtained simulation results demonstrate satisfactory effectiveness of the proposed approach.

Neural coding of space by time.

The intertwining of space and time poses a significant scientific challenge, transcending disciplines from philosophy and physics to neuroscience. Deciphering neural coding, marked by its inherent spatial and temporal dimensions, has proven to be a complex task. In this paper, we present insights into temporal and spatial modes of neural coding and their intricate interplay, drawn from neuroscientific findings. We illustrate the conversion of a purely spatial input into the temporal form of a singular spike train, demonstrating storage, transmission to remote locations, and recall through spike bursts corresponding to Sharp Wave Ripples. Moreover, the converted temporal representation can be transformed back into a spatiotemporal pattern. The principles of the transformation process are illustrated using a simple feed-forward spiking neural network. The frequencies and phases of Subthreshold Membrane potential Oscillations play a pivotal role in this framework. The model offers insights into information multiplexing and phenomena such as stretching or compressing time of spike patterns.

DenRAM: neuromorphic dendritic architecture with RRAM for efficient temporal processing with delays

Neuroscience findings emphasize the role of dendritic branching in neocortical pyramidal neurons for non-linear computations and signal processing. Dendritic branches facilitate temporal feature detection via synaptic delays that enable coincidence detection (CD) mechanisms. Spiking neural networks highlight the significance of delays for spatio-temporal pattern recognition in feed-forward networks, eliminating the need for recurrent structures. Here, we introduce DenRAM, a novel analog electronic feed-forward spiking neural network with dendritic compartments. Utilizing 130 nm technology integrated with resistive RAM (RRAM), DenRAM incorporates both delays and synaptic weights. By configuring RRAMs to emulate bio-realistic delays and exploiting their heterogeneity, DenRAM mimics synaptic delays and efficiently performs CD for pattern recognition. Hardware-aware simulations on temporal benchmarks show DenRAM’s robustness against hardware noise, and its higher accuracy over recurrent networks. DenRAM advances temporal processing in neuromorphic computing, optimizes memory usage, and marks progress in low-power, real-time signal processing

Learnable axonal delay in spiking neural networks improves spoken word recognition.

Spiking neural networks (SNNs), which are composed of biologically plausible spiking neurons, and combined with bio-physically realistic auditory periphery models, offer a means to explore and understand human auditory processing-especially in tasks where precise timing is essential. However, because of the inherent temporal complexity in spike sequences, the performance of SNNs has remained less competitive compared to artificial neural networks (ANNs). To tackle this challenge, a fundamental research topic is the configuration of spike-timing and the exploration of more intricate architectures. In this work, we demonstrate a learnable axonal delay combined with local skip-connections yields state-of-the-art performance on challenging benchmarks for spoken word recognition. Additionally, we introduce an auxiliary loss term to further enhance accuracy and stability. Experiments on the neuromorphic speech benchmark datasets, NTIDIDIGITS and SHD, show improvements in performance when incorporating our delay module in comparison to vanilla feedforward SNNs. Specifically, with the integration of our delay module, the performance on NTIDIDIGITS and SHD improves by 14% and 18%, respectively. When paired with local skip-connections and the auxiliary loss, our approach surpasses both recurrent and convolutional neural networks, yet uses 10 × fewer parameters for NTIDIDIGITS and 7 × fewer for SHD.

A Low-Power Analog Cell for Implementing Spiking Neural Networks in 65 nm CMOS

A Spiking Neural Network (SNN) is realized within a 65 nm CMOS process to demonstrate the feasibility of its constituent cells. Analog hardware neural networks have shown improved energy efficiency in edge computing for real-time-inference applications, such as speech recognition. The proposed network uses a leaky integrate and fire neuron scheme for computation, interleaved with a Spike Timing Dependent Plasticity (STDP) circuit for implementing synaptic-like weights. The low-power, asynchronous analog neurons and synapses are tailored for the VLSI environment needed to effectively make use of hardware SSN systems. To demonstrate functionality, a feed-forward Spiking Neural Network composed of two layers, the first with ten neurons and the second with six, is implemented. The neuron design operates with 2.1 pJ of power per spike and 20 pJ per synaptic operation.

Surrogate-Assisted Cooperative Co-evolutionary Reservoir Architecture Search for Liquid State Machines

Liquid state machines (LSMs) are biologically more plausible than feedforward spiking neural networks for brain-inspired computing and neuromorphic engineering. However, optimizing and training complex recurrent network architectures in the reservoir of LSMs remains challenging. Most existing algorithms aim to adjust the synaptic strength only, without fundamentally modifying the reservoir architecture of LSMs. Recently, it has become popular to simultaneously optimize the architecture and parameters of the reservoir in LSMs. However, most existing architecture representation schemes are too restricted to discover more powerful architectures of LSMs. To address the above issue, this paper proposes a generative liquid state machine, whose reservoir architecture is evolved using a cooperative co-evolutionary algorithm whose weights are tuned by synaptic plasticity rules. To reduce the computation time for evolving the reservoir, random forest is adopted to assist the cooperative co-evolutionary algorithm, together with a data parallelism strategy. The proposed algorithm is assessed on three sequence classification benchmarks and our experimental results show that the proposed algorithm outperforms the state-of-the-art on the benchmark problems. Meanwhile, our analysis shows that the data parallelism strategy is effective in speeding up the evaluation process.

One-Pass Online Learning Based on Gradient Descent for Multilayer Spiking Neural Networks

Online supervised learning algorithms update synaptic weights in real-time during the running process of spiking neural networks (SNNs), which are important for modeling the behavior and cognitive process of the brain. This article proposes an online supervised learning algorithm based on gradient descent for multilayer feedforward SNNs, where precisely timed spike trains are used to represent neural information. The online learning rule is derived from the real-time error function and backpropagation mechanism. The synaptic weights are adjusted when an output neuron fires a spike. Results of spike train learning demonstrate that the proposed online learning algorithm can achieve higher learning accuracy and requires fewer learning epochs than the corresponding offline learning method and other typical supervised learning algorithms. Furthermore, the proposed algorithm is used for solving pattern classification problems, where the one-pass learning approach is employed for training SNNs. Results show that the proposed algorithm can obtain comparable classification accuracy compared with other state-of-the-art algorithms even in the case of only one iteration. It indicates that the proposed algorithm is effective for solving spatio-temporal pattern recognition problems.

A shallow hybrid classical–quantum spiking feedforward neural network for noise-robust image classification

Deep Convolutional Neural Network (CNN)-based image classification systems are often susceptible to noise interruption, i.e., minor image noise may significantly impact the outcome. On the contrary, classical Spiking Neural Network (SNN) is known for handling noisy data due to the stochastic and temporal behaviour of the spiking neuron signals. However, it is not always feasible to train the weights of the classical SNN due to stochastic and non-differentiable spiking signals. Recent applications of quantum machine learning to stochastic modelling have predicted that quantum computing would prove to be a significant advantage, hence catapulting the study of quantum and neuromorphic computing to a new level of development. Motivated by these observations, this paper introduces a shallow hybrid classical–quantum spiking feedforward neural network referred to as Spiking Quantum Neural Network (SQNN) for dealing with a robust image classification task in the presence of noise and adversarial attacks. The proposed SQNN offers the inherent capabilities of processing unforeseen noisy test images due to its spatial and temporal information. An efficient variational quantum circuit training with the help of a standard back-propagation algorithm obviates the classical Spiking Time-Dependent Plasticity (STDP) and Spike-Prop algorithms, which are often found inefficient in training the feedforward SNN. The proposed SQNN is extensively tested and benchmarked on the PennyLane Quantum Simulator. Experimental results show that the proposed SQNN model supersedes the feedforward SNN (SFNN), Random Quantum Neural Networks (RQNN), AlexNet and ResNet-18 on unseen test images with added noise from the FashionMNIST, MNIST, KMNIST, CIFAR10, and ImageNet datasets.

Neuromorphic Functional Modules of a Spiking Neural Network

In the current era, the design and development of artificial neural networks exploiting the architecture of the human brain have evolved rapidly. Artificial neural networks (ANN) effectively solve a wide range of common artificial-intelligence tasks involving data classification and recognition, prediction, forecasting and adaptive control of the behavior of an object. The biologically inspired underlying principles of ANN operation have certain advantages over the conventional von Neumann architecture including unsupervised learning, architectural flexibility and adaptability to environmental change and high performance under significantly reduced power consumption due to large parallel and asynchronous data processing. In this paper, we present a circuit design of main functional blocks (neurons and synapses) intended for the hardware implementation of a perceptron-based feedforward spiking neural network. As the third generation of artificial neural networks, spiking neural networks perform data processing utilizing spikes, which are discrete events (or functions) that take place at points in time. Neurons in spiking neural networks initiate precisely timed spikes and communicate with each other via spikes transmitted through synaptic connections or synapses with adaptable scalable weight. One of the prospective approaches to emulating synaptic behavior in hardware implemented spiking neural networks is to use nonvolatile-memory devices with analog conduction modulation (or memristive structures). Here we propose a circuit design for functional analogues of memristive structures to mimic synaptic plasticity, pre- and postsynaptic neurons which could be used for developing a circuit design of spiking-neural-network architectures with different training algorithms including the spike-timing-dependent-plasticity learning rule. Two different circuits of an electrical synapse are developed. The first one is an analog synapse with a photoresistor optocoupler used to ensure the tunable conductivity for synaptic-plasticity emulation. While the second one is a digital synapse, in which the synaptic weight is stored in a digital code with its direct conversion into conductivity (without a digital-to-analog converter and photoresistor optocoupler). The results of prototyping the developed circuits for electronic analogs of synapses, pre- and postsynaptic neurons and the study of transient processes are presented. The developed approach could provide a basis for the ASIC (application-specific integrated circuit) design of spiking neural networks based on CMOS (complementary metal—oxide–semiconductor) design technology.

Enhancing spiking neural networks with hybrid top-down attention.

As the representatives of brain-inspired models at the neuronal level, spiking neural networks (SNNs) have shown great promise in processing spatiotemporal information with intrinsic temporal dynamics. SNNs are expected to further improve their robustness and computing efficiency by introducing top-down attention at the architectural level, which is crucial for the human brain to support advanced intelligence. However, this attempt encounters difficulties in optimizing the attention in SNNs largely due to the lack of annotations. Here, we develop a hybrid network model with a top-down attention mechanism (HTDA) by incorporating an artificial neural network (ANN) to generate attention maps based on the features extracted by a feedforward SNN. The attention map is then used to modulate the encoding layer of the SNN so that it focuses on the most informative sensory input. To facilitate direct learning of attention maps and avoid labor-intensive annotations, we propose a general principle and a corresponding weakly-supervised objective, which promotes the HTDA model to utilize an integral and small subset of the input to give accurate predictions. On this basis, the ANN and the SNN can be jointly optimized by surrogate gradient descent in an end-to-end manner. We comprehensively evaluated the HTDA model on object recognition tasks, which demonstrates strong robustness to adversarial noise, high computing efficiency, and good interpretability. On the widely-adopted CIFAR-10, CIFAR-100, and MNIST benchmarks, the HTDA model reduces firing rates by up to 50% and improves adversarial robustness by up to 10% with comparable or better accuracy compared with the state-of-the-art SNNs. The HTDA model is also verified on dynamic neuromorphic datasets and achieves consistent improvements. This study provides a new way to boost the performance of SNNs by employing a hybrid top-down attention mechanism.

Encrypted internet traffic classification using a supervised spiking neural network

Internet traffic recognition is essential for access providers since it helps them define adapted priorities in order to enhance user experience, e.g., a high priority for an audio conference and a low priority for a file transfer. As internet traffic becomes increasingly encrypted, the main classic traffic recognition technique, payload inspection, is rendered ineffective. Hence this paper uses machine learning techniques looking only at packet size and time of arrival. For the first time, Spiking neural networks (SNNs), which are inspired by biological neurons, were used for this task for two reasons. Firstly, they can recognize time-related data packet features. Secondly, they can be implemented efficiently on neuromorphic hardware. Here we used a simple feedforward SNN, with only one fully connected hidden layer, and trained in a supervised manner using the new method known as Surrogate Gradient Learning. Surprisingly, such a simple SNN reached an accuracy of 95.9% on ISCX datasets, outperforming previous approaches. Besides better accuracy, there is also a significant improvement in simplicity: input size, the number of neurons, trainable parameters are all reduced by one to four orders of magnitude. Next, we analyzed the reasons for this good performance. It turns out that, beyond spatial (i.e., packet size) features, the SNN also exploits temporal ones, mainly the nearly synchronous (i.e., within a 200 ms range) arrival times of packets with specific sizes. Taken together, these results show that SNNs are an excellent fit for encrypted internet traffic classification: they can be more accurate than conventional artificial neural networks (ANN), and they could be implemented efficiently on low-power embedded systems.

Neuromorphic Functional Modules of Spiking Neural Network

In the current era, design and development of artificial neural networks exploiting the architecture of the human brain have evolved rapidly. Artificial neural networks effectively solve a wide range of common for artificial intelligence tasks involving data classification and recognition, prediction, forecasting and adaptive control of object behavior. Biologically inspired underlying principles of ANN operation have certain advantages over the conventional von Neumann architecture including unsupervised learning, architectural flexibility and adaptability to environmental change and high performance under significantly reduced power consumption due to heavy parallel and asynchronous data processing. In this paper, we present the circuit design of main functional blocks (neurons and synapses) intended for hardware implementation of a perceptron-based feedforward spiking neural network. As the third generation of artificial neural networks, spiking neural networks perform data processing utilizing spikes, which are discrete events (or functions) that take place at points in time. Neurons in spiking neural networks initiate precisely timing spikes and communicate with each other via spikes transmitted through synaptic connections or synapses with adaptable scalable weight. One of the prospective approach to emulate the synaptic behavior in hardware implemented spiking neural networks is to use non-volatile memory devices with analog conduction modulation (or memristive structures). Here we propose a circuit design for functional analogues of memristive structure to mimic a synaptic plasticity, pre- and postsynaptic neurons which could be used for developing circuit design of spiking neural network architectures with different training algorithms including spike-timing dependent plasticity learning rule. Two different circuits of electronic synapse were developed. The first one is an analog synapse with photoresistive optocoupler used to ensure the tunable conductivity for synaptic plasticity emulation. While the second one is a digital synapse, in which the synaptic weight is stored in a digital code with its direct conversion into conductivity (without digital-to-analog converter andphotoresistive optocoupler). The results of the prototyping of developed circuits for electronic analogues of synapses, pre- and postsynaptic neurons and the study of transient processes are presented. The developed approach could provide a basis for ASIC design of spiking neural networks based on CMOS (complementary metal oxide semiconductor) design technology.

Supervised Learning Algorithm for Multilayer Spiking Neural Networks with Long-Term Memory Spike Response Model.

As a new brain-inspired computational model of artificial neural networks, spiking neural networks transmit and process information via precisely timed spike trains. Constructing efficient learning methods is a significant research field in spiking neural networks. In this paper, we present a supervised learning algorithm for multilayer feedforward spiking neural networks; all neurons can fire multiple spikes in all layers. The feedforward network consists of spiking neurons governed by biologically plausible long-term memory spike response model, in which the effect of earlier spikes on the refractoriness is not neglected to incorporate adaptation effects. The gradient descent method is employed to derive synaptic weight updating rule for learning spike trains. The proposed algorithm is tested and verified on spatiotemporal pattern learning problems, including a set of spike train learning tasks and nonlinear pattern classification problems on four UCI datasets. Simulation results indicate that the proposed algorithm can improve learning accuracy in comparison with other supervised learning algorithms.

Necessary conditions for STDP-based pattern recognition learning in a memristive spiking neural network

This work is aimed to study experimental and theoretical approaches for searching effective local training rules for unsupervised pattern recognition by high-performance memristor-based Spiking Neural Networks (SNNs). First, the possibility of weight change using Spike-Timing-Dependent Plasticity (STDP) is demonstrated with a pair of hardware analog neurons connected through a (CoFeB)x(LiNbO3)1−x nanocomposite memristor. Next, the learning convergence to a solution of binary clusterization task is analyzed in a wide range of memristive STDP parameters for a single-layer fully connected feedforward SNN. The memristive STDP behavior supplying convergence in this simple task is shown also to provide it in the handwritten digit recognition domain by the more complex SNN architecture with a Winner-Take-All competition between neurons. To investigate basic conditions necessary for training convergence, an original probabilistic generative model of a rate-based single-layer network with independent or competing neurons is built and thoroughly analyzed. The main result is a statement of “correlation growth-anticorrelation decay” principle which prompts near-optimal policy to configure model parameters. This principle is in line with requiring the binary clusterization convergence which can be defined as the necessary condition for optimal learning and used as the simple benchmark for tuning parameters of various neural network realizations with population-rate information coding. At last, a heuristic algorithm is described to experimentally find out the convergence conditions in a memristive SNN, including robustness to a device variability. Due to the generality of the proposed approach, it can be applied to a wide range of memristors and neurons of software- or hardware-based rate-coding single-layer SNNs when searching for local rules that ensure their unsupervised learning convergence in a pattern recognition task domain.

Memristive Logic Design of Multifunctional Spiking Neural Network with Unsupervised Learning

We report a prospective approach to neural network modeling based on implementation of metal-oxide heterostructures with non-volatile memory behavior and multilevel resistive switching. These structures could be used as artificial synapses in neural networks, providing a modulation of synaptic strength in the range of seven orders of magnitude. Together with leaky integrated-and-fire neurons, it allows to organize a feed-forward spiking neural network with embedded spike-timing-dependent plasticity mechanism at the hardware level. The results of computer simulation demonstrate an ability of reconstructed networks based on electronic multilevel synapses to unsupervised learning and processing of asynchronous stream of spikes. Following the results of computer simulation, the biologically inspired circuit design for artificial neural network with memristive synapses was developed. The idea underlying the circuit design is based on the analogue approach and implies the hybrid CMOS-neurons/memristive synapses network, where operational amplifiers used as elements of current control through the memristive devices.

SpiFoG: an efficient supervised learning algorithm for the network of spiking neurons

There has been a lot of research on supervised learning in spiking neural network (SNN) for a couple of decades to improve computational efficiency. However, evolutionary algorithm based supervised learning for SNN has not been investigated thoroughly which is still in embryo stage. This paper introduce an efficient algorithm (SpiFoG) to train multilayer feed forward SNN in supervised manner that uses elitist floating point genetic algorithm with hybrid crossover. The evidence from neuroscience claims that the brain uses spike times with random synaptic delays for information processing. Therefore, leaky-integrate-and-fire spiking neuron is used in this research introducing random synaptic delays. The SpiFoG allows both excitatory and inhibitory neurons by allowing a mixture of positive and negative synaptic weights. In addition, random synaptic delays are also trained with synaptic weights in an efficient manner. Moreover, computational efficiency of SpiFoG was increased by reducing the total simulation time and increasing the time step since increasing time step within the total simulation time takes less iteration. The SpiFoG is benchmarked on Iris and WBC dataset drawn from the UCI machine learning repository and found better performance than state-of-the-art techniques.

Training multi-layer spiking neural networks using NormAD based spatio-temporal error backpropagation

Spiking neural networks (SNNs) have garnered a great amount of interest for supervised and unsupervised learning applications. This paper deals with the problem of training multi-layer feedforward SNNs. The non-linear integrate-and-fire dynamics employed by spiking neurons make it difficult to train SNNs to generate desired spike trains in response to a given input. To tackle this, first the problem of training a multi-layer SNN is formulated as an optimization problem such that its objective function is based on the deviation in membrane potential rather than the spike arrival instants. Then, an optimization method named Normalized Approximate Descent (NormAD), hand-crafted for such non-convex optimization problems, is employed to derive the iterative synaptic weight update rule. Next, it is reformulated to efficiently train multi-layer SNNs, and is shown to be effectively performing spatio-temporal error backpropagation. The learning rule is validated by training 2-layer SNNs to solve a spike based formulation of the XOR problem as well as training 3-layer SNNs for generic spike based training problems. Thus, the new algorithm is a key step towards building deep spiking neural networks capable of efficient event-triggered learning.

Design Partition in a Spiking Neural Arrange of Hippocampus Vigorous to Imbalanced Excitation/Inhibition in Mesh Network

Proficient design division in dentate gyrus plays an imperative part in putting away data within the hippocampus. Current information of the structure and work of the hippocampus, entorhinal cortex, and dentate gyrus, in design partition are joined in this work. A three-layer feed-forward spiking neural network inspired by the rodent hippocampus an equipped with simplified synaptic and molecular mechanisms is developed. The aim of the study is to make a spiking neural network capable of pattern separation in imbalanced excitationinhibition ratios caused by different levels of stimulations or network damage. This work presents a novel theory on the cellular mechanisms of robustness to damages to synapses and connectivity of neurons in dentate gyrus that results in imbalanced excitation-inhibition activity of neurons. This spiking neural network uses simplified molecular and cellular hypothetical mechanisms and demonstrates efficient storing of information in different levels of stimulation and can be implemented in cognitive robotics.

Representation learning using event-based STDP

Although representation learning methods developed within the framework of traditional neural networks are relatively mature, developing a spiking representation model remains a challenging problem. This paper proposes an event-based method to train a feedforward spiking neural network (SNN) layer for extracting visual features. The method introduces a novel spike-timing-dependent plasticity (STDP) learning rule and a threshold adjustment rule both derived from a vector quantization-like objective function subject to a sparsity constraint. The STDP rule is obtained by the gradient of a vector quantization criterion that is converted to spike-based, spatio-temporally local update rules in a spiking network of leaky, integrate-and-fire (LIF) neurons. Independence and sparsity of the model are achieved by the threshold adjustment rule and by a softmax function implementing inhibition in the representation layer consisting of WTA-thresholded spiking neurons. Together, these mechanisms implement a form of spike-based, competitive learning. Two sets of experiments are performed on the MNIST and natural image datasets. The results demonstrate a sparse spiking visual representation model with low reconstruction loss comparable with state-of-the-art visual coding approaches, yet our rule is local in both time and space, thus biologically plausible and hardware friendly.

Fast unsupervised learning for visual pattern recognition using spike timing dependent plasticity

Real-time learning needs algorithms operating in a fast speed comparable to human or animal, however this is a huge challenge in processing visual inputs. Research shows a biological brain can process complicated real-life recognition scenarios at milliseconds scale. Inspired by biological system, in this paper, we proposed a novel real-time learning method by combing the spike timing-based feed-forward spiking neural network (SNN) and the fast unsupervised spike timing dependent plasticity learning method with dynamic post-synaptic thresholds. Fast cross-validated experiments using MNIST database showed the high efficiency of the proposed method at an acceptable accuracy.