Implementation Of Spiking Neural Networks Research Articles

Columnar Learning Networks for Multisensory Spatiotemporal Learning

Network features found in the brain may help implement more efficient and robust neural networks. Spiking neural networks (SNNs) process spikes in the spatiotemporal domain and can offer better energy efficiency than deep neural networks. However, most SNN implementations rely on simple point neurons that neglect the rich neuronal and dendritic dynamics. Herein, a bio‐inspired columnar learning network (CLN) structure that employs feedforward, lateral, and feedback connections to make robust classification with sparse data is proposed. CLN is inspired by the mammalian neocortex, comprising cortical columns each containing multiple minicolumns formed by interacting pyramidal neurons. A column continuously processes spatiotemporal signals from its sensor, while learning spatial and temporal correlations between features in different regions of an object along with the sensor's movement through sensorimotor interaction. CLN can be implemented using memristor crossbars with a local learning rule, spiking timing‐dependent plasticity (STDP), which can be natively obtained in second‐order memristors. CLN allows inputs from multiple sensors to be simultaneously processed by different columns, resulting in higher classification accuracy and better noise tolerance. Analysis of networks implemented on memristor crossbars shows that the system can operate at very low power and high throughput, with high accuracy and robustness to noise.

Magnetic Skyrmion-Based Spiking Neural Network for Pattern Recognition

Spiking neural network (SNN) has emerged as one of the most powerful brain-inspired computing paradigms in complex pattern recognition tasks that can be enabled by neuromorphic hardware. However, owing to the fundamental architecture mismatch between biological and Boolean logic, CMOS implementation of SNN is energy inefficient. A low-power approach with novel “neuro-mimetic” devices offering a direct mapping to synaptic and neuronal functionalities is still an open area. In this paper, SNN constructed with novel magnetic skyrmion-based leaky-integrate-fire (LIF) spiking neuron and the skyrmionic synapse crossbar is proposed. We perform a systematic device-circuit-architecture co-design for pattern recognition to evaluate the feasibility of our proposal. The simulation results demonstrated that our device has superior lower switching voltage and high energy efficiency, two times lower programming energy efficiency in comparison with CMOS devices. This work paves a novel pathway for low-power hardware design using full-skyrmion SNN architecture, as well as promising avenues for implementing neuromorphic computing schemes.

An organic synaptic circuit: toward flexible and biocompatible organic neuromorphic processing

In the nervous system synapses play a critical role in computation. In neuromorphic systems, biologically inspired hardware implementations of spiking neural networks, electronic synaptic circuits pass signals between silicon neurons by integrating pre-synaptic voltage pulses and converting them into post-synaptic currents, which are scaled by the synaptic weight parameter. The overwhelming majority of neuromorphic systems are implemented using inorganic, mainly silicon, technology. As such, they are physically rigid, require expensive fabrication equipment and high fabrication temperatures, are limited to small-area fabrication, and are difficult to interface with biological tissue. Organic electronics are based on electronic properties of carbon-based molecules and polymers and offer benefits including physical flexibility, low cost, low temperature, and large-area fabrication, as well as biocompatibility, all unavailable to inorganic electronics. Here, we demonstrate an organic differential-pair integrator synaptic circuit, a biologically realistic synapse model, implemented using physically flexible complementary organic electronics. The synapse is shown to convert input voltage spikes into output current traces with biologically realistic time scales. We characterize circuit’s responses based on various synaptic parameters, including gain and weighting voltages, time-constant, synaptic capacitance, and circuit response due to inputs of different frequencies. Time constants comparable to those of biological synapses and the neurons are critical in processing real-world sensory signals such as speech, or bio-signals measured from the body. For processing even slower signals, e.g., on behavioral time scales, we demonstrate time constants in excess of two seconds, while biologically plausible time constants are achieved by deploying smaller synaptic capacitors. We measure the circuit synaptic response to input voltage spikes and present the circuit response properties using custom-made circuit simulations, which are in good agreement with the measured behavior.

Toward the Optimal Design and FPGA Implementation of Spiking Neural Networks.

The performance of a biologically plausible spiking neural network (SNN) largely depends on the model parameters and neural dynamics. This article proposes a parameter optimization scheme for improving the performance of a biologically plausible SNN and a parallel on-field-programmable gate array (FPGA) online learning neuromorphic platform for the digital implementation based on two numerical methods, namely, the Euler and third-order Runge-Kutta (RK3) methods. The optimization scheme explores the impact of biological time constants on information transmission in the SNN and improves the convergence rate of the SNN on digit recognition with a suitable choice of the time constants. The parallel digital implementation leads to a significant speedup over software simulation on a general-purpose CPU. The parallel implementation with the Euler method enables around 180× ( 20× ) training (inference) speedup over a Pytorch-based SNN simulation on CPU. Moreover, compared with previous work, our parallel implementation shows more than 300× ( 240× ) improvement on speed and 180× ( 250× ) reduction in energy consumption for training (inference). In addition, due to the high-order accuracy, the RK3 method is demonstrated to gain 2× training speedup over the Euler method, which makes it suitable for online training in real-time applications.

Advancements in Algorithms and Neuromorphic Hardware for Spiking Neural Networks.

Artificial neural networks (ANNs) have experienced a rapid advancement for their success in various application domains, including autonomous driving and drone vision. Researchers have been improving the performance efficiency and computational requirement of ANNs inspired by the mechanisms of the biological brain. Spiking neural networks (SNNs) provide a power-efficient and brain-inspired computing paradigm for machine learning applications. However, evaluating large-scale SNNs on classical von Neumann architectures (central processing units/graphics processing units) demands a high amount of power and time. Therefore, hardware designers have developed neuromorphic platforms to execute SNNs in and approach that combines fast processing and low power consumption. Recently, field-programmable gate arrays (FPGAs) have been considered promising candidates for implementing neuromorphic solutions due to their varied advantages, such as higher flexibility, shorter design, and excellent stability. This review aims to describe recent advances in SNNs and the neuromorphic hardware platforms (digital, analog, hybrid, and FPGA based) suitable for their implementation. We present that biological background of SNN learning, such as neuron models and information encoding techniques, followed by a categorization of SNN training. In addition, we describe state-of-the-art SNN simulators. Furthermore, we review and present FPGA-based hardware implementation of SNNs. Finally, we discuss some future directions for research in this field.

Memristive synaptic device based on a natural organic material—honey for spiking neural network in biodegradable neuromorphic systems

Spiking neural network (SNN) in future neuromorphic architectures requires hardware devices to be not only capable of emulating fundamental functionalities of biological synapse such as spike-timing dependent plasticity (STDP) and spike-rate dependent plasticity (SRDP), but also biodegradable to address current ecological challenges of electronic waste. Among different device technologies and materials, memristive synaptic devices based on natural organic materials have emerged as the favourable candidate to meet these demands. The metal–insulator-metal structure is analogous to biological synapse with low power consumption, fast switching speed and simulation of synaptic plasticity, while natural organic materials are water soluble, renewable and environmental friendly. In this study, the potential of a natural organic material—honey-based memristor for SNNs was demonstrated. The device exhibited forming-free bipolar resistive switching, a high switching speed of 100 ns set time and 500 ns reset time, STDP and SRDP learning behaviours, and dissolving in water. The intuitive conduction models for STDP and SRDP were proposed. These results testified that honey-based memristive synaptic devices are promising for SNN implementation in green electronics and biodegradable neuromorphic systems.

Fault Injection Attacks in Spiking Neural Networks and Countermeasures

Spiking Neural Networks (SNN) are fast emerging as an alternative option to Deep Neural Networks (DNN). They are computationally more powerful and provide higher energy-efficiency than DNNs. While exciting at first glance, SNNs contain security-sensitive assets (e.g., neuron threshold voltage) and vulnerabilities (e.g., sensitivity of classification accuracy to neuron threshold voltage change) that can be exploited by the adversaries. We explore global fault injection attacks using external power supply and laser-induced local power glitches on SNN designed using common analog neurons to corrupt critical training parameters such as spike amplitude and neuron’s membrane threshold potential. We also analyze the impact of power-based attacks on the SNN for digit classification task and observe a worst-case classification accuracy degradation of −85.65%. We explore the impact of various design parameters of SNN (e.g., learning rate, spike trace decay constant, and number of neurons) and identify design choices for robust implementation of SNN. We recover classification accuracy degradation by 30–47% for a subset of power-based attacks by modifying SNN training parameters such as learning rate, trace decay constant, and neurons per layer. We also propose hardware-level defenses, e.g., a robust current driver design that is immune to power-oriented attacks, improved circuit sizing of neuron components to reduce/recover the adversarial accuracy degradation at the cost of negligible area, and 25% power overhead. We also propose a dummy neuron-based detection of voltage fault injection at ∼1% power and area overhead each.

Supervised training of spiking neural networks for robust deployment on mixed-signal neuromorphic processors

Mixed-signal analog/digital circuits emulate spiking neurons and synapses with extremely high energy efficiency, an approach known as “neuromorphic engineering”. However, analog circuits are sensitive to process-induced variation among transistors in a chip (“device mismatch”). For neuromorphic implementation of Spiking Neural Networks (SNNs), mismatch causes parameter variation between identically-configured neurons and synapses. Each chip exhibits a different distribution of neural parameters, causing deployed networks to respond differently between chips. Current solutions to mitigate mismatch based on per-chip calibration or on-chip learning entail increased design complexity, area and cost, making deployment of neuromorphic devices expensive and difficult. Here we present a supervised learning approach that produces SNNs with high robustness to mismatch and other common sources of noise. Our method trains SNNs to perform temporal classification tasks by mimicking a pre-trained dynamical system, using a local learning rule from non-linear control theory. We demonstrate our method on two tasks requiring temporal memory, and measure the robustness of our approach to several forms of noise and mismatch. We show that our approach is more robust than common alternatives for training SNNs. Our method provides robust deployment of pre-trained networks on mixed-signal neuromorphic hardware, without requiring per-device training or calibration.

Memristor-based multi-synaptic spiking neuron circuit for spiking neural network

Spiking neural networks (SNNs) are widely used in many fields because they work closer to biological neurons. However, due to its computational complexity, many SNNs implementations are limited to computer programs. First, this paper proposes a multi-synaptic circuit (MSC) based on memristor, which realizes the multi-synapse connection between neurons and the multi-delay transmission of pulse signals. The synapse circuit participates in the calculation of the network while transmitting the pulse signal, and completes the complex calculations on the software with hardware. Secondly, a new spiking neuron circuit based on the leaky integrate-and-fire (LIF) model is designed in this paper. The amplitude and width of the pulse emitted by the spiking neuron circuit can be adjusted as required. The combination of spiking neuron circuit and MSC forms the multi-synaptic spiking neuron (MSSN). The MSSN was simulated in PSPICE and the expected result was obtained, which verified the feasibility of the circuit. Finally, a small SNN was designed based on the mathematical model of MSSN. After the SNN is trained and optimized, it obtains a good accuracy in the classification of the IRIS-dataset, which verifies the practicability of the design in the network.

A Spiking Stochastic Neuron Based on Stacked InGaZnO Memristors

AbstractSpiking encoded stochastic neural network is believed to be energy efficient and biologically plausible and an increasing effort has been made recently to translate its great cognitive power into hardware implementations. Here, a stacked indium–gallium–zinc–oxide (IGZO)‐based threshold switching memristor with essential properties as a spiking stochastic neuron is introduced. Such IGZO spiking stochastic neuron shows a sigmoid firing probability that can be tuned by the amplitude, width, and frequency of the applied pulse sequence. More importantly, the stacked configuration is experimentally demonstrated with eliminated switching variation compared to one single memristor and a narrow relative deviation (≤6.8%) of the firing probability can be achieved. The IGZO stochastic neuron is applied to perform probabilistic unsupervised learning for handwritten digit reconstruction based on a restricted Boltzmann machine and a recognition accuracy of 91.2% can be achieved. Such IGZO stochastic neuron with reproducible firing probability emulates probabilistic computing in the brain, which is of significant importance to hardware implementation of spiking neural network to analyze sensory stimuli, produce adequate motor control, and make reasonable inference.

Benchmarking Highly Parallel Hardware for Spiking Neural Networks in Robotics.

Animal brains still outperform even the most performant machines with significantly lower speed. Nonetheless, impressive progress has been made in robotics in the areas of vision, motion- and path planning in the last decades. Brain-inspired Spiking Neural Networks (SNN) and the parallel hardware necessary to exploit their full potential have promising features for robotic application. Besides the most obvious platform for deploying SNN, brain-inspired neuromorphic hardware, Graphical Processing Units (GPU) are well capable of parallel computing as well. Libraries for generating CUDA-optimized code, like GeNN and affordable embedded systems make them an attractive alternative due to their low price and availability. While a few performance tests exist, there has been a lack of benchmarks targeting robotic applications. We compare the performance of a neural Wavefront algorithm as a representative of use cases in robotics on different hardware suitable for running SNN simulations. The SNN used for this benchmark is modeled in the simulator-independent declarative language PyNN, which allows using the same model for different simulator backends. Our emphasis is the comparison between Nest, running on serial CPU, SpiNNaker, as a representative of neuromorphic hardware, and an implementation in GeNN. Beyond that, we also investigate the differences of GeNN deployed to different hardware. A comparison between the different simulators and hardware is performed with regard to total simulation time, average energy consumption per run, and the length of the resulting path. We hope that the insights gained about performance details of parallel hardware solutions contribute to developing more efficient SNN implementations for robotics.

Toward Robust Cognitive 3D Brain-Inspired Cross-Paradigm System.

Spiking Neuromorphic systems have been introduced as promising platforms for energy-efficient spiking neural network (SNNs) execution. SNNs incorporate neuronal and synaptic states in addition to the variant time scale into their computational model. Since each neuron in these networks is connected to many others, high bandwidth is required. Moreover, since the spike times are used to encode information in SNN, a precise communication latency is also needed, although SNN is tolerant to the spike delay variation in some limits when it is seen as a whole. The two-dimensional packet-switched network-on-chip was proposed as a solution to provide a scalable interconnect fabric in large-scale spike-based neural networks. The 3D-ICs have also attracted a lot of attention as a potential solution to resolve the interconnect bottleneck. Combining these two emerging technologies provides a new horizon for IC design to satisfy the high requirements of low power and small footprint in emerging AI applications. Moreover, although fault-tolerance is a natural feature of biological systems, integrating many computation and memory units into neuromorphic chips confronts the reliability issue, where a defective part can affect the overall system's performance. This paper presents the design and simulation of R-NASH-a reliable three-dimensional digital neuromorphic system geared explicitly toward the 3D-ICs biological brain's three-dimensional structure, where information in the network is represented by sparse patterns of spike timing and learning is based on the local spike-timing-dependent-plasticity rule. Our platform enables high integration density and small spike delay of spiking networks and features a scalable design. R-NASH is a design based on the Through-Silicon-Via technology, facilitating spiking neural network implementation on clustered neurons based on Network-on-Chip. We provide a memory interface with the host CPU, allowing for online training and inference of spiking neural networks. Moreover, R-NASH supports fault recovery with graceful performance degradation.

Reconfigurable field‐programmable gate array‐based on‐chip learning neuromorphic digital implementation for nonlinear function approximation

SummaryHardware implementations of spiking neural networks, which are known as neuromorphic architectures, provide an explicit understanding of brain performance. As a result, biological features of the brain may well inspire the next generation of computers and electronic systems used in such areas as signal processing, image processing, function approximation, and pattern recognition. Approximating nonlinear functions has many uses in computer science and applied mathematics. The sigmoid is the most universal activation function in neural networks by which the relationship between biological and artificial neurons is defined. It is a suitable option for predicting the probability of anything from 0 to 1 as output. In this paper, a spiking neural network using Izhikevich neurons and a gradient descent learning algorithm are propounded to approximate the sigmoid and other nonlinear functions. The flexibility of the spiking network is demonstrated by showing the average relative errors in the approximation process. A time‐ and cost‐efficient digital neuromorphic implementation on the base of on‐chip learning method for approximating the sigmoid function is also discussed. The paper reports the results of the hardware synthesis and the spiking network's physical implementation on a field‐programmable gate array. The maximum frequency and throughput of the implemented network were 83.209 MHz and 9.86 Mb/s, respectively.

Detection of COVID-19 from CT scan images: A spiking neural network-based approach

The outbreak of a global pandemic called coronavirus has created unprecedented circumstances resulting into a large number of deaths and risk of community spreading throughout the world. Desperate times have called for desperate measures to detect the disease at an early stage via various medically proven methods like chest computed tomography (CT) scan, chest X-Ray, etc., in order to prevent the virus from spreading across the community. Developing deep learning models for analysing these kinds of radiological images is a well-known methodology in the domain of computer based medical image analysis. However, doing the same by mimicking the biological models and leveraging the newly developed neuromorphic computing chips might be more economical. These chips have been shown to be more powerful and are more efficient than conventional central and graphics processing units. Additionally, these chips facilitate the implementation of spiking neural networks (SNNs) in real-world scenarios. To this end, in this work, we have tried to simulate the SNNs using various deep learning libraries. We have applied them for the classification of chest CT scan images into COVID and non-COVID classes. Our approach has achieved very high F1 score of 0.99 for the potential-based model and outperforms many state-of-the-art models. The working code associated with our present work can be found here.

A biomimetic neural encoder for spiking neural network

Spiking neural networks (SNNs) promise to bridge the gap between artificial neural networks (ANNs) and biological neural networks (BNNs) by exploiting biologically plausible neurons that offer faster inference, lower energy expenditure, and event-driven information processing capabilities. However, implementation of SNNs in future neuromorphic hardware requires hardware encoders analogous to the sensory neurons, which convert external/internal stimulus into spike trains based on specific neural algorithm along with inherent stochasticity. Unfortunately, conventional solid-state transducers are inadequate for this purpose necessitating the development of neural encoders to serve the growing need of neuromorphic computing. Here, we demonstrate a biomimetic device based on a dual gated MoS2 field effect transistor (FET) capable of encoding analog signals into stochastic spike trains following various neural encoding algorithms such as rate-based encoding, spike timing-based encoding, and spike count-based encoding. Two important aspects of neural encoding, namely, dynamic range and encoding precision are also captured in our demonstration. Furthermore, the encoding energy was found to be as frugal as ≈1–5 pJ/spike. Finally, we show fast (≈200 timesteps) encoding of the MNIST data set using our biomimetic device followed by more than 91% accurate inference using a trained SNN.

Minimally buffered deflection router for spiking neural network hardware implementations

Spiking neural networks (SNNs) have the potential to closely mimic the information processing of biological brains, by using massive neurons that are interconnected in a complex network. Recent researches have considered using electronic hardware circuits to SNN implementations to meet real-time processing requirements. Network-on-Chips (NoCs) have been widely used to develop such SNN circuits as their interconnections can offer stable interconnectivity for neuron communications with high throughput and real-time execution. However, its scalability is limited due to expensive and complex NoC routers which leads to high energy consumption and large area utilization. Therefore, a minimally buffered deflection router (MBDR) is proposed in this work to address the scalability challenge of the hardware SNNs. It employs a deflection router technique to remove most of the inter-buffers and other expensive components of the conventional routers. Moreover, a novel flow controller is developed in MBDR to further reduce power consumption. Compared to existing approaches, experimental results show that based on 90-nm CMOS technology the area and power consumption of the proposed router are reduced by ~ 86% and ~ 88%, respectively. In the meantime, system throughput is maintained at a high level.

Spiking Neural Network (SNN) With Memristor Synapses Having Non-linear Weight Update.

Among many artificial neural networks, the research on Spike Neural Network (SNN), which mimics the energy-efficient signal system in the brain, is drawing much attention. Memristor is a promising candidate as a synaptic component for hardware implementation of SNN, but several non-ideal device properties are making it challengeable. In this work, we conducted an SNN simulation by adding a device model with a non-linear weight update to test the impact on SNN performance. We found that SNN has a strong tolerance for the device non-linearity and the network can keep the accuracy high if a device meets one of the two conditions: 1. symmetric LTP and LTD curves and 2. positive non-linearity factors for both LTP and LTD. The reason was analyzed in terms of the balance between network parameters as well as the variability of weight. The results are considered to be a piece of useful prior information for the future implementation of emerging device-based neuromorphic hardware.

The Heidelberg Spiking Data Sets for the Systematic Evaluation of Spiking Neural Networks.

Spiking neural networks are the basis of versatile and power-efficient information processing in the brain. Although we currently lack a detailed understanding of how these networks compute, recently developed optimization techniques allow us to instantiate increasingly complex functional spiking neural networks in-silico. These methods hold the promise to build more efficient non-von-Neumann computing hardware and will offer new vistas in the quest of unraveling brain circuit function. To accelerate the development of such methods, objective ways to compare their performance are indispensable. Presently, however, there are no widely accepted means for comparing the computational performance of spiking neural networks. To address this issue, we introduce two spike-based classification data sets, broadly applicable to benchmark both software and neuromorphic hardware implementations of spiking neural networks. To accomplish this, we developed a general audio-to-spiking conversion procedure inspired by neurophysiology. Furthermore, we applied this conversion to an existing and a novel speech data set. The latter is the free, high-fidelity, and word-level aligned Heidelberg digit data set that we created specifically for this study. By training a range of conventional and spiking classifiers, we show that leveraging spike timing information within these data sets is essential for good classification accuracy. These results serve as the first reference for future performance comparisons of spiking neural networks.

Organic electronics Axon-Hillock neuromorphic circuit: towards biologically compatible, and physically flexible, integrate-and-fire spiking neural networks

Spiking neural networks (SNNs) have emerged as a promising computational paradigm to emulate the features of natural neural tissue physiology. While hardware implementations of SNNs are being conceived to emulate biological systems, they typically rely on hard and rigid silicon electronics that are not bio-compatible. In the physical, or materials realm, organic electronics offer mechanical flexibility and bio-compatibility, allowing for the construction of neural processing systems that can be directly interfaced to biological networks. This study introduces an organic electronics implementation of an Integrate-and-Fire spiking neuron based on the Axon-Hillock CMOS circuit. The circuit employs organic p-type and n-type field effective transistors and reproduces the behavior of the CMOS neuromorphic counterpart. We demonstrate its operating characteristics measuring its spike rate output as a function of its input current. We show how it properly integrates input currents and demonstrate its computing abilities in a basic current summing experiment. The static and dynamic power dissipation is calculated to be less than 0.4 and 40 µW, respectively. This is the first demonstration of the spiking Axon-Hillock neuromorphic circuit using organic materials.

Optimization of Spiking Neural Networks Based on Binary Streamed Rate Coding

Spiking neural networks (SNN) increasingly attract attention for their similarity to the biological neural system. Hardware implementation of spiking neural networks, however, remains a great challenge due to their excessive complexity and circuit size. This work introduces a novel optimization method for hardware friendly SNN architecture based on a modified rate coding scheme called Binary Streamed Rate Coding (BSRC). BSRC combines the features of both rate and temporal coding. In addition, by employing a built-in randomizer, the BSRC SNN model provides a higher accuracy and faster training. We also present SNN optimization methods including structure optimization and weight quantization. Extensive evaluations with MNIST SNNs demonstrate that the structure optimization of SNN (81-30-20-10) provides 183.19 times reduction in hardware compared with SNN (784-800-10), while providing an accuracy of 95.25%, a small loss compared with 98.89% and 98.93% reported in the previous works. Our weight quantization reduces 32-bit weights to 4-bit integers leading to further hardware reduction of 4 times with only 0.56% accuracy loss. Overall, the SNN model (81-30-20-10) optimized by our method shrinks the SNN’s circuit area from 3089.49 mm2 for SNN (784-800-10) to 4.04 mm2—a reduction of 765 times.