Hardware Spiking Neural Networks Research Articles

Recognizing Spatiotemporal Features by a Neuromorphic Network with Highly Reliable Ferroelectric Capacitors on Epitaxial GeSn Film.

Epitaxial GeSn (epi-GeSn) shows the capability to form ferroelectric capacitors (FeCAPs) with a higher remanent polarization (Pr) than epi-Ge. With the interface engineering by a high-k AlON, the reliability of the epi-GeSn-based FeCAPs is enhanced. Using the highly reliable FeCAP in series with a resistor as the synapse and axon, a simplified neuromorphic network based on a differentiator circuit is proposed. The network not only holds the leaky integrate-and-fire (LIF) function but is also capable of recognizing the spatiotemporal features, which sets it apart from other LIF neurons arising from the FeCAP-modulated leaky behavior of the potential on the axon by spiking-time-dependent plasticity. Furthermore, it is more energy efficient to operate, nondestructive to read, and simpler to fabricate by employing FeCAPs, making it eligible for emergent spiking neural network hardware accelerators.

Hybrid memristor-CMOS neurons for in-situ learning in fully hardware memristive spiking neural networks

Spiking neural network, inspired by the human brain, consisting of spiking neurons and plastic synapses, is a promising solution for highly efficient data processing in neuromorphic computing. Recently, memristor-based neurons and synapses are becoming intriguing candidates to build spiking neural networks in hardware, owing to the close resemblance between their device dynamics and the biological counterparts. However, the functionalities of memristor-based neurons are currently very limited, and a hardware demonstration of fully memristor-based spiking neural networks supporting in-situ learning is very challenging. Here, a hybrid spiking neuron combining a memristor with simple digital circuits is designed and implemented in hardware to enhance neuron functions. The hybrid neuron with memristive dynamics not only realizes the basic leaky integrate-and-fire neuron function but also enables the in-situ tuning of the connected synaptic weights. Finally, a fully hardware spiking neural network with the hybrid neurons and memristive synapses is experimentally demonstrated for the first time, and in-situ Hebbian learning is achieved with this network. This work opens up a way towards the implementation of spiking neurons, supporting in-situ learning for future neuromorphic computing systems.

Minimally buffered deflection router for spiking neural network hardware implementations

Spiking neural networks (SNNs) have the potential to closely mimic the information processing of biological brains, by using massive neurons that are interconnected in a complex network. Recent researches have considered using electronic hardware circuits to SNN implementations to meet real-time processing requirements. Network-on-Chips (NoCs) have been widely used to develop such SNN circuits as their interconnections can offer stable interconnectivity for neuron communications with high throughput and real-time execution. However, its scalability is limited due to expensive and complex NoC routers which leads to high energy consumption and large area utilization. Therefore, a minimally buffered deflection router (MBDR) is proposed in this work to address the scalability challenge of the hardware SNNs. It employs a deflection router technique to remove most of the inter-buffers and other expensive components of the conventional routers. Moreover, a novel flow controller is developed in MBDR to further reduce power consumption. Compared to existing approaches, experimental results show that based on 90-nm CMOS technology the area and power consumption of the proposed router are reduced by ~ 86% and ~ 88%, respectively. In the meantime, system throughput is maintained at a high level.

Quantized Weight Transfer Method Using Spike-Timing-Dependent Plasticity for Hardware Spiking Neural Network

A hardware-based spiking neural network (SNN) has attracted many researcher’s attention due to its energy-efficiency. When implementing the hardware-based SNN, offline training is most commonly used by which trained weights by a software-based artificial neural network (ANN) are transferred to synaptic devices. However, it is time-consuming to map all the synaptic weights as the scale of the neural network increases. In this paper, we propose a method for quantized weight transfer using spike-timing-dependent plasticity (STDP) for hardware-based SNN. STDP is an online learning algorithm for SNN, but we utilize it as the weight transfer method. Firstly, we train SNN using the Modified National Institute of Standards and Technology (MNIST) dataset and perform weight quantization. Next, the quantized weights are mapped to the synaptic devices using STDP, by which all the synaptic weights connected to a neuron are transferred simultaneously, reducing the number of pulse steps. The performance of the proposed method is confirmed, and it is demonstrated that there is little reduction in the accuracy at more than a certain level of quantization, but the number of pulse steps for weight transfer substantially decreased. In addition, the effect of the device variation is verified.

NUMERICAL SIMULATION AND EXPERIMENTAL STUDY OF A HARDWARE PULSE NEURAL NETWORK WITH MEMRISTOR SYNAPSES

This article presents the results of a numerical simulation and an experimental study of the electrical circuit of a hardware spiking perceptron based on a memristor-diode crossbar. That has required developing and manufacturing a measuring bench, the electrical circuit of which consists of a hardware perceptron circuit and an input peripheral electrical circuit to implement the activation functions of the neurons and ensure the operation of the memory matrix in a spiking mode. The authors have performed a study of the operation of the hardware spiking neural network with memristor synapses in the form of a memory matrix in the mode of a single-layer perceptron synapses. The perceptron can be considered as the first layer of a biomorphic neural network that performs primary processing of incoming information in a biomorphic neuroprocessor. The obtained experimental and simulation learning curves show the expected increase in the proportion of correct classifications with an increase in the number of training epochs. The authors demonstrate generating a new association during retraining caused by the presence of new input information. Comparison of the results of modeling and an experiment on training a small neural network with a small crossbar will allow creating adequate models of hardware neural networks with a large memristor-diode crossbar. The arrival of new unknown information at the input of the hardware spiking neural network can be related with the generation of new associations in the biomorphic neuroprocessor. With further improvement of the neural network, this information will be comprehended and, therefore, will allow the transition from weak to strong artificial intelligence.

E3NE: An End-to-End Framework for Accelerating Spiking Neural Networks with Emerging Neural Encoding on FPGAs

Compiler frameworks are crucial for the widespread use of FPGA-based deep learning accelerators. They allow researchers and developers, who are not familiar with hardware engineering, to harness the performance attained by domain-specific logic. There exists a variety of frameworks for conventional artificial neural networks. However, not much research effort has been put into the creation of frameworks optimized for spiking neural networks (SNNs). This new generation of neural networks becomes increasingly interesting for the deployment of AI on edge devices, which have tight power and resource constraints. Our end-to-end framework E3NE automates the generation of efficient SNN inference logic for FPGAs. Based on a PyTorch model and user parameters, it applies various optimizations and assesses trade-offs inherent to spike-based accelerators. Multiple levels of parallelism and the use of an emerging neural encoding scheme result in an efficiency superior to previous SNN hardware implementations. For a similar model, E3NE uses less than 50\% of hardware resources and 20\% less power, while reducing the latency by an order of magnitude. Furthermore, scalability and generality allowed the deployment of the large-scale SNN models AlexNet and VGG.

IMPULSE: A 65-nm Digital Compute-in-Memory Macro With Fused Weights and Membrane Potential for Spike-Based Sequential Learning Tasks

The inherent dynamics of the neuron membrane potential in spiking neural networks (SNNs) allows the processing of sequential learning tasks, avoiding the complexity of recurrent neural networks. The highly sparse spike-based computations in such spatiotemporal data can be leveraged for energy efficiency. However, the membrane potential incurs additional memory access bottlenecks in current SNN hardware. To that effect, we propose a 10T-SRAM compute-in-memory (CIM) macro, specifically designed for state-of-the-art SNN inference. It consists of a fused weight ( W <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> MEM</sub> ) and membrane potential ( V <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> MEM</sub> ) memory and inherently exploits sparsity in input spikes leading to ~97.4% reduction in energy-delay product (EDP) at 85% sparsity (typical of SNNs considered in this work) compared to the case of no sparsity. We propose staggered data mapping and reconfigurable peripherals for handling different bit precision requirements of W <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> MEM</sub> and V <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> MEM</sub> , while supporting multiple neuron functionalities. The proposed macro was fabricated in 65-nm CMOS technology, achieving energy efficiency of 0.99 TOPS/W at 0.85-V supply and 200-MHz frequency for signed 11-bit operations. We evaluate the SNN for sentiment classification from the IMDB dataset of movie reviews and achieve within ~1% accuracy difference and ~ 5× higher energy efficiency compared to a corresponding long short-term memory network.

Digital Multiplier-Less Spiking Neural Network Architecture of Reinforcement Learning in a Context-Dependent Task

Neuromorphic engineers develop event-based spiking neural networks (SNNs) in hardware. These SNNs closer resemble the dynamics of biological neurons than conventional artificial neural networks and achieve higher efficiency thanks to the event-based, asynchronous nature of the processing. Learning in the hardware SNNs is a more challenging task, however. The conventional supervised learning methods cannot be directly applied to SNNs due to the non-differentiable event-based nature of their activation. For this reason, learning in SNNs is currently an active research topic. Reinforcement learning (RL) is a particularly promising learning method for neuromorphic implementation, especially in the field of autonomous agents' control. An SNN realization of a bio-inspired RL model is in the focus of this work. In particular, in this article, we propose a new digital multiplier-less hardware implementation of an SNN with RL capability. We show how the proposed network can learn stimulus-response associations in a context-dependent task. The task is inspired by biological experiments that study RL in animals. The architecture is described using the standard digital design flow and uses power- and space-efficient cores. The proposed hardware SNN model is compared both to data from animal experiments and to a computational model. We perform a comparison to the behavioral experiments using a robot, to show the learning capability in hardware in a closed sensory-motor loop.

An memristor-based synapse implementation using BCM learning rule

A novel memristive synapse model based on the HP memristor is proposed in this paper, which can address the problem of synaptic weight infinite modulations. The sliding threshold mechanism of the Bienenstock-Cooper-Munro rule (BCM) is used to redefine the memristance (i.e. synaptic weight) adjustment process of the memristive synapse model. Based on the proposed memristor-based synapse and Leaky Integrate-and-Fire neurons, a spiking neural network (SNN) hardware fragment is constructed, where spike trains with different frequencies are used to evaluate the stability performance of the proposed SNN hardware. Results show that compared to other approaches, the network is stable under different stimuli due to the characteristics of the memristor-based synapse model, and prove that the proposed synapse model is able to mimic biological synaptic behaviour and the problem of synaptic weight infinite modulations is addressed.

Case Study-Spiking Neural Network Hardware System for Structural Health Monitoring.

This case study provides feasibility analysis of adapting Spiking Neural Networks (SNN) based Structural Health Monitoring (SHM) system to explore low-cost solution for inspection of structural health of damaged buildings which survived after natural disaster that is, earthquakes or similar activities. Various techniques are used to detect the structural health status of a building for performance benchmarking, including different feature extraction methods and classification techniques (e.g., SNN, K-means and artificial neural network etc.). The SNN is utilized to process the sensory data generated from full-scale seven-story reinforced concrete building to verify the classification performances. Results show that the proposed SNN hardware has high classification accuracy, reliability, longevity and low hardware area overhead.

ASIE

Neuromorphic computing based on spiking neural network (SNN) shows good energy-efficiency. However, it is inefficient for SNN to perform the convolution based on frame. It may contain a lot of redundant information in the frame. The output of Dynamic Vision Sensors (DVS) is a stream event based on Address Event Representation (AER). The asynchronous nature of AER events makes the event-based convolution reflect the characteristics of SNN low energy consumption. This article presents an SNN hardware inference engine based on an asynchronous Processing Element (PE) array with AER events as input. The engine uses a convolution algorithm based on AER events. This design also uses distributed storage in the PE array to store the state of neurons to reduce the cost of memory access. The experimental results show that the design can achieve a recognition accuracy of 98.0% for the MNIST AER dataset. The design can perform the reference process more efficiently in the case where the accuracy of the loss is negligible. During the filling and draining processes of the systolic array, the number of active PE units in our PE array is reduced and, thus, the average power consumption per PE unit is drastically decreased.

Supervised Learning in All FeFET-Based Spiking Neural Network: Opportunities and Challenges.

The two possible pathways toward artificial intelligence (AI)—(i) neuroscience-oriented neuromorphic computing [like spiking neural network (SNN)] and (ii) computer science driven machine learning (like deep learning) differ widely in their fundamental formalism and coding schemes (Pei et al., 2019). Deviating from traditional deep learning approach of relying on neuronal models with static nonlinearities, SNNs attempt to capture brain-like features like computation using spikes. This holds the promise of improving the energy efficiency of the computing platforms. In order to achieve a much higher areal and energy efficiency compared to today’s hardware implementation of SNN, we need to go beyond the traditional route of relying on CMOS-based digital or mixed-signal neuronal circuits and segregation of computation and memory under the von Neumann architecture. Recently, ferroelectric field-effect transistors (FeFETs) are being explored as a promising alternative for building neuromorphic hardware by utilizing their non-volatile nature and rich polarization switching dynamics. In this work, we propose an all FeFET-based SNN hardware that allows low-power spike-based information processing and co-localized memory and computing (a.k.a. in-memory computing). We experimentally demonstrate the essential neuronal and synaptic dynamics in a 28 nm high-K metal gate FeFET technology. Furthermore, drawing inspiration from the traditional machine learning approach of optimizing a cost function to adjust the synaptic weights, we implement a surrogate gradient (SG) learning algorithm on our SNN platform that allows us to perform supervised learning on MNIST dataset. As such, we provide a pathway toward building energy-efficient neuromorphic hardware that can support traditional machine learning algorithms. Finally, we undertake synergistic device-algorithm co-design by accounting for the impacts of device-level variation (stochasticity) and limited bit precision of on-chip synaptic weights (available analog states) on the classification accuracy.

Band-to-Band Tunneling Based Ultra-Energy-Efficient Silicon Neuron

The human brain comprises about a hundred billion neurons connected through quadrillion synapses. Spiking Neural Networks (SNNs) take inspiration from the brain to model complex cognitive and learning tasks. Neuromorphic engineering implements SNNs in hardware, aspiring to mimic the brain at scale (i.e., 100 billion neurons) with biological area and energy efficiency. The design of ultra-energy efficient and compact neurons is essential for the large-scale implementation of SNNs in hardware. In this work, we have experimentally demonstrated a Partially Depleted (PD) Silicon-On-Insulator (SOI) MOSFET based Leaky-Integrate & Fire (LIF) neuron where energy-and area-efficiency is enabled by two elements of design - first tunneling based operation and second compact sub-threshold SOI control circuit design. Band-to-Band Tunneling (BTBT) induced hole storage in the body is used for the "Integrate" function of the neuron. A compact control circuit "Fires" a spike when the body potential exceeds the firing threshold. The neuron then "Resets" by removing the stored holes from the body contact of the device. Additionally, the control circuit provides "Leakiness" in the neuron which is an essential property of biological neurons. The proposed neuron provides 10x higher area efficiency compared to CMOS design with equivalent energy/spike. Alternatively, it has 10^4x higher energy efficiency at area-equivalent neuron technologies. Biologically comparable energy- and area-efficiency along with CMOS compatibility make the proposed device attractive for large-scale hardware implementation of SNNs.

A biomorphic neuroprocessor based on a composite memristor-diode crossbar

A concept of biomorphic neuroprocessor that implements hardware spiking neural network for traditional tasks of information processing and can simulate operation of brain cortical column or its fragment is proposed. The key units of hardware neural network are memory and logic matrices, previously developed on the basis of composite memristor-diode crossbar. These matrices provide high element integration and energy efficiency compared to known neuroprocessors and individual matrices. Such efficiency is achieved by application of mixed analog-digital computations, including those that use memristors integrated in composite memristor-diode crossbars. Neuron electrical model was constructed on the basis of these matrices and the Hodgkin-Huxley biomorphic model for neuron membrane. Unlike existing neural networks with synapses based on discrete memristors, the generation of new association was demonstrated in memristor-diode crossbar by SPICE modeling of associative self-learning processes. The adaptation procedure for biomorphic neural network program to neuroprocessor hardware is defined. In essence, presented neuroprocessor is a prototype of new generation computers with artificial intelligence.

Analog Complementary Metal-Oxide-Semiconductor Integrate-and-Fire Neuron Circuit for Overflow Retaining in Hardware Spiking Neural Networks.

The spiking neural network (SNN) is regarded as the third generation of an artificial neural network (ANN). In order to realize a high-performance SNN, an integrate-and-fire (I&F) neuron, one of the key elements in an SNN, must retain the overflow in its membrane after firing. This paper presents an analog CMOS I&F neuron circuit for overflow retaining. Compared with the conventional I&F neuron circuit, the basic operation of the proposed circuit is confirmed in a circuit-level simulation. Furthermore, a single-layer SNN simulation was also performed to demonstrate the effect of the proposed circuit on neural network applications by comparing the raster plots from the circuit-level simulation with those from a high-level simulation. These results demonstrate the potential of the I&F neuron circuit with overflow retaining characteristics to be utilized in upcoming high-performance hardware SNN systems.

Impact of the Sub-Resting Membrane Potential on Accurate Inference in Spiking Neural Networks

Spiking neural networks (SNNs) are considered as the third generation of artificial neural networks, having the potential to improve the energy efficiency of conventional computing systems. Although the firing rate of a spiking neuron is an approximation of rectified linear unit (ReLU) activation in an analog-valued neural network (ANN), there remain many challenges to be overcome owing to differences in operation between ANNs and SNNs. Unlike actual biological and biophysical processes, various hardware implementations of neurons and SNNs do not allow the membrane potential to fall below the resting potential—in other words, neurons must allow the sub-resting membrane potential. Because there occur an excitatory post-synaptic potential (EPSP) as well as an inhibitory post-synaptic potential (IPSP), negatively valued synaptic weights in SNNs induce the sub-resting membrane potential at some time point. If a membrane is not allowed to hold the sub-resting potential, errors will accumulate over time, resulting in inaccurate inference operations. This phenomenon is not observed in ANNs given their use of only spatial synaptic integration, but it can cause serious performance degradation in SNNs. In this paper, we demonstrate the impact of the sub-resting membrane potential on accurate inference operations in SNNs. Moreover, several important considerations for a hardware SNN that can maintain the sub-resting membrane potential are discussed. All of the results in this paper indicate that it is essential for neurons to allow the sub-resting membrane potential in order to realize high-performance SNNs.

Boosting Throughput and Efficiency of Hardware Spiking Neural Accelerators Using Time Compression Supporting Multiple Spike Codes.

Spiking neural networks (SNNs) are the third generation of neural networks and can explore both rate and temporal coding for energy-efficient event-driven computation. However, the decision accuracy of existing SNN designs is contingent upon processing a large number of spikes over a long period. Nevertheless, the switching power of SNN hardware accelerators is proportional to the number of spikes processed while the length of spike trains limits throughput and static power efficiency. This paper presents the first study on developing temporal compression to significantly boost throughput and reduce energy dissipation of digital hardware SNN accelerators while being applicable to multiple spike codes. The proposed compression architectures consist of low-cost input spike compression units, novel input-and-output-weighted spiking neurons, and reconfigurable time constant scaling to support large and flexible time compression ratios. Our compression architectures can be transparently applied to any given pre-designed SNNs employing either rate or temporal codes while incurring minimal modification of the neural models, learning algorithms, and hardware design. Using spiking speech and image recognition datasets, we demonstrate the feasibility of supporting large time compression ratios of up to 16×, delivering up to 15.93×, 13.88×, and 86.21× improvements in throughput, energy dissipation, the tradeoffs between hardware area, runtime, energy, and classification accuracy, respectively based on different spike codes on a Xilinx Zynq-7000 FPGA. These results are achieved while incurring little extra hardware overhead.

Spike time dependent plasticity (STDP) enabled learning in spiking neural networks using domain wall based synapses and neurons

We have implemented a Spiking Neural Network (SNN) architecture using a combination of spin orbit torque driven domain wall devices and transistor based peripheral circuits as both synapses and neurons. Learning in the SNN hardware is achieved both under completely unsupervised mode and partially supervised mode through mechanisms, incorporated in our spintronic synapses and neurons, that have biological plausibility, e.g., Spike Time Dependent Plasticity (STDP) and homoeostasis. High classification accuracy is obtained on the popular Iris dataset for both modes of learning.

Design of a Power Efficient Artificial Neuron Using Superconducting Nanowires.

With the rising societal demand for more information-processing capacity with lower power consumption, alternative architectures inspired by the parallelism and robustness of the human brain have recently emerged as possible solutions. In particular, spiking neural networks (SNNs) offer a bio-realistic approach, relying on pulses, analogous to action potentials, as units of information. While software encoded networks provide flexibility and precision, they are often computationally expensive. As a result, hardware SNNs based on the spiking dynamics of a device or circuit represent an increasingly appealing direction. Here, we propose to use superconducting nanowires as a platform for the development of an artificial neuron. Building on an architecture first proposed for Josephson junctions, we rely on the intrinsic non-linearity of two coupled nanowires to generate spiking behavior, and use electrothermal circuit simulations to demonstrate that the nanowire neuron reproduces multiple characteristics of biological neurons. Furthermore, by harnessing the non-linearity of the superconducting nanowire’s inductance, we develop a design for a variable inductive synapse capable of both excitatory and inhibitory control. We demonstrate that this synapse design supports direct fan-out, a feature that has been difficult to achieve in other superconducting architectures, and that the nanowire neuron’s nominal energy performance is competitive with that of current technologies.

Electrical Tunability of Partially Depleted Silicon on Insulator (PD-SOI) Neuron

The hardware realization of spiking neural network (SNN) requires a compact and energy efficient electronic analog to the biological neuron. A knob to tune the response of the as-fabricated neuron allows the network to perform various functioning without altering the hardware. Earlier, our group has experimentally demonstrated an LIF (leaky integrate & fire) neuron on a highly matured 32 nm SOI CMOS technology. In this work, we have experimentally demonstrated electrical tunability of the same through its intrinsic charge dynamics based on impact ionization (II) enabled floating body effect. First, a tunable input threshold (Vth) is achieved by changing the drain bias. Second, above threshold, a firing frequency (f) to input (V) sensitivity (df/dV) tuning is successfully demonstrated by controlling the SOI-MOSFET’s current threshold. We show that both the independent control of sensitivity and threshold is fundamentally enabled by the non-linearity of the impact ionization based carrier dynamics. The SOI neuron provides equivalent electrical tunability to Resistor-Capacitor (RC) based LIF neurons without degrading its original area and power advantages for clock-less, asynchronous SNNs. Further, we show that the neuronal behavior (threshold and sensitivity) is a key determinant of network performance, specifically the learning accuracy. Such flexibility based on post-fabrication electrical tuning will be an attractive enabler for the SNN hardware.