Spiking Neural Network System Research Articles

Unidirectional and hierarchical on-chip interconnected architecture for large-scale hardware spiking neural networks

Spiking Neural Networks (SNNs) exhibit the strong capability to address spatiotemporal dynamic problems. Recent research has explored the hardware SNN systems to solve the spatiotemporal problems in real-time. The Network-on-Chip (NoC) is an effective scheme for building large-scale hardware SNNs. However, for the existing NoC-based hardware SNNs, large area overhead and hardware power are consumed by their interconnections, because of complex topologies and router structures. Therefore, in this work a novel Unidirectional and Hierarchical on-Chip Interconnected Architecture (UHCIA) is proposed to address this problem. The proposed UHCIA mainly combines the novel hybrid topology of unidirectional multiple loops and rings, and uses a deflection router technique. Experimental results show that compared to other works, the UHCIA achieves ∼23.6X of area reduction and ∼6.4X of power reduction, with high system throughput and biological real-time computations.

Nonlinear Dynamics in HfO2/SiO2-Based Interface Dipole Modulation Field-Effect Transistors for Synaptic Applications

In the pursuit of energy-efficient spiking neural network (SNN) hardware, synaptic devices leveraging emerging memory technologies hold significant promise. This study investigates the application of the recently proposed HfO2/SiO2-based interface dipole modulation (IDM) memory for synaptic spike timing-dependent plasticity (STDP) learning. Firstly, through pulse measurements of IDM metal–oxide–semiconductor (MOS) capacitors, we demonstrate that IDM exhibits an inherently nonlinear and near-symmetric response. Secondly, we discuss the drain current response of a field-effect transistor (FET) incorporating a multi-stack IDM structure, revealing its nonlinear and asymmetric pulse response, and suggest that the degree of the asymmetry depends on the modulation current ratio. Thirdly, to emulate synaptic STDP behavior, we implement double-pulse-controlled drain current modulation of IDMFET using a simple bipolar rectangular pulse. Additionally, we propose a double-pulse-controlled synaptic depression that is valuable for optimizing STDP-based unsupervised learning. Integrating the pulse response characteristics of IDMFETs into a two-layer SNN system for synaptic weight updates, we assess training and classification performance on handwritten digits. Our results demonstrate that IDMFET-based synaptic devices can achieve classification accuracy comparable to previously reported simulation-based results.

SHIP: a computational framework for simulating and validating novel technologies in hardware spiking neural networks.

Investigations in the field of spiking neural networks (SNNs) encompass diverse, yet overlapping, scientific disciplines. Examples range from purely neuroscientific investigations, researches on computational aspects of neuroscience, or applicative-oriented studies aiming to improve SNNs performance or to develop artificial hardware counterparts. However, the simulation of SNNs is a complex task that can not be adequately addressed with a single platform applicable to all scenarios. The optimization of a simulation environment to meet specific metrics often entails compromises in other aspects. This computational challenge has led to an apparent dichotomy of approaches, with model-driven algorithms dedicated to the detailed simulation of biological networks, and data-driven algorithms designed for efficient processing of large input datasets. Nevertheless, material scientists, device physicists, and neuromorphic engineers who develop new technologies for spiking neuromorphic hardware solutions would find benefit in a simulation environment that borrows aspects from both approaches, thus facilitating modeling, analysis, and training of prospective SNN systems. This manuscript explores the numerical challenges deriving from the simulation of spiking neural networks, and introduces SHIP, Spiking (neural network) Hardware In PyTorch, a numerical tool that supports the investigation and/or validation of materials, devices, small circuit blocks within SNN architectures. SHIP facilitates the algorithmic definition of the models for the components of a network, the monitoring of states and output of the modeled systems, and the training of the synaptic weights of the network, by way of user-defined unsupervised learning rules or supervised training techniques derived from conventional machine learning. SHIP offers a valuable tool for researchers and developers in the field of hardware-based spiking neural networks, enabling efficient simulation and validation of novel technologies.

Circuit implementation of on-chip trainable spiking neural network using CMOS based memristive STDP synapses and LIF neurons

Computation on a large volume of data at high speed and low power requires energy-efficient architectures for edge computing applications. As a result, scientists focus on memristive circuits and systems for area and energy efficiency. Spiking neural network (SNN) with bio-inspired spike-timing-dependent plasticity learning (STDP) is a promising solution for energy-efficient neuromorphic systems than conventional artificial neural network (ANN). Previous works on SNN with STDP learning primarily use memristor macro models, which are software-based and cannot give complete insight into circuit implementation challenges. Some reported works on SNN use memristive devices, which require additional fabrication steps. This article presents a full circuit-level implementation of the SNN system featuring on-chip training and classification using memristive STDP synapse in standard CMOS technology. A new learning rule using the modified STDP is implemented to simplify the weight modification process. It does not involve FPGAs, CPUs, or GPUs to train the neural network. The approach used in this paper to modify the weights does not require any additional combinational or digital circuits attached to the memristive synapse resulting in less consumption of area, energy and time. We demonstrated the complete circuit-level design, implementation and simulation of SNN with on-chip training and pattern classification using 180 nm CMOS technology. A comprehensive comparison of the proposed SNN circuit with the previous related work is also presented. To demonstrate the versatility of the CMOS synapse circuit for application scenarios requiring rate-based learning, we have tuned the pair-based STDP circuit to obtain Bienenstock–Cooper–Munro (BCM) characteristics and applied it to heart rate classification.

Indium-Gallium-Zinc Oxide-Based Synaptic Charge Trap Flash for Spiking Neural Network-Restricted Boltzmann Machine.

Recently, neuromorphic computing has been proposed to overcome the drawbacks of the current von Neumann computing architecture. Especially, spiking neural network (SNN) has received significant attention due to its ability to mimic the spike-driven behavior of biological neurons and synapses, potentially leading to low-power consumption and other advantages. In this work, we designed the indium-gallium-zinc oxide (IGZO) channel charge-trap flash (CTF) synaptic device based on a HfO2/Al2O3/Si3N4/Al2O3 layer. Our IGZO-based CTF device exhibits synaptic functions with 128 levels of synaptic weight states and spike-timing-dependent plasticity. The SNN-restricted Boltzmann machine was used to simulate the fabricated CTF device to evaluate the efficiency for the SNN system, achieving the high pattern-recognition accuracy of 83.9%. We believe that our results show the suitability of the fabricated IGZO CTF device as a synaptic device for neuromorphic computing.

MTJ-based random number generation and its application in SNN handwritten digits recognition

Spiking Neural Networks (SNNs) that require synapse weight initialization using random numbers have been widely used in the neural morphological system. However, the random numbers generated by traditional digital circuits have certain repeatability, and the entire computing architecture has issues such as high resource consumption and low integration. In this letter, a hardware system for true random number generation is realized through integrating a magnetic tunnel junction, a memory cell of MRAM (magnetic random access memory) chips, with an interface circuit and using the same mechanism as writing data in spin transfer torque MRAM. The generated true random numbers are evaluated using NIST SP800-22 standard and are used for synapse weight initialization in an SNN system. The recognition rate of the system initialized by the generated true random numbers is about 84% for an MNIST handwritten digit dataset, which is 2%–3% higher than that using a traditional linear feedback shift register. The reported work provides a new approach for better SNN performance.

Design of a Configurable Spike-Encoding Circuit Based on Focal Plane Array

Spiking neural networks inspired by biological models are gaining popularity in artificial intelligence due to their ability to solve diverse problems while reducing energy consumption. As a result of the trade-off between the need to transmit large amounts of data and the power consumption of hardware deployment, artificial vision systems are particularly well-suited to construction using spiking neural networks (SNNs). How to communicate with the neuromorphic network effectively is one of the challenges associated with building systems that utilize SNN systems. It is necessary to convert the data to spike form before they can be processed by an SNN as input, unless neuromorphic or event-triggered sensing systems are employed. We present a configurable circuit based on a focal plane array (FPA) capable of providing spike-encoded readout data at the pixel level. With this type of circuit, the current signal of the photoelectric sensor can be encoded into two spike encodings with different precision, which are sent for processing to SNNs. This provides image information at two different scales for the artificial vision system based on SNNs. With this feature, we can use this circuit and different SNN structures to build an artificial target recognition system that is closer to the biological visual system.

Supervised learning and pattern recognition in photonic spiking neural networks based on MRR and phase-change materials

Traditional computers are limited by the separation of memory and processor units, is difficult to achieve fast, efficient, and low-power computing. While photonic spiking neural networks (SNNs) can overcome these shortcomings, they encounter limitations in large-scale integration. Silicon photonics platform, compatible with mature Complementary Metal Oxide Semiconductor (CMOS) platforms, is a promising candidate for realizing large-scale photonic SNNs. In this work, we proposed an integrated photonic SNN by exploiting the photonic properties of phase-change material (PCM) Ge2Sb2Te5 (GST) and micro-ring resonators (MRR), and demonstrated its integrate-and-fire (IF) behavior. Based on a system-level behavioral model, we adopt an improved Tempotron-like ReSuMe supervised learning algorithm to train the proposed photonic SNNs and complete a pattern recognition task for the clock's 12 clockwise directions. Then the influence of different noise levels is considered, and the accuracy is close to 1 when the noise level is less than 0.2. We propose a photonic implementation of such an SNN system, and use wavelength division multiplexing to achieve a scalable architecture for the pattern recognition task. The collaborative design and optimization of hardware architecture and algorithm are realized, providing a theoretical basis for the realization of photonic SNN based on MRRs and PCM.

Pattern Recognition Accuracy Optimization of Unsupervised Spiking Neural Network Using Y-Doped AlN Memristors

Inspired by the biological nervous system, the unsupervised spiking neural network (SNN) with the spike-timing-dependent plasticity (STDP) learning rule has been considered as the next-generation artificial neural network. To construct an SNN with high pattern recognition accuracy, hardware with balance synaptic behavior needs to be developed. Here, yttrium (Y)-doped aluminum nitride (AlN) memristors were proposed as artificial synapses in SNNs to investigate the dependence between the doping concentration and the pattern recognition accuracy. With the doping of Y in AlN films, both the memory characteristics and synaptic behaviors of the AlN memristors were optimized. In addition, the STDP parameters of the memristors were extracted and fed into the SNN system to simulate the pattern recognition capability. The optimized pattern recognition accuracies of 75.89% and 60.21% for the MNIST and ETH-80 datasets, respectively, were achieved for the AlN memristors with a Y-doping concentration of 3.4%, which is promising for implementation in future neuromorphic computing system and artificial intelligence.

Auditory perception architecture with spiking neural network and implementation on FPGA

Spike-based perception brings up a new research idea in the field of neuromorphic engineering. A high-performance biologically inspired flexible spiking neural network (SNN) architecture provides a novel method for the exploration of perception mechanisms and the development of neuromorphic computing systems . In this article, we present a biological-inspired spike-based SNN perception digital system that can realize robust perception. The system employs a fully paralleled pipeline scheme to improve the performance and accelerate the processing of feature extraction. An auditory perception system prototype is realized on ten Intel Cyclone field-programmable gate arrays, which can reach the maximum frequency of 107.28 MHz and the maximum throughput of 5364 Mbps. Our design also achieves the power of 5. 148 W/system and energy efficiency of 845.85 μJ. Our auditory perception implementation is also proved to have superior robustness compared with other SNN systems. We use TIMIT digit speech in noise in accuracy testing. Result shows that it achieves up to 85.75% speech recognition accuracy under obvious noise conditions (signal-to-noise ratio of 20 dB) and maintain small accuracy attenuation with the decline of the signal-to-noise ratio. The overall performance of our proposed system outperforms the state-of-the-art perception system on SNN.

A TTFS-based energy and utilization efficient neuromorphic CNN accelerator.

Spiking neural networks (SNNs), which are a form of neuromorphic, brain-inspired AI, have the potential to be a power-efficient alternative to artificial neural networks (ANNs). Spikes that occur in SNN systems, also known as activations, tend to be extremely sparse, and low in number. This minimizes the number of data accesses typically needed for processing. In addition, SNN systems are typically designed to use addition operations which consume much less energy than the typical multiply and accumulate operations used in DNN systems. The vast majority of neuromorphic hardware designs support rate-based SNNs, where the information is encoded by spike rates. Generally, rate-based SNNs can be inefficient as a large number of spikes will be transmitted and processed during inference. One coding scheme that has the potential to improve efficiency is the time-to-first-spike (TTFS) coding, where the information isn't presented through the frequency of spikes, but instead through the relative spike arrival time. In TTFS-based SNNs, each neuron can only spike once during the entire inference process, and this results in high sparsity. The activation sparsity of TTFS-based SNNs is higher than rate-based SNNs, but TTFS-based SNNs have yet to achieve the same accuracy as rate-based SNNs. In this work, we propose two key improvements for TTFS-based SNN systems: (1) a novel optimization algorithm to improve the accuracy of TTFS-based SNNs and (2) a novel hardware accelerator for TTFS-based SNNs that uses a scalable and low-power design. Our work in TTFS coding and training improves the accuracy of TTFS-based SNNs to achieve state-of-the-art results on the MNIST and Fashion-MNIST datasets. Meanwhile, our work reduces the power consumption by at least 2.4×, 25.9×, and 38.4× over the state-of-the-art neuromorphic hardware on MNIST, Fashion-MNIST, and CIFAR10, respectively.

Ultralow Power Always-On Intelligent and Connected SNN-Based System for Multimedia IoT-Enabled Applications

The recent advances in artificial neural networks (ANNs) have created immense opportunities to achieve excellent results on the Internet of Things (IoT), which serve the uses of various real-time smart applications, such as in computer vision and speech recognition. However, the efficiency of ANNs comes at the expense of a huge number of computational resources. This tends to necessitate larger and wider ANNs unapplicable for embedded systems with limited hardware resources, e.g., mobile and wearable devices. To that end, biologically realistic spiking neural networks (SNNs) are first employed to build an always-on intelligent and connected integrated IoT-enabled system with ultralow power consumption, where we encode the temporal dynamic stimuli into effective, efficient, and reconstructable spike patterns to facilitate the subsequent processing. Herein, neural encoding plays a key role in faithfully describing the temporally rich patterns for downstream cognitive tasks. Therefore, a novel nonlinear piecewise latency coding approach for a fully event-driven SNN system is developed. Moreover, a surrogate postsynaptic potential kernel function is utilized to address the nondifferential nature of the spike generation scheme when using the error backpropagation learning method. The effectiveness of the proposal tandem with SNNs has been corroborated by indicative empirical results on different data sets serving cognitive tasks.

System-Level Simulation for Integrated Phase-Change Photonics

Conventional computing systems are limited in performance by the well-known von Neumann bottleneck, arising from the physical separation of processor and memory units. The use of electrical signals in such systems also limits computing speeds and introduces significant energy losses. There is thus a pressing need for unconventional computing approaches, ones that can exploit the high bandwidths/speeds and low losses intrinsic to photonics. A promising platform for such a purpose is that offered by integrated phase-change photonics. Here, chalcogenide phase-change materials are incorporated into standard integrated photonics devices to deliver wide-ranging computational functionality, including non-volatile memory and fast, low-energy arithmetic and neuromorphic processing. We report the development of a compact behavioral model for integrated phase-change photonic devices, one which is fast enough to allow system level simulations to be run in a reasonable timescale with basic computing resources, while also being accurate enough to capture the key operating characteristics of real devices. Moreover, our model is readily incorporated with commercially available simulation software for photonic integrated circuits, thereby enabling the design, simulation and optimization of large-scale phase-change photonics systems. We demonstrate such capabilities by exploring the optimization and simulation of the operating characteristics of two important phase-change photonic systems recently reported, namely a spiking neural network system and a matrix-vector photonic crossbar array (photonic tensor core). Results show that use of our behavioral model can significantly facilitate the design and optimization at the system level, as well as expediting exploration of the capabilities of novel phase-change computing architectures.

Normalized approximate descent used for spike based automatic bird species recognition system

The paper involves a study on the application of spiking neural network (SNN), which is also known as the third generation of neural networks for automatic bird species recognition. Spiking neural network has worked incredibly well in Neuromorphic computing, such as sequence identification, character identification, etc. This neural network architecture is a consistent model, which comprises of sensory encrypt, training and decrypt functional parts. This decrypt layer utilizes an algorithm based on the normalized approximate gradient descent (NormAD) for synaptic weight adjustment. The result reflected in this paper is for processing spatiotemporal pattern, and has higher computational power over perceptron, along with biological plausibility of spiking neurons. The proposed representation uses segmentation of audio frame based on the energy of the bird call separating voiced, unvoiced and a silence portion of it. This is passed to the attribute extrication method i.e. Mel frequency Cepstral coefficient and the by-product is thus fed to the SNN system for bird species classification. The research output recognizes 14 bird species with 90.10% accuracy in comparison to the perceptron system with an accuracy of 78%.

On-Chip Training Spiking Neural Networks Using Approximated Backpropagation With Analog Synaptic Devices.

Hardware-based spiking neural networks (SNNs) inspired by a biological nervous system are regarded as an innovative computing system with very low power consumption and massively parallel operation. To train SNNs with supervision, we propose an efficient on-chip training scheme approximating backpropagation algorithm suitable for hardware implementation. We show that the accuracy of the proposed scheme for SNNs is close to that of conventional artificial neural networks (ANNs) by using the stochastic characteristics of neurons. In a hardware configuration, gated Schottky diodes (GSDs) are used as synaptic devices, which have a saturated current with respect to the input voltage. We design the SNN system by using the proposed on-chip training scheme with the GSDs, which can update their conductance in parallel to speed up the overall system. The performance of the on-chip training SNN system is validated through MNIST data set classification based on network size and total time step. The SNN systems achieve accuracy of 97.83% with 1 hidden layer and 98.44% with 4 hidden layers in fully connected neural networks. We then evaluate the effect of non-linearity and asymmetry of conductance response for long-term potentiation (LTP) and long-term depression (LTD) on the performance of the on-chip training SNN system. In addition, the impact of device variations on the performance of the on-chip training SNN system is evaluated.

Analog Complementary Metal-Oxide-Semiconductor Integrate-and-Fire Neuron Circuit for Overflow Retaining in Hardware Spiking Neural Networks.

The spiking neural network (SNN) is regarded as the third generation of an artificial neural network (ANN). In order to realize a high-performance SNN, an integrate-and-fire (I&F) neuron, one of the key elements in an SNN, must retain the overflow in its membrane after firing. This paper presents an analog CMOS I&F neuron circuit for overflow retaining. Compared with the conventional I&F neuron circuit, the basic operation of the proposed circuit is confirmed in a circuit-level simulation. Furthermore, a single-layer SNN simulation was also performed to demonstrate the effect of the proposed circuit on neural network applications by comparing the raster plots from the circuit-level simulation with those from a high-level simulation. These results demonstrate the potential of the I&F neuron circuit with overflow retaining characteristics to be utilized in upcoming high-performance hardware SNN systems.

Selection and Optimization of Temporal Spike Encoding Methods for Spiking Neural Networks.

Spiking neural networks (SNNs) receive trains of spiking events as inputs. In order to design efficient SNN systems, real-valued signals must be optimally encoded into spike trains so that the task-relevant information is retained. This paper provides a systematic quantitative and qualitative analysis and guidelines for optimal temporal encoding. It proposes a methodology of a three-step encoding workflow: method selection by signal characteristics, parameter optimization by error metrics between original and reconstructed signals, and validation by comparison of the original signal and the encoded spike train. Four encoding methods are analyzed: one stimulus estimation [Ben's Spiker algorithm (BSA)] and three temporal contrast [threshold-based, step-forward (SW), and moving-window (MW)] encodings. A short theoretical analysis is provided, and the extended quantitative analysis is carried out applying four types of test signals: step-wise signal, smooth (sinusoid) signal with added noise, trended smooth signal, and event-like smooth signal. Various time-domain and frequency spectrum properties are explored, and a comparison is provided. BSA, the only method providing unipolar spikes, was shown to be ineffective for step-wise signals, but it can follow smoothly changing signals if filter coefficients are scaled appropriately. Producing bipolar (positive and negative) spike trains, SW encoding was most effective for all types of signals as it proved to be robust and easy to optimize. Signal-to-noise ratio (SNR) can be recommended as the error metric for parameter optimization. Currently, only a visual check is available for final validation.

Comprehensive Analytic Performance Assessment and K-means based Multicast Routing Algorithm and Architecture for 3D-NoC of Spiking Neurons

Spiking neural networks (SNNs) are artificial neural network models that more closely mimic biological neural networks. In addition to neuronal and synaptic state, SNNs incorporate the variant time scale into their computational model. Since each neuron in these networks is connected to thousands of others, high bandwidth is required. Moreover, since the spike times are used to encode information in SNN, very low communication latency is also needed. The 2D-NoC was used as a solution to provide a scalable interconnection fabric in large-scale parallel SNN systems. The 3D-ICs have also attracted a lot of attention as a potential solution to resolve the interconnect bottleneck. The combination of these two emerging technologies provides a new horizon for IC designs to satisfy the high requirements of low power and small footprint in emerging AI applications. In this work, we first present a comprehensive analytical model to analyze the performance of 3D mesh NoC over variants of different SNN topologies and communications protocols. Second, we present an architecture and a low-latency spike routing algorithm, named shortest path K-means based multicast (SP-KMCR), for three-dimensional NoC of spiking neurons (3DNoC-SNN). The proposed system was validated based on an RTL-level implementation, while area/power analysis was performed using 45nm CMOS technology.

A Low-Power Actor-Critic Framework Based on Memristive Spiking Neural Network

Traditional deep reinforcement learning (DRL) algorithms consume much energy. Energy-efficient spiking neural networks (SNNs) are promising technologies to bulid a low-power reinforcement learning architecture. In this paper, an actor-critic framework based on memrisitive SNN is proposed. To convey and process information in SNN, spike encoding and decoding systems are created. Then, an improved learning algorithm based on spike-timing-dependent plasticity (STDP) learning rule is designed to combine actor-critic method with SNN. Moreover, this learning algorithm is also hardware-friendly. Besides, memristive synapse is designed to accelerate this learning algorithm. Finally, a continuous control problem is applied to illustrate the effectiveness of the proposed framework. The results show the proposed framework is prior to traditional methods.

Analytical performance assessment and high-throughput low-latency spike routing algorithm for spiking neural network systems

Large-scale artificial neural networks (ANNs) have been used to mimic the information processing function of the brain. Spiking neural networks (SNNs) are a kind of ANN, which mimic real biological neural networks, conveying information through the communication of short pulses between neurons. Since each neuron in these networks is connected to thousands of others, high bandwidth is required. Moreover, since the spike times are used to encode information in SNN, very low communication latency is also required. The 2D-NoC was used as a solution to provide a scalable interconnection fabric in large-scale parallel SNN systems. The 3D-ICs have also attracted a lot of attention as a potential solution to resolve the interconnect bottleneck. The combination of these two emerging technologies provides a new horizon for IC designs to satisfy the high requirements of low-power and small footprint in emerging AI applications. This paper first presents an analytical model to analyze the performance of different neural network topologies and compare it with a system-level simulation. Second, we present an architecture and a low-latency routing algorithm for spike traffic routing in 3D-NoC of spiking neurons (3DNoC-SNN). The 3DNoC-SNN is validated based on an RTL-level implementation, while area/power analysis is performed using 45-nm CMOS technology.