Large-scale Neuromorphic Systems Research Articles

SC-IZ: A Low-Cost Biologically Plausible Izhikevich Neuron for Large-Scale Neuromorphic Systems Using Stochastic Computing

Neurons are crucial components of neural networks, but implementing biologically accurate neuron models in hardware is challenging due to their nonlinearity and time variance. This paper introduces the SC-IZ neuron model, a low-cost digital implementation of the Izhikevich neuron model designed for large-scale neuromorphic systems using stochastic computing (SC). Simulation results show that SC-IZ can reproduce the behaviors of the original Izhikevich neuron. The model is synthesized and implemented on an FPGA. Comparative analysis shows improved hardware efficiency; reduced resource utilization, which is a 56.25% reduction in slices, 57.61% reduction in Look-Up Table (LUT) usage, and a 58.80% reduction in Flip-Flop (FF) utilization; and a higher operating frequency compared to state-of-the-art Izhikevich implementation.

Advancements in spiking neural network communication and synchronization techniques for event-driven neuromorphic systems

Neuromorphic event-driven systems emulate the computational mechanisms of the brain through the utilization of spiking neural networks (SNN). Neuromorphic systems serve two primary application domains: simulating neural information processing in neuroscience and acting as accelerators for cognitive computing in engineering applications. A distinguishing characteristic of neuromorphic systems is their asynchronous or event-driven nature, but even event-driven systems require some synchronous time management of the neuron populations to guarantee sufficient time for the proper delivery of spiking messages. In this study, we assess three distinct algorithms proposed for adding a synchronization capability to asynchronous event-driven compute systems. We run these algorithms on POETS (Partially Ordered Event-Triggered Systems), a custom-built FPGA-based hardware platform, as a neuromorphic architecture. This study presents the simulation speed of SNNs of various sizes. We explore essential aspects of event-driven neuromorphic system design that contribute to efficient computation and communication. These aspects include varying degrees of connectivity, routing methods, mapping techniques onto hardware components, and firing rates. The hardware mapping and simulation of up to eight million neurons, where each neuron is connected to up to one thousand other neurons, are presented in this work using 3072 reconfigurable processing cores, each of which has 16 hardware threads. Using the best synchronization and communication methods, our architecture design demonstrates 20-fold and 16-fold speedups over the Brian simulator and one 48-chip SpiNNaker node, respectively. We conclude with a brief comparison between our platform and existing large-scale neuromorphic systems in terms of synchronization, routing, and communication methods, to guide the development of future event-driven neuromorphic systems.

An Analytical MOS Device Model With Mismatch and Temperature Variation for Subthreshold Circuits

Subthreshold analog circuits are attractive for low-power, large-scale neuromorphic systems. However, subthreshold currents are exponentially sensitive to temperature and device mismatch, and a compact model that accounts for these effects is needed. We develop an analytical compact model with mismatch and temperature variation for subthreshold MOS devices. The model only requires an initial set of Monte Carlo (MC) simulations on individual devices for parameter extraction. Then the designer can use its parameterized analytical expressions for circuit design, instead of running repeated MC simulations on large circuits. We apply this model to a subthreshold current mirror design example. Good agreement between the developed model and Spectre simulations is achieved in a 28-nm fully-depleted silicon-on-insulator (FDSOI) process. The model is general and can also guide the design of other subthreshold circuits, such as low-power silicon neurons. It has been used to design Braindrop, the first neuromorphic chip programmed at a high level of abstraction.

Exploring Neuromorphic Computing Based on Spiking Neural Networks: Algorithms to Hardware

Neuromorphic Computing, a concept pioneered in the late 1980s, is receiving a lot of attention lately due to its promise of reducing the computational energy, latency, as well as learning complexity in artificial neural networks. Taking inspiration from neuroscience, this interdisciplinary field performs a multi-stack optimization across devices, circuits, and algorithms by providing an end-to-end approach to achieving brain-like efficiency in machine intelligence. On one side, neuromorphic computing introduces a new algorithmic paradigm, known as Spiking Neural Networks (SNNs), which is a significant shift from standard deep learning and transmits information as spikes (“1” or “0”) rather than analog values. This has opened up novel algorithmic research directions to formulate methods to represent data in spike-trains, develop neuron models that can process information over time, design learning algorithms for event-driven dynamical systems, and engineer network architectures amenable to sparse, asynchronous, event-driven computing to achieve lower power consumption. On the other side, a parallel research thrust focuses on development of efficient computing platforms for new algorithms. Standard accelerators that are amenable to deep learning workloads are not particularly suitable to handle processing across multiple timesteps efficiently. To that effect, researchers have designed neuromorphic hardware that rely on event-driven sparse computations as well as efficient matrix operations. While most large-scale neuromorphic systems have been explored based on CMOS technology, recently, Non-Volatile Memory (NVM) technologies show promise toward implementing bio-mimetic functionalities on single devices. In this article, we outline several strides that neuromorphic computing based on spiking neural networks (SNNs) has taken over the recent past, and we present our outlook on the challenges that this field needs to overcome to make the bio-plausibility route a successful one.

High Dynamic Range Digital Neuron Core With Time-Embedded Floating-Point Arithmetic

Recently, many large-scale neuromorphic systems that emulate spiking neural networks have been presented. Biological evidence emphasizes the importance of the log-normal distribution of biological neural and synaptic parameters in the brain; however, this fact is easily ignored sometimes, and the parameters are excessively optimized to scale up a system. This is because high-precision parameters require floating-point arithmetic–an operation known to consume high-energy and result in a high implementation cost in digital hardware. In this study, we propose a novel neuron implementation model that enhances neural and synaptic dynamics using the time-embedded floating-point arithmetic for better biological plausibility and low-power consumption. The proposed algorithm enables sharing temporal information with a membrane potential by time-embedded floating-point arithmetic, thus minimizing the memory usage of the neural state. In addition, this method need not access the static random-access memory at every time step, thus reducing the dynamic power consumption, even with a floating-point precision neural and synaptic dynamics. Using the proposed model, we implemented a core group with a total of 8,192 neurons on a field-programmable gate array device, Xilinx XC7K160T. The core group is designed for use in large-scale neuromorphic systems. We tested the neuron model in a core under various experimental conditions.

Is neuromorphic computing disruptive enough to 1) advance our understanding of the brain and 2) make the design and working of (bio)electronic devices efficient and scalable?

Neuromorphic computing is an approach to computing that is inspired by the structure and function of the brain. In terms of basic research, it aims at improving our understanding of biological intelligence by replicating aspects of its physical substrate – spiking neurons, synaptic plasticity etc. – and harness them towards also replicating its function. With respect to technological advances, it aims to inherit the brain’s combination of computational prowess and extreme energy efficiency; this is thought to foster a plethora of applications, from large-scale neuromorphic systems for machine learning to small-scale edge devices for signal processing and control, for example in the form of wearables for healthcare or adaptive sensors/processors for autonomous agents. The demand and usefulness of neuromorphic computing in bioelectronics is likely to increase in the future as researchers continue to explore its capabilities and develop new applications.

Low Energy, Non-Cortical, Graphene Nanoribbon-Based STDP Plastic Synapses

The realization of energy efficient, low area, and fast processing neuron and synapse circuits is of prime importance for unleashing neuromorphic computing full potential. In this paper, we introduce a graphene-based synapse, which can emulate Spike Timing Dependent Plasticity (STDP) and Short/Long Term Plasticity (STP/LTP) with variable signal amplitude and temporal dynamics. The synapse operation is validated by means of SPICE simulations, and its synaptic modulation ability is showcased through reinforcement learning within a Spiking Neural Network for robotic navigation with obstacles avoidance. Besides its functional versatility, the proposed graphene-based synapse can potentially occupy low active area ($ \approx 170{\kern 1pt} {\mathrm{n}}{{\mathrm{m}}^2}$≈170nm2) and operate at low voltage ($200{\kern 1pt} {\mathrm{mV}}$200 mV ). When compared with a biological brain synapse, its energy consumption per spike for a weight update operation ($0.5{\kern 1pt} {\mathrm{fJ}}$0.5 fJ ) is $20 \times $20× lower, while the processing speed is increased by six orders of magnitude. Such properties are essential desiderata for the realization of large scale neuromorphic systems, making the proposed graphene-based synapse an outstanding candidate for this purpose.

(Digital Presentation) Electrochemical Metallization Cell Based Memristive Neuron Chip Fabricated with 28nm CMOS Process for Real-Time Unsupervised Learning and Pattern Recognition

Spiking neurons communicate with other neurons using sparse and binary signals in a human brain, so they can achieve real-time processing of the information with ultra-low power consumption[1,2]. Thereby, spiking neurons are essential elements for building an energy-efficient biomimetic spatiotemporal system. Recently, to emulate the behavior of biological neuron, many researches for memristive device-based neurons with peripheral circuits (i.e., sense-amplifier or reset circuit)[3] and complementary metal-oxide-semiconductor (CMOS) neurons with capacitors have been reported[4]. Most of the reported memristive device-based neurons required a high operation voltage (>1.2 V) for emulating integrate function of a biological neuron. In addition, complementary metal-oxide-semiconductor-based neurons could not achieve high neuronal density due to using a capacitor in emulating integrate function.In this study, therefore, we propose an electrochemical metallization cell based memristive neuron chip fabricated with 28-nm CMOS process having a low operation voltage (<0.7 V) for emulating integrate function. In addition, since the proposed electrochemical metallization cell based memristive neuron chip does not require a capacitor for emulating integrate function, it can realize a high neuronal density. The memristive neuron chip exhibited a typical integrate-and-fire function; particularly, the frequency of generating a spiking output signal exponentially increased with the input voltage amplitude. Moreover, a spiking neural network was designed using the memristive neuron chip and the software program consisting of a crossbar synaptic memristor array, simplified spike-timing-dependent-plasticity learning rule, and current-to-voltage converters. Using the co-designed spiking neural network with software and hardware, real-time unsupervised learning was realized. Finally, using the trained spiking neural network, a real-time classification for the MNIST hand-written image taken by a live webcam was successfully performed in an inference process. Acknowledgement This research was supported by National R&D Program through the National Research Foundation of Korea(NRF) funded by Ministry of Science and ICT(2021M3F3A2A01037733). Reference Mead, C. Neuromorphic Electronic Systems. Encycl. Comput. Neurosci. 78, 1979–1979 (2015).Douglas, R. Neuromorphic Analogue VLSI. Annu. Rev. Neurosci. 18, 255–281 (1995).Tuma, T., Pantazi, A., Le Gallo, M., Sebastian, A. & Eleftheriou, E. Stochastic phase-change neurons. Nat. Nanotechnol. 11, 693–699 (2016).Aamir, S. A., Müller, P., Hartel, A., Schemmel, J. & Meier, K. A highly tunable 65-nm CMOS LIF neuron for a large scale neuromorphic system. Eur. Solid-State Circuits Conf. 2016-Octob, 71–74 (2016). Figure 1

Hierarchical Network Connectivity and Partitioning for Reconfigurable Large-Scale Neuromorphic Systems.

We present an efficient and scalable partitioning method for mapping large-scale neural network models with locally dense and globally sparse connectivity onto reconfigurable neuromorphic hardware. Scalability in computational efficiency, i.e., amount of time spent in actual computation, remains a huge challenge in very large networks. Most partitioning algorithms also struggle to address the scalability in network workloads in finding a globally optimal partition and efficiently mapping onto hardware. As communication is regarded as the most energy and time-consuming part of such distributed processing, the partitioning framework is optimized for compute-balanced, memory-efficient parallel processing targeting low-latency execution and dense synaptic storage, with minimal routing across various compute cores. We demonstrate highly scalable and efficient partitioning for connectivity-aware and hierarchical address-event routing resource-optimized mapping, significantly reducing the total communication volume recursively when compared to random balanced assignment. We showcase our results working on synthetic networks with varying degrees of sparsity factor and fan-out, small-world networks, feed-forward networks, and a hemibrain connectome reconstruction of the fruit-fly brain. The combination of our method and practical results suggest a promising path toward extending to very large-scale networks and scalable hardware-aware partitioning.

Robustness of Binary Stochastic Neurons Implemented With Low Barrier Nanomagnets Made of Dilute Magnetic Semiconductors

Binary stochastic neurons (BSNs) are excellent hardware accelerators for machine learning. A popular platform for implementing them are low- or zero-energy barrier nanomagnets possessing in-plane magnetic anisotropy (e.g. circular disks or quasi-elliptical disks with very small eccentricity). Unfortunately, small geometric variations in the lateral shapes of such nanomagnets can produce large changes in the BSN response times if the nanomagnets are made of common metallic ferromagnets (Co, Ni, Fe) with large saturation magnetization. Additionally, the response times are also very sensitive to initial conditions. Here, we show that if the nanomagnets are made of dilute magnetic semiconductors with much smaller saturation magnetization, then the variability in their response times (due to shape variations and variation in the initial condition) is drastically suppressed. This significantly reduces the device-to-device variation, which is a serious challenge for large scale neuromorphic systems.

BrainFreeze: Expanding the Capabilities of Neuromorphic Systems Using Mixed-Signal Superconducting Electronics.

Superconducting electronics (SCE) is uniquely suited to implement neuromorphic systems. As a result, SCE has the potential to enable a new generation of neuromorphic architectures that can simultaneously provide scalability, programmability, biological fidelity, on-line learning support, efficiency and speed. Supporting all of these capabilities simultaneously has thus far proven to be difficult using existing semiconductor technologies. However, as the fields of computational neuroscience and artificial intelligence (AI) continue to advance, the need for architectures that can provide combinations of these capabilities will grow. In this paper, we will explain how superconducting electronics could be used to address this need by combining analog and digital SCE circuits to build large scale neuromorphic systems. In particular, we will show through detailed analysis that the available SCE technology is suitable for near term neuromorphic demonstrations. Furthermore, this analysis will establish that neuromorphic architectures built using SCE will have the potential to be significantly faster and more efficient than current approaches, all while supporting capabilities such as biologically suggestive neuron models and on-line learning. In the future, SCE-based neuromorphic systems could serve as experimental platforms supporting investigations that are not feasible with current approaches. Ultimately, these systems and the experiments that they support would enable the advancement of neuroscience and the development of more sophisticated AI.

Considerations for Neuromorphic Supercomputing in Semiconducting and Superconducting Optoelectronic Hardware.

Any large-scale spiking neuromorphic system striving for complexity at the level of the human brain and beyond will need to be co-optimized for communication and computation. Such reasoning leads to the proposal for optoelectronic neuromorphic platforms that leverage the complementary properties of optics and electronics. Starting from the conjecture that future large-scale neuromorphic systems will utilize integrated photonics and fiber optics for communication in conjunction with analog electronics for computation, we consider two possible paths toward achieving this vision. The first is a semiconductor platform based on analog CMOS circuits and waveguide-integrated photodiodes. The second is a superconducting approach that utilizes Josephson junctions and waveguide-integrated superconducting single-photon detectors. We discuss available devices, assess scaling potential, and provide a list of key metrics and demonstrations for each platform. Both platforms hold potential, but their development will diverge in important respects. Semiconductor systems benefit from a robust fabrication ecosystem and can build on extensive progress made in purely electronic neuromorphic computing but will require III-V light source integration with electronics at an unprecedented scale, further advances in ultra-low capacitance photodiodes, and success from emerging memory technologies. Superconducting systems place near theoretically minimum burdens on light sources (a tremendous boon to one of the most speculative aspects of either platform) and provide new opportunities for integrated, high-endurance synaptic memory. However, superconducting optoelectronic systems will also contend with interfacing low-voltage electronic circuits to semiconductor light sources, the serial biasing of superconducting devices on an unprecedented scale, a less mature fabrication ecosystem, and cryogenic infrastructure.

Graphene-Based Artificial Synapses with Tunable Plasticity

Design and implementation of artificial neuromorphic systems able to provide brain akin computation and/or bio-compatible interfacing ability are crucial for understanding the human brain’s complex functionality and unleashing brain-inspired computation’s full potential. To this end, the realization of energy-efficient, low-area, and bio-compatible artificial synapses, which sustain the signal transmission between neurons, is of particular interest for any large-scale neuromorphic system. Graphene is a prime candidate material with excellent electronic properties, atomic dimensions, and low-energy envelope perspectives, which was already proven effective for logic gates implementations. Furthermore, distinct from any other materials used in current artificial synapse implementations, graphene is biocompatible, which offers perspectives for neural interfaces. In view of this, we investigate the feasibility of graphene-based synapses to emulate various synaptic plasticity behaviors and look into their potential area and energy consumption for large-scale implementations. In this article, we propose a generic graphene-based synapse structure, which can emulate the fundamental synaptic functionalities, i.e., Spike-Timing-Dependent Plasticity (STDP) and Long-Term Plasticity . Additionally, the graphene synapse is programable by means of back-gate bias voltage and can exhibit both excitatory or inhibitory behavior. We investigate its capability to obtain different potentiation/depression time scale for STDP with identical synaptic weight change amplitude when the input spike duration varies. Our simulation results, for various synaptic plasticities, indicate that a maximum 30% synaptic weight change and potentiation/depression time scale range from [-1.5 ms, 1.1 ms to [-32.2 ms, 24.1 ms] are achievable. We further explore the effect of our proposal at the Spiking Neural Network (SNN) level by performing NEST-based simulations of a small SNN implemented with 5 leaky-integrate-and-fire neurons connected via graphene-based synapses. Our experiments indicate that the number of SNN firing events exhibits a strong connection with the synaptic plasticity type, and monotonously varies with respect to the input spike frequency. Moreover, for graphene-based Hebbian STDP and spike duration of 20ms we obtain an SNN behavior relatively similar with the one provided by the same SNN with biological STDP. The proposed graphene-based synapse requires a small area (max. 30 nm 2 ), operates at low voltage (200 mV), and can emulate various plasticity types, which makes it an outstanding candidate for implementing large-scale brain-inspired computation systems.

Digital and analog functionality in monolayer AlOx-based memristors with various oxidizer sources

Memristors with the outstanding advantages are beneficial for neuromorphic computing and next-generation storage. Realizing various resistive switching behaviors in monolayer memristors is essential for understanding the device physics and fabricating fully memristive devices. In this paper, a simple and feasible method was proposed to achieve the digital and analog resistive switching in Cu/AlOx/Ag memristors by using ozone and water precursors in atomic layer deposition. According to the characterization results of surface topography, Raman spectrum and electrical measurement, the transition between the abrupt and gradual resistive switching was ascribed to the migration and diffusion of active electrode metal ions in the sparser, rougher and more amorphous AlOx dielectric films. The key features of biological synapses including long-term potentiation/depression, paired-pulse facilitation and learning-experience behaviors were emulated in the analog monolayer memristors. This study makes an important step towards the development of the sophisticated, multi-functional, and large-scale integrated neuromorphic devices and systems.

Computing Primitive of Fully VCSEL-Based All-Optical Spiking Neural Network for Supervised Learning and Pattern Classification.

We propose computing primitive for an all-optical spiking neural network (SNN) based on vertical-cavity surface-emitting lasers (VCSELs) for supervised learning by using biologically plausible mechanisms. The spike-timing-dependent plasticity (STDP) model was established based on the dynamics of the vertical-cavity semiconductor optical amplifier (VCSOA) subject to dual-optical pulse injection. The neuron-synapse self-consistent unified model of the all-optical SNN was developed, which enables reproducing the essential neuron-like dynamics and STDP function. Optical character numbers are trained and tested by the proposed fully VCSEL-based all-optical SNN. Simulation results show that the proposed all-optical SNN is capable of recognizing ten numbers by a supervised learning algorithm, in which the input and output patterns as well as the teacher signals of the all-optical SNN are represented by spatiotemporal fashions. Moreover, the lateral inhibition is not required in our proposed architecture, which is friendly to the hardware implementation. The system-level unified model enables architecture-algorithm codesigns and optimization of all-optical SNN. To the best of our knowledge, the computing primitive of an all-optical SNN based on VCSELs for supervised learning has not yet been reported, which paves the way toward fully VCSEL-based large-scale photonic neuromorphic systems with low power consumption.

Ultrahigh drive current and large selectivity in GeS selector

Selector devices are indispensable components of large-scale nonvolatile memory and neuromorphic array systems. Besides the conventional silicon transistor, two-terminal ovonic threshold switching device with much higher scalability is currently the most industrially favored selector technology. However, current ovonic threshold switching devices rely heavily on intricate control of material stoichiometry and generally suffer from toxic and complex dopants. Here, we report on a selector with a large drive current density of 34 MA cm−2 and a ~106 high nonlinearity, realized in an environment-friendly and earth-abundant sulfide binary semiconductor, GeS. Both experiments and first-principles calculations reveal Ge pyramid-dominated network and high density of near-valence band trap states in amorphous GeS. The high-drive current capacity is associated with the strong Ge-S covalency and the high nonlinearity could arise from the synergy of the mid-gap traps assisted electronic transition and local Ge-Ge chain growth as well as locally enhanced bond alignment under high electric field.

Presynaptic Spike-Driven Spike Timing-Dependent Plasticity With Address Event Representation for Large-Scale Neuromorphic Systems

Learning plays an important role in the brain to make it adaptive to dynamical environments. This paper presents a presynaptic spike-driven spike timing-dependent plasticity (STDP) learning rule in the address domain for a neuromorphic architecture using a synaptic connectivity table in an external memory at a local routing node. We contribute two aspects to the implementation of the learning rule for extended large-scale neuromorphic systems. First, we reduced buffer sizes required for tracing a spike train which is required to pair all presynaptic and postsynaptic spike for an STDP time window. This method implements an exponential decay STDP function with two parameters: the latest timestamp and the synaptic modification rate at the latest timestamp. It reduces the required buffer size compared to previous works. Second, we resolve a lack of reverse lookup table issue with the presynaptic spike-driven algorithm. The proposed algorithm holds causal updates at postsynaptic spikes until a next presynaptic spike arrival. This approach removes the need of a reverse lookup table required at a postsynaptic spike. We show the implementation of the proposed algorithm in an FPGA device and validate it with a spiking neural network configuration. The experiment results show the proposed algorithm is comparable qualitatively with a conventional STDP learning rule.

Low-Cost Adaptive Exponential Integrate-and-Fire Neuron Using Stochastic Computing.

Neurons are the primary building block of the nervous system. Exploring the mysteries of the brain in science or building a novel brain-inspired hardware substrate in engineering are inseparable from constructing an efficient biological neuron. Balancing the functional capability and the implementation cost of a neuron is a grand challenge in neuromorphic field. In this paper, we present a low-cost adaptive exponential integrate-and-fire neuron, called SC-AdEx, for large-scale neuromorphic systems using stochastic computing. In the proposed model, arithmetic operations are performed on stochastic bit-streams with small and low-power circuitry. To evaluate the proposed neuron, we perform biological behavior analysis, including various firing patterns. Furthermore, the model is synthesized and implemented physically on FPGA as a proof of concept. Experimental results show that our model can precisely reproduce wide range biological behaviors as the original model, with higher computational performance and lower hardware cost against state-of-the-art AdEx hardware neurons.

Reliability of analog resistive switching memory for neuromorphic computing

As artificial intelligence calls for novel energy-efficient hardware, neuromorphic computing systems based on analog resistive switching memory (RSM) devices have drawn great attention recently. Different from the well-studied binary RSMs, the analog RSMs are featured by a continuous and controllable conductance-tuning ability and thus are capable of combining analog computing and data storage at the device level. Although significant research achievements on analog RSMs have been accomplished, there have been few works demonstrating large-scale neuromorphic systems. A major bottleneck lies in the reliability issues of the analog RSM, such as endurance and retention degradation and read/write noises and disturbances. Owing to the complexity of resistive switching mechanisms, studies on the origins of reliability degradation and the corresponding optimization methodology face many challenges. In this article, aiming on the high-performance neuromorphic computing applications, we provide a comprehensive review on the status of reliability studies of analog RSMs, the reliability requirements, and evaluation criteria and outlook for future reliability research directions in this field.

A flashback for resistive memory

Purely electronic memristive devices based on CMOS flash memory technology could lead to large-scale neuromorphic systems.