Neuromorphic Processor Research Articles

3D-aCortex: an ultra-compact energy-efficient neurocomputing platform based on commercial 3D-NAND flash memories

We first propose an ultra-compact energy-efficient time-domain vector-by-matrix multiplier (VMM) based on commercial 3D-NAND flash memory structure. The proposed 3D-VMM uses a novel resistive successive integrate and re-scaling (RSIR) scheme to eliminate the stringent requirement of a bulky load capacitor which otherwise dominates the area- and energy-landscape of the conventional time-domain VMMs. Our rigorous analysis, performed at the 55 nm technology node, shows that RSIR-3D-VMM achieves a record-breaking area efficiency of ∼0.02 μm2/Byte and the energy efficiency of ∼6 f J/Op for a 500 × 500 4-bit VMM, representing 5× and 1.3× improvements over the previously reported 3D-VMM approach. Moreover, unlike the previous approach, the proposed VMM can be efficiently tailored to work in a smaller current output range. Our second major contribution is the development of 3D-aCortex, a multi-purpose neuromorphic inference processor that utilizes the proposed 3D-VMM block as its core processing unit. Rigorous performance modeling of the 3D-aCortex targeting several state-of-the-art neural network benchmarks has shown that it may provide a record-breaking 30.7 MB mm−2 storage efficiency, 113.3 TOp/J peak energy efficiency, and 10.66 TOp/s computational throughput. The system-level analysis indicates that the gain in the area-efficiency of RSIR leads to a smaller data transfer delay, which compensates for the reduction in the VMM throughput due to an increased input time window.

Adaptive Extreme Edge Computing for Wearable Devices.

Wearable devices are a fast-growing technology with impact on personal healthcare for both society and economy. Due to the widespread of sensors in pervasive and distributed networks, power consumption, processing speed, and system adaptation are vital in future smart wearable devices. The visioning and forecasting of how to bring computation to the edge in smart sensors have already begun, with an aspiration to provide adaptive extreme edge computing. Here, we provide a holistic view of hardware and theoretical solutions toward smart wearable devices that can provide guidance to research in this pervasive computing era. We propose various solutions for biologically plausible models for continual learning in neuromorphic computing technologies for wearable sensors. To envision this concept, we provide a systematic outline in which prospective low power and low latency scenarios of wearable sensors in neuromorphic platforms are expected. We successively describe vital potential landscapes of neuromorphic processors exploiting complementary metal-oxide semiconductors (CMOS) and emerging memory technologies (e.g., memristive devices). Furthermore, we evaluate the requirements for edge computing within wearable devices in terms of footprint, power consumption, latency, and data size. We additionally investigate the challenges beyond neuromorphic computing hardware, algorithms and devices that could impede enhancement of adaptive edge computing in smart wearable devices.

In-Memory Computing with Resistive Memory Circuits: Status and Outlook

In-memory computing (IMC) refers to non-von Neumann architectures where data are processed in situ within the memory by taking advantage of physical laws. Among the memory devices that have been considered for IMC, the resistive switching memory (RRAM), also known as memristor, is one of the most promising technologies due to its relatively easy integration and scaling. RRAM devices have been explored for both memory and IMC applications, such as neural network accelerators and neuromorphic processors. This work presents the status and outlook on the RRAM for analog computing, where the precision of the encoded coefficients, such as the synaptic weights of a neural network, is one of the key requirements. We show the experimental study of the cycle-to-cycle variation of set and reset processes for HfO2-based RRAM, which indicate that gate-controlled pulses present the least variation in conductance. Assuming a constant variation of conductance σG, we then evaluate and compare various mapping schemes, including multilevel, binary, unary, redundant and slicing techniques. We present analytical formulas for the standard deviation of the conductance and the maximum number of bits that still satisfies a given maximum error. Finally, we discuss RRAM performance for various analog computing tasks compared to other computational memory devices. RRAM appears as one of the most promising devices in terms of scaling, accuracy and low-current operation.

A Surrogate of Slow-Wave Sleep is Necessary for Stable Unsupervised Learning in a Model of Cortical Development

Sparse coding has been used to model the acquisition of V1 simple cell receptive fields. The Locally Competitive Algorithm (LCA) provides a biologically plausible implementation of sparse coding based on lateral inhibition. LCA can be reformulated to support dictionary learning via an online local Hebbian rule that reduces predictive coding error. Although originally formulated in terms of leaky integrator rate-coded neurons, LCA based on lateral inhibition between leaky integrate-and-fire (LIF) neurons has been implemented on spiking neuromorphic processors but such implementations preclude local online learning.

Advancing Neuromorphic Computing With Loihi: A Survey of Results and Outlook

Deep artificial neural networks apply principles of the brain’s information processing that led to breakthroughs in machine learning spanning many problem domains. Neuromorphic computing aims to take this a step further to chips more directly inspired by the form and function of biological neural circuits, so they can process new knowledge, adapt, behave, and learn in real time at low power levels. Despite several decades of research, until recently, very few published results have shown that today’s neuromorphic chips can demonstrate quantitative computational value. This is now changing with the advent of Intel’s Loihi, a neuromorphic research processor designed to support a broad range of spiking neural networks with sufficient scale, performance, and features to deliver competitive results compared to state-of-the-art contemporary computing architectures. This survey reviews results that are obtained to date with Loihi across the major algorithmic domains under study, including deep learning approaches and novel approaches that aim to more directly harness the key features of spike-based neuromorphic hardware. While conventional feedforward deep neural networks show modest if any benefit on Loihi, more brain-inspired networks using recurrence, precise spike-timing relationships, synaptic plasticity, stochasticity, and sparsity perform certain computation with orders of magnitude lower latency and energy compared to state-of-the-art conventional approaches. These compelling neuromorphic networks solve a diverse range of problems representative of brain-like computation, such as event-based data processing, adaptive control, constrained optimization, sparse feature regression, and graph search.

Fast Post-Hoc Normalization for Brain Inspired Sparse Coding on a Neuromorphic Device

Exploration of novel computational platforms is critical for the advancement of artificial intelligence as we approach the physical limitations of traditional hardware. Biologically accurate, energy efficient neuromorphic systems are particularly promising for enabling future breakthroughs because of their ability to process information in parallel and to scale using extremely low power. Sparse coding is a signal processing technique which has been known to model the information encoding in the primary visual cortex. When sparse solutions are solved using local neuron competition along with the unsupervised dictionary learning that mimics cortical development, we can build an end to end, hardware to software, brain inspired solution to a machine learning problem. In this article, we perform a detailed comparison of sparse coding solutions generated classically by orthogonal matching pursuit (OMP) implemented on a conventional digital processor with spike-based solutions obtained using the Intel Loihi neuromorphic processor. A novel “post-hoc” normalization technique to shorten simulation time for Loihi is presented along with analysis of optimal parameter selection, reconstruction errors, and unsupervised dictionary learning for Loihi approaches and their classical counterparts. Preliminary results show that both the Loihi full simulation approach and the post-hoc normalization approach are well suited to neuromorphic processors and operate in a size, weight and power regime that is not accessible by classical approaches. Ultimately, the use of this normalization technique allows for faster and, often, better solutions than demonstrated previously.

Multi-Tone Phase Coding of Interaural Time Difference for Sound Source Localization With Spiking Neural Networks

Mammals exhibit remarkable capability of detecting and localizing sound sources in complex acoustic environments by using binaural cues in the spiking manner. Emulating the auditory process for sound source localization (SSL) by mammals, we propose a computational model for accurate and robust SSL under the neuromorphic spiking neural network (SNN) framework. The center of this model is a Multi-Tone Phase Coding (MTPC) scheme, which encodes the interaural time difference (ITD) between binaural pure tones into discriminative spike patterns that can be directly classified by SNNs. As such, SSL can be implemented as an event-driven task on highly efficient, neuromorphic parallel processors. We evaluate the proposed computational model on a directional audio dataset recorded from a microphone array in a realistic acoustic environment with background noise, obstruction, reflection, and other interferences. We report superior localization capability with a mean absolute error (MAE) of 1.02 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">°</sup> or 100% classification accuracy with an angle resolution of 5 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">°</sup> , which surpasses other SNN-based biologically plausible neuromorphic approaches by a relatively large margin and on par with human performance in similar tasks. This study opens up many application opportunities in human-robot interaction where energy efficiency is crucial. As a case study, we successfully deploy the proposed SSL system in a robotic platform to track the speaker and orient the robot's attention.

Multi-channel optical neuromorphic processor for frequency-multiplexed signals

Here we develop the first optical neuromorphic processor for frequency-multiplexed and multi-channel signals and demonstrate its application for orthogonal frequency-division multiplexing and wavelength-division multiplexing. The presented architecture supports multichannel operation through incorporation of the optical Fourier transform and dispersion-managed fiber echo state network analogue enabling processing of dual-quadrature high bandwidth signals. We demonstrate applicability of the design for prediction and equalization tasks. The proposed technology paves the way to multi-channel neuromorphic signal processing.

A perovskite retinomorphic sensor

Designed to outperform conventional computers when performing machine-learning tasks, neuromorphic computation is the principle whereby certain aspects of the human brain are replicated in hardware. While great progress has been made in this field in recent years, almost all input signals provided to neuromorphic processors are still designed for traditional (von Neumann) computer architectures. Here, we show that a simple photosensitive capacitor will inherently reproduce certain aspects of biological retinas. We found that capacitors based on metal halide perovskites will output a brief voltage spike in response to changes in incident light intensity, but output zero voltage under constant illumination. Such a sensor is not only optimized for use with spiking neuromorphic processors but also anticipated to have broad appeal from fields such as light detection and ranging, autonomous vehicles, facile recognition, navigation, and robotics.

Hardware Implementation of Deep Network Accelerators Towards Healthcare and Biomedical Applications.

The advent of dedicated Deep Learning (DL) accelerators and neuromorphic processors has brought on new opportunities for applying both Deep and Spiking Neural Network (SNN) algorithms to healthcare and biomedical applications at the edge. This can facilitate the advancement of medical Internet of Things (IoT) systems and Point of Care (PoC) devices. In this paper, we provide a tutorial describing how various technologies including emerging memristive devices, Field Programmable Gate Arrays (FPGAs), and Complementary Metal Oxide Semiconductor (CMOS) can be used to develop efficient DL accelerators to solve a wide variety of diagnostic, pattern recognition, and signal processing problems in healthcare. Furthermore, we explore how spiking neuromorphic processors can complement their DL counterparts for processing biomedical signals. The tutorial is augmented with case studies of the vast literature on neural network and neuromorphic hardware as applied to the healthcare domain. We benchmark various hardware platforms by performing a sensor fusion signal processing task combining electromyography (EMG) signals with computer vision. Comparisons are made between dedicated neuromorphic processors and embedded AI accelerators in terms of inference latency and energy. Finally, we provide our analysis of the field and share a perspective on the advantages, disadvantages, challenges, and opportunities that various accelerators and neuromorphic processors introduce to healthcare and biomedical domains.

1225-Channel Neuromorphic Retinal-Prosthesis SoC With Localized Temperature-Regulation.

A 1225-Channel Neuromorphic Retinal Prosthesis (RP) SoC is presented. Existing RP SoCs directly convert light intensity to electrical stimulus, which limit the adoption of delicate stimulus patterns to increase visual acuity. Moreover, a conventional centralized image processor leads to the local hot spot that poses a risk to the nearby retinal cells. To solve these issues, the proposed SoC adopts a distributed Neuromorphic Image Processor (NMIP) located within each pixel that extracts the outline of the incoming image, which reduces current dispersion and stimulus power compared with light-intensity proportional stimulus pattern. A spike-based asynchronous digital operation results in the power consumption of 56.3 nW/Ch without local temperature hot spot. At every 5×5 pixels, the localized (49-point) temperature-regulation circuit limits the temperature increase of neighboring retinal cells to less than 1°C, and the overall power consumption of the SoC to be less than that of the human eye. The 1225-channel SoC fabricated in 0.18μm 1P6M CMOS occupies 15mm2 while consuming 2.7 mW, and is successfully verified with image reconstruction demonstration.

Closed-Loop Spiking Control on a Neuromorphic Processor Implemented on the iCub

Despite neuromorphic engineering promises the deployment of low latency, adaptive and low power systems that can lead to the design of truly autonomous artificial agents, the development of a fully neuromorphic artificial agent is still missing. While neuromorphic sensing and perception, as well as decision-making systems, are now mature, the control and actuation part is lagging behind. In this paper, we present a closed-loop motor controller implemented on mixed-signal analog-digital neuromorphic hardware using a spiking neural network. The network performs a proportional control action by encoding target, feedback, and error signals using a spiking relational network. It continuously calculates the error through a connectivity pattern, which relates the three variables by means of feed-forward connections. Recurrent connections within each population are used to speed up the convergence, decrease the effect of mismatch and improve selectivity. The neuromorphic motor controller is interfaced with the iCub robot simulator. We tested our spiking P controller in a single joint control task, specifically for the robot head yaw. The spiking controller sends the target positions, reads the motor state from its encoder, and sends back the motor commands to the joint. The performance of the spiking controller is tested in a step response experiment and in a target pursuit task. In this work, we optimize the network structure to make it more robust to noisy inputs and device mismatch, which leads to better control performances.

EMG-Based Gestures Classification Using a Mixed-Signal Neuromorphic Processing System

The rapid increase of wearable sensor devices poses new challenges for implementing continuous real-time processing of physiological data. Neuromorphic sensory-processing devices can enable both the measurement of bio-signals and their processing locally in compact embedded wearable systems. In particular, mixed-signal spiking neural networks implemented on neuromorphic processors can be integrated directly with the sensors to extract temporal data-streams in real-time with very low power consumption. In this work, we present a neuromorphic approach for classifying spatio-temporal data from electromyography (EMG) signals, which paves the way toward the realization of compact wearable solutions for neuroprosthetic control. Here we extend previously proposed delta-encoding methods to transform bio-signals into spike trains and use a spiking Recurrent Neural Network (SRNN) architecture to extract features from them. The SRNN was first simulated in software to find the optimal set of hyperparameters, and then validated on the neuromorphic hardware, with a difference in the performance of less than 2%. We describe how biologically plausible mechanisms such as Spike-Timing Dependent Plasticity (STDP) and soft Winner-Take-All (WTA) networks can be exploited to classify the EMG signals and show how their combined use in EMG data classification achieves competitive results with different datasets. Specifically, the classification performance for the Roshambo EMG dataset, which has three different classes, is above 85%, and for the basic finger movements dataset from the Ninapro database, which has eight different classes, reaches 55% accuracy.

A Spike-Based Neuromorphic Architecture of Stereo Vision.

The problem of finding stereo correspondences in binocular vision is solved effortlessly in nature and yet it is still a critical bottleneck for artificial machine vision systems. As temporal information is a crucial feature in this process, the advent of event-based vision sensors and dedicated event-based processors promises to offer an effective approach to solving the stereo matching problem. Indeed, event-based neuromorphic hardware provides an optimal substrate for fast, asynchronous computation, that can make explicit use of precise temporal coincidences. However, although several biologically-inspired solutions have already been proposed, the performance benefits of combining event-based sensing with asynchronous and parallel computation are yet to be explored. Here we present a hardware spike-based stereo-vision system that leverages the advantages of brain-inspired neuromorphic computing by interfacing two event-based vision sensors to an event-based mixed-signal analog/digital neuromorphic processor. We describe a prototype interface designed to enable the emulation of a stereo-vision system on neuromorphic hardware and we quantify the stereo matching performance with two datasets. Our results provide a path toward the realization of low-latency, end-to-end event-based, neuromorphic architectures for stereo vision.

Ultra-Low-Power FDSOI Neural Circuits for Extreme-Edge Neuromorphic Intelligence

Recent years have seen an increasing interest in the development of artificial intelligence circuits and systems for edge computing applications. In-memory computing mixed-signal neuromorphic architectures provide promising ultra-low-power solutions for edge-computing sensory-processing applications, thanks to their ability to emulate spiking neural networks in real-time. The fine-grain parallelism offered by this approach allows such neural circuits to process the sensory data efficiently by adapting their dynamics to the ones of the sensed signals, without having to resort to the time-multiplexed computing paradigm of von Neumann architectures. To reduce power consumption even further, we present a set of mixed-signal analog/digital circuits that exploit the features of advanced Fully-Depleted Silicon on Insulator (FDSOI) integration processes. Specifically, we explore the options of advanced FDSOI technologies to address analog design issues and optimize the design of the synapse integrator and of the adaptive neuron circuits accordingly. We present circuit simulation results and demonstrate the circuit's ability to produce biologically plausible neural dynamics with compact designs, optimized for the realization of large-scale spiking neural networks in neuromorphic processors.

Neuromorphic analog spiking-modulator for audio signal processing

While CMOS scaling is currently reaching its limits in power dissipation and circuit density, the analogy between biology and silicon is emerging as a solution to ultra-low-power signal processing. Urgent applications involving artificial vision and audition, including intelligent sensing, appeal original energy efficient and ultra-miniaturized silicon-based solutions. While state-of-the-art is focusing on digital-oriented solutions, this paper proposes a neuromorphic analog signal processor using Izhikevich-based artificial neurons in an analog spiking modulator. A varicap-based artificial neuron is explored reducing the silicon area to $$98.6\;\upmu{\text{m}}^{2}$$ and the substrate leakage to a $$1.95 \;{\text{fJ}}/{\text{spike}}$$ efficiency. Post-layout simulation results are presented to investigate the high-resolution, high-speed, and full-scale dynamic range for audio signal processing applications. The proposal demonstrates a $$9 \;{\text{bits}}$$ spiking-modulator resolution, a maximum of $$8\;{\text{fJ}}/{\text{conv}}$$ efficiency, and a root–mean–square error of $$0.63\;{\text{mV}}_{RMS}$$ .

Online Few-Shot Gesture Learning on a Neuromorphic Processor

We present the Surrogate-gradient Online Error-triggered Learning (SOEL) system for online few-shot learning on neuromorphic processors. The SOEL learning system uses a combination of transfer learning and principles of computational neuroscience and deep learning. We show that partially trained deep Spiking Neural Networks (SNNs) implemented on neuromorphic hardware can rapidly adapt online to new classes of data within a domain. SOEL updates trigger when an error occurs, enabling faster learning with fewer updates. Using gesture recognition as a case study, we show SOEL can be used for online few-shot learning of new classes of pre-recorded gesture data and rapid online learning of new gestures from data streamed live from a Dynamic Active-pixel Vision Sensor to an Intel Loihi neuromorphic research processor.

Multidimensional fiber echo state network analogue

Optical neuoromorphic technologies enable neural network-based signal processing through a specifically designed hardware and may confer advantages in speed and energy. However, the advances of such technologies in bandwidth and/or dimensionality are often limited by the constraints of the underlying material. Optical fiber presents a well-studied low-cost solution with unique advantages for low-loss high-speed signal processing. The fiber echo state network analogue (FESNA), fiber-based neuromorphic processor, has been the first technology suitable for multichannel high bandwidth (including THz) and dual-quadrature signal processing. Here we propose the multidimensional FESNA (MD-FESNA) processing by utilizing multi-mode fiber non-linearity. Thus, the developed MD-FESNA is the first neuromorphic technology which augments all aforementioned advantages of FESNA with multidimensional spatio-temporal processing. We demonstrate the performance and flexibility of the technology on the example of prediction tasks for hyperchaotic systems. These results will pave the way for a high-speed neuromorphic processing of multidimensional tasks, hardware for spatio-temporal neural networks and open new application venues for fiber-based spatio-temporal multiplexing.

Primer on silicon neuromorphic photonic processors: architecture and compiler

AbstractMicroelectronic computers have encountered challenges in meeting all of today’s demands for information processing. Meeting these demands will require the development of unconventional computers employing alternative processing models and new device physics. Neural network models have come to dominate modern machine learning algorithms, and specialized electronic hardware has been developed to implement them more efficiently. A silicon photonic integration industry promises to bring manufacturing ecosystems normally reserved for microelectronics to photonics. Photonic devices have already found simple analog signal processing niches where electronics cannot provide sufficient bandwidth and reconfigurability. In order to solve more complex information processing problems, they will have to adopt a processing model that generalizes and scales.Neuromorphic photonicsaims to map physical models of optoelectronic systems to abstract models of neural networks. It represents a new opportunity for machine information processing on sub-nanosecond timescales, with application to mathematical programming, intelligent radio frequency signal processing, and real-time control. The strategy of neuromorphic engineering is to externalize the risk of developing computational theory alongside hardware. The strategy of remaining compatible with silicon photonics externalizes the risk of platform development. In this perspective article, we provide a rationale for a neuromorphic photonics processor, envisioning its architecture and a compiler. We also discuss how it can be interfaced with a general purpose computer, i.e. a CPU, as a coprocessor to target specific applications. This paper is intended for a wide audience and provides a roadmap for expanding research in the direction of transforming neuromorphic photonics into a viable and useful candidate for accelerating neuromorphic computing.

Hand-Gesture Recognition Based on EMG and Event-Based Camera Sensor Fusion: A Benchmark in Neuromorphic Computing.

Hand gestures are a form of non-verbal communication used by individuals in conjunction with speech to communicate. Nowadays, with the increasing use of technology, hand-gesture recognition is considered to be an important aspect of Human-Machine Interaction (HMI), allowing the machine to capture and interpret the user's intent and to respond accordingly. The ability to discriminate between human gestures can help in several applications, such as assisted living, healthcare, neuro-rehabilitation, and sports. Recently, multi-sensor data fusion mechanisms have been investigated to improve discrimination accuracy. In this paper, we present a sensor fusion framework that integrates complementary systems: the electromyography (EMG) signal from muscles and visual information. This multi-sensor approach, while improving accuracy and robustness, introduces the disadvantage of high computational cost, which grows exponentially with the number of sensors and the number of measurements. Furthermore, this huge amount of data to process can affect the classification latency which can be crucial in real-case scenarios, such as prosthetic control. Neuromorphic technologies can be deployed to overcome these limitations since they allow real-time processing in parallel at low power consumption. In this paper, we present a fully neuromorphic sensor fusion approach for hand-gesture recognition comprised of an event-based vision sensor and three different neuromorphic processors. In particular, we used the event-based camera, called DVS, and two neuromorphic platforms, Loihi and ODIN + MorphIC. The EMG signals were recorded using traditional electrodes and then converted into spikes to be fed into the chips. We collected a dataset of five gestures from sign language where visual and electromyography signals are synchronized. We compared a fully neuromorphic approach to a baseline implemented using traditional machine learning approaches on a portable GPU system. According to the chip's constraints, we designed specific spiking neural networks (SNNs) for sensor fusion that showed classification accuracy comparable to the software baseline. These neuromorphic alternatives have increased inference time, between 20 and 40%, with respect to the GPU system but have a significantly smaller energy-delay product (EDP) which makes them between 30× and 600× more efficient. The proposed work represents a new benchmark that moves neuromorphic computing toward a real-world scenario.