A New ANN-SNN Conversion Method with High Accuracy, Low Latency and Good Robustness

A universal ANN-to-SNN framework for achieving high accuracy and low latency deep Spiking Neural Networks

Spiking Neural Network (SNN) is a promising energy-efficient AI model when implemented on neuromorphic hardware. However, it is a challenge to efficiently train SNNs due to their non-differentiability. Most existing methods either suffer from high latency (i.e., long simulation time steps), or cannot achieve as high performance as Artificial Neural Networks (ANNs). In this paper, we propose the Differentiation on Spike Representation (DSR) method, which could achieve high performance that is competitive to ANNs yet with low latency. First, we encode the spike trains into spike representation using (weighted) firing rate coding. Based on the spike representation, we systematically derive that the spiking dynamics with common neural models can be represented as some sub-differentiable mapping. With this viewpoint, our proposed DSR method trains SNNs through gradients of the mapping and avoids the common non-differentiability problem in SNN training. Then we analyze the error when representing the specific mapping with the forward computation of the SNN. To reduce such error, we propose to train the spike threshold in each layer, and to introduce a new hyperparameter for the neural models. With these components, the DSR method can achieve state-of-the-art SNN performance with low latency on both static and neuromorphic datasets, including CIFAR-10, CIFAR-100, ImageNet, and DVS-CIFAR10.

Spiking neural networks (SNNs) have shown advantages in computation and energy efficiency over traditional artificial neural networks (ANNs) thanks to their event-driven representations. SNNs also replace weight multiplications in ANNs with additions, which are more energy-efficient and less computationally intensive. However, it remains a challenge to train deep SNNs due to the discrete spiking function. A popular approach to circumvent this challenge is ANN-to-SNN conversion. However, due to the quantization error and accumulating error, it often requires lots of time steps (high inference latency) to achieve high performance, which negates SNN's advantages. To this end, this paper proposes Fast-SNN that achieves high performance with low latency. We demonstrate the equivalent mapping between temporal quantization in SNNs and spatial quantization in ANNs, based on which the minimization of the quantization error is transferred to quantized ANN training. With the minimization of the quantization error, we show that the sequential error is the primary cause of the accumulating error, which is addressed by introducing a signed IF neuron model and a layer-wise fine-tuning mechanism. Our method achieves state-of-the-art performance and low latency on various computer vision tasks, including image classification, object detection, and semantic segmentation. Codes are available at: https://github.com/yangfan-hu/Fast-SNN.

Spiking neural networks (SNNs) have recently emerged as the low-power alternative to artificial neural networks (ANNs) because of their sparse, asynchronous, and binary event-driven processing. Due to their energy efficiency, SNNs have a high possibility of being deployed for real-world, resource-constrained systems such as autonomous vehicles and drones. However, owing to their non-differentiable and complex neuronal dynamics, most previous SNN optimization methods have been limited to image recognition. In this paper, we explore the SNN applications beyond classification and present semantic segmentation networks configured with spiking neurons. Specifically, we first investigate two representative SNN optimization techniques for recognition tasks (i.e., ANN-SNN conversion and surrogate gradient learning) on semantic segmentation datasets. We observe that, when converted from ANNs, SNNs suffer from high latency and low performance due to the spatial variance of features. Therefore, we directly train networks with surrogate gradient learning, resulting in lower latency and higher performance than ANN-SNN conversion. Moreover, we redesign two fundamental ANN segmentation architectures (i.e., Fully Convolutional Networks and DeepLab) for the SNN domain. We conduct experiments on three semantic segmentation benchmarks including PASCAL VOC2012 dataset, DDD17 event-based dataset, and synthetic segmentation dataset combined CIFAR10 and MNIST datasets. In addition to showing the feasibility of SNNs for semantic segmentation, we show that SNNs can be more robust and energy-efficient compared to their ANN counterparts in this domain.

Communication by rare, binary spikes is a key factor for the energy efficiency of biological brains. However, it is harder to train biologically-inspired spiking neural networks than artificial neural networks. This is puzzling given that theoretical results provide exact mapping algorithms from artificial to spiking neural networks with time-to-first-spike coding. In this paper we analyze in theory and simulation the learning dynamics of time-to-first-spike-networks and identify a specific instance of the vanishing-or-exploding gradient problem. While two choices of spiking neural network mappings solve this problem at initialization, only the one with a constant slope of the neuron membrane potential at threshold guarantees the equivalence of the training trajectory between spiking and artificial neural networks with rectified linear units. For specific image classification architectures comprising feed-forward dense or convolutional layers, we demonstrate that deep spiking neural network models can be effectively trained from scratch on MNIST and Fashion-MNIST datasets, or fine-tuned on large-scale datasets, such as CIFAR10, CIFAR100 and PLACES365, to achieve the exact same performance as that of artificial neural networks, surpassing previous spiking neural networks. Our approach accomplishes high-performance classification with less than 0.3 spikes per neuron, lending itself for an energy-efficient implementation. We also show that fine-tuning spiking neural networks with our robust gradient descent algorithm enables their optimization for hardware implementations with low latency and resilience to noise and quantization.

Spiking Neural Networks (SNNs) mimic the behavior of biological neurons. Unlike traditional Artificial Neural Networks (ANNs) that operate in a continuous time domain and use activation functions to process information, SNNs operate discrete event-driven, where data is encoded and communicated through spikes or discrete events. This unique approach offers several advantages, such as efficient computation and lower power consumption, making SNNs particularly attractive for energy-constrained and neuromorphic applications. However, training SNNs poses significant challenges due to the discrete nature of spikes and the non-differentiable behavior they exhibit. As a result, converting pre-trained ANNs into SNNs has gained attention as a convenient approach. While this approach simplifies the training process, it introduces certain drawbacks, including high latency. The conversion of ANNs to SNNs typically leads to a loss of accuracy, which can be attributed to various factors, including quantization, clipping, and timing errors. Previous studies have proposed techniques to mitigate quantization and clipping errors during the conversion process. However, they do not consider timing errors, degrading SNN accuracies at low latency conditions. This work introduces the MiCE conversion method, which offers a comprehensive joint optimization strategy to simultaneously alleviate quantization, clipping, and timing errors. At a moderate latency of 8 time-steps, our converted ResNet-20 achieves classification accuracies of 79.02% and 95.74% on the CIFAR-100 and CIFAR-10 datasets, respectively.

Spiking neural networks (SNNs) hold significant promise as energy-efficient alternatives to conventional artificial neural networks (ANNs). However, SNNs require computations across multiple timesteps, resulting in increased latency, heightened energy consumption, and additional memory access overhead. Techniques to reduce SNN latency down to a unit timestep have emerged to realize true superior energy efficiency over ANNs. Nonetheless, this latency reduction often comes at the expense of noticeable accuracy degradation. Therefore, achieving an optimal balance in the tradeoff between accuracy and energy consumption by adjusting the latency of multiple timesteps remains a significant challenge. This work leverages an additional dimension to enhance the accuracy-energy tradeoff space using a novel one-hot multi-level leaky integrate-and-fire (M-LIF) neuron model. The proposed one-hot M-LIF model represents the inputs and outputs of hidden layers as a set of one-hot binary-weighted spike lanes to find better tradeoff points while still being able to model conventional SNNs. For image classification on static datasets, we demonstrate one-hot M-LIF SNNs outperform iso-architecture conventional LIF SNNs in terms of accuracy (2% higher than VGG16 SNN on ImageNet) while still being energy-efficient (20× lower energy than VGG16 ANN on ImageNet). For dynamic vision datasets, we demonstrate the ability of M-LIF SNNs to reduce latency by 3× compared to conventional LIF SNNs while limiting accuracy degradation (< 1%).

In recent years, the spiking neural network (SNN) has gained widespread attention for its low latency and low power consumption characteristics. Customizing accelerators to exploit the parallelism in SNN fully has become a current research hotspot. Because the calculations between different time steps in SNN have dependencies, the existing accelerators update the neuron state strictly in time sequence, which results in the calculation of SNN even more than that of the artificial neural network (ANN). In addition, traditional event-driven SNN accelerators cannot support inhibitory SNN models, which severely limits their scope of application. Based on this observation, we combined the characteristics of the most commonly used neuron model and spike code to design an event-driven SNN accelerator (FEAS). FEAS realizes the acceleration for SNN (both the excitatory SNN and the inhibitory SNN) models through algorithm approximation and novel hardware modules. We use FEAS to accelerate a fully connected SNN model for classification on the MNIST dataset. The results show that compared with the state-of-the-art technology, our accelerator can increase the throughput by an order of magnitude with the model's accuracy as high as 97.54%.

Spiking Neural Networks (SNNs) are a promising alternative to traditional deep learning methods since they perform event-driven information processing. However, a major drawback of SNNs is high inference latency. The efficiency of SNNs could be enhanced using compression methods such as pruning and quantization. Notably, SNNs, unlike their non-spiking counterparts, consist of a temporal dimension, the compression of which can lead to latency reduction. In this paper, we propose spatial and temporal pruning of SNNs. First, structured spatial pruning is performed by determining the layer-wise significant dimensions using principal component analysis of the average accumulated membrane potential of the neurons. This step leads to 10-14X model compression. Additionally, it enables inference with lower latency and decreases the spike count per inference. To further reduce latency, temporal pruning is performed by gradually reducing the timesteps while training. The networks are trained using surrogate gradient descent based backpropagation and we validate the results on CIFAR10 and CIFAR100, using VGG architectures. The spatiotemporally pruned SNNs achieve 89.04% and 66.4% accuracy on CIFAR10 and CIFAR100, respectively, while performing inference with 3-30X reduced latency compared to state-of-the-art SNNs. Moreover, they require 8-14X lesser compute energy compared to their unpruned standard deep learning counterparts. The energy numbers are obtained by multiplying the number of operations with energy per operation. These SNNs also provide 1–4% higher robustness against Gaussian noise corrupted inputs. Furthermore, we perform weight quantization and find that performance remains reasonably stable up to 5-bit quantization.

In recent years, Artificial neural network has made great progress in image, machine perception and other aspects, and has a very good performance in the scope of deep learning. As a highly intensive neural network, Artificial neural network's performance has gradually reached saturation in today's increasing network demand, but its efficiency and consumption are still relatively large. Therefore, more and more attention has been paid to the peak neural network with low energy consumption in operating equipment. Spiking neural networks shows good performance of low power consumption when running on hardware. More and more researchers begin to use Spiking neural networks to study the performance of image recognition and other aspects. Although Spiking neural network has many limitations in accuracy and training difficulty, it has stimulated the research enthusiasm of many researchers. Spiking neural networks has developed rapidly, and many training methods can achieve the same or even higher accuracy than Artificial neural networks. In this paper, we further understand the advantages and framework of Spiking neural network through its development.

With the adoption of smart systems, artificial neural networks (ANNs) have become ubiquitous. Conventional ANN implementations have high energy consumption, limiting their use in embedded and mobile applications. Spiking neural networks (SNNs) mimic the dynamics of biological neural networks by distributing information over time through binary spikes. Neuromorphic hardware has emerged to leverage the characteristics of SNNs, such as asynchronous processing and high activation sparsity. Therefore, SNNs have recently gained interest in the machine learning community as a brain-inspired alternative to ANNs for low-power applications. However, the discrete representation of the information makes the training of SNNs by backpropagation-based techniques challenging. In this survey, we review training strategies for deep SNNs targeting deep learning applications such as image processing. We start with methods based on the conversion from an ANN to an SNN and compare these with backpropagation-based techniques. We propose a new taxonomy of spiking backpropagation algorithms into three categories, namely, spatial, spatiotemporal, and single-spike approaches. In addition, we analyze different strategies to improve accuracy, latency, and sparsity, such as regularization methods, training hybridization, and tuning of the parameters specific to the SNN neuron model. We highlight the impact of input encoding, network architecture, and training strategy on the accuracy-latency tradeoff. Finally, in light of the remaining challenges for accurate and efficient SNN solutions, we emphasize the importance of joint hardware-software codevelopment.

Artificial neural networks (ANNs) have experienced a rapid advancement for their success in various application domains, including autonomous driving and drone vision. Researchers have been improving the performance efficiency and computational requirement of ANNs inspired by the mechanisms of the biological brain. Spiking neural networks (SNNs) provide a power-efficient and brain-inspired computing paradigm for machine learning applications. However, evaluating large-scale SNNs on classical von Neumann architectures (central processing units/graphics processing units) demands a high amount of power and time. Therefore, hardware designers have developed neuromorphic platforms to execute SNNs in and approach that combines fast processing and low power consumption. Recently, field-programmable gate arrays (FPGAs) have been considered promising candidates for implementing neuromorphic solutions due to their varied advantages, such as higher flexibility, shorter design, and excellent stability. This review aims to describe recent advances in SNNs and the neuromorphic hardware platforms (digital, analog, hybrid, and FPGA based) suitable for their implementation. We present that biological background of SNN learning, such as neuron models and information encoding techniques, followed by a categorization of SNN training. In addition, we describe state-of-the-art SNN simulators. Furthermore, we review and present FPGA-based hardware implementation of SNNs. Finally, we discuss some future directions for research in this field.

Spiking Neural Networks (SNNs) are receiving increasing attention due to their biological plausibility and the potential for ultra-low-power event-driven neuromorphic hardware implementation. Due to the complex temporal dynamics and discontinuity of spikes, training SNNs directly usually suffers from high computing resources and a long training time. As an alternative, SNN can be converted from a pre-trained artificial neural network (ANN) to bypass the difficulty in SNNs learning. However, the existing ANN-to-SNN methods neglect the inconsistency of information transmission between synchronous ANNs and asynchronous SNNs. In this work, we first analyze how the asynchronous spikes in SNNs may cause conversion errors between ANN and SNN. To address this problem, we propose a signed neuron with memory function, which enables almost no accuracy loss during the conversion process, and maintains the properties of asynchronous transmission in the converted SNNs. We further propose a new normalization method, named neuron-wise normalization, to significantly shorten the inference latency in the converted SNNs. We conduct experiments on challenging datasets including CIFAR10 (95.44% top-1), CIFAR100 (78.3% top-1) and ImageNet (73.16% top-1). Experimental results demonstrate that the proposed method outperforms the state-of-the-art works in terms of accuracy and inference time. The code is available at https://github.com/ppppps/ANN2SNNConversion_SNM_NeuronNorm.

Spiking Neural Networks (SNN) promise extremely low-power and low-latency inference on neuromorphic hardware. Recent studies demonstrate the competitive performance of SNNs compared with Artificial Neural Networks (ANN) in conventional classification tasks.In this work, we present an energy-efficient implementation of a Reinforcement Learning (RL) algorithm using SNNs to solve an obstacle avoidance task performed by an Unmanned Aerial Vehicle (UAV), taking a Dynamic Vision Sensor (DVS) as event-based input. We train the SNN directly, improving upon state-of-art implementations based on hybrid (not directly trained) SNNs. For this purpose, we devise an adaptation of the Spatio-Temporal Backpropagation algorithm (STBP) for RL. We then compare the SNN with a state-of-art Convolutional Neural Network (CNN) designed to solve the same task. To this aim, we train both networks by exploiting a photorealistic training pipeline based on AirSim. To achieve a realistic latency and throughput assessment for embedded deployment, we designed and trained three different embedded SNN versions to be executed on state-of-art neuromorphic hardware, targeting state-of-the-art.We compared SNN and CNN in terms of obstacle avoidance performance showing that the SNN algorithm achieves better results than the CNN with a factor of 6× less energy. We also characterize the different SNN hardware implementations in terms of energy and spiking activity.

Over the past three decades, the field of neuromorphic engineering has produced sensors and processors that show great promise as efficient, brain-inspired systems. In parallel to this development, tremendous advances in Deep Learning ( DL ) have supplied highly accurate algorithms for computer vision. Unfortunately, these algorithms are not directly compatible with neuromorphic hardware. The present work bridges this gap by developing algorithms that leverage the power of Deep Learning while being suited for operation on neuro-inspired hardware. The Dynamic Vision Sensor ( DVS ), a neuromorphic sensor used in this thesis, differs radically from conventional cameras by producing a stream of asynchronous pixel-events rather than regularly spaced light-intensity frames. These events signal changes in local brightness at high temporal resolution and wide dynamic range, which makes the sensor well suited for applications with spatio-temporal redundancy, difficult lighting conditions, or fast reaction time. However, the event-based nature of the sensor output impedes the application of standard computer vision techniques like Deep Neural Networks ( DNNs ). In this thesis we demonstrate that DNNs can be adapted to operate in an event-based fashion, similar to how neural networks in the brain use discrete spikes for signal transmission. By converting Artificial Neural Networks ( ANNs ) trained with DL into Spiking Neural Networks ( SNNs ), we achieve some of the largest and most accurate spiking models for object classification to date. While information in SNNs for computer vision tasks is often encoded in the form of firing rates, the nervous system is known to employ other spike codes optimized for the requirements of a particular sensory pathway. Motivated by evidence of visual processing in humans occurring within milliseconds, we explore encoding schemes that make use of the precise timing of individual spikes to represent information. We show that high classification accuracy can be achieved in artificial systems based on few spikes per neuron.&#13;\n&#13;\nPart of the widespread success of DL can be attributed to the ease with which algorithms and hardware are made accessible today. A beginner can readily find an online notebook that enables them to build, train, and run a full-scale DNN on a remote GPU within minutes, without any overhead setting up software and hardware. To achieve the same on a neuromorphic platform requires intricate understanding of the underlying hardware constraints, and a great deal of manual, low-level programming. One theme in this thesis is the reduction of these obstacles by providing automated tools to convert DNNs to the spike-domain, and to deploy them on neuromorphic hardware. Here, we mainly consider the Intel research processor Loihi, for which we developed a DNN compilation framework. Much of the previous work on SNNs is confined to simulations on general-purpose hardware, which allow no reliable characterization of the actual latency and power consumption of SNNs on dedicated hardware. By means of the toolchain developed here, we are able to perform such benchmarking on standard tasks from computer vision. viiThough SNNs operate on spike events internally, they may receive conventional image frames as input. A more consistent approach is to use event-based input, e. g. from a DVS . In this work we discuss some of the benefits and challenges one can expect when thus combining event-based sensing and processing. In applications ranging from data compression to optical flow and Spiking Neural Networks, we demonstrate computational savings when operating on sparse, informative events rather than dense, redundant frames. Finally, we turn to a biological vision system, the retina, and show that a Dynamic Vision Sensor can be used to drive mouse Retinal Ganglion Cells in vitro - thereby opening a door for applications in retinal prostheses.&#13;\n&#13;\nOne recurrent theme in this thesis is the reduction of computational cost of neural networks. In a final study we ask whether the principle of sparse, event-driven updating can be transferred to standard ANNs without the use of spiking neurons. Inspired by how the DVS removes spatio-temporal redundancy from video, we apply a dynamic masking scheme to the layers of a DNN to reduce the number of operations during inference. The algorithm is shown to produce equivalent accuracy results at reduced computational cost on a range of vision tasks including human pose estimation and object detection in static and dynamic scenes. This thesis contributes overall to a fruitful exchange between conventional computer vision on one hand and neuromorphic sensors and processors on the other. Both fields, to various degrees, share the motive for ever increasing efficiency, and many of the seeming restrictions of dedicated hardware, like reduced numeric precision, turn out to be desirable from an algorithmic perspective. Ultimately, we cherish the hope that in building massively constrained neuromorphic systems, we will one day understand more clearly how our brain accomplishes its tasks within a minimal space and energy budget.