Abstract
Spiking neural networks (SNNs), with their inherent capability to learn sparse spike-based input representations over time, offer a promising solution for enabling the next generation of intelligent autonomous systems. Nevertheless, end-to-end training of deep SNNs is both compute- and memory-intensive because of the need to backpropagate error gradients through time. We propose BlocTrain, which is a scalable and complexity-aware incremental algorithm for memory-efficient training of deep SNNs. We divide a deep SNN into blocks, where each block consists of few convolutional layers followed by a classifier. We train the blocks sequentially using local errors from the classifier. Once a given block is trained, our algorithm dynamically figures out easy vs. hard classes using the class-wise accuracy, and trains the deeper block only on the hard class inputs. In addition, we also incorporate a hard class detector (HCD) per block that is used during inference to exit early for the easy class inputs and activate the deeper blocks only for the hard class inputs. We trained ResNet-9 SNN divided into three blocks, using BlocTrain, on CIFAR-10 and obtained 86.4% accuracy, which is achieved with up to 2.95× lower memory requirement during the course of training, and 1.89× compute efficiency per inference (due to early exit strategy) with 1.45× memory overhead (primarily due to classifier weights) compared to end-to-end network. We also trained ResNet-11, divided into four blocks, on CIFAR-100 and obtained 58.21% accuracy, which is one of the first reported accuracy for SNN trained entirely with spike-based backpropagation on CIFAR-100.
Highlights
Deep neural networks have achieved remarkable success and redefined the state-of-the-art performance for a variety of artificial intelligence tasks including image recognition (He et al, 2016), action recognition in videos (Simonyan and Zisserman, 2014a), and natural language processing (Bahdanau et al, 2014; Sutskever et al, 2014), among other tasks
We demonstrate the capability of BlocTrain to provide improved accuracy as well as higher training and inference efficiency relative to endto-end approaches for deep spiking neural networks (SNNs) on the CIFAR-10 and the CIFAR-100 datasets
We demonstrate the efficacy of BlocTrain for ResNet-9, and ResNet-11 and VGG-16, which are among the deepest models trained entirely using spike-based backpropagation through time (BPTT) algorithms (Lee et al, 2020)
Summary
Deep neural networks have achieved remarkable success and redefined the state-of-the-art performance for a variety of artificial intelligence tasks including image recognition (He et al, 2016), action recognition in videos (Simonyan and Zisserman, 2014a), and natural language processing (Bahdanau et al, 2014; Sutskever et al, 2014), among other tasks. Error backpropagation algorithms, which are the workhorse for training deep ANNs with millions of parameters, suffer from scalability limitations when adapted for SNNs. It is well known that end-to-end training of feed-forward ANNs, using backpropagation, requires the activations of all the layers to be stored in memory for computing the weight updates. The ensuing weight update computation requires the spiking neuronal activation and state ( known as membrane potential) to be stored across time-steps for the entire network. End-to-end training of SNN using backpropagation through time (BPTT) requires much higher memory footprint over that incurred for training sized ANN on Graphics Processing Units (GPUs) (Gruslys et al, 2016)
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
