Exploiting memristive autapse and temporal distillation for training spiking neural networks

Abstract

Spiking neural networks (SNNs) have attracted widespread attention due to their brain-inspired information processing mechanism and low power, sparse accumulation computation on neuromorphic chips. The surrogate gradient method makes it possible to train deep SNNs using backpropagation and shows satisfactory performance on some tasks. However, as the network structure becomes deeper, the spike information may fail to transmit to deeper layers, thus causing the output layer to make wrong predictions in recognition tasks. Inspired by the autaptic structure in the cerebral cortex, which is formed by axons connecting to their own dendrites and capable of modulating neuronal activity, we use discrete memristors to build feedback-connected autapses to adaptively regulate the precision of the spikes. Further, to prevent outlier at a certain time step from affecting the overall output, we distill the averaged knowledge into sub-models at each time step to correct potential errors. By combining these two proposed methods, we propose a deep SNNs optimized by Leaky Integrate-and-Fire (LIF) model with memristive autapse and temporal distillation, referred to as MA-SNN. A series of experiments on static datasets (CIFAR10 and CIFAR100) as well as neuromorphic datasets (DVS-CIFAR10 and N-Caltech101) demonstrated the competitiveness of the proposed model and validated the effectiveness of its components. Code for MA-SNN is available at: https://github.com/CHNtao/MA-SNN.