Unsupervised Adaptive Weight Pruning for Energy-Efficient Neuromorphic Systems.

Wenzhe Guo,Khaled Nabil Salama,Hasan Erdem Yantir,Ahmed M Eltawil,Mohammed E Fouda

doi:10.3389/fnins.2020.598876

Abstract

To tackle real-world challenges, deep and complex neural networks are generally used with a massive number of parameters, which require large memory size, extensive computational operations, and high energy consumption in neuromorphic hardware systems. In this work, we propose an unsupervised online adaptive weight pruning method that dynamically removes non-critical weights from a spiking neural network (SNN) to reduce network complexity and improve energy efficiency. The adaptive pruning method explores neural dynamics and firing activity of SNNs and adapts the pruning threshold over time and neurons during training. The proposed adaptation scheme allows the network to effectively identify critical weights associated with each neuron by changing the pruning threshold dynamically over time and neurons. It balances the connection strength of neurons with the previous layer with adaptive thresholds and prevents weak neurons from failure after pruning. We also evaluated improvement in the energy efficiency of SNNs with our method by computing synaptic operations (SOPs). Simulation results and detailed analyses have revealed that applying adaptation in the pruning threshold can significantly improve network performance and reduce the number of SOPs. The pruned SNN with 800 excitatory neurons can achieve a 30% reduction in SOPs during training and a 55% reduction during inference, with only 0.44% accuracy loss on MNIST dataset. Compared with a previously reported online soft pruning method, the proposed adaptive pruning method shows 3.33% higher classification accuracy and 67% more reduction in SOPs. The effectiveness of our method was confirmed on different datasets and for different network sizes. Our evaluation showed that the implementation overhead of the adaptive method regarding speed, area, and energy is negligible in the network. Therefore, this work offers a promising solution for effective network compression and building highly energy-efficient neuromorphic systems in real-time applications.

Highlights

In recent years, as the prediction of Moore’s law slows down prominently, neuromorphic computing has been widely regarded as a promising approach for large-scale computing
− v) − ge (v − Eexc) − gi (v − Einh) where τm is the time constant, vr is the resting membrane potential, ge is the excitatory conductance associated with an excitatory channel, Eexc is the reverse potential of the channel, gi is the conductance associated with an inhibitory channel, Einh is the reverse potential of the channel
We proposed an online adaptive weight pruning method that adapts the pruning threshold over time and neurons during training in an unsupervised spiking neural network (SNN)

Summary

Introduction

As the prediction of Moore’s law slows down prominently, neuromorphic computing has been widely regarded as a promising approach for large-scale computing. Neuromorphic systems are constructed following biological principles existing in our central nervous systems, which features in massive parallelism, collocated memory, and processors, and asynchronous event-driven computation (Mead, 1990; Furber et al, 2014; Davies et al, 2018). With energy-efficient computation and parallel information processing features, SNNs are widely adopted for building neuromorphic hardware systems (Thakur et al, 2018). In such systems, information is transmitted through synapses from a presynaptic neuron to a postsynaptic neuron on the occurrence of an event (or a spike).

Objectives

Results

Conclusion