Activation-Based Weight Significance Criterion for Pruning Deep Neural Networks

Abstract

Due to the massive amount of network parameters and great demand for computational resources, large-scale neural networks, especially deep convolutional neural networks (CNNs), can be inconvenient to implement for many real world applications. Therefore, sparsifying deep and densely connected neural networks is becoming a more and more important topic in the computer vision field for addressing these limitations. This paper starts from a very deep CNN trained for face recognition, then explores sparsifying neuron connections for network compression. We propose an activation-based weight significance criterion which estimates the contribution that each weight makes in the activations of the neurons in the next layer, then removes those weights that make least contribution first. A concise but effective procedure is devised for pruning parameters of densely connected neural networks. In this procedure, one neuron is sparsified at a time, and a requested amount of parameters related to this neuron is removed. Applying the proposed method, we greatly compressed the size of a large-scale neural network without causing any loss in recognition accuracy. Furthermore, our experiments show that this procedure can work with different weight significance criterions for different expectations.