Self-distillation enhanced adaptive pruning of convolutional neural networks

Abstract

Convolutional neural networks (CNNs) suffer from issues of large parameter size and high computational complexity. To address this, we propose an adaptive pruning algorithm based on self-distillation. The algorithm introduces a trainable parameter for each channel to control channel pruning and integrates the pruning process into network training, enabling pruning and fine-tuning in a single training iteration to derive the final pruned model. Moreover, this framework requires only a single overall pruning rate to achieve adaptive pruning for each layer, avoiding tedious hyperparameter tuning for a less iterative, simple and efficient pruning process. Additionally, self-distillation is utilized in the pruning algorithm, leveraging the knowledge from the pretrained CNN to guide its own pruning process, facilitating the recovery from performance degradation caused by pruning and achieving higher accuracy. Extensive pruning experiments on various CNN models over different datasets demonstrate that at least 75% of redundant parameters can be reduced without sacrificing model accuracy.