An efficient training-from-scratch framework with BN-based structural compressor

Abstract

Channel pruning is an effective way of compressing convolutional neural networks (CNNs) under constrained resources. The current pruning methods follow a progressive pretrain-prune-finetune pipeline, which is inefficient and computationally expensive. In this paper, we bypass the pretrain-prune-finetune pipeline and propose a novel and efficient model training framework based on online channel pruning, which automatically produces a compact well-performed sub-network in one training-from-scratch pass under a given budget condition. Specifically, we introduce a novel BN-based indicator and a sparsity regularization strategy in the early training stage to iteratively and greedily shrink the model layers, which encourages a high-quality architecture with low channel redundancy. To ensure training stability and promote the generalization ability of the resultant pruned network, we also skillfully incorporate a simple self-distillation framework into our training and pruning pipeline. Extensive experiments indicate that our method can effectively achieve competitive performance on the image classification task compared with the state-of-the-arts.