Abstract
Neural network pruning is critical to alleviating the high computational cost of deep neural networks on resource-limited devices. Conventional network pruning methods compress the network based on the hand-crafted rules with a pre-defined pruning ratio (PR), which fails to consider the variety of channels among different layers, thus, resulting in a sub-optimal pruned model. To alleviate this issue, this study proposes a genetic wavelet channel search (GWCS) based pruning framework, where the pruning process is modeled as a multi-stage genetic optimization procedure. Its main ideas are 2-fold: (1) it encodes all the channels of the pertained network and divide them into multiple searching spaces according to the different functional convolutional layers from concrete to abstract. (2) it develops a wavelet channel aggregation based fitness function to explore the most representative and discriminative channels at each layer and prune the network dynamically. In the experiments, the proposed GWCS is evaluated on CIFAR-10, CIFAR-100, and ImageNet datasets with two kinds of popular deep convolutional neural networks (CNNs) (ResNet and VGGNet). The results demonstrate that GNAS outperforms state-of-the-art pruning algorithms in both accuracy and compression rate. Notably, GNAS reduces more than 73.1% FLOPs by pruning ResNet-32 with even 0.79% accuracy improvement on CIFAR-100.
Highlights
Deep convolutional neural networks (CNNs) have achieved substantial progress in many research fields, such as computer vision (Wang et al, 2019), natural language processing (Giménez et al, 2020), and information recommendation (Wu et al, 2021a,b)
It contains three steps which are shown in Figure 2: (1) Training a large CNNs, (2) Using genetic wavelet channel search (GWCS) to prune the channels in pre-trained network M layer by layer, (3) Knowledge distilling (KD) the pruned network to recover the model accuracy
Results on CIFAR-10 and CIFAR-100 The pruning result of ResNet series networks on CIFAR-10 and CIFAR-100 are shown in Tables 1, 2
Summary
Deep convolutional neural networks (CNNs) have achieved substantial progress in many research fields, such as computer vision (Wang et al, 2019), natural language processing (Giménez et al, 2020), and information recommendation (Wu et al, 2021a,b). One solution is deleting the weights with small absolute values (Liu et al, 2017) under the presumption that the smaller value of a weight parameter is, the less impact it has on the final result This intuitive assumption has been proved invalid in some cases (Ye et al, 2018). Many other pruning algorithms have been developed, such as judging the influence of parameter clipping on training loss (Molchanov et al, 2016) or the reconstruction errors of feature outputs (He et al, 2017) Such algorithms mainly rely on human expert knowledge and hand-crafted pruning rules
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
