PCA-Pruner: Filter pruning by principal component analysis

Abstract

Deep Convolutional Neural Networks (CNNs) have been widely used in various domains due to their outstanding performance. However, they simultaneously bring enormous computational overhead, making it difficult to deploy to mobile and edge devices. Therefore, researchers use network compression techniques such as quantization, knowledge distillation and neural network pruning to alleviate this problem. Among network pruning, filter pruning has received broad attention. At present, most of the filter pruning methods need to define pruning rates manually, which is a trial-and-error process and requires rich experimental experience. Some methods obtain global optimal network parameters by Neural Architecture Search (NAS) or Evolutionary Algorithms (EA) to overcome this difficulty. However, they also introduce huge computational burden. To mitigate the above problems, this study proposes a pruning strategy based on Principal Component Analysis (PCA) called PCA-Pruner. Filter weights of a layer is regarded as a set of features, and the number of filters responding to feature dimension. Then, the number of reserved filters in each layer can be determined by PCA which is a classical dimensionality reduction technology. After that, we calculate the L1 norm of each filter in each layer and use it as an importance measurement to prune filters. Experimental results show that PCA-Pruner achieve performance improvements over the state-of-the-arts algorithms. For example, we compress the FLOPs and parameters of ResNet-56 on CIFA-10 by 45.8% and 47.1%, with an increase in accuracy of 0.27%. For ResNet-110 on CIFAR-10, we improve the accuracy by 0.58% and reduce the FLOPs and Params of the model by 58.3% and 56.2%, respectively. Towards ResNet-56 on CIFAR-100 dataset, we achieve a 38.8% FLOPs decrease and 38.0% Params reduction with only 0.69% accuracy loss.