DNN compression by ADMM-based joint pruning

Abstract

The success of deep neural networks (DNNs) has motivated pursuit of both computationally and memory efficient models for applications in resource-constrained systems such as embedded devices. In line with this trend, network pruning methods reducing redundancy in over-parameterized models are being studied actively. Previous works on this research have demonstrated the ability to learn a compact network by imposing sparsity constraints on the parameters, but most of them have difficulty not only in identifying both connections and neurons to be pruned, but also in converging to optimal solutions. We propose a systematic DNN compression method where weights and network architectures are jointly optimized. We solve the joint problem using alternating direction method of multipliers (ADMM), a powerful technique capable of handling non-convex separable programming. Additionally, we provide a holistic pruning approach, an integrated form of our method, for automatically pruning networks without specific layer-wise hyper-parameters. To verify our work, we deployed the proposed method to a variety of state-of-the-art convolutional neural networks (CNNs) on three image classification benchmark datasets: MNIST, CIFAR-10, and ImageNet. Results show that the proposed pruning method effectively compresses the network parameters and reduces the computation cost while preserving prediction accuracy.