Deep Compression with Low Rank and Sparse Integrated Decomposition

Abstract

This work aims to compress the deep neural networks (DNNs) by learning the low rank and sparsity characteristics of weight filters. Most of the compression models consider the low rank or sparse property of weights independently, however, they cannot capture the features accurately and thus become inefficient. It is revealed that the low rank component of the weights also tends to be sparse, therefore, in this work, we propose an extreme neural network compression method by embedding sparsifying operations after the low rank decomposition. In addition, a global sparse component is proposed to compensate the performance loss of the compressed model. Experiments are also included to prove the benefits of the proposed algorithm.