An Analysis of Low-Rank Decomposition Selection for Deep Convolutional Neural Networks

Abstract

AbstractDeep convolutional neural networks have achieved state of the art results in many image classification tasks. However, the large amount of parameters of the network limit its deployment to storage space limited situations. Low-rank decomposition methods are effective to compress the network, such as Canonical Polyadic decomposition and Tucker decomposition. However, most low-rank decomposition based approaches cannot achieve a satisfactory balance between the classification accuracy and compression ratio of the network. In this paper, we analyze the advantages and disadvantages of Canonical Polyadic and Tucker decomposition and give a selection guidance to take full advantage of both. And we recommend to use Tucker decomposition for shallow layers and Canonical Polyadic decomposition for deep layers of a deep convolutional neural network. The experiment results show that our approach achieves the best trade-off between accuracy and parameter compression ratio, which validates our point of view.KeywordsLow-rank decompositionCanonical Polyadic decompositionTucker decompositionDeep convolutional neural networksImage classification