Bayesian Optimization-Based Global Optimal Rank Selection for Compression of Convolutional Neural Networks

Jieun Lee,Yoonsik Choe,Taehyeon Kim

doi:10.1109/access.2020.2968357

Abstract

Recently, convolutional neural network (CNN) compression via low-rank decomposition has achieved remarkable performance. Finding the optimal rank is a crucial problem because rank is the only hyperparameter for controlling computational complexity and accuracy in compressed CNNs. In this paper, we propose a global optimal rank selection method based on Bayesian optimization (BayesOpt), which is a machine learning based global optimization technique. By utilizing both a simple objective function and a proper optimization scheme, the proposed method produces a global optimal rank that provides a good trade-off between computational complexity and accuracy degradation. In addition, our method also reflects the correlation of each rank in multi-rank selection, and is able to flexibly yield an optimal rank with a given fixed compression ratio. Experimental results indicate that the proposed algorithm can identify the global optimal rank regardless of the huge size of dataset or the various structural features of CNNs. In all experiments on multi-rank selection, the proposed method produces the rank with higher accuracy and lower computational complexity than the state-of-the-art rank selection method, variational Bayesian matrix factorization (VBMF).

Highlights

Convolutional neural networks (CNNs) with a large number of parameters and complex structures have achieved remarkable performance in various machine learning applications [1]–[5], there are serious implementation challenges caused by over-parameterized convolutional neural network (CNN) on resource constrained devices, such as mobile phones and embedded systems
We evaluated the performance of rank selection in terms of top-1 accuracy, top-5 accuracy, computational complexity, and memory complexity in the decomposed CNNs
In this paper, a novel global optimal rank selection method of low-rank decomposition for CNN compression based on BayesOpt was proposed

Summary

Introduction

Convolutional neural networks (CNNs) with a large number of parameters and complex structures have achieved remarkable performance in various machine learning applications [1]–[5], there are serious implementation challenges caused by over-parameterized CNNs on resource constrained devices, such as mobile phones and embedded systems To address these problems, CNN compression via low-rank decomposition has been proposed with various decomposition methods including singular value decomposition (SVD) [6], canonical polyadic decomposition (CPD) [7], tensor-train decomposition (TT decomposition) [8], [9], Tucker-2 decomposition [7], and block term decomposition [10]. The rank selection of SVD and Tucker-2 decomposition are considered as single- and multi-rank selection, respectively, because SVD decomposes the data with one rank r ∈ R and Tucker-2 decomposition decomposes the data with two ranks r ∈ R2

Methods

Results

Conclusion