Multi-Head Convolutional Neural Network Compression based on High-Order Principal Component Analysis

Abstract

A multi-head convolutional neural network performs remarkably in various multi-task learning-based computer vision applications. Behind these achievements, a multi-head convolutional neural network utilizes significantly huge parameters and complex neural architecture. This peculiarity of the multi-head convolutional neural networks can make them represent and capture versatile features from images; however, it also creates serious implementation problems when deploying the multi-head convolutional neural network on resource-constrained systems. To handle this problem, we propose a novel neural network compression algorithm that can maintain the core features and remove redundant features in the convolutional layer as an aspect of multi-head convolutional neural network architecture. The proposed neural network compression algorithm computes multidimensional principal components on the convolutional layer of a multi-head convolutional neural network with statistically guaranteed hyper-parameter optimization. Experiments show that the proposed algorithm is able to produce an efficient multi-head convolutional neural network with low computational complexity and negligible performance degradation.