Structured Sparse Fully-Connected Layers in the CNNs and Its GPU Acceleration

Abstract

Convolutional Neural Networks (CNNs) have become one of the most important and popular artificial intelligent techniques recently. While capabilities of CNNs have improved dramatically, enlargement of the network has become a problem. Thus far, several studies about network model compression for CNNs have been devoted. However, most of the compressed networks are unsuitable for parallel computing platforms since the compressed networks are unstructured. To address this problem, we propose a new sparse network architecture for CNNs. We compress the fully-connected layers (FLs) in the CNNs that occupying most of the weights. By considering the characteristic of the output of convolutional layers, we can highly compress the network structurally with almost the same accuracy. In addition, using the structure of the compressed network, we propose an efficient implementation of the inference computation on the GPU. We have applied the proposed approach to AlexNet for the classification problem for ILSVRC-2012 dataset. The experimental results show that the number of weights in FLs is reduced from 58.6 million to 6.8 million with 0.68% and 0.19% loss of top-1 and top-5 accuracy, respectively. Also, the GPU implementation for the compressed FLs can run 5.86 times faster than that for non-compressed FLs.