Abstract
In recent years, deep learning models have achieved remarkable successes in various applications, such as pattern recognition, computer vision, and signal processing. However, high-performance deep architectures are often accompanied by a large storage space and long computational time, which make it difficult to fully exploit many deep neural networks (DNNs), especially in scenarios in which computing resources are limited. In this paper, to tackle this problem, we introduce a method for compressing the structure and parameters of DNNs based on neuron agglomerative clustering (NAC). Specifically, we utilize the agglomerative clustering algorithm to find similar neurons, while these similar neurons and the connections linked to them are then agglomerated together. Using NAC, the number of parameters and the storage space of DNNs are greatly reduced, without the support of an extra library or hardware. Extensive experiments demonstrate that NAC is very effective for the neuron agglomeration of both the fully connected and convolutional layers, which are common building blocks of DNNs, delivering similar or even higher network accuracy. Specifically, on the benchmark CIFAR-10 and CIFAR-100 datasets, using NAC to compress the parameters of the original VGGNet by 92.96% and 81.10%, respectively, the compact network obtained still outperforms the original networks.
Highlights
In order to solve challenging deep learning problems, such as pattern recognition and computer vision [1,2,3], researchers tend to design deep neural networks (DNNs) with complex structures and many neurons
We introduce a systematic DNN compression method built on neuron agglomerative clustering (NAC), which is mainly applied to the neurons/feature maps of fully connected layers and convolutional layers
We report the results of the experiments separately, where the best results shown in the tables are highlighted in boldface
Summary
In order to solve challenging deep learning problems, such as pattern recognition and computer vision [1,2,3], researchers tend to design deep neural networks (DNNs) with complex structures and many neurons. Some existing network compression methods can only work on fully connected layers [6], while some approaches to compress convolutional layers require an additional sparse BLASlibrary or special hardware support [7,8,9]. We introduce a systematic DNN compression method built on neuron agglomerative clustering (NAC), which is mainly applied to the neurons/feature maps of fully connected layers and convolutional layers. We attain similar neurons/feature maps in the neural network through agglomerative clustering and respectively agglomerate them and their related connections together. Last but not the least, during the process of agglomerative clustering, NAC does not need to use the original training data, but only the weights and biases connected to the neurons/feature maps.
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
