Merging Neurons for Structure Compression of Deep Networks

Abstract

Deep neural networks are increasingly used in many fields, such as pattern recognition, computer vision, and natural language processing. However, how to apply deep neural networks in mobile settings has become an urgent issue, as mobile devices are getting more and more popularity. This is mainly due to the fact that mobile devices usually have very limited computation and storage resources, which prevents from running a large-scale deep network. This paper proposes a novel method for structure compression of deep neural networks. The main idea is to merge the neurons and connections of the original network using clustering methods. To the end, the new network after compression possesses much less parameters, which leads to reduced requirements for computation and storage resources. Experiments on benchmark data sets demonstrate that the proposed method can greatly improve the efficiency of deep neural networks, while retain their learning capability.