Layer sparsity in neural networks

Abstract

Sparsity has become popular in machine learning because it can save computational resources, facilitate interpretations, and prevent overfitting. This paper discusses sparsity in the framework of neural networks. In particular, we formulate a new notion of sparsity, called layer sparsity, that concerns the networks’ layers and, therefore, aligns particularly well with the current trend toward deep networks. We then introduce corresponding regularization and refitting schemes that can complement standard deep-learning pipelines to generate more compact and accurate networks.