First-Order Sensitivity Analysis for Hidden Neuron Selection in Layer-Wise Training of Networks

Abstract

Multilayer neural networks are current trends in machine learning. Although complex architectures bring high performance, having sparse neurons in each layer can save memory, energy, and computational resources. In this paper, we aim to balance benefits between the complexity of architectures and the sparsity of neurons. An algorithm is proposed to prune neurons in multilayer neural networks through the global sensitivity analysis. Motivated by layer-wise training, we construct autoencoders with linear decoders, so mathematical models of multilayer neural networks can be considered as additive models. Hence, a first-order sensitivity analysis method, called random balance designs (RBD), is employed to select redundant neurons in hidden layers. This paper provides a novel framework to apply RBD in multilayer neural networks. Multiple experimental results demonstrate the generality and effectiveness of the proposed approach on structural learning of neural networks. After removing superfluous hidden neurons, higher accuracy can be obtained in most cases with less computation.