Statistically Based Weight Pruning in Feed-Forward Neural Networks

Abstract

A statistically-based algorithm for pruning weights from feed-forward networks is presented. This algorithm relies upon the Generalized Wald and t-test statistics to determine which weights to remove from the network. Because both of these tests use the exact Hessian matrix, an algorithm for learning the exact Hessian matrix for a feed-forward neural network using a single backward pass through the data is presented when the L2 norm is minimized in the energy function. The pruning algorithm is then applied in two simulations: The first simulation investigates the relationship between neural networks and linear regression (Ordinary Least Squares), and the weight covariance matrix is found to be asymptotically equivalent to the White (1980) standard error corrections for heterogeneity of variance. The final simulation applies the algorithm to a network solving the sunspot data and compares the results to those found in the literature, with mixed results.