Spectrum Analysis for Fully Connected Neural Networks.

Abstract

This article studies the meaning of parameters of fully connected neural networks with single hidden layer from the perspective of spectrum. Under the constraints of numerical range, the corresponding relationship between parameters and the spectrum of network function can be established by the Fourier series coefficients of the activation function, which is truncated and periodically extended. This work is substantiated on the Mixed National Institute of Standards and Technology (MNIST) handwritten dataset and two illustrative examples with certain spectra. The simulations complete the conversion between spectrum and parameters with high precision and give the significance of hidden nodes to the spectrum of network function. Some algorithms derived from these properties, such as the parameter initialization method using spectrum and the pruning method by sorting amplification weights, are also presented to introduce how spectrum analysis affects neural network decision-making. Thus, spectrum analysis has great potential in network interpretation.