Abstract
Neural Networks (NN) provide state-of-the-art performance in many problem domains. They can accommodate a vast number of parameters and still perform well when classic machine learning techniques provided with the same number of parameters would tend to overfit. To further the understanding of such incongruities, we develop a metric called the Expected Spanning Dimension (ESD) which allows one to measure the intrinsic flexibility of a NN. We analyze NNs from the small, in which the ESD can be exactly computed, to large real-world networks with millions of parameters, in which we demonstrate how the ESD can be numerically approximated efficiently. The small NNs we study can be understood in detail, their ESD can be computed analytically, and they provide opportunities for understanding their performance from a theoretical perspective. On the other hand, applying the ESD to large-scale NNs sheds light on their relative generalization performances and provides suggestions as to how such NNs may be improved.
Highlights
Neural networks (NN) have wide applications in machine learning [1]
We have defined the idea of an expected spanning dimension (ESD) for NNs and have demonstrated how maximizing this ESD improves the performance of real-world NNs
A small ESD can be thought of as a regularization of the NN, and increasing the ESD is an example of the bias–variance trade-off in machine learning
Summary
Neural networks (NN) have wide applications in machine learning [1]. For example, in the field of computer vision, convolutional NNs, a specific type of feed-forward NN, greatly outperform any previous traditional computer vision method [2]. We develop a method to measure the flexibility of an NN, independent of any input data, that is both computed and is highly correlated with the testing accuracy of the NNs on standard benchmark datasets We accomplish this by considering a basis of functions to represent the NN and analyzing the dimension of the range of the function of the inputs of the NN to the coefficients of the output of the NN with respect to that basis. Analyzing the rank of appropriately defined Jacobians, which arise naturally from NNs, is one of our key analytical tools Such Jacobians can be efficiently computed using the automatic differentiation capabilities of deep learning libraries like PyTorch and TensorFlow. We can expand this polynomial to obtain the coefficients of the basis 1, x, x2, x3,
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
