Conceptual complexity of neural networks

Abstract

We propose a complexity measure of a neural network mapping function based on the order and diversity of the set of tangent spaces from different inputs. Treating each tangent space as a linear PAC concept we use an entropy-based measure of the bundle of concepts to estimate the conceptual capacity of the network. The theoretical maximal capacity of a ReLU network is equivalent to the number of its neurons. In practice, however, due to correlations between neuron activities within the network, the actual capacity can be remarkably small, even for very big networks. We formulate a new measure of conceptual complexity by normalising the capacity of the network by the degree of separation of concepts related to different classes. Empirical evaluations show that this new measure is correlated with the generalisation capabilities of the corresponding network. It captures the effective, as opposed to the theoretical, complexity of the network function. We also showcase some uses of the proposed measures for analysis and comparison of trained neural network models.