Explicit symmetries and the capacity of multilayer neural networks

Abstract

Calculating the capacity and generalization capabilities of feed-forward multilayer neural networks requires the use of replica-symmetry-breaking methods, making the calculation practically unfeasible. Replica symmetry is broken because the configuration space is disconnected, which is clearly the case in the capacity limit where the configuration space shrinks to isolated points. Moreover, there is no knowledge about the number of replica-symmetry-breaking steps required to obtain reliable results. Novel approaches to tackle the capacity calculation of feed-forward neural networks avoiding the use of replica-symmetry-breaking methods are presented in this paper. The basic idea behind these approaches is that breaking explicit symmetries of the network prior to the capacity calculation itself restores order-parameter symmetry, at least to a good approximation, and therefore enables the use of the replica-symmetry ansatz. Two methods are presented for breaking the explicit symmetries and restoring replica symmetry; one restricts relations between the various weight elements while the other restricts the values of the order parameters. These methods, which are demonstrated in this work via the capacity calculation of feed-forward neural networks, are applicable to a variety of capacity, learning and generalization capability calculations of such nets. We examine an approximation for carrying out the multi-dimensional Gaussian integrals appearing during the calculation as well as exact results for some simple cases. Numerical results obtained for nets with one to six hidden neurons using the downhill simplex and adaptive simulated-annealing optimization algorithms are in good agreement with simulation results.