On optimal neural network learning

Abstract

We introduce optimal learning with a neural network, which we define as building a network with a minimal expectation generalisation error. This procedure may be analysed exactly in idealized problems by exploiting the relationship between sampling a space of hypotheses and the replica method of statistical physics. We find that the optimally trained spherical perceptron may learn a linearly separable rule as well as any possible computer, and present simulation results supporting our conclusions. Optimal learning of a well-known unlearnable problem, the “mismatched weight” problem, gives better asymptotic learning than conventional techniques, and may be simulated more easily. Unlike many other perceptron learning schemes, optimal learning extends to more general networks learning more complex rules.