Parameter convergence and learning curves for neural networks

Abstract

We revisit the oft-studied asymptotic (in sample size) behavior of the parameter or weight estimate returned by any member of a large family of neural network training algorithms. By properly accounting for the characteristic property of neural networks that their empirical and generalization errors possess multiple minima, we establish conditions under which the parameter estimate converges strongly into the set of minima of the generalization error. These results are then used to derive learning curves for generalization and empirical errors that leads to bounds on rates of convergence.