Asymptotic probability density of the generalization error

Abstract

The generalization error-or risk-of a model, which is defined as the expectation of some loss function, quantifies the model performance from the user's point of view. The model parameters are assumed to be determined by minimizing the sample risk (e.g., the sum of squared errors). Since the training set is random and finite, the risk is a random variable. In this paper, the asymptotic probability density function (PDF) of the generalization error is derived. This expression is a second-order approximation valid when the size of the training set is large compared to the number of model parameters. In the particular case of models linear in the parameters with square loss, the PDF reduces to a chi-square density, which is a well-known property in linear regression. On the other hand, the average risk, which is at the origin of the AIC, GPE and NIC criteria, follows as an immediate corollary. By calculating the variance of the PDF, one obtains a measure of the robustness of generalization.