Comparing the predictive powers of alternative multiple regression models

Abstract

Mean squared error of prediction is used as the criterion for determining which of two multiple regression models (not necessarily nested) is more predictive. We show that an unrestricted (or true) model witht parameters should be chosen over a restricted (or misspecified) model withm parameters if (Pt2−Pm2)>(1−Pt2)(t−m)/n, wherePt2 andPm2 are the population coefficients of determination of the unrestricted and restricted models, respectively, andn is the sample size. The left-hand side of the above inequality represents the squared bias in prediction by using the restricted model, and the right-hand side gives the reduction in variance of prediction error by using the restricted model. Thus, model choice amounts to the classical statistical tradeoff of bias against variance. In practical applications, we recommend thatP2 be estimated by adjustedR2. Our recommendation is equivalent to performing theF-test for model comparison, and using a critical value of 2−(m/n); that is, ifF>2−(m/n), the unrestricted model is recommended; otherwise, the restricted model is recommended.