Evaluating the tradeoff between bias and variance through use of prior probabilities

Abstract

Listen

Translate article

Increasing the complexity of a model generally decreases the bias but increases the variance of estimates of features of the true distribution. This finding, known as the bias-variance tradeoff, indicates that fitting an extra parameter to a model may either increase or decrease the mean squared error of estimates. In this paper we adopt a Bayesian framework from which to evaluate the bias-variance tradeoff. Using the balanced ANOVA model to illustrate the approach, we define a loss function based on the squared error of the cell mean estimates. Then, using the prior on the parameter of interest θ, we calculate an expected loss under the decisions to fit or not fit 0. Fitting θ is defined as using the conventional frequentist least squares estimate; not fitting θ means setting it equal to 0, The criterion of minimizing expected loss determines, for each outcome, whether we fit θ (Ml decision) or not (MO decision). For different prior distributions, e.g., uniform, normal, we determine the set of outcomes corresponding to choosing MO. When θ has one degree of freedom we find for a zero mean normal prior that, depending on the variance of the prior, either all outcomes favor MO or all outcomes favor Ml. For a zero mean uniform prior all outcomes favor MO for a compact enough prior; for a more diffuse prior we find the paradoxical result that MO is favored when the test statistic for θ is large and Ml is favored when the test statistic is small. We compare this method with other common methods of model selection, both Bayesian and non-Bayesian.