Visualizing Estimation Error with Bias-Variance Parameter Space Diagrams

Abstract

Who would want to be biased? Bias seems obviously, inherently bad. An example of a biased estimator is one that excludes explanatory variables, such as a model of school dropout rates that excludes science test scores. Another example is a model that restricts the allowed parameter values, such as a financial investment model that has a maximum amount that can be invested in any single mutual fund. But there are empirical and analytical demonstrations that biased estimators like these can outperform unbiased estimators. Yet data modelers remain reluctant to use biased estimators because these findings seem so unintuitive.This paper provides a visual demonstration of how biased estimators work so even people with minimal statistics can feel comfortable using them. The bias-variance parameter space in this paper are a simplified version of the parameter space diagrams in standard machine learning and statistics textbooks such as Hastie et. al (2001). The baseline case shows an unbiased estimator using a large parameter space, fit with a large sample or small sample. The bias and variance achieved in this case is compared with a) a small, constrained parameter space such as a unit-weight linear model and b) ridge regression, an example of shrinkage. We then consider these cases again in highly collinear data, a situation in which biased models have an even bigger advantage over unbiased models. The visualization provides a deeper understanding of what it means for an estimator to reduce sampling variance at the cost of additional bias. This should give the reader more confidence in using biased estimators in situations where they are expected to outperform unbiased estimators.