A closer look at the bias-variance trade-off in multivariate calibration

Abstract

Principal component regression (PCR) and partial least squares (PLS) avoid the high variance often associated with ordinary least squares (OLS) results by allowing a small bias in the model. This paper presents a closer look at this bias–variance trade-off by discussing three practical aspects: (1) variance increases relatively slowly with increasing model complexity; (2) bias may be zero for the optimum model; (3) variance does not necessarily increase with increasing model complexity. While the first aspect is well known, the last two aspects are not. The second aspect implies that so-called biased regression methods do not necessarily yield biased predictions, while the third aspect, which is only encountered with non-linear estimation methods such as PLS, even contradicts the concept of bias–variance trade-off. The possibility of having both variance and bias decreasing with increasing PLS model complexity is illustrated using a near-infrared data set published by Fearn. Copyright © 1999 John Wiley & Sons, Ltd.