On the choice of TLS versus OLS in climate signal detection regression

Abstract

Total least squares (TLS) or multivariate orthogonal regression is widely used as a remedy for attenuation bias in climate signal detection or “optimal fingerprinting” regression. But under some circumstances it overcorrects and imparts an upward bias, as well as generating extremely unstable and imprecise coefficient estimates. While there has been increasing attention paid recently to the validity of TLS-based confidence intervals, there has been no corresponding examination of coefficient bias problems. This note explains why they are pertinent and presents a Monte Carlo simulation to illustrate the hazards of using TLS in a signal detection application without testing whether the modeling context makes it a suitable choice. TLS is not automatically preferred over OLS even when explanatory variables are believed to contain random errors. Notably it can be sufficiently biased to cause false positives when explanatory signals are negatively correlated, and the bias gets worse as the signal-noise ratio on the explanatory variables rises. Additionally TLS should not be used on its own for climate signal detection inferences since if the no-signal null is true, TLS is generally inconsistent whereas OLS attenuation bias disappears.