On Fitting a Straight Line to Data when the “Noise” in Both Variables Is Unknown*

Abstract

Abstract In meteorology and oceanography, and other fields, it is often necessary to fit a straight line to some points and estimate its slope. If both variables corresponding to the points are noisy, the slope as estimated by the ordinary least squares regression coefficient is biased low; that is, for a large enough sample, it always underestimates the true regression coefficient between the variables. In the common situation when the relative size of the noise in the variables is unknown, an appropriate regression coefficient is plus or minus the ratio of the standard deviations of the variables, the sign being determined by the sign of the correlation coefficient. For this case of unknown noise, the authors here obtain the probability density function (pdf) for the true regression coefficient divided by the appropriate regression coefficient just mentioned. For the case when the number of data is very large, a simple analytical expression for this pdf is obtained; for a finite number of data points the relevant pdfs are obtained numerically. The pdfs enable the authors to provide tables for confidence intervals for the true regression coefficient. Using these tables, the end result of this analysis is a simple practical way to estimate the true regression coefficient between two variables given their standard deviations, the sample correlation, and the number of independent data.