Doubtful Significance: Can an Amorphous Cloud of Points Really Illustrate a Significant Relationship?

Abstract

Can an amorphous cloud of points really illustrate a statistically significant relationship between two variables? We examine one particular scatter-plot from 402 observations which appears to show no relationship whatever between the Y and X variables, and yet, the conventional OLS regression finds a t-statistic of 3.0. To explain this puzzle, we derive a formula showing that the conventional t-statistic is the product of a signal-to-noise ratio and the square root of the regression degrees of freedom. This signal-to-noise ratio is a convenient measure of the visual clarity of any relationship in the scatter-plot. If the ratio is 1, then the relationship is reasonably clear to the eye. If the ratio is 10, then it is very clear indeed. But if the ratio is 0.1, or less, then the relationship is more or less invisible to the eye. However, even if the signal-to-noise ratio is very low, our formula shows that we can still obtain a large t-statistic, so long as we have enough observations. But how it can make sense to say that an amorphous cloud of points is ‘significant’? The standard independence assumption of regression analysis is absolutely critical here, and robustness checks show that without it, the relationship between y and x could be almost anything. Finally, we examine a sample of 2220 parameter estimates taken from recent articles in 20 leading journals, and find that for most of these – even those that are reported as statistically significant – the implicit signal to noise ratio is very low indeed (median value 0.033). The implication is that in the absence of the independence assumption, these parameter estimates tell us very little indeed. We conclude with a brief reexamination of this critical independence assumption and whether it is really justified.