Defending the logic of significance testing: a response to Gorard

Abstract

Recently, Stephen Gorard has outlined strong objections to the use of significance testing in social research. He has argued, first, that as the samples used in social research are almost always non-random it is not possible to use inferential statistical techniques and, second, that even if a truly random sample were achieved, the logic behind the calculation and interpretation of p-values is fundamentally flawed. Arguments against Gorard's position have focused almost exclusively on the first point (the non-random nature of samples) despite the fact that it is the second point which is the more important: if the logic of significance testing is indeed flawed, then whether non-random samples are a problem or not becomes irrelevant. This article aims to show that the logic of significance testing is not flawed in the ways which Gorard claims because: 1) samples do contain real information about the population from which they are derived; and 2) under certain assumptions the p-value can reflect—or at least be a reasonable proxy for—the probability of the null hypothesis being true given the data.