Scientific versus statistical inference.

Abstract

Abstract We argue that the goals of scientists in data analysis and scientific communication do not match the logic of hypothesis testing as it is typically taught in introductory statistics courses. The method has a number of well-known logical fallacies and the results of the procedures are often misinterpreted. However, we suspect that these problems do not directly compromise researchers' inferences concerning their data because researchers implicitly use a different method for identifying the implications of data. In particular, we argue that researchers are typically interested in comparing the relative adequacy of different accounts or theoretical explanations rather than rejecting an artificially constructed null hypothesis; a simple analysis of recent journal articles supports this view. An alternative approach to data analysis and scientific communication is to present the strength of the evidence provided by an experiment in the form of a maximum likelihood ratio. This approach is more consistent with the logic of comparing alternative accounts that scientists commonly use. Likelihood ratios are simple to calculate and provide a simple and intuitive summary of the results relevant to evaluating the alternative explanations. Further, reporting likelihood ratios avoids the logical fallacies and interpretational problems of the Intro Stats method, allows one to identify compelling evidence for null effects, provides a simple method for evaluating failures to replicate, and can be readily used to aggregate results across experiments. In the present article, we argue that there is a mismatch between what psychologists do on one hand when they design experiments, analyze data, and communicate results and conclusions, and what is ostensibly done on the other when results sections are filled out with F, t, and p values. What is ostensibly done is captured in its essential elements by what undergraduates are taught in introductory statistics courses; we refer to this approach as the method. As it is commonly taught, the Intro Stats method involves assessing the likelihood of the data on the assumption that an artificially constructed null hypothesis is true; if the likelihood is sufficiently small, a mutually exclusive and exhaustive alternative hypothesis can be embraced by default. It is interesting that this approach to data analysis was never seriously proposed by anyone in the development of methods of statistical inference; instead, it arose more or less by default. We review some of this history below. However, regardless of its etiology, our central point is that this approach is fundamentally different from the logic that is generally adopted by experienced researchers in designing experiments and interpreting results. In particular, we argue that in many cases, researchers are interested in comparing the adequacy of different theoretical accounts; they are rarely concerned with accepting or rejecting the so-called null hypothesis. We propose an alternative approach to inference that seems more in line with the actual logic used by researchers. The Development of Statistical Inference Historically, there have been four dominant approaches to hypotheses testing: the Bayesian approach, the Fisherian approach, the Neyman-Pearson approach, and the current Intro Stats method, which is a combination of the Fisherian and Neyman-Pearson approach (Chow, 1996). IMAGE FORMULA6IMAGE FORMULA7 The Bayesian approach to hypothesis testing has been criticized for its reliance on prior probabilities. In many situations there is no clear theoretical basis for calculating prior probabilities, and the researcher is left to estimate these values (e.g., Edwards, Lindman, & Savage, 1963). This can become problematic, for example, when one compares results across different experiments (Loftus & Masson, 1994). …