Abstract
Obtaining evidence that something does not exist requires knowing how big it would be were it to exist. Testing a theory that predicts an effect thus entails specifying the range of effect sizes consistent with the theory, in order to know when the evidence counts against the theory. Indeed, a theoretically relevant effect size must be specified for power calculations, equivalence testing, and Bayes factors in order that the inferential statistics test the theory. Specifying relevant effect sizes for power, or the equivalence region for equivalence testing, or the scale factor for Bayes factors, is necessary for many journal formats, such as registered reports, and should be necessary for all articles that use hypothesis testing. Yet there is little systematic advice on how to approach this problem. This article offers some principles and practical advice for specifying theoretically relevant effect sizes for hypothesis testing.
Highlights
If there is no way for data from a study to count against a theory, the putative test of the theory is not a test at all
This paper addresses what might be called the pragmatics of statistical inference: How theory relates to inferential statistics in order to severely test a theory
The heuristic of using the effect size of an original study is useful for Bayes factors but not for power or inference by intervals; the effect obtained in a previous study does not in itself provide a minimally interesting effect size
Summary
If there is no way for data from a study to count against a theory, the putative test of the theory is not a test at all. The smallest effect size of interest instantiates a value in that model (in this case, the bounds of the null interval) It is in this way the theory itself can be tested: The model represents predictions that data can count for or against. The heuristic of using the effect size of an original study is useful for Bayes factors but not for power or inference by intervals; the effect obtained in a previous study does not in itself provide a minimally interesting effect size. Palfi and Dienes (2019; version 3 Table 2) provide very quick heuristics for working this probability out, without needing simulations In both cases, one uses the expected scale of effect, not the minimally interesting effect size. Could demand characteristics make similar predictions as an alternative substantial theory (cf Dienes, Lush & Palfi, in press)? Lush (2020; Orne, 1962) illustrates a way of obtaining quantitative predictions from the theory of demand characteristics which could be compared to a theory one wished to test
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
