Power, Dominance, and Constraint: A Note on the Appeal of Different Design Traditions

Abstract

The recent field-wide emphasis on power has brought the number of participants used in psychological experiments into focus. Social psychology typically follows a tradition in which many participants perform a small number of trials each; in psychophysics, the tradition is to include only a few participants, who perform many trials each; and the tradition in cognitive psychology falls in between, balancing the number of participants and trials. We ask whether it is better to add trials or to add participants if one wishes to increase power. The answer is straightforward—greatest power is achieved by using more people, and the gain from adding people is greater than the gain from adding trials. In light of these results, the design parameters in the social psychology tradition seem ideal. Yet there are conditions in which one may trade people for trials with only a minor decrement in power. Under these conditions, the limiting factor is the trial-to-trial variability rather than the variability across people in the population. These conditions are highly plausible, and we present a theoretical argument as to why. We think that most cognitive effects are characterized by stochastic dominance; that is, everyone’s true effect is in the same direction. For example, it is plausible that when performing the Stroop task, all people truly identify congruent colors faster than incongruent ones. When dominance holds, small mean effects imply a small degree of variability across the population. It is this degree of homogeneity, the consequence of dominance, that licenses the design parameters of the cognitive psychology and psychophysics traditions.