Exploring Small, Confirming Big: An alternative system to The New Statistics for advancing cumulative and replicable psychological research

Abstract

While outlining his vision of The New Statistics, Cumming (2014) proposes that a more rigorous and cumulative psychological science will be built, in part, by having psychologists abandon traditional null-hypothesis significance testing (NHST) approaches, and conducting small-scale meta-analyses on their data whenever possible. In the present paper, I propose an alternative system for conducting rigorous and replicable psychological investigations, which I describe as Exploring Small, Confirming Big. I begin with a critical evaluation of the merits of NHST and small-scale meta-analyses, and argue that NHST does have a valuable role in the scientific process, whereas small-scale meta-analyses will do little to advance a cumulative science. I then present an overview of an alternative system for producing cumulative and replicable psychological research: Exploring Small, Confirming Big. It involves a two-step process to psychological research, consisting of (1) small N investigation(s), in which psychologists use NHST to develop exploratory models; and (2) strong, confirmatory tests of exploratory models, by analyzing new and/or existing large N datasets with variables that capture the effect(s) of interest from the Exploring Small stage. I conclude by discussing several anticipated benefits and challenges of adopting the Exploring Small, Confirming Big approach.