Understanding the exploratory/confirmatory data analysis continuum: Moving beyond the "replication crisis".

Abstract

In light of the "replication crisis," some advocate for stricter standards and greater transparency in research methods. These efforts push toward a data analysis approach called "confirmatory data analysis" (CDA; see Wagenmakers et al., 2012). However, some (e.g., Baumeister, 2016; Goldin-Meadow, 2016) suggest that emphasizing CDA may restrict creativity and discovery. These scholars argued (sometimes inadvertently) for greater freedom to pursue "exploratory data analysis" (EDA; see Tukey, 1977). Ironically and unfortunately, many who push against stricter CDA standards do not realize EDA exists, or misunderstand the philosophy and proper tools for exploration. In this article, the meaning, tools, philosophy, and ethics associated with EDA, CDA, and a relatively unknown but important approach called "rough CDA" are clarified. Important distinctions are developed between EDA/rough CDA/CDA and other (some problematic) analysis activities including p-hacking, HARKing, and data mining, which are situated in a (graphical) framework that clarifies relationships and ethical boundaries with each. In short, the proper data analytic approach depends on (a) intentions and (b) transparency. Most psychological research is not at a maturity level to justify CDA; researchers have historically used tools mismatched to their research agenda. In the conclusion, recommendations are presented about how these typologies can be integrated into graduate training programs and how a cumulative research program can help psychology move beyond the replication crisis. (PsycInfo Database Record (c) 2022 APA, all rights reserved).