Crafting statistical analysis plans: A cross‐discipline approach

Abstract

AbstractDeveloping a plan for data analysis at the beginning of a study is an important practice that is underutilized in many scientific fields. Several funding agencies and journals now require submission of statistical analysis plans in advance of scientific studies, particularly in the clinical sciences. Even when a plan is not required, it can be advantageous to the scientific process by improving reproducibility. An analysis plan allows researchers to organize their knowledge about their research questions and experimental design to more easily recognize and choose the appropriate statistical analyses. An analysis plan provides a roadmap for the analyses: Researchers can think through potential statistical decisions (e.g. to transform or not to transform? how to handle missing or censored data?) in advance and thoroughly document the justifications and trade‐offs for their intended analyses. Such decisions are not influenced by data when made before data collection, thus preventing pitfalls like p‐hacking, HARKing and non‐replicability of results. We describe a general framework for crafting an analysis plan—including essential components of any plan—and provide an example template that can be used by researchers. The analysis plan framework is presented for broad appeal to experienced statisticians, quantitative researchers and everyone in between.