A checklist to guide sensitivity analyses and replications of impact evaluations

Abstract

ABSTRACT The transparency, reproducibility, and ethics movement has helped promote efforts to improve the rigour and validity of social science research. However, replication research, where the original data is used to check the validity and robustness of the estimations and recommendations in the original paper, has not been widely used as a tool in this field. This article outlines a checklist and standardised process that can guide replication researchers in creating and justifying their replication plans. To create the checklist, we reviewed 31 resources that were identified through unstructured keyword searches. We also reviewed and coded replication studies from 3ie’s Replication Program which includes more than 20 3ie-funded replication papers and guidance on our replication process. There were five categories of potential validation tests and robustness checks that researchers could include. The checklist attributes range from testing specific assumptions for select impact evaluation methods to data transformations or dealing with heterogeneity. We hope that this checklist helps stimulate demand for replication research and promotes the use of replication research as a tool within transparency, reproducibility and ethics.