Does preregistration improve the credibility of research findings?

Mark Rubin

doi:10.20982/tqmp.16.4.p376

Abstract

Preregistration entails researchers registering their planned research hypotheses, methods, and analyses in a time-stamped document before they undertake their data collection and analyses. This document is then made available with the published research report to allow readers to identify discrepancies between what the researchers originally planned to do and what they actually ended up doing. This historical transparency is supposed to facilitate judgments about the credibility of the research findings. The present article provides a critical review of 17 of the reasons behind this argument. The article covers issues such as HARKing, multiple testing, p-hacking, forking paths, optional stopping, researchers' biases, selective reporting, test severity, publication bias, and replication rates. It is concluded that preregistration's historical transparency does not facilitate judgments about the credibility of research findings when researchers provide contemporary transparency in the form of (a) clear rationales for current hypotheses and analytical approaches, (b) public access to research data, materials, and code, and (c) demonstrations of the robustness of research conclusions to alternative interpretations and analytical approaches.

Highlights

Preregistration has recently become popular in psychology and other disciplines
As Nosek et al (2018) concluded in their Preregistration Revolution article, “preregistration improves the interpretability and credibility of research findings” (p. 2605). Is this really the case?1 To address this question, I distinguish between two forms of transparency: historical transparency and contemporary transparency
Preregistration has been proposed as a method of identifying researchers’ biases, selective reporting, test severity, and null results. It has been proposed as a method of reducing publication bias and improving replication rates

Summary

Introduction

Preregistration has recently become popular in psychology and other disciplines. Preregistration entails researchers registering their planned research hypotheses, methods, and analyses in a time-stamped document before they undertake their data collection and analyses (Nosek, Ebersole, DeHaven, & Mellor, 2018; Wagenmakers, Wetzels, Borsboom, van der Maas, & Kievit, 2012). P values lose their meaning in exploratory analyses because they cannot be compared to an appropriately adjusted alpha level Preregistration solves this problem by providing a clearly defined plan of the number of tests in the data analysis. A researcher might test the joint null hypothesis that “men do not have better self-esteem than women on either SelfEsteem Measure 1 or Self-Esteem Measure 2.” In this case, the associated p values do not lose their meaning in exploratory analyses, because alpha adjustments can be applied relative to the specific joint null hypothesis in question (e.g., α/2 in the current example) A researcher might test the joint null hypothesis that “men do not have better self-esteem than women on either SelfEsteem Measure 1 or Self-Esteem Measure 2.” In this case, the associated p values do not lose their meaning in exploratory analyses, because alpha adjustments can be applied relative to the specific joint null hypothesis in question (e.g., α/2 in the current example)

Research Conclusions

Conclusion

Findings

Key Points