Analytic reproducibility in articles receiving open data badges at the journal Psychological Science: an observational study.

Tom E Hardwicke,Kyle Macdonald,Michèle B Nuijten,Michael C Frank,Erica J Yoon,Benjamin E Demayo,Bria Long,Benjamin N Peloquin,Manuel Bohn,Emily Hembacher

doi:10.1098/rsos.201494

Abstract

For any scientific report, repeating the original analyses upon the original data should yield the original outcomes. We evaluated analytic reproducibility in 25 Psychological Science articles awarded open data badges between 2014 and 2015. Initially, 16 (64%, 95% confidence interval [43,81]) articles contained at least one ‘major numerical discrepancy' (>10% difference) prompting us to request input from original authors. Ultimately, target values were reproducible without author involvement for 9 (36% [20,59]) articles; reproducible with author involvement for 6 (24% [8,47]) articles; not fully reproducible with no substantive author response for 3 (12% [0,35]) articles; and not fully reproducible despite author involvement for 7 (28% [12,51]) articles. Overall, 37 major numerical discrepancies remained out of 789 checked values (5% [3,6]), but original conclusions did not appear affected. Non-reproducibility was primarily caused by unclear reporting of analytic procedures. These results highlight that open data alone is not sufficient to ensure analytic reproducibility.

Highlights

A minimum quality standard expected of all scientific manuscripts is that any reported numerical values can be reproduced if the original analyses are repeated upon the original data [1]
The study protocol was pre-registered on 18 October 2017. All deviations from this protocol or additional exploratory analyses are explicitly acknowledged in electronic supplementary material, section D
The open badges scheme introduced at Psychological Science has been associated with an increase in data availability [15], the current findings suggest that additional efforts may be required in order to ensure analytic reproducibility

Summary

Introduction

A minimum quality standard expected of all scientific manuscripts is that any reported numerical values can be reproduced if the original analyses are repeated upon the original data [1]. This concept is known as analytic reproducibility ([2]; or relatedly, computational reproducibility,1 [3]). When a number cannot be reproduced, this minimally indicates that the process by which it was calculated has not been sufficiently documented. Non-reproducibility may indicate that an error has occurred, either during the original calculation or subsequent reporting. The integrity of the analysis pipeline that transforms raw data into reported results cannot be guaranteed. Nonreproducibility can undermine data reuse [5], complicate replication attempts [7], and create uncertainty about the provenance and veracity of scientific evidence, potentially undermining the credibility of any associated inferences [2]

Objectives

Results

Discussion

Conclusion