Calibrating the Scientific Ecosystem Through Meta-Research

Tom E Hardwicke,Valentin Danchev,John P.A Ioannidis,Stylianos Serghiou,Sophia Crüwell,Steven N Goodman,Perrine Janiaud

doi:10.1146/annurev-statistics-031219-041104

Abstract

While some scientists study insects, molecules, brains, or clouds, other scientists study science itself. Meta-research, or research-on-research, is a burgeoning discipline that investigates efficiency, quality, and bias in the scientific ecosystem, topics that have become especially relevant amid widespread concerns about the credibility of the scientific literature. Meta-research may help calibrate the scientific ecosystem toward higher standards by providing empirical evidence that informs the iterative generation and refinement of reform initiatives. We introduce a translational framework that involves ( a) identifying problems, ( b) investigating problems, ( c) developing solutions, and ( d) evaluating solutions. In each of these areas, we review key meta-research endeavors and discuss several examples of prior and ongoing work. The scientific ecosystem is perpetually evolving; the discipline of meta-research presents an opportunity to use empirical evidence to guide its development and maximize its potential.

Highlights

MethodsIdentified a ‘straightforward’ and substantive finding from a pseudo-randomly selected subset of 35 articles from Stage 1 judged to have re-usable data in principle
Research departments Academic societies Universities Publishers Librarians Journals Funders Scientists Media Students Politicians Practitioners General public
Completeness

Summary

Methods

Identified a ‘straightforward’ and substantive finding from a pseudo-randomly selected subset of 35 articles from Stage 1 judged to have re-usable data in principle. Always sought author assistance/clarification when issues arose. “Major error” is a >= 10% discrepancy between reported and obtained value. An article is ‘not fully reproducible’ if it has one or more major errors. Important caveat: No cases where reproducibility issues appeared to seriously undermine substantive conclusions (3 unclear cases). 64 “major numerical errors” (5% error rate)

Conclusions

Final Report not yet published