No evidence of important difference in summary treatment effects between COVID-19 preprints and peer-reviewed publications: a meta-epidemiological study

Abstract

ObjectivesPreprints became a major source of research communication during the COVID-19 pandemic. We aimed to evaluate whether summary treatment effect estimates differ between preprint and peer-reviewed journal trials. Study Design and SettingA meta-epidemiological study. Data were derived from the COVID-NMA living systematic review (covid-nma.com) up to July 20, 2022. We identified all meta-analyses evaluating pharmacological treatments vs. standard of care or placebo for patients with COVID-19 that included at least one preprint and one peer-reviewed journal article. Difference in effect estimates between preprint and peer-reviewed journal trials were estimated by the ratio of odds ratio (ROR); ROR <1 indicated larger effects in preprint trials. ResultsThirty-seven meta-analyses including 114 trials (44 preprints and 70 peer-reviewed publications) were selected. The median number of randomized controlled trials (RCTs) per meta-analysis was 2 (interquartile range [IQR], 2–4; maximum, 11), median sample size of RCTs was 199 (IQR, 99–478). Overall, there was no statistically significant difference in summary effect estimates between preprint and peer-reviewed journal trials (ROR, 0.88; 95% CI, 0.71–1.09; I2 = 17.8%; τ2 = 0.06). ConclusionWe did not find an important difference between summary treatment effects of preprints and summary treatment effects of peer-reviewed publications. Systematic reviewers and guideline developers should assess preprint inclusion individually, accounting for risk of bias and completeness of reporting.