Journal and disciplinary variations in academic open peer review anonymity, outcomes, and length

Abstract

Understanding more about variations in peer review is essential to ensure that editors and reviewers harness it effectively in existing and new formats, including for mega-journals and when published online. This article analyzes open reviews from the MDPI suite of journals to identify commonalities and differences from a simplistic quantitative perspective, focusing on reviewer anonymity, review length, and review outcomes. The sample contained 45,385 first round open reviews from published standard journal articles in 288 MDPI journals classified into one or more Scopus disciplinary areas (Health Sciences; Life Sciences; Physical Sciences; Social Sciences). The eight main findings include substantial differences between journals and disciplines in review lengths, reviewer anonymity, review outcomes, and the use of attachments. In particular, Physical Sciences journal reviews tended to be stricter and were more likely to be anonymous. Life Sciences and Social Sciences reviews were the longest overall. Signed reviews tend to be 15% longer (perhaps to be more careful or polite) but gave similar decisions to anonymous reviews. Finally, reviews with major revision outcomes tended to be 68% longer than reviews with for minor revision outcomes, except in a few journals. In conclusion, signing reviews does not seem to threaten the validity of peer review outcomes and authors, editors, and reviewers of multidisciplinary articles should be aware of substantial field differences in what constitutes an appropriate review.