Measuring the effect of reviewers on manuscript change: A study on a sample of submissions to Royal Society journals (2006–2017)

Abstract

Peer review is key for public trust of academic journals. It ensures that only rigorous research is published but also helps authors to increase the value of their manuscripts through feedback from reviewers. However, measuring the developmental value of peer review is difficult as it requires fine-grained manuscript data on various stages of the editorial process, which are rarely available. To fill this gap, we accessed complete data from Royal Society journals from 2006 to 2017, and measured manuscript changes during peer review from their initial submissions. We then estimated the effect of the number of reviewers and the evaluation of reviewers on manuscript development and their citations after publication. We found that the number of reviewers had an almost linear effect on manuscript change although with decreasing marginal effects whenever more than two reviewers were involved. This effect did not depend on the initial quality of manuscripts. We also found that changes due to reviewers tended to increase a manuscript’s probability of being cited at least once after publication. While our findings show the multiple functions of peer review for manuscript development, research with larger and more representative journal samples is needed to develop more systematic measures that reflect the complexity of peer review.