Complex systems approach to scientific publication and peer-review system: development of an agent-based model calibrated with empirical journal data.

Michail Kovanis,Raphaël Porcher,Philippe Ravaud,Ludovic Trinquart

doi:10.1007/s11192-015-1800-6

Abstract

Scientific peer-review and publication systems incur a huge burden in terms of costs and time. Innovative alternatives have been proposed to improve the systems, but assessing their impact in experimental studies is not feasible at a systemic level. We developed an agent-based model by adopting a unified view of peer review and publication systems and calibrating it with empirical journal data in the biomedical and life sciences. We modeled researchers, research manuscripts and scientific journals as agents. Researchers were characterized by their scientific level and resources, manuscripts by their scientific value, and journals by their reputation and acceptance or rejection thresholds. These state variables were used in submodels for various processes such as production of articles, submissions to target journals, in-house and external peer review, and resubmissions. We collected data for a sample of biomedical and life sciences journals regarding acceptance rates, resubmission patterns and total number of published articles. We adjusted submodel parameters so that the agent-based model outputs fit these empirical data. We simulated 105 journals, 25,000 researchers and 410,000 manuscripts over 10 years. A mean of 33,600 articles were published per year; 19 % of submitted manuscripts remained unpublished. The mean acceptance rate was 21 % after external peer review and rejection rate 32 % after in-house review; 15 % publications resulted from the first submission, 47 % the second submission and 20 % the third submission. All decisions in the model were mainly driven by the scientific value, whereas journal targeting and persistence in resubmission defined whether a manuscript would be published or abandoned after one or many rejections. This agent-based model may help in better understanding the determinants of the scientific publication and peer-review systems. It may also help in assessing and identifying the most promising alternative systems of peer review.

Highlights

Background and significanceThe burden associated with the worldwide scientific production has recently generated much debate and criticism about the sustainability of the established system of scientific publication
Scientific peer-review and publication systems incur a huge burden in terms of costs and time
We developed an agent-based model by adopting a unified view of peer review and publication systems and calibrating it with empirical journal data in the biomedical and life sciences

Summary

Introduction

The burden associated with the worldwide scientific production has recently generated much debate and criticism about the sustainability of the established system of scientific publication. The exponential increase in number of manuscripts submitted for publication is much higher than the increase in number of researchers and overburdens the ability of available qualified referees (Ware and Mabe 2015; Gannon 2001; Laakso et al 2011; Bohannon 2014; Arns 2014; Alberts et al 2008). For the UK higher education institutions alone, peer review would cost more than £110 million annually (Look and Sparks 2010). Most researchers believe that peer review is vital to scientific publication, but it needs to be improved to address all the challenges that arise (Mulligan et al 2013; Nicholas et al 2015)

Objectives

Findings

Discussion

Conclusion