Assessment of Variables that Influence Agreement Amongst Reviewers for Foot & Ankle International

Abstract

Category: Other Introduction/Purpose: The peer-review process is a rigorous process under which manuscripts are assessed for their overall scientific quality and is generally accepted as the highest standard of scientific scrutiny with regard to medical publishing. A common criticism regards the often disparate nature of reviewer recommendations when a decision is rendered which belies the supposed uniformity of the review process. The purpose of this investigation was to: (1) examine the historic level of agreement amongst reviewers for Foot & Ankle International (FAI) and (2) to assess variables which may influence agreement in order to improve the peer-review process. Methods: Approval to conduct this study was obtained from the Editorial Board of FAI. All manuscripts submitted to FAI during 2015 which underwent peer-review were included in the analysis. For each reviewed manuscript, demographic data was collected regarding specific reviewer and manuscript characteristics in a de-identified manner. Univariate analysis was performed. Results: 442 manuscripts underwent peer-review by 198 reviewers during the study period. During this time period, other papers were reviewed by the Editor-in-Chief and rejected prior to being sent out for review. Of the 884 reviews performed, 339 (38%) recommended rejection, 353 (40%) recommended revision and resubmission and191(22%) recommended accept. Only 199 manuscripts (45%) had a decision rendered in which both reviewers agreed on the initial recommendation.The most common initial decision was rejection (52.7%) followed by revise and resubmit (42.8%). Only 20 manuscripts (4.5%) received an outright acceptance upon initial review. Comparing the agreeing versus disagreeing reviewers, there was no difference in demographic data such as reviewer age or experience. When examining key words (designated by reviewers as a particular area of interest within foot and ankle), there was no association between shared interests and level of reviewer agreement. Overall, for all reviewers, mean acceptance rate was19% (+/- 16%), mean reject rate 37% (+/- 20%) and mean revise 44% (+/- 19%). Conclusion: Regarding initial decision for publication in FAI, there was only 45% agreement amongst reviewers for manuscripts which underwent peer-review in 2015. However, no reviewer-specific variables examined in this investigation were found to correlate with agreement. Despite reviewers having similar interests in various aspects of foot and ankle surgery, this did not lead to an increased likelihood of agreement. Agreement and more uniform assessment of manuscripts by reviewers may be increased by specific education.