Reviewing the review: a qualitative assessment of the peer review process in surgical journals

Catherine H Davis,Jeffrey E Lee,O James Garden,Thomas A Aloia,Charles M Balch,Kevin E Behrns,Keith D Lillemoe,Barbara L Bass,Mark S Roh

doi:10.1186/s41073-018-0048-0

Abstract

BackgroundDespite rapid growth of the scientific literature, no consensus guidelines have emerged to define the optimal criteria for editors to grade submitted manuscripts. The purpose of this project was to assess the peer reviewer metrics currently used in the surgical literature to evaluate original manuscript submissions.MethodsManuscript grading forms for 14 of the highest circulation general surgery-related journals were evaluated for content, including the type and number of quantitative and qualitative questions asked of peer reviewers. Reviewer grading forms for the seven surgical journals with the higher impact factors were compared to the seven surgical journals with lower impact factors using Fisher’s exact tests.ResultsImpact factors of the studied journals ranged from 1.73 to 8.57, with a median impact factor of 4.26 in the higher group and 2.81 in the lower group. The content of the grading forms was found to vary considerably. Relatively few journals asked reviewers to grade specific components of a manuscript. Higher impact factor journal manuscript grading forms more frequently addressed statistical analysis, ethical considerations, and conflict of interest. In contrast, lower impact factor journals more commonly requested reviewers to make qualitative assessments of novelty/originality, scientific validity, and scientific importance.ConclusionSubstantial variation exists in the grading criteria used to evaluate original manuscripts submitted to the surgical literature for peer review, with differential emphasis placed on certain criteria correlated to journal impact factors.

Highlights

IntroductionDespite rapid growth of the scientific literature, no consensus guidelines have emerged to define the optimal criteria for editors to grade submitted manuscripts
The main findings of the study are that there is a significant variability in the specific criteria that surgical journals use to evaluate submitted original manuscripts and that the use of certain criteria is associated with impact factor levels
Multiple of the specific points from the guidelines published by the Council of Biology Editors (CBE) [11] and Frank [12] were identified in the reviewer grading forms, but there was considerable inconsistency between different journals, and almost none addressed the majority of the suggested guidelines

Summary

Introduction

Despite rapid growth of the scientific literature, no consensus guidelines have emerged to define the optimal criteria for editors to grade submitted manuscripts. The purpose of this project was to assess the peer reviewer metrics currently used in the surgical literature to evaluate original manuscript submissions. While an optimal peer review process aims to maintain a high level of research integrity and a literature that supports the practice of evidence-based medicine, these systems are imperfect [1, 3]. Not all peer review is currently performed to a high standard, and there is concern that manuscripts published after inadequate peer review will negatively affect future literature reviews, meta-analyses, and most importantly, medical practice [1, 4]. Where there are additional hurdles to collection of high quality evidence in the form of randomized

Objectives

Methods

Results

Conclusion