A content analysis of referees’ comments: how do comments on manuscripts rejected by a high-impact journal and later published in either a low- or high-impact journal differ?

Abstract

Using the data of a comprehensive evaluation study on the peer review process of Angewandte Chemie International Edition (AC-IE), we examined in this study the way in which referees’ comments differ on manuscripts rejected at AC-IE and later published in either a low-impact journal (Tetrahedron Letters, n = 54) or a high-impact journal (Journal of the American Chemical Society, n = 42). For this purpose, a content analysis was performed of comments which led to the rejection of the manuscripts at AC-IE. For the content analysis, a classification scheme with thematic areas developed by Bornmann et al. (2008) was used. As the results of the analysis demonstrate, a large number of negative comments from referees in the areas “Relevance of contribution” and “Design/Conception” are clear signs that a manuscript rejected at AC-IE will not be published later in a high-impact journal. The number of negative statements in the areas “Writing/Presentation,” “Discussion of results,” “Method/Statistics,” and “Reference to the literature and documentation,” on the other hand, had no statistically significant influence on the probability that a rejected manuscript would later be published in a low- or high-impact journal. The results of this study have various implications for authors, journal editors and referees.