A reviewer-reputation ranking algorithm to identify high-quality papers during the review process

Abstract

With the exponential growth in the number of academic researchers, it is crucial for editors of scientific journals to identify the highest-quality papers. While several measures exist to evaluate a paper’s impact post-publication, the challenge of determining the potential impact of a manuscript during the review process remains an understudied issue. In this paper, we propose a reviewer-reputation ranking algorithm to identify high-quality papers based on paper citations, where a reviewer’s reputation is computed from the correlation between their past ratings and the current number of citations received by the papers they have evaluated. During the review process, reviewers with high reputation scores are given more weight to determine the quality of papers. We test the algorithm on an artificial network with 200 reviewers and 600 papers, as well as on the American Physical Society (APS) data set, including in the analysis 308,243 papers and 274,154 mutual citations. We compare our approach with two existing methods, demonstrating that our algorithm significantly outperforms the others in identifying manuscripts with the highest quality. Our findings can help improve the impact of scientific journals, thereby contributing to academic and scientific progress.