Do Journal Administrators solve the reviewer assignment problem as well as editors? Consideration of reviewer rigour and timeliness

Abstract

The study aimed to find whether journal editorial office administrators were as effective as editors at assigning rigorous reviewers. We analysed four journals using blended systems of editor‐selected reviewers (ESR) and Journal Administrator‐selected reviewers (JASR) from four disciplines: medicine, sociology, education, and business/applied psychology. In blended journals, both editors and Journal Administrators select authors to review using web‐based expert‐finding tools. All reviewers selected were both authors and reviewers. We primarily wanted to assess the quality of reviews from both ESR and JASR reviewer selection methodologies. Reviewer rigour was defined as differences between editor decisions and reviewer recommendations. Timeliness data were also recorded and analysed separately as an indicator of efficiency. Reviewer rigour, the quality of reviewers’ evaluations, was estimated from the level of agreement between editors and reviewers. Timeliness was not considered a direct measure of rigour. For two journals, no statistically significant differences were observed; for two, in a small proportion of cases, ESR reviews were more negative. One journal showed some statistically significant major differences but only in 2% of reviews. Timeliness data indicated some statistically significant trends that JASR return reviews more promptly. Therefore, where editors rely on at least two reviewers’ recommendations, JASR is equally rigorous as ESR.