Exploring the effects of journal article features: Implications for automated prediction of scholarly impact

Abstract

Listen

Translate article

ABSTRACT Purpose Scholars face an unprecedented ever increasing demand for acting as reviewers for journals, recruitment and promotion committees, granting agencies, and research assessment agencies. Consequently, journal editors face an ever increasing scarcity of experts willing to act as reviewers. It is not infrequent that reviews diverge, which forces editors to recur to additional reviewers or make a final decision on their own. The purpose of the proposed bibliometric system is to support of editors’ accept/reject decisions in such situations. Design/methodology/approach We analyse nearly two million 2017 publications and their scholarly impact, measured by normalized citations. Based on theory and previous literature, we extrapolated the publication traits of text, byline, and bibliographic references expected to be associated with future citations. We then fitted a regression model with the outcome variable as the scholarly impact of the publication and the independent variables as the above non-scientific traits, controlling for fixed effects at the journal level. Findings Non-scientific factors explained more than 26% of the paper’s impact, with slight variation across disciplines. On average, OA articles have a 7% greater impact than non-OA articles. A 1% increase in the number of references was associated with an average increase of 0.27% in impact. Higher-impact articles in the reference list, the number of authors and of countries in the byline, the article length, and the average impact of co-authors’ past publications all show a positive association with the article’s impact. Female authors, authors from English-speaking countries, and the average age of the article’s references show instead a negative association. Research limitations The selected non-scientific factors are the only observable and measurable ones to us, but we cannot rule out the presence of significant omitted variables. Using citations as a measure of impact has well-known limitations and overlooks other forms of scholarly influence. Additionally, the large dataset constrained us to one year’s global publications, preventing us from capturing and accounting for time effects. Practical implications This study provides journal editors with a quantitative model that complements peer reviews, particularly when reviewer evaluations diverge. By incorporating non-scientific factors that significantly predict a paper’s future impact, editors can make more informed decisions, reduce reliance on additional reviewers, and improve the efficiency and fairness of the manuscript selection process. Originality/value To the best of our knowledge, this study is the first one to specifically address the problem of supporting editors in any field in their decisions on submitted manuscripts with a quantitative model. Previous works have generally investigated the relationship between a few of the above publication traits and their impact or the agreement between peer-review and bibliometric evaluations of publications.