Defining, Measuring, and Rewarding Scholarly Impact: It’s the Level of Analysis, Stupid

Abstract

We demonstrate that the pervasive practice of using journals’ impact factor (JIF) as a proxy to assess researcher and article scholarly impact is inappropriate because it results in grossly incorrect conclusions about the impact of both individual researchers and individual articles. For example, using JIF underestimates the impact of some researchers by 123%, while overestimating it for others by 64%. Similarly, it underestimates the impact of some articles by 1,800%, while overestimating the impact of others by 1,100%. Using the multilevel paradigm, we explain that this practice is invalid because of confusion about the definition and measurement of impact at different levels of analysis. Specifically, JIF is a journal-level measure of impact, computed by aggregating citations of individual articles (i.e., upward effect), and is therefore not appropriate to use when measuring impact at lower levels of analysis such as that of individual researchers or individual articles published in a particular journal (i.e., downward effect). Accordingly, we advocate an immediate moratorium on the use of JIF and other journal-level (i.e., higher level of analysis) measures when assessing the impact of individual researchers and individual articles (i.e., lower level of analysis). Given the importance and interest in assessing scholarly impact of researchers and articles, we delineate measures that are level-appropriate and readily available. We discuss implications for theory and future research, the careers of researchers and educators and the administration and future of business schools, and conclude with actionable recommendations for internal and external stakeholders regarding the assessment of scholarly impact.