Abstract
Quantitative research evaluation requires measures that are transparent, relatively simple, and free of disciplinary and temporal bias. We document and provide solution to a hitherto unaddressed temporal bias – citation inflation – which arises from the basic fact that scientific publication is steadily growing at roughly 4% per year. Moreover, because the total production of citations grows by a factor of 2 every 12 years, this means that the real value of a citation depends on when it was produced. Consequently, failing to convert nominal citation values into real citation values produces significant mis-measurement of scientific impact. To address this problem, we develop a citation deflator method, outline the steps to generalize and implement it using the Web of Science portal, and analyze a large set of researchers from biology and physics to demonstrate how two common evaluation metrics – total citations and h-index – can differ by a remarkable amount depending on whether the underlying citation counts are deflated or not. In particular, our results show that the scientific impact of prior generations is likely to be significantly under- estimated when citations are not deflated, often by 100% or more of the nominal value. Thus, our study points to the need for a systemic overhaul of the counting methods used evaluating citation impact – especially in the case of researchers, journals, and institutions – which can span several decades and thus several doubling periods.
Highlights
IntroductionWhether for merit review, tenure and promotion of academics or for the assessment of national research systems, the evaluation of scientific productivity and impact increasingly relies on quantitative measures (Moed et al, 1985; Luukkonen, 1991; Moed, 2006; Vinkler, 2010; Hicks et al, 2015; Wildson, 2015; Wilsdon et al, 2015)
Here we address a more fundamental and under-appreciated bias in bibliometric evaluation – ‘citation inflation’ – a systematic measurement problem that arises from the persistent secular growth of the scientific system (Lariviere et al, 2008; Althouse et al, 2009)
Our results show that measurement errors upwards of 100% of the traditional nominal citation value can arise when citations are not deflated properly, which is especially exacerbated when comparing researchers from different age cohorts, as is customary in “all-time” lists (ACUMEN, 2018)
Summary
Whether for merit review, tenure and promotion of academics or for the assessment of national research systems, the evaluation of scientific productivity and impact increasingly relies on quantitative measures (Moed et al, 1985; Luukkonen, 1991; Moed, 2006; Vinkler, 2010; Hicks et al, 2015; Wildson, 2015; Wilsdon et al, 2015). Recent efforts to predict researchers’ future bibliometric impact (Acuna et al, 2012) have been shown to suffer from cohort and autocorrelation bias (Penner et al, 2013a,b), making the proposed predictive methods unreliable for quantitative faculty evaluation. Against this backdrop, here we address a more fundamental and under-appreciated bias in bibliometric evaluation – ‘citation inflation’ – a systematic measurement problem that arises from the persistent secular growth of the scientific system (Lariviere et al, 2008; Althouse et al, 2009). We conclude with a discussion of our results and policy recommendations
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
