Misuse of Journal Impact Factors in Scientific Assessment

Abstract

The editors of Traffic support the view that journal impact factors should not be used as tools to assess the quality of research papers or their authors. In 1972, Garfield proposed the use of a metric to evaluate journal impact 1. Thomson Reuters now reports this metric, the Journal Impact Factor (JIF), which is calculated as the number of citations to papers published in a particular journal over a 2-year period divided by the total number of papers published by the journal in those 2 years. The intent is that this metric provides a standard through which librarians can compare journals to help in making decisions for purchasing subscriptions. Librarians embraced this metric for the intended purpose. However, as most of us are only too well aware and despite numerous warnings 2-5, many academics and others have also adopted the JIF not only as a measure of journal “quality” but also of the caliber of research and researchers. Many investigators use the JIF in deciding where to submit their work for publication, opting to submit to “high impact” journals over journals, which though more appropriate, have a lower JIF value. More insidiously, many academics, employers, funding agencies and others, across the world have come to use the JIF of journals in which individuals publish to make decisions about hiring, academic promotion, funding, etc. Rather than judging the value of a paper based on its significance and the quality of the work presented, or judging our peers based on the aggregate of their individual merits, we judge them based on the average number of citations received to papers published in journals in which they publish. Publishers and editors of the journals themselves have taken note of this trend, and anxiously await the release of the journal JIF every June; many journals “game” the JIF by opting to favor publishing article types that tend to accumulate citations or by classifying some articles that will be cited by others (and thus will be in the numerator of the JIF calculation) as non-source items (therefore not included in the denominator) 6, 7. There are some reports that as few as 20% of papers published in some journals are considered source items in calculating the JIF 8. At the 2012 meeting of the American Society for Cell Biology (ASCB) a group of concerned editors of bioscience journals (The San Francisco group), including the editors of Traffic, met to discuss the misuse of JIFs and to generate an action plan to stem this misuse. The concerns are outlined in detail in the San Francisco Declaration on Research Assessment (www.ascb.org/SFdeclaration.html), which has been signed by a number of preeminent scientists, scientific organizations, and journals. The Traffic editors and editorial team completely support the recommendations of the Declaration, which are summarized below. In addition to this editorial, the Declaration is being highlighted in editorials in a number of other journals, including Science, Molecular Biology of the Cell, EMBO Journal and the Journal of Cell Biology, to emphasise the concerns of the editors of these journals about the misuse of the JIF. This is being highlighted in editorials in several journals. There are three main threads in the recommendations set out in the Declaration. The first two implore the scientific community to stop using journal metrics as a means to evaluate science and scientists. While the San Francisco group initially focused their attention on the deficits of the Thomson Reuters 2-year JIF—which have been well described in the literature 9-14—the Declaration takes a broader approach and recommends against the use of journal metrics in general as a means to evaluate science and scientists. By its very nature, any journal wide metric is an inaccurate reflection of the number of citations of any one particular paper published in the journal. While a journal metric that accounted for publication type and field-specific publication trends (neither of which is accounted for by the Thomson Reuters JIF, but is addressed by several other available metrics) might provide a general relative measure of a journal's impact on a field, it should not be applied to all papers published in that journal. By example, the 2012 JIF for The Journal of Biological Chemistry is below 5 (compared to 32.4 for Cell), but several Nobel prizes have been awarded ostensibly for work published in The JBC. We, and many of our colleagues, openly discuss the deficits of many of the papers published in journals with high JIF values, but nevertheless continue to consider these journals—and their constituent papers—to be the most sought after in the field. We must stop insisting that our colleagues publish in these journals in order to be hired, promoted or funded. Rather, the quality of the work itself should be considered, and impact measured either by metrics that focus on the individual paper rather than the journal (such as h-factors for established scientists) or, preferably, reputation or colleague references. The third recommendation aims to exploit the advantages afforded by on-line publication to reverse citation trends that negatively impact primary papers. In accordance with an editorial that we published in January 15, this recommendation supports the relaxation of citation limits and encourages the citation of primary literature over reviews, where possible, thus giving credit for original findings rather than a third party reviewer. Traffic has never had citation limits and continues to promote primary literature citations over reviews both in the instructions to authors and to referees. The editors of Traffic strongly endorse all of the recommendations of the San Francisco Declaration. We believe that the JIF and similar journal metrics undervalue the true impact of field specific “workhorse” journals, such as Traffic, Molecular Biology of the Cell, The Journal of Biological Chemistry, The Journal of Cell Science, and many others, in which many primary research findings are reported. A solid paper in Molecular Biology of the Cell that impacts the field in an important manner should be more highly considered than a paper that meets the “wow” factor of a high JIF journal but has only a marginal overall impact on the field. Moreover, research reporting is frequently substantially delayed by attempts to publish work in journals with higher JIF values and research careers are often sidelined. We advocate that research findings should be published in the journals in which they are most likely to be reviewed by the most appropriate reviewers and seen by the most relevant readers. If you agree, please go to the web site www.ascb.org/SFdeclaration.html and add your name to those of us who support the San Francisco Declaration.