What is the most appropriate citation metric for a clinical journal?

Abstract

The decision of which journals researchers submit their manuscripts to is based on a range of factors, including the quality of reviews, acceptance probability, turnaround time, target audience and fit, but an increasing one is the journal's reputation or perceived ‘quality’ as most commonly currently quantified by its impact factor.1 In an earlier editorial, it was argued that the impact factor, although not without flaws, provides a reasonable ball park assessment of the journal's impact or usefulness.2 It was, however, suggested that a small array of metrics would be more appropriate and could include a second citation metric that included citations to textbooks (are papers from the journal being used to provide evidence for undergraduate and postgraduate texts?), a download metric (how many people are reading the articles?) and a journal's acceptance rate.2 Because of its recognised flaws, new citation metrics are being introduced to rival the impact factor3-7 and these new metrics have been highlighted by their use in recent international Government research assessments.7 For example, Elsevier's Scopus database, which provides the citation metrics SCImago Journal Ranking Indicator and CiteScore, has been used in a number of national assessment exercises, including the UK Research Excellence Framework (2014), The Excellence in Research for Australia (ERA) assessments (2010, 2012, 2015), the Italian Abilitazione Scientifica Nazionale's researcher assessment (2012, 2013) and the Portuguese Fundação para a Ciência e aTecnologia (FCT) assessment (2013/14).7 In light of these developments, it is timely to describe the various citation metrics and consider what is the most appropriate metric(s) for a clinical journal such as Ophthalmic & Physiological Optics (OPO). The most well-known, currently used citation metric is the Thomson-Reuters (now Clarivate Analytics) Institute of Scientific Information (ISI)'s 2-year journal impact factor.2, 8-10 The 2016 impact factor was calculated by the number of citations to the journal's articles published in 2014 and 2015, divided by the number of ‘papers’ published in the journal during those years. Only citations from journals that are included in Clarivate's Journal Citation Reports (JCR) on the online subscription-based service Web of Science (previously Web of Knowledge) count. This severely limits what is counted as a citation, although it could be argued that this keeps the quality of citations high. The denominator includes research papers (original articles, review papers, etc.) and does not include editorials, letters and news items, but JCR citations to all articles count (including those to editorials and letters). Case reports appear to be judged on a journal-by-journal basis by Web of Science, sometimes they are included in the denominator (and as they are rarely cited, this would lower impact factors) and sometimes not. This lack of transparency is also a flaw in the use of the impact factor and it is difficult to predict impact factors as it is not easy to forecast which documents will be included in calculations and which will not. OPO has an impact factor of 2.30 with 290 citations in 2014 and 2015 to 126 research papers (290/126 = 2.30). Web of Science also provides several other citation metrics, including an immediacy index, a 5-year impact factor and an Eigenfactor score (see Table 1).3 The 2016 immediacy index was determined by the number of citations in 2016 from papers published in 2016 (for OPO, 57/60 = 0.95) and is heavily dependent on papers published earlier in the year (e.g. the January OPO issue) as the papers published in the later issues (e.g. the November OPO issue) are highly unlikely to be cited in that year, given the lag in time from submission to publication of any papers that could cite a November publication. The immediacy index is sometimes considered an indicator for the following year's impact factor. This also highlights the fact that the 2-year window for impact factors is less (1.5 years?) for papers published in the November and December issues due to this time lag between submission and publication of citing papers. This is another reason why the 2-year impact factor is flawed. The 2016 5-year impact factor was determined by the number of citations in 2016 from papers published in 2011, 2012, 2013, 2014 and 2015 (for OPO, 777/323 = 2.41). It is similar to the 2-year impact factor in that includes citations to editorials and letters in the numerator, but does not count them in the denominator. Web of Science also includes an indicator of the longevity of citations from journals, the journal half-life. This indicates the median age of papers cited in the given year, so that OPO's 2016 cited half-life of 9.9 years, indicates that half of the citations to OPO in 2016 were from papers 9.9 years old or younger. This citation metric is rarely discussed or used, but is a useful indicator of the longevity of papers in a journal. For example, our website includes a list of OPO's classic papers (http://onlinelibrary.wiley.com/journal/10.1111/(ISSN)1475-1313/homepage/classic_papers.htm) that remain relevant and still cited despite being up to 25 years old. The Eigenfactor is produced for Web of Science by Jevin West and Carl Bergstrom at the University of Washington, www.eigenfactor.com).3 It provides 5-year citation data and weights the prestige of citations using an algorithm similar to Google's PageRank, so that citations from high ranking journals such as Nature and Science count more than lower ranking journals. It does not include journal self-citations (e.g. Journal of Cataract & Refractive Surgery citations in Journal of Cataract & Refractive Surgery papers; author self-citations in different journals are counted), which works against highly specialised journals. However, its algorithm attempts to adjust for citation differences between different research fields.3 Unlike impact factors, eigenfactors are based on a total score of citations and are not divided by the number of papers published. This means that if two journals have the same impact factor, but one publishes 1000 papers per year and the other 500 papers, the former will have twice the eigenfactor. This may be partly because the eigenfactor is provided as a reference for libraries, in that journals can be ranked using a ‘Cost-Effectiveness Search’ based on the citation value per dollar that they provide. From the perspective of libraries, the 1000 paper volume journal provides twice as many articles for their readers compared to a 500 paper journal of equal impact factor. SCImago and CiteScore are the principal citation metrics developed within Elsevier's online subscription-based service Scopus to rival impact factors provided by Clarivate Analytics’ Web of Science.6 SCImago is the most well-established of the Scopus metrics. It is based on 3-year citation data, so that the 2016 SCImago Journal Rank Indicator (JRI) is based on citations in 2016 from papers published in 2013, 2014 and 2015.11 Like Eigenfactors, it weights the prestige of citations using an algorithm similar to Google's PageRank. It also differs from impact factors in that editorials and letters are included in the denominator. This means that journals which include a large number of editorials and letters, such as the Nature and Lancet series of journals, tend to have significantly lower SCImago JRIs (and CiteScores) than would be expected from their impact factors.5 CiteScore is a more recently introduced metric (December 2016) and is similar to SCImago, but differs in that it doesn't weight citations (Table 1). Both SCImago and CiteScore consider ‘documents’ beyond just journal articles, such as books and conference proceedings. Indeed, the Scopus database includes far more (they claim ‘double’) journals than Web of Science. For example, Web of Science includes four optometry journals: Ophthalmic & Physiological Optics, Optometry & Vision Science, Contract Lens & Anterior Eye and Clinical & Experimental Optometry. Scopus includes these four journals, plus the Journal of Optometry and Clinical & Refractive Optometry and Scopus includes a subcategory of optometry journals11 unlike Web of Science where Optometry journals are considered within the Ophthalmology subcategory or Google Scholar which has a joint Ophthalmology and Optometry subcategory.12 Google Scholar provides two citation metrics, the 5-year h-index and 5-year median h-index.12 These h-indices are Journal indices (h-indices can also be calculated for individual researchers and will be the topic for a future editorial). A h-index of N is calculated if N papers published in a journal have been cited N or more times. The median h-index is the median number of times that those N articles have been cited. In these ways the h-index avoids being distorted by one or a small number of very highly cited articles, which is a problem for the other metrics. Another strength is that it provides citations from all sources available on the web and can include citations in textbooks and important websites for example. This is also a potential downfall in that some of those sources are non-scholarly and some citations can be double (or more) counted if they appear in different places on the web. In addition, the h-index is typically driven by the papers in the two earliest years as these have the largest citations, so it provides potentially out-of-date information. For example, Optometry & Vision Science's 2017 h-index of 36 is based on 34 papers from 2012 and 2013, two from 2014 and none from 2015 or 2016. The 2017 h-index is therefore indicating the citation levels of the 2012 and 2013 volumes. It also provides a total score of citations and does not divide by the number of papers a journal publishes in the 5-year period, so that it is highly dependent on the number of papers. For example, although Ophthalmic & Physiological Optics (OPO) has much higher average citation metrics (Impact factor, SCImago and CiteScore) than Optometry & Vision Science, OPO's Google Scholar h-index (25) is lower than Optometry & Vision Science (36) as it published about three times fewer papers in 2012–2016 (391 vs 1207). Of the six main citation metrics, three are 5-year, two 3-year and one 2-year. Eigenfactors chose 5-years as ‘in many research areas, articles are not frequently cited until several years after publication. Therefore, measures that only look at citations in the first 2 years after publication can be misleading’.3 Scopus’ metrics use 3-years as ‘The time window that is chosen for a metric calculated for all serial titles in a multi-disciplinary database like Scopus should be a compromise that is the best fit across all subject areas……3 years is long enough to capture the citation peak of the majority of disciplines. A shorter time window, such as 2 years, is unfair to slower moving fields such as Physics and Astronomy. A longer time window, such as 5 years, is unfair to more rapidly moving fields such as Biochemistry and Molecular Biology.’6 The champions for a 2-year citation metric are, not surprisingly, those that this metric favours such as researchers in the biochemistry and molecular biology fields. Indeed, it has been suggested that the importance of the 2-year impact factor in rating research quality has led to the loss of clinical staff in UK academic departments of medicine in favour of basic scientists who are likely to provide higher impact factor papers (and thereby perhaps gain greater funding for their University) for the UK Research Assessment Exercise.10 Basic research is cited by both basic research and clinical research papers, while clinical research is cited by other clinical research papers, textbooks, professional guidelines etc. and much of the latter does not get included in impact factor calculations. Certainly, researchers in the biochemistry and molecular biology fields can readily publish in high impact factor journals and this can affect inter-discipline comparisons within Universities and with granting agencies as well as Government research assessments. Journals receiving numerous citations soon after publication might suggest that they have immediate and high impact and therefore must be high quality. However, papers in fields that cite a high volume of recent work tend to do so because older work rapidly becomes obsolete because of rapid advances in that field.13 Taking the example in Table 2, a hypothetical journal in which papers were cited highly in the first few years but then tails off, could have an impact factor of 10 but minimal impact after about 5 years. A journal in a discipline that is more of a slow burner would have citations in the other direction. Its impact factor could be 2.5, but its greatest impact is after 5 years and remains highly cited in subsequent years given the broader and sustained relevance of the research. The former journal does not appear to be providing papers that are four-times more impactful or higher quality and it could be argued that its overall impact is considerably less in the longer term. The latter example is not fanciful. For example, OPO's top cited paper, with 324 citations at an average of 25 citations per year, received 2 and 6 citations in its first 2 years after publication! The top cited papers in OPO tend to peak in terms of citations in years 3 and 4. For example, the top 10 cited OPO papers from the last 8 years that have a full 5-year history had mean citations rates of 12 and 13 in years 1 and 2, yet 15 and 17 in years 3 and 4, then drop to 13 in year 5. The lower cited papers follow this pattern even more strongly. Given that our impact factor is 2.3 currently, this highlights that impact factors tend to be greatly influenced by a relatively small number of highly cited papers rather than the breadth of papers published.2, 9 A 5-year window appears to be better, but has other problems. As discussed previously the Google Scholar h-index tends to be out-of-date and often reflects the two oldest volumes included, so that a 2017 h-index is essentially based on citations from 2012 and 2013. The issue is also present for other 5-year metrics such as the Eigenfactor and 5-year impact factor, although to a lesser extent. The 5-year impact factor score also takes many years to change in that a new editorial team would have to wait about 7 years before they could determine whether changes to the journal had influenced the 5-year citation metrics. Three or 4 year metrics may therefore be a good compromise. The impact factor's inclusion of citations from editorial and letters without including them in the denominator during its calculation seems inappropriate. I am not aware of any rationale for doing this and no other citation metric does so. The variable inclusion of case reports and other items means that there is a lack of transparency in the calculation of the impact factor and it is difficult to predict despite access to Web of Science. All journal articles other than news items should be included in both the numerator and denominator of citation metric calculations. Although the inclusion of journal self-citation is open to citation ‘game playing’ by journals, their removal as in Eigenfactors, seems harsh on specialised journals (such as Cataract & Refractive Surgery and the Journal of Refractive Surgery) which are bound to have a high level of journal self-citations. The weighting of citations in Eigenfactors and SCImago seems reasonable, but again leads to questions about transparency: what is the algorithm? Who decides what it is? I assume this is why Scopus have recently introduced the simpler CiteScore. Finally, should the main citation metric be an average or total score of citations? Is a journal of higher quality because it produces a good number of highly cited papers (but many more poorly cited ones) because it publishes a large number of papers? Or is it a journal whose average quality is high? I clearly prefer the latter, but this is likely due to the fact that OPO is a relatively small journal. The use of the total number of citations without division by the number of papers published means that Google Scholar and Eigenfactor citation metrics are heavily influenced by how many papers the journal publishes: they do not measure quantity or quality but a variable mixture of both. An earlier editorial argued that several citation metrics should be used, the standard 2-year impact factor plus another.1 That argument was predicated on the perception that the impact factor is so well established it is irreplaceable. It is not. Not too many years ago, the suggestion of Snellen visual acuity being replaced would have been laughed out of clinics. However, the Snellen chart is flawed and visual acuity using logMAR-based charts is now the standard in research studies, clinical trials and in computer-based systems and is quickly becoming the standard in everyday clinics.14 Similarly, the impact factor (i.e. the standard 2-year impact factor) is flawed, mainly because the 2-year window is inappropriate for so many journals and disciplines,2, 3, 5, 9 but also because of its illogical calculation, restriction to the relatively small range of journals included in Web of Science and lack of transparency. A range of metrics may well be ideal,2, 4, 6 but the 2-year impact factor being used as the standard surely needs to be challenged. To answer the question in the title, the best citation metric for a clinical journal (that provides an average citation score) currently available appears to be Scopus’ SCImago Journal Ranking Indicator or CIteScore. This may be best accompanied by a range of other metrics2, 4 that could include the Web of Science's 5-year impact factor and cited half-life, download metrics and acceptance rate. Thank you to Prof. Joanne Wood for helpful comments on an earlier version of this editorial. As the Editor-in-Chief of OPO, I am likely to favour the citation metrics that best portrays OPO; similarly as a researcher I am likely to favour citation metrics that favour my own clinical vision science research and highlight inadequacies in metrics that do not (such as the 2-year impact factor). OPO is listed in Web of Science, Scopus and Google Scholar.