Impact Factors versus Actual Citations

Abstract

Accessible online at: www.karger.com/journals/pps In the past decade, there has been increasing interest in the use of impact factors of journals. The number of citations that are obtained in a specific year by articles published in a journal in the previous 2 years divided by the number of these articles has become the most popular tool to gauge the scientific value of the journal. It is certainly helpful to monitor the progress of a journal. For instance, Psychotherapy and Psychosomatics’ impact factor has grown steadily in the past several years (1.05 in 1995; 1.58 in 1996; 1.80 in 1997; 2.10 in 1998; 2.26 in 1999). Journal impact factors, however, depend on the research field; very high impact factors are likely in journals covering large areas of basic research with a rapidly expanding and short-lived literature that uses many references per article [1]. Further, substantial criticism has been raised [2] as to the fact that the Institute for Scientific Information (ISI) counts all types of documents (including letters and editorials) in the numerator of the impact factor, whereas it includes only regular articles and reviews in the denominator. This creates a substantial advantage to journals such as Nature, Lancet and New England Journal of Medicine, which publish many letters and editorials. Even though the impact factor was originally meant for comparison of journals, it is also used for the assessment of the quality of individual papers, scientists and departments [3]. For instance, it has become increasingly common in Europe to assess the research performance of a scientist by summing the impact factors of the journals where he or she has published. This calculation is made on the assumption that journals’ impact factors provide an approximate but valid indicator of the number of citations that individual articles have attracted. Is there any basis for this assumption? A citation analysis, using a database of the ISI, was performed on all scientific papers with at least one Italian author which were published between 1981 and 1998 in psychiatric and psychological journals in the Science Citation Index and Social Sciences Citation Index [4]. This analysis generated a ranking of the 20 most cited Italian investigators, based on actual numbers of citations in 1981–1998. For each investigator of this ranking, we have made the sum of the impact factors of the journals where the articles that were used in this analysis appeared. We used both the Science Citation Index and the Social Sciences Citation Index Journal Citation Reports. The ranking which was obtained was rather different from the one based on actual citations. For instance, the most cited Italian investigator was only 14th according to the sum of impact factors. The product moment correlation coefficient between number of citations and sum of impact factors of the 20 investigators only approached statistical significance (r = 0.44; p = 0.0505). This indicates that, even though a certain degree of correlation between the two methods exists, journals’ impact factors cannot substitute actual citation analysis. The results are not surprising in view of the fact that the impact factors are based on a short-term span of citations