Evaluation by citation: An imperfect system

Abstract

Bibliometric measures such as impact factor, citation rate, and h-index are gaining influence on the funding of research units and the hiring and promotion of individual scientists. Strengths and weaknesses of such a system are being widely discussed. However, appropriate evaluation of research and researchers requires a reliable and complete bibliographic database. Currently, most bibliometric evaluations are based on the source index of the ISI Web of Knowledge, which has a user-friendly basic search option that delivers publication information, citation rates, and h-indices within seconds. Almost anyone can retrieve information within minutes, so evaluation based on the source index seemingly requires no special knowledge.Weaknesses of the system become evident when one goes to the cited-reference search. That database contains many more bibliographic data, including citations to papers published in journals not included in the source index; to books and conference proceedings not in the source index; and, most important, to papers cited with numerous typographical errors or in an unofficial bibliographic style.We wanted to check whether the data inconsistencies between the two databases would be distributed randomly and thus not affect relative trends. We chose a prominent example: Danish physicist Jens Lindhard (1922–97), a Nobel Prize nominee well known for contributions to condensed-matter physics, ion–solid interactions, and other areas.The basic source-index search shows 3735 citations as of 13 January 2009 and an h-index of 14 for Lindhard—a nice achievement if he were a postdoc in physics. However, the cited reference search shows more than 10 000 citations and an h-index of roughly 25, an appropriate number for a professor. Lindhard published 49 items total, not all of which are expected to generate citations, so his h-index would never be much higher.The table shows four of Lindhard’s highly cited papers. The first paper provided the theoretical basis for the entire field of ion implantation, a key technique in microelectronics. The citation total of 4008 indicates top impact, and the paper has rightly been named a citation classic. Yet the paper does not even appear in the source index! The second paper shows that such omission is not general for articles in that journal. The third, the central paper in the physics of channeling, a phenomenon in the interaction of swift particles with crystals, is still highly cited every year, yet of more than 1800 citations, only 19 are mentioned in the source index.PaperBasic search (source index)Cited reference search (citation index)104008215841890319184544 (4)4 (1615)A particularly spectacular case is the fourth paper, another classic. Both the source and citation indexes show 4 citations when one searches using Lindhard’s name. Under Niels Bohr, the first author, the source index likewise shows 4 citations, but the citation index has 1615. We have no way of understanding the discrepancy or evaluating its significance. Citation counts of Nobel laureates were studied with special care in the beginnings of bibliometry, to support the postulate that high citation rate indicates high research quality.One could argue that Bohr and Lindhard are no longer alive and, therefore, no longer subject to research evaluation. We looked at similar data for one of us (Sigmund). The source index delivers an h-index of 46, yet 11 highly cited papers are not mentioned at all, including a paper with 1025 citations.Our observations confirm the repeated claim of Eugene Garfield, founder of the Science Citation Index, that citation analysis with the aim of evaluating researchers or research groups should only be performed with a complete list of publications at hand. Our examples show that despite significant development of the two data-bases, Garfield’s statement is still valid.We conclude that the source index, accessed through the basic search function, cannot be used uncritically in research evaluation. The citation index contains most of the pertinent information, but it is not user friendly. And a fundamental weakness is that it underestimates the citations to journals not included in the source index.A valid citation analysis is time consuming; it requires insight and, therefore, needs to be performed by experts.SAMPLE REFERENCESSection:ChooseTop of pageSAMPLE REFERENCES <<CITING ARTICLES1. J. Lindhard, M. Scharff, H. E. Schiøtt, Mat. Fys. Medd. Dan. Vid. Selsk. 33(14), 1 (1963). Google Scholar2. J. Lindhard, Mat. Fys. Medd. Dan. Vid. Selsk. 28(8), 1 (1954). Google Scholar3. J. Lindhard, Mat. Fys. Medd. Dan. Vid. Selsk. 34(14), 1 (1965). Google Scholar4. N. Bohr, J. Lindhard, Mat. Fys. Medd. Dan. Vid. Selsk. 28(7), 1 (1954). Google Scholar© 2009 American Institute of Physics.