Comment from the editor

Abstract

An argument for identifying highly cited papers on biographical sketchesBiographical sketches are widely used by scientific reviewers and administrators when judging the performance of investigators. These sketches include information about an investigator’s educational background, past and current positions, honors, research funding, and selected publications. This commentary proposes that these biographical sketches could be made more useful by routinely providing citation counts for each of the investigator’s most important research papers.This proposal is based on the fact that the impact of individual papers can now be conveniently measured using an on-line database. This database compiles information extracted from the bibliography of each published paper to identify prior papers as they are subsequently cited. This information can be useful because influential papers tend to be cited frequently by others in later papers.In our field, as in all of biomedical research, groundbreaking discoveries are uncommon. When papers report breakthrough discoveries, they change the research agenda for the field and they sometimes change clinical practice. These groundbreaking papers have impact, and this impact correlates with how often the papers are cited. Thus, the importance of an influential paper can be quantitated by counting how many times the paper is cited.Citation counts for individual papers can now be performed very conveniently on-line. To illustrate this, here is a step-by-step protocol: 1.Contact your medical library for access to the database (on-line at http://isi4.isiknowledge.com).2.Select “full search” and “general search.”3.In the author category, enter the last name and initial(s), and select “search.”4.Select “mark all,” “submit marks,” and then “marked list.”5.Check off “times cited” (in step 1); sort by “times cited” (drop box below step 2); select “bibliographic” (first drop box in step 2); click on “format for print.”6.Refine search as needed to exclude papers by others with similar names.7.This generates a list of the author’s most-cited papers with citation counts.By following these steps, quantitative information about cumulative impact can be obtained within a few minutes for nearly all papers published since 1978. In addition, this search strategy provides the data needed to compare investigators based on how many highly cited papers they have published.A few years ago, I began to routinely check citations when I reviewed biographical sketches. On many occasions, the citation analyses provided useful and otherwise unavailable information.One important feature of the citation data is that it is skewed. This applies to the number of citations per peer-reviewed paper, the number of citations per paper for papers in a specific journal, the number of citations per investigator, and the number of highly cited papers per investigator. The skewed nature of this data implies that we should pay special attention to outliers-namely the occasional papers (and investigators) that had very high impact.Seasoned scientific reviewers recognize that peer-reviewed papers vary widely in their importance. To compensate for this, they often assign extra weight to papers in competitive journals. However, a journal’s impact factor says little about the impact of a single paper in the journal because the distribution of citation counts for the individual papers in a journal is skewed. How skewed is this distribution? For papers published in Gastroenterology, the median number of citations per paper is well below the mean value, and the mean is less than the standard deviation.The skewed distribution of citations per paper for papers in a single journal implies that it is usually misleading to use a journal’s impact factor to estimate the importance of individual papers in the journal. For most papers, the actual citation count is below what would be expected based on the impact factor (the median is less than the mean). However, journals like Gastroenterology publish a handful of highly cited papers each year. In the absence of paper-specific citation data, the importance of these papers will be underestimated by wide margins (5-fold or more). Reviewers need to see paper-specific citation data to judge the degree to which these particular papers are outliers.Citation data also may be useful for asking broader questions. My casual reviews of citation data suggest that surprisingly few investigators seem to account for most of the high-impact papers in our field. This seems important enough to justify a systematic study.We could begin by compiling demographic data for U.S. gastroenterologists: How many investigators have produced large numbers of high-impact papers (e.g., a half-dozen papers with 100+ citations, excluding work done as a trainee)? Who are these individuals and what qualities do they share? Can we improve our training programs by studying these investigators? Can we correlate cumulative impact versus N.I.H. funding for all investigators to identify outliers who may warrant special consideration? More generally, how can past performance help us predict who is likely to produce high-impact discoveries in the future?Citation counts capture information about cumulative impact that is not now included in biographical sketches. Although much remains to be learned about the uses and limitations of this data, information about impact clearly can be pertinent to scientific review. An argument for identifying highly cited papers on biographical sketchesBiographical sketches are widely used by scientific reviewers and administrators when judging the performance of investigators. These sketches include information about an investigator’s educational background, past and current positions, honors, research funding, and selected publications. This commentary proposes that these biographical sketches could be made more useful by routinely providing citation counts for each of the investigator’s most important research papers.This proposal is based on the fact that the impact of individual papers can now be conveniently measured using an on-line database. This database compiles information extracted from the bibliography of each published paper to identify prior papers as they are subsequently cited. This information can be useful because influential papers tend to be cited frequently by others in later papers.In our field, as in all of biomedical research, groundbreaking discoveries are uncommon. When papers report breakthrough discoveries, they change the research agenda for the field and they sometimes change clinical practice. These groundbreaking papers have impact, and this impact correlates with how often the papers are cited. Thus, the importance of an influential paper can be quantitated by counting how many times the paper is cited.Citation counts for individual papers can now be performed very conveniently on-line. To illustrate this, here is a step-by-step protocol: 1.Contact your medical library for access to the database (on-line at http://isi4.isiknowledge.com).2.Select “full search” and “general search.”3.In the author category, enter the last name and initial(s), and select “search.”4.Select “mark all,” “submit marks,” and then “marked list.”5.Check off “times cited” (in step 1); sort by “times cited” (drop box below step 2); select “bibliographic” (first drop box in step 2); click on “format for print.”6.Refine search as needed to exclude papers by others with similar names.7.This generates a list of the author’s most-cited papers with citation counts.By following these steps, quantitative information about cumulative impact can be obtained within a few minutes for nearly all papers published since 1978. In addition, this search strategy provides the data needed to compare investigators based on how many highly cited papers they have published.A few years ago, I began to routinely check citations when I reviewed biographical sketches. On many occasions, the citation analyses provided useful and otherwise unavailable information.One important feature of the citation data is that it is skewed. This applies to the number of citations per peer-reviewed paper, the number of citations per paper for papers in a specific journal, the number of citations per investigator, and the number of highly cited papers per investigator. The skewed nature of this data implies that we should pay special attention to outliers-namely the occasional papers (and investigators) that had very high impact.Seasoned scientific reviewers recognize that peer-reviewed papers vary widely in their importance. To compensate for this, they often assign extra weight to papers in competitive journals. However, a journal’s impact factor says little about the impact of a single paper in the journal because the distribution of citation counts for the individual papers in a journal is skewed. How skewed is this distribution? For papers published in Gastroenterology, the median number of citations per paper is well below the mean value, and the mean is less than the standard deviation.The skewed distribution of citations per paper for papers in a single journal implies that it is usually misleading to use a journal’s impact factor to estimate the importance of individual papers in the journal. For most papers, the actual citation count is below what would be expected based on the impact factor (the median is less than the mean). However, journals like Gastroenterology publish a handful of highly cited papers each year. In the absence of paper-specific citation data, the importance of these papers will be underestimated by wide margins (5-fold or more). Reviewers need to see paper-specific citation data to judge the degree to which these particular papers are outliers.Citation data also may be useful for asking broader questions. My casual reviews of citation data suggest that surprisingly few investigators seem to account for most of the high-impact papers in our field. This seems important enough to justify a systematic study.We could begin by compiling demographic data for U.S. gastroenterologists: How many investigators have produced large numbers of high-impact papers (e.g., a half-dozen papers with 100+ citations, excluding work done as a trainee)? Who are these individuals and what qualities do they share? Can we improve our training programs by studying these investigators? Can we correlate cumulative impact versus N.I.H. funding for all investigators to identify outliers who may warrant special consideration? More generally, how can past performance help us predict who is likely to produce high-impact discoveries in the future?Citation counts capture information about cumulative impact that is not now included in biographical sketches. Although much remains to be learned about the uses and limitations of this data, information about impact clearly can be pertinent to scientific review. Biographical sketches are widely used by scientific reviewers and administrators when judging the performance of investigators. These sketches include information about an investigator’s educational background, past and current positions, honors, research funding, and selected publications. This commentary proposes that these biographical sketches could be made more useful by routinely providing citation counts for each of the investigator’s most important research papers. This proposal is based on the fact that the impact of individual papers can now be conveniently measured using an on-line database. This database compiles information extracted from the bibliography of each published paper to identify prior papers as they are subsequently cited. This information can be useful because influential papers tend to be cited frequently by others in later papers. In our field, as in all of biomedical research, groundbreaking discoveries are uncommon. When papers report breakthrough discoveries, they change the research agenda for the field and they sometimes change clinical practice. These groundbreaking papers have impact, and this impact correlates with how often the papers are cited. Thus, the importance of an influential paper can be quantitated by counting how many times the paper is cited. Citation counts for individual papers can now be performed very conveniently on-line. To illustrate this, here is a step-by-step protocol: 1.Contact your medical library for access to the database (on-line at http://isi4.isiknowledge.com).2.Select “full search” and “general search.”3.In the author category, enter the last name and initial(s), and select “search.”4.Select “mark all,” “submit marks,” and then “marked list.”5.Check off “times cited” (in step 1); sort by “times cited” (drop box below step 2); select “bibliographic” (first drop box in step 2); click on “format for print.”6.Refine search as needed to exclude papers by others with similar names.7.This generates a list of the author’s most-cited papers with citation counts. By following these steps, quantitative information about cumulative impact can be obtained within a few minutes for nearly all papers published since 1978. In addition, this search strategy provides the data needed to compare investigators based on how many highly cited papers they have published. A few years ago, I began to routinely check citations when I reviewed biographical sketches. On many occasions, the citation analyses provided useful and otherwise unavailable information. One important feature of the citation data is that it is skewed. This applies to the number of citations per peer-reviewed paper, the number of citations per paper for papers in a specific journal, the number of citations per investigator, and the number of highly cited papers per investigator. The skewed nature of this data implies that we should pay special attention to outliers-namely the occasional papers (and investigators) that had very high impact. Seasoned scientific reviewers recognize that peer-reviewed papers vary widely in their importance. To compensate for this, they often assign extra weight to papers in competitive journals. However, a journal’s impact factor says little about the impact of a single paper in the journal because the distribution of citation counts for the individual papers in a journal is skewed. How skewed is this distribution? For papers published in Gastroenterology, the median number of citations per paper is well below the mean value, and the mean is less than the standard deviation. The skewed distribution of citations per paper for papers in a single journal implies that it is usually misleading to use a journal’s impact factor to estimate the importance of individual papers in the journal. For most papers, the actual citation count is below what would be expected based on the impact factor (the median is less than the mean). However, journals like Gastroenterology publish a handful of highly cited papers each year. In the absence of paper-specific citation data, the importance of these papers will be underestimated by wide margins (5-fold or more). Reviewers need to see paper-specific citation data to judge the degree to which these particular papers are outliers. Citation data also may be useful for asking broader questions. My casual reviews of citation data suggest that surprisingly few investigators seem to account for most of the high-impact papers in our field. This seems important enough to justify a systematic study. We could begin by compiling demographic data for U.S. gastroenterologists: How many investigators have produced large numbers of high-impact papers (e.g., a half-dozen papers with 100+ citations, excluding work done as a trainee)? Who are these individuals and what qualities do they share? Can we improve our training programs by studying these investigators? Can we correlate cumulative impact versus N.I.H. funding for all investigators to identify outliers who may warrant special consideration? More generally, how can past performance help us predict who is likely to produce high-impact discoveries in the future? Citation counts capture information about cumulative impact that is not now included in biographical sketches. Although much remains to be learned about the uses and limitations of this data, information about impact clearly can be pertinent to scientific review.