Comparing the scientific impact of conference and journal publications in computer science

Abstract

The impact of scientific publications is often estimated by the number of citations they receive, i.e. how frequently they are referenced by other publications. Since publications have associated authors, originating institutions and publication venues (e.g. journals, conference proceedings) citations have also been used to compare their scientific impact. For instance, one commonly considered indicator of the quality of a journal is its impact factor [1]. The impact factors are published yearly by Thomson ISI in the Journal Citation Report (JCR) by counting the citations from articles of thousands of journals. However, research results in computer science are often published in high-quality conferences which are not covered by the JCR citation databases [2]. Other commercial citation data sources such as Elsevier Scopus also focus on journals and contain comparatively few conference publications. Hence these data sources cover only a fraction of quality scientific publications in computer science. Furthermore, they miss many citations even for journal articles since all references to them are not captured which originate from conference papers or other papers not included in the publication database. Several recent system developments capture citation numbers for both journal and conference publications especially in computer science, e.g. Citeseer, the ACM Digital Library, Microsoft Libra (Libra) and Google Scholar (GS). For example, Libra holds more than 900,000 computer science publications and more than 3.5 million citations to them as of December 2007. As shown in Table 1, the majority of papers appeared in conferences and workshops, not in journals. Furthermore, the total number of citations is higher for conferences and workshops than for journals. While there are many workshops and conferences with comparatively little scientific impact the top-cited conferences are highly significant and need to be considered for a meaningful citation analysis in computer science. For example, in the Libra dataset the average number of citations per paper is similar for the 100 most cited conference series than for the 100 most cited journals. These 200 venues account for 78% of all citations. In [4] we used cleaned citation data from GS for an in-depth citation analysis for database research, a subfield of computer science research. We analyzed all publications over a period of 10 years (1994–2003) which appeared in top database conferences and top database journals. It turned out that the two top conferences (ACM Sigmod, VLDB) not only publish many more papers than the top journals (ACM TODS, VLDB Journal) but that they receive many more citations in total and per paper. The original study used GS data from August 2005. We recently confirmed the findings with GS citation data from December 2007.