Traditional Citation Analysis Research Articles

Research on the evaluation of academic journals from a multidimensional fusion measurement perspective

PurposeThis paper aims to conduct a comprehensive analysis to evaluate the current situation of journals, examine the factors that influence their development, and establish an evaluation index system and model. The objective is to enhance the theory and methodologies used for journal evaluation and provide guidance for their positive development.Design/methodology/approachThis study uses empirical data from economics journals to analyse their evaluation dimensions, methods, index system and evaluation framework. This study then assigns weights to journal data using single and combined evaluations in three dimensions: influence, communication and novelty. It calculates several evaluation metrics, including the explanation rate, information entropy value, difference coefficient and novelty degree. Finally, this study applies the concept of fuzzy mathematics to measure the final results.FindingsThe use of affiliation degree and fuzzy Borda number can synthesize ranking and score differences among evaluation methods. It combines internal objective information and improves model accuracy. The novelty of journal topics positively correlates with both the journal impact factor and social media mentions. In addition, journal communication power indicators compensate for the shortcomings of traditional citation analysis. Finally, the three-dimensional representative evaluation index serves as a reminder to academic journals to avoid the vortex of the Matthew effect.Originality/valueThis paper proposes a journal evaluation model comprising academic influence, communication power and novelty dimensions. It uses fuzzy Borda evaluation to address issues related to the weighing of single evaluation methods. This study also analyses the relationship of the three dimensions and offers insights for journal development in the new media era.

Main path analysis considering citation structure and content: Case studies in different domains

Main path analysis (MPA) is an effective method widely accepted in science and technology for extracting knowledge diffusion paths. Traditional citation analysis assumes that all citations are treated equally. In contrast, this paper proposes a new MPA framework from the perspective of citation structure and content. Three indicators are considered to adjust edge weight: (1) Structural similarity, (2) Topic similarity and (3) Sentiment analysis. This study takes the bullwhip effect and the Internet of Things domain as examples to verify the reliability and feasibility of improved MPA. The results show that the improved main path uncovers the knowledge trajectories appropriately, which has an ability to distinguish citations and detect important papers. This research enriches MPA theory and provides future research directions from perspective of citation structure and content.

Assessing books’ academic impacts via integrated computation of multi-level citation information

PurposeCitations have been used as a common basis to measure the academic accomplishments of scientific books. However, traditional citation analysis ignored content mining and without consideration of citation equivalence, which may lead to the decline of evaluation reliability. Hence, this paper aims to integrate multi-level citation information to conduct multi-dimensional analysis.Design/methodology/approachIn this paper, books’ academic impacts were measured by integrating multi-level citation resources, including books’ citation frequencies and citation-related contents. Specifically, firstly, books’ citation frequencies were counted as the frequency-level metric. Secondly, content-level metrics were detected from multi-dimensional citation contents based on finer-grained mining, including topic extraction on the metadata and citation classification on the citation contexts. Finally, differential metric weighting methods were compared with integrate the multi-level metrics and computing books’ academic impacts.FindingsThe experimental results indicate that the integration of multiple citation resources is necessary, as it can significantly improve the comprehensiveness of the evaluation results. Meanwhile, compared with the type differences of books, disciplinary differences need more attention when evaluating the academic impacts of books.Originality/valueAcademic impact assessment of books via integrating multi-level citation information can provide more detailed evaluation information and cover shortcomings of methods based on single citation data. Moreover, the method proposed in this paper is publication independent, which can be used to measure other publications besides books.

Is Self-Citation Biased? An Investigation via the Lens of Citation Polarity, Density, and Location

Traditional citation analysis methods have been criticized because their theoretical base of statistical counts does not reflect the motive or judgment of citing authors. In particular, self-citations may give undue credits to a cited article or mislead scientific development. This research aims to answer the question of whether self-citation is biased by probing into the motives and context of citations. It takes an integrated and fine-grained view of self-citations by examining them via multiple lenses — polarity, density, and location of citations. In addition, it explores potential moderating effects of citation level and associations among location contexts of citations to the same references for the first time. We analyzed academic publications across different topics and disciplines using both qualitative and quantitative methods. The results provide evidence that self-citations are free of bias in terms of citation density and polarity uncertainty, but they can be biased with respect to positivity and negativity of citations. Furthermore, this study reveals impacts of self-citing behavior on some citation patterns involving citation density, location concentration, and associations. The examination of self-citing behavior from those new perspectives shed new lights on the nature and function of self-citing behavior.

Are scientific memes inherited differently from gendered authorship?

This paper seeks to build upon the previous literature on gender aspects in research collaboration and knowledge diffusion. Our approach adds the meme inheritance notion to traditional citation analysis, as we investigate if scientific memes are inherited differently from gendered authorship. Since authors of scientific papers inherit knowledge from their cited authors, once authorship is gendered we are able to characterize the inheritance process with respect to the frequencies of memes and their propagation scores depending on the gender of the authors. By applying methods that enable the gender disambiguation of authors, missing data on the gender of citing and cited authors is dealt with. Our empirically based approach allows for investigating the combined effect of meme inheritance and gendered transmission. Results show that scientific memes do not spread differently from either male or female cited authors. Likewise, the memes that we analyse were not found to propagate more easily via male or female inheritance.

Identifying NIST Impacts on Patenting: A Novel Data Set and Potential Uses.

The National Institute of Standards and Technology's (NIST's) mission is to "promote U.S. innovation and industrial competitiveness." To meet this mission, NIST scientists produce a great variety of scientific and technical outputs. This paper presents results from a novel effort to measure usage and impact of a more complete set of outputs, including patents, publications, research data, software, reference materials, and a variety of additional formal and informal scientific outputs. This effort captures a significantly broader set of scientific outputs than traditional citation analysis which typically examines patent-to-patent citations or more recently patent-to-(peer-reviewed) paper citations. This may be of significant importance to NIST as NIST scientists produce a wide variety of scientific and technical outputs beyond patents and papers. Our results indicate that metrics that solely rely on patents issued to NIST inventors understate NIST's true impact on invention and do not capture usage of much of NIST's scientific output by other inventors. Thus, identifying the magnitude and varied usage of different types of NIST outputs represents a significant improvement in NIST impact metrics. The results clearly indicate that different companies, industries and technologies rely on different types of NIST outputs. Therefore, reliance on a limited set of technology transfer tools by either researchers or policy makers creates a risk that NIST knowledge and capabilities will not be transferred to and adopted by businesses and other organizations. Finally, the data developed here suggest a number of new technology transfer metrics that promote shared technology transfer responsibilities and may focus attention on activities that increase the impact of current research without fundamentally altering the infrastructural character of this research.

The Productivity Commission: a different engine for law reform?

This empirical study analyses the extent to which the Productivity Commission (Commission) relies on different types of evidence in formulating recommendations in a sample of reports. It goes deeper than traditional citation analysis; rather than classifying and counting all material cited in each sample report, it includes only citations that influenced the final recommendations. The findings, which run counter to the rhetoric employed in relation to the Commission's work, reveal the extent to which the Commission relies on non-quantitative forms of evidence, including bare assertions, personal experience, and logical and legal arguments, particularly in reports addressing broader questions of social policy. It concludes with a discussion of the significance of these findings, linking them to Graycar's critique of law reform commissions. As such, it provides a more accurate, but still preliminary, basis for understanding the Commission's methods than that otherwise appearing in the discourse surrounding the Commission's work.

A survey of Chinese interpreting studies: who influences who …and why?

This paper describes how scholars in Chinese Interpreting Studies (CIS) interact with each other and form discrete circles of influence. It also discusses what it means to be an influential scholar in the community and the relationship between an author’s choice of research topic and his academic influence. The study examines an all-but-exhaustive collection of 59,303 citations from 1,289 MA theses, 32 doctoral dissertations and 2,909 research papers, combining traditional citation analysis with the newer Social Network Analysis to paint a panorama of CIS. It concludes that the community cannot be broadly divided into Liberal Arts and Empirical Science camps; rather, it comprises several distinct communities with various defining features. The analysis also reveals that the top Western influencers have an array of academic backgrounds and research interests across many different disciplines, whereas their Chinese counterparts are predominantly focused on Interpreting Studies. Last but not least, there is found to be a positive correlation between choosing non-mainstream research topics and having a high level of academic influence in the community.

Citation analysis and discoverability: a critical challenge for disability studies

Disability is an identity, a culture/community/cause/power struggle; a rights movement demanding civil, ethical, political and social positions at tables of power, equality and social justice. This perspective forms the core of changes in the past 50 years in legal, social and political arenas that have taken place and is essential to understanding the growth, depth and nature of Disability Studies as a separate and legitimate field of inquiry. This study focuses primarily on studying the evolving literature in the core aspects of Disability Studies, using citation and text analyses to study trends in publishing patterns and the field’s impact on the wider academic enterprise. Lack of thorough indexing to key journals made traditional citation analysis impossible, leading to the use of dissertations as a means to tease out useful information on the state of Disability Studies today. The most significant finding of the study is the lack of adequate indexing of key journals in standard scholarly databases, which can only limit the potential development of both the field and its impact/influence on related disciplines and professions. The study also found the significant role that dissertations can play in tracking new research directions.

Content‐based citation analysis: The next generation of citation analysis

Traditional citation analysis has been widely applied to detect patterns of scientific collaboration, map the landscapes of scholarly disciplines, assess the impact of research outputs, and observe knowledge transfer across domains. It is, however, limited, as it assumes all citations are of similar value and weights each equally. Content‐based citation analysis (CCA) addresses a citation's value by interpreting each one based on its context at both the syntactic and semantic levels. This paper provides a comprehensive overview of CAA research in terms of its theoretical foundations, methodical approaches, and example applications. In addition, we highlight how increased computational capabilities and publicly available full‐text resources have opened this area of research to vast possibilities, which enable deeper citation analysis, more accurate citation prediction, and increased knowledge discovery.

In‐text author citation analysis: Feasibility, benefits, and limitations

This article explores the feasibility, benefits, and limitations of in‐text author citation analysis and tests how well it works compared with traditional author citation analysis using citation databases. In‐text author citation analysis refers to author‐based citation analysis using in‐text citation data from full‐text articles rather than reference data from citation databases. It has the potential to help with the application of citation analysis to research fields such as the social sciences that are not covered well by citation databases and to support weighted citation and cocitation counting for improved citation analysis results. We found that in‐text author citation analysis can work as well as traditional citation analysis using citation databases for both author ranking and mapping if author name disambiguation is performed properly. Using in‐text citation data without any author name disambiguation, ranking authors by citations is useless, whereas cocitation analysis works well for identifying major specialties and their interrelationships with cautions required for the interpretation of small research areas and some authors' memberships in specialties.

Tweeting biomedicine: An analysis of tweets and citations in the biomedical literature

Data collected by social media platforms have been introduced as new sources for indicators to help measure the impact of scholarly research in ways that are complementary to traditional citation analysis. Data generated from social media activities can be used to reflect broad types of impact. This article aims to provide systematic evidence about how often Twitter is used to disseminate information about journal articles in the biomedical sciences. The analysis is based on 1.4 million documents covered by both PubMed and Web of Science and published between 2010 and 2012. The number of tweets containing links to these documents was analyzed and compared to citations to evaluate the degree to which certain journals, disciplines, and specialties were represented on Twitter and how far tweets correlate with citation impact. With less than 10% of PubMed articles mentioned on Twitter, its uptake is low in general but differs between journals and specialties. Correlations between tweets and citations are low, implying that impact metrics based on tweets are different from those based on citations. A framework using the coverage of articles and the correlation between Twitter mentions and citations is proposed to facilitate the evaluation of novel social‐media‐based metrics.

New perspectives on article-level metrics: developing ways to assess research uptake and impact online

Altmetrics were born from a desire to see and measure research impact differently. Complementing traditional citation analysis, altmetrics are intended to reflect more broad views of research impact by taking into account the use of digital scholarly communication tools. Aggregating online attention paid to individual scholarly articles and data sets is the approach taken by Altmetric LLP, an altmetrics tool provider. Potential uses for article-level metrics collected by Altmetric include: 1) the assessment of an article's impact within a particular community, 2) the assessment of the overall impact of a body of scholarly work, and 3) the characterization of entire author and reader communities that engage with particular articles online. Although attention metrics are still being refined, qualitative altmetrics data are beginning to illustrate the rich new world of scholarly communication, and are emerging as ways to highlight the immediate societal impacts of research.

JASIST 2001–2010

AbstractEditor's SummaryThe Journal of the American Society for Information Science and Technology (JASIST) started in 2001 after being published since 1950 under two other titles. Prior bibliometric analyses of JASIS focused on author and article characteristics and trends and on geographic and keyword distributions. The current study examines article citations from 2001 through 2010, drawing on three major citation databases and readership counts. Of 1,459 articles, 14 were cited at the top of at least one database, and seven were among both the top‐cited papers and those with the highest readership counts. The top‐cited papers focused on the web, informetrics, link analysis, theory and knowledge management. The most often read were on the web as a topic, theory, link analysis, informetrics and databases. Though not used in this research, alternative metrics such as mention counts in social media, Slideshare, Wikipedia and ReaderMeter can complement traditional citation analysis.

How and why scholars cite on Twitter

Scholars are increasingly using the microblogging service Twitter as a communication platform. Since citing is a central practice of scholarly communication, we investigated whether and how scholars cite on Twitter. We conducted interviews and harvested 46,515 tweets from a sample of 28 scholars and found that they do cite on Twitter, though often indirectly. Twitter citations are part of a fast-moving conversation that participants believe reflects scholarly impact. Twitter citation metrics could augment traditional citation analysis, supporting a “scientometrics 2.0.”

Beyond citation analysis: a model for assessment of research impact

Is there a means of assessing research impact beyond citation analysis? The case study took place at the Washington University School of Medicine Becker Medical Library. This case study analyzed the research study process to identify indicators beyond citation count that demonstrate research impact. The authors discovered a number of indicators that can be documented for assessment of research impact, as well as resources to locate evidence of impact. As a result of the project, the authors developed a model for assessment of research impact, the Becker Medical Library Model for Assessment of Research. Assessment of research impact using traditional citation analysis alone is not a sufficient tool for assessing the impact of research findings, and it is not predictive of subsequent clinical applications resulting in meaningful health outcomes. The Becker Model can be used by both researchers and librarians to document research impact to supplement citation analysis.

Library Catalog Analysis as a tool in studies of social sciences and humanities: An exploratory study of published book titles in Economics

This paper explores the use of Library Catalog Analysis (LCA), defined as the application of bibliometric or informetric techniques to a set of library online catalogs, to describe quantitatively a scientific-scholarly field on the basis of published book titles. It focuses on its value as a tool in studies of Social Sciences and Humanities, especially its cognitive structures, main book publishers and the research performance of its actors. The paper proposes an analogy model between traditional citation analysis of journal articles and Library Catalog Analysis of book titles. It presents the outcomes of an exploratory study of book titles in Economics included in 42 academic library catalogs from 7 countries. It describes the process of data collection and cleaning, and applies a series of indicators and thematic mapping techniques. It illustrates how LCA can be fruitfully used to assess book production and research performance at the level of an individual researcher, a research department, an entire country and a book publisher. It discusses a number of issues that should be addressed in follow-up studies and concludes that LCA of published book titles can be developed into a powerful and useful tool in studies of Social Sciences and Humanities.

Visualizing latent domain knowledge

Knowledge discovery and data mining commonly rely on finding salient patterns of association from a vast amount of data. Traditional citation analysis of scientific literature draws insights from strong citation patterns. Latent domain knowledge, in contrast to the mainstream domain knowledge, often consists of highly relevant but relatively infrequently cited scientific works. Visualizing latent domain knowledge presents a significant challenge to knowledge discovery and quantitative studies of science. We build upon a citation-based knowledge visualization procedure and develop an approach that not only captures knowledge structures from prominent and highly cited works, but also traces latent domain knowledge through low-frequency citation chains. We apply this approach to two cases: (1) identifying cross-domain applications of Pathfinder networks (PFNETs) and (2) clarifying the current status of scientific inquiry of a possible link between Bovine spongiform encephalopathy (BSE), also known as mad cow disease, and a new variant Creutzfeldt-Jakob disease (vCJD), a type of brain disease in humans.