Citation-based Impact Research Articles

Novel Journal Metrics in Cardiothoracic Surgery: How Different Are Contemporary Metrics From the Impact Factor?

Despite shortcomings, impact factor (IF) remains the "gold standard" metric for journal quality. However, novel metrics including the h-index, g-index, and Altmetric Attention Score (AAS; mentions in mainstream/social media) are gaining traction. We assessed correlations between these metrics among cardiothoracic surgery journals. For all cardiothoracic surgery journals with a 2021 Clarivate IF (N = 20), the 2-year IF (2019 to 2020) and 5-year IF (2016 to 2020), h-index, and g-index were obtained. Two-year journal-level AAS (2019 to 2020) was also calculated. Journal Twitter presence and activity was sourced from Twitter and the Twitter application programming interface. Correlations were assessed using Spearman correlation, and coefficients of determination were calculated. IF demonstrated a moderate-strong positive correlation with the h-index (rs = 0.48 to 0.77) and g-index (rs = 0.49 to 0.79) and a moderate correlation with AAS (rs = 0.53 to 0.58). The 2-year IF accounted for 25% to 49% of variability in the h-index, 27% to 55% of variability in the g-index, and 32% of variability in the AAS. Among journals with a Twitter account (N = 10), IF was strongly correlated with Twitter following (rs = 0.81 to 0.86), which was in turn strongly correlated with journal AAS (rs = 0.79). Article-level AAS was moderately correlated with citation count (rs = 0.47). IF accounted for only between 25% and 55% of variability in the h-index and g-index, indicating that these newer metrics measure unique dimensions of citation-based impact. Thus, the academic community must familiarize itself with these newer journal metrics. Social media attention may be associated with scholarly impact, although further work is needed to understand these relationships.

Beyond views, productivity, and citations: measuring geopolitical differences of scientific impact in communication research

Scientometric analyses applying critical sociological frameworks have previously shown that high-prestige research output—with regards to both quantity and impact—is typically clustered in a few core countries and world regions, indicating uneven power relations and systematic biases within global academia. Although citation count is a common formula in these analyses, only a handful of studies investigated altmetrics (impact measures beyond citation-based metrics) in communication science. In this paper, we explore geopolitical biases of impact amongst the most productive scholars in the field of communication from 11 countries and 3 world regions. Drawing on SCOPUS data, we test three formulas that measure scholarly performance (citations per document; views per document; and citations per view) to investigate how geographical location affects the impact of scholars. Our results indicate a strong US-dominance with regard to citation-based impact, emphasizing a further need for de-Westernization within the field. Moreover, the analysis of altmetric formulas revealed that research published by Eastern European and Spanish scholars, although accessed similarly or even more often than American or Western European publications, is less cited than those. Country-level comparisons are also discussed.

Analyzing the Relationship between Citation-Based Impact Metrics and Electronic Journal Usage: A Case Study

This study examines the relationship between Journal Impact Factors (JIF) and e-journal usage in collection management. We focus on the impact of major JIFs on local e-journal usage and propose an alternative approach to conventional methods for collection selectors. By treating journal usage patterns as panel data and employing fixed-effects regression models, we find that journal popularity has the greatest influence on local e-journal usage and the effects of impact factors on academic article usage can vary across different disciplines. The methodology and findings of our study offer valuable insights and practical implications for academic librarians overseeing collection development.

The Twitter accounts of scientific journals: a dataset

Twitter harbours dense networks of academics, but to what extent do scientific journals use that platform? This article introduces a dataset of 3,485 Twitter accounts pertaining to a sample of 13,821 journals listed in Web of Science’s three major indices (SCIE, SSCI and AHCI). The summary statistics indicate that 25.2% of the journals have a dedicated Twitter presence. This number is likely to grow, as, on average, every one and a half days sees yet another journal setting up a new profile. The share of Twitter presence, however, varies strongly by publisher and discipline. The most active discipline is political science, which has almost 75% of its journals on Twitter, while other research categories have zero. The median account issues 116 messages a year and it interacts with distinct other users once in two to three Tweets. Approximately 600 journals refer to themselves as ‘peer-reviewed’, while 263 journals refer to their citation-based impact (like the impact factor) in their profile description. All in all, the data convey immense heterogeneity with respect to the Twitter behaviour of scientific journals. As there are numerous deceptive Twitter profile names established by predatory publishers, it is recommended that journals establish their official accounts lest bogus journals mislead the public about scientific findings. The dataset is available for use for further scientometric analyses.

Use of Research Organizations Registry (ROR) identifiers in author academic profiles: the case of Google Scholar Profiles

Research organizations' persistent identifiers allow for reducing affiliation ambiguities, enable accurate institutional analyses and favor the design of modern online scholarly databases suited for research discovery and research evaluation. However, few studies have attempted to quantify their degree of use. Precisely, the purpose of this work is to determine the use of Research Organizations Registry (ROR) IDs in author academic profiles, specifically in Google Scholar Profiles (GSP). To do this, all the Google Scholar profiles including the term ROR in any of the public descriptive fields were collected and analyzed. The results evidence a low use of ROR IDs (1,033 profiles), mainly from a few institutions (e.g. Pontificia Universidad Javeriana in Colombia, and Escuela Superior Politécnica del Litoral in Ecuador hold 55.7% of all profiles), from low citation-based impact authors (45.1% of profiles attain less than 100 citations each), belonging mainly to Social Sciences (26.3%), Engineering fields (25.3%), and Natural Sciences (22.2%). Although Google Scholar does not facilitate the inclusion of identifiers, it seems that the world's leading research institutions are not recommending their researchers include these identifiers in their profiles yet.

Scientific rewards for biomedical specialization are large and persistent

BackgroundWhile specialization plays an essential role in how scientific research is pursued, we understand little about its effects on a researcher’s impact and career. In particular, the extent to which one specializes within their chosen fields likely has complex relationships with productivity, career stage, and eventual impact. Here, we develop a novel and fine-grained approach for measuring a researcher’s level of specialization at each point in their career and apply it to the publication data of almost 30,000 established biomedical researchers to measure the effect that specialization has on the impact of a researcher’s publications.ResultsUsing a within-researcher, panel-based econometric framework, we arrive at several important results. First, there are significant scientific rewards for specialization—25% more citations per standard deviation increase in specialization. Second, these benefits are much higher early in a researcher’s career—as large as 75% per standard deviation increase in specialization. Third, rewards are higher for researchers who publish few papers relative to their peers. Finally, we find that, all else equal, researchers who make large changes in their research direction see generally increased impact.ConclusionsThe extent to which one specializes, particularly at the early stages of a biomedical research career, appears to play a significant role in determining the citation-based impact of their publications. When this measure of impact is, implicitly or explicitly, an input into decision-making processes within the scientific system (for example, for job opportunities, promotions, or invited talks), these findings lead to some important implications for the system-level organization of scientific research and the incentives that exist therein. We propose several mechanisms within modern scientific systems that likely lead to the scientific rewards we observe and discuss them within the broader context of reward structures in biomedicine and science more generally.

Reference publication year spectroscopy (RPYS) in practice: a software tutorial

In course of the organization of Workshop III entitled “Cited References Analysis Using CRExplorer” at the International Conference of the International Society for Scientometrics and Informetrics (ISSI2021), we have prepared three reference publication year spectroscopy (RPYS) analyses: (i) papers published in Journal of Informetrics; (ii) papers regarding the topic altmetrics; and (iii) papers published by Ludo Waltman (we selected this researcher because he received the Derek de Solla Price Memorial Medal during the ISSI2021 conference). The first RPYS analysis has been presented live at the workshop and the second and third RPYS analyses have been left to the participants for undertaking after the workshop. Here, we present the results for all three RPYS analyses. The three analyses have shown quite different seminal papers with a few overlaps. Many of the foundational papers in the field of scientometrics (e.g., distributions of publications and citations, citation network and co-citation analyses, and citation analysis with the aim of impact measurement and research evaluation) were retrieved as seminal papers of the papers published in Journal of Informetrics. Mainly papers with discussions of the deficiencies of citation-based impact measurements and comparisons between altmetrics and citations were retrieved as seminal papers of the topic altmetrics. The RPYS analysis of the paper set published by Ludo Waltman mainly retrieved papers about network analyses, citation relations, and citation impact measurement.

514 Using text mining approaches to identify research trends – an exploratory analysis of Miami Clinical and Translational Science Institute publications from 2013-2021

OBJECTIVES/GOALS: The goal of this project was to perform an exploratory analysis of the research themes of scientific publications from the Miami Clinical and Translational Science Institute using text mining techniques and using bibliometric characterization and network analysis to further assess research trends. METHODS/STUDY POPULATION: Publications were identified from the Web of Science database using Miami CTSI grant numbers as search criteria for the period 2013-2021 and KL2 scholar publications. Following data pre-processing, topic modeling was performed using the Latent Dirichlet Allocation algorithm and cluster analysis in the R programming language. The resulting themes will be further analyzed by employing a citation-based impact assessment approach to identify trends over time. Network analysis of publications will be performed using the VOSviewer package to visualize publication networks using citation and co-authorship relations within each major theme and their evolution over time. Findings will be evaluated for alignment with institutional research strategy. RESULTS/ANTICIPATED RESULTS: About 400 CTSI publications from 2013-2021 to date were used for analysis. Twenty topics and five major research themes were identified among the Miami CTSI publications – neuroscience, cancer, community and public health, metabolics, and HIV/infectious diseases. Top keywords within each topic were aligned with the most frequent author-assigned keywords for that topic. The CTSI research themes were also well-aligned with the institutional vision for research and focus areas. Trends using citations and networks for each research theme are currently being analyzed and results will be included in the overall findings post analysis. DISCUSSION/SIGNIFICANCE: Text mining was successfully used in identifying topics and research themes for clinical and translational research publications of the Miami CTSI, and in combination with bibliometric characterization, will be helpful in shaping CTSI strategy and alignment with the universitys research priorities.

The scholarly impact of private sector research: A multivariate analysis

• We investigate the scientific impact of Italian private research as compared to public research. • We also delve into the relation between different types of research collaboration and impact of resulting publications. • We examine Italian 2010–2017 publications indexed in WoS. • 70% of publications by the Italian private sector are in collaboration public research institutions. • Industry only publications show a lower impact than that recorded for public only publications. • Public-private joint research leads to publications of greater impact. Literature about the scholarly impact of scientific research offers very few contributions on private sector research, and the comparison with public sector. In this work, we try to fill this gap examining the citation-based impact of Italian 2010–2017 publications distinguishing authorship by the private sector from the public sector. In particular, we investigate the relation between different forms of collaboration and impact: how intra-sector private publications compare to public, and how private-public joint publications compare to intra-sector extramural collaborations. Finally, we assess the different effect of international collaboration on private and public research impact, and whether there occur differences across research fields.

Impact-Based Ranking of Scientific Publications: A Survey and Experimental Evaluation

As the rate at which scientific work is published continues to increase, so does the need to discern high-impact publications. In recent years, there have been several approaches that seek to rank publications based on their expected citation-based impact. Despite this level of attention, this research area has not been systematically studied. Past literature often fails to distinguish between short-term impact, the current popularity of an article, and long-term impact, the overall influence of an article. Moreover, the evaluation methodologies applied vary widely and are inconsistent. In this work, we aim to fill these gaps, studying impact-based ranking theoretically and experimentally. First, we provide explicit definitions for short-term and long-term impact, and introduce the associated ranking problems. Then, we identify and classify the most important ideas employed by state-of-the-art methods. After studying various evaluation methodologies of the literature, we propose a specific benchmark framework that can help us better differentiate effectiveness across impact aspects. Using this framework we investigate: (1) the practical difference between ranking by short- and long-term impact, and (2) the effectiveness and efficiency of ranking methods in different settings. To avoid reporting results that are discipline-dependent, we perform our experiments using four datasets from different scientific disciplines.

Bibliometrics versus altmetrics: Researchers’ attitudes in Slovakia

Objective. The paper presents principal theoretical starting points and an overview of attitudes to altmetrics-based research evaluation among scientists of various disciplines in Slovakia, more specifically in comparison with traditional, bibliometric approaches.
 Design/Methodology/Approach. We used a questionnaire and asked researchers from various disciplines about their knowledge of principle characteristics defining the altmetrics and about their attitudes towards the application of bibliometrics and altmetrics approaches in the processes of research evaluation. The questionnaire covered a wide range of issues, predominantly associated with interpretation and perception of the importance of alternative indicators, including the factors influencing the interpretation. In this analysis, we concentrate on those parts of research reflecting the willingness of respondents to accept alternative indicators as part of research/science assessment.
 Results/Discussion. The attitude to the application of alternative indicators in research assessment is not significantly more negative among Slovak scientists if we compare it to other countries. The citations are still a respected indicator perceived as a tool for creating bonds within science. If they are perceived as obsolete or insufficient, it relates to the experience of researchers whose results are predominantly aimed at non-academic target group or if they are active in the research of social phenomena and problems. Due to pragmatic reasons connected with research financing, however, registering the citations is a generally accepted priority. Familiarity and practical experience with social media were identified as one of the factors influencing the interpretation of alternative indicators in the environment of both public and academic media.
 Conclusions. Traditional bibliometric methods, especially in the area of research evaluation, are still preferred over the altmetrics. The scientists realize that the kind of attention captured by the indicators based on social media is not comparable with the citation-based impact. Or, more precisely, that altmetrics reflects a different kind of impact whose correspondence with a traditional world of bibliographic citations remains to be explored.
 Originality/Value. The article presents the results of original research carried out in the community of scientists in Slovakia, applying quantitative research methods. It contributes to the results of previous studies on the attitudes and acceptance of altmetrics in the research community, with a special focus on research assessment.

Use of relative extra citation counts and uncited publications to enhance the discriminatory power of the h-index

Although the conventional h-index attempts to combine a researcher’s publications and citation-based impact, it requires threshold citation counts and cited papers to award a score. The consequence is that the h-index fails to consider extra citation counts as well as uncited publications, both of which reveal a researcher’s performance output. Recently, an apparent h-index was presented to extend the conventional h-index to account for the uncited publications of a researcher; unfortunately, the apparent h-index also fails to account for a researcher’s extra citation counts. In this paper, we extend the apparent h-index to account for a researcher’s extra citation counts. In particular, contrary to the conventional h-index and its variants, the proposed author-level metric uses relative extra citation counts and uncited publications to discriminate among researchers. The relative extra citation count is defined as the ratio of the absolute extra citation counts to the total number of citations. The proposed index, called a comprehensive h-index (c-index), is formulated by incorporating the relative extra citation counts and the fraction of the cited publications into the h-index. The advantages of the c-index are two-folds: first, it enhances the discriminatory power of the h-index as it considers researcher’s entire publications, whether cited or not, and all the citation counts; and second, it preserves the desirable features of the conventional h-index, namely robustness and simplicity. Case studies for researchers in Chemistry, Physics, Material Science, Engineering and Medicine have been presented. Results show that researchers could have equal h-index, but unequal c-index due to their unequal relative extra citation counts, as well as their uncited publications. Furthermore, a comparison of the c-index with the h- and g-indices have been made; results show that the c-index is more discriminatory than both the h- and g-indices. The proposed metric may contribute to the ongoing discussions on the improvement of the conventional h-index.

Do Online Readerships Offer Useful Assessment Tools? Discussion Around the Practical Applications of Mendeley Readership for Scholarly Assessment

This methods report illustrates the relevance of Mendeley readership as a tool for research assessment. Readership indicators offer new possibilities to inform the evaluation of publications and outputs either poorly covered in citation indexes (e.g. non-English language outputs, Global South publications, Social sciences and humanities), or typically excluded from citation analysis (e.g. letters, editorial material, etc.). Mendeley readership can also inform the earlier impact of scientific outputs, as well as the impact among wider non-academic audiences. All these features are discussed in this report and the relevance of readership indicators to extend the concept of research impact beyond specific acts (e.g. citations) is highlighted. Best practical recommendations on how Mendeley readership can be used for assessment purposes are discussed.Policy highlights:This paper illustrates practical possibilities of readership indicators for research evaluation.Readership indicators inform impact of publications poorly covered in bibliometrics databases or excluded from citation analysis.Readership indicators inform early impact and non-academic impact of publications.Readership indicators can be used to inform, support, and complement (citation-based impact) decisions on research evaluation exercises.

Which journal ranking list? A case study in business and economics

PurposeWith over 3,000 academic journals in the fields of Business and Economics, most academics face a hard time selecting an adequate journal to submit their work to. In today's demanding academic environment and with the presence of different journal ranking lists (JRLs), the selection becomes more difficult when considering employment, promotion and funding. The purpose of this paper is to explore key differences among multiple JRLs pertinent to the latter common objectives. An extensive analysis is conducted to compare the content of journals in the Australian Business Deans Council (ABDC) Journal Quality list, Scopus and Web of Science (WoS) in the fields of Business and Economics. Then, a case of a university with medium research output is considered where scholarly performance evaluation is based on the ABDC Journal Quality List.Design/methodology/approachAfter ranking journals in the fields of Business and Economics based on SCImago Journal Rank (SJR) indicator, JCR's Journal Impact Factor (JIF) and JCR's Eigenfactor (EF), a methodology is proposed to categorize journals in the three JRLs into the same categorization adopted by ABDC. The latter establishes a way to compare the four JRLs under consideration and serves as a basis to compare and analyze the content of journals in the ABDC Journal Quality list, Scopus and WoS. As a proxy impact metric, a normalized citation count is associated with each article based on Google Scholar. The publications of the considered university are then evaluated from the perspective of the four JRLs in terms of citation-based impact and quality while considering the exposure to popular world university ranking tables.FindingsFor journals classified under fourth tier by ABDC, over 53 and 59% are not indexed by Scopus and WoS, respectively. In this case study, over 42% of the publications appear in journals that are not listed in JCR despite the fact that over 94% of them are listed by the SJR list. Generally, publications that appear in journals listed by JCR achieve, on a yearly average, significantly higher citation rates when compared to those that appear in journals listed in ABDC and SJR Lists.Originality/valueA four-tier mapping is proposed for consistent comparison among JRLs. Normalized citation count associated with each article based on Google Scholar is employed for evaluation. The findings provide recommendations for scholars, administrators and global universities, including Euro-Med Universities, on which JRL can be more influential for both faculty development and positioning of the university.

An examination of citation-based impact of the computational biology conferences.

An examination of citation-based impact of the computational biology conferences.

Understanding the intellectual structure and evolution of Competitive Intelligence: a bibliometric analysis from 1984 to 2017

ABSTRACTCompetitive Intelligence (CI) is a relatively novel discipline that is generating a growing interest in the field of strategic management. Its novelty, approach and origin have meant that the definition of the CI entails different interpretations and utilities depending on the audience. This research develops a conceptual analysis of CI in literature by quantifying the bibliometric performance indicators, identifying the main authors, countries, journals and research areas and evaluating the intellectual structure and evolution of the discipline using SciMAT as bibliometric analysis software. The bibliometric performance analysis is focused on the citation-based impact of the scientific output, while the science mapping illustrates the evolution of the research themes that build the discipline through the use of bibliometric network analysis techniques. To this purpose, the publications related to CI from 1984 to 2017 available at Scopus have been retrieved (5,275 publications). Finally, it offers a framework to support future researches.

Topics and trends in finance research: What is published, who publishes it and what gets cited?

In this study we provide a detailed examination of the subject matter of finance research and its institutional features as it has evolved over the past two decades. Drawing on novel approaches from data science, we examine the content of more than 30,000 published papers. Overall, we find a striking lack of diversity in the topics investigated and the methodological approaches used. Almost all finance research is conducted using techniques from economics and mathematics, with virtually no use made of qualitative methods or interdisciplinary approaches. Looking at the developments of the discipline over time, we document an increase in the volume of corporate finance research and a variation in the topics covered following the financial crisis, although these changes appear to be reactive and trivial rather than paradigm shifts. We also provide a cartography of research in finance and its citation-based impact by the location of authors. Leading finance research is concentrated in elite US institutions, and has a disproportionately strong citation-based impact. Compared with other business and management sub-fields, citations in finance are heavily skewed towards the top journals as the latter generate almost twice as much impact as the lower rated outlets.

Topics and Trends in Finance Research: What is Published, Who Publishes it and What Gets Cited?

In this study we provide a detailed examination of the subject matter of finance research and its institutional features as it has evolved over the past two decades. Drawing on novel approaches from data science, we examine the content of more than 30,000 published papers. Overall, we find a striking lack of diversity in the topics investigated and the methodological approaches used. Almost all finance research is conducted using techniques from economics and mathematics, with virtually no use made of qualitative methods or interdisciplinary approaches. Looking at the developments of the discipline over time, we document an increase in the volume of corporate finance research and a variation in the topics covered following the financial crisis, although these changes appear to be reactive and trivial rather than paradigm shifts. We also provide a cartography of research in finance and its citation-based impact by the location of authors. Leading finance research is concentrated in elite US institutions, and has a disproportionately strong citation-based impact. Compared with other business and management sub-fields, citations in finance are heavily skewed towards the top journals as the latter generate almost twice as much impact as the lower rated outlets.

Quantity versus impact of software engineering papers: a quantitative study

According to the data from the Scopus publication database, as analyzed in several recent studies, more than 70,000 papers have been published in the area of Software Engineering (SE) since late 1960’s. According to our recent work, 43% of those papers have received no citations at all. Since citations are the most commonly used metric for measuring research (academic) impact, these figures raise questions (doubts) about the (non-existing) impact of such a large set of papers. It is a reality that typical academic reward systems encourage researchers to publish more papers and do not place a major emphasis on research impact. To shed light on the issue of volume (quantity) versus citation-based impact of SE research papers, we conduct and report in this paper a quantitative bibliometrics assessment in four aspects: (1) quantity versus impact of different paper types (e.g., conference versus journal papers), (2) ratios of uncited (non-impactful) papers, (3) quantity versus impact of papers originating from different countries, and (4) quantity versus impact of papers by each of the top-10 authors (in terms of number of papers). To achieve the above objective, we conducted a quantitative exploratory bibliometrics assessment, comprised of four research questions, to assess quantity versus impact of SE papers with respect to the aspects discussed above. We extracted the data through a systematic, automated and repeatable process from the Scopus paper database, which we also used in two previous papers. Our results show that the distribution of SE publications has a major inequality in terms of impact overall, and also when categorized in terms of the above four aspects. The situation in the SE literature is similar to the other areas of science as studied by previous bibliometrics studies. Also, among our results is the fact that journal articles and conference papers have been cited 12.6 and 3.6 times on average, confirming the expectation that journal articles have more impact, in general, than conference papers. Also, papers originated from English-speaking countries have in general more visibility and impact (and consequently citations) when compared to papers originated from non-English-speaking countries. Our results have implications for improvement of academic reward systems, which nowadays mainly encourage researchers to publish more papers and usually neglect research impact. Also, our results can help researchers in non-English-speaking countries to consider improvements to increase their research impact of their upcoming papers.

The citation-based impact of complex innovation systems scales with the size of the system

The aim of this paper is to explore the power-law relationship between the publishing size of complex innovation systems and their citation-based impact. We analyzed articles and reviews from InCites™ database published by six complex innovation systems. We found scale-invariant properties in the complex innovation systems which were analyzed. These properties are evidenced in the power law correlation between complex innovation systems' citation-based impact and their size with a scaling exponent ź ź 1.19 ± 0.01. The results suggest citations to a complex innovation system tend to increase 21.19 or 2.28 times when the system doubles its size over time. These scale-invariant emergent properties are a common property of a complex innovation system. These properties can be quantified using the parameters of scale-invariant correlation. These parameters can be used to formulate measures and models useful for informing public policy about scale-invariant emerging properties of a complex innovation system, making comparisons of citation impact between complex systems of vastly different sizes, evaluating citation impact of complex innovation systems according to their size, long range planning, and elaborating rankings across complex innovation systems.