Mean Normalized Citation Score Research Articles

Is the scientific impact of the LIS themes gender-biased? A bibliometric analysis of the evolution, scientific impact, and relative contribution by gender from 2007 to 2022

AbstractGender equity remains a challenge both globally and within academia, despite recent efforts to change it. Moreover, beyond the authors’ productivity, studies have reported that women often achieve lower scientific impact than their peers. To shed light on this complex relationship between the scientific impact and the themes addressed, this study conducts a comprehensive analysis of Library and Information Science field from 2007 to 2022 in four consecutive slides, identifying the principal themes covered in the field, analyzing the relative gender contribution rate, employing strategic diagrams, and assessing impact metrics, such as mean normalized citation score, 1% of most cited papers, and H-Classic. We employed science mapping analysis to explore a core of 45,650 documents from the Web of Science, with gender identification in 94.25% of cases. Our findings revealed a slight increase in the percentage of women authors within the field across the time, and a decline during the COVID-19 pandemic, demonstrating consistency in the themes addressed over the years. Women were overrepresented in the classic themes of LIS, human, and health-related fields, with these themes displaying lower performance rates. In contrast, men authors were overrepresented in STEM-related fields and innovation themes, associated with higher metric values. Our findings underlined the potential association between research themes and scientific performance, and provide societal and structural explanations for these observations. This study contributes valuable insights into the relationship between research themes and the scientific impact achieved by researchers in LIS, highlighting the importance of encouraging women’s participation in diverse knowledge domains and challenging prevailing stereotypes within academic and professional spheres.

How well does I3 perform for impact measurement compared to other bibliometric indicators? The convergent validity of several (field-normalized) indicators

Recently, the integrated impact indicator (I3) was introduced where citations are weighted in accordance with the percentile rank class of each publication in a set of publications. I3 can also be used as a field-normalized indicator. Field-normalization is common practice in bibliometrics, especially when institutions and countries are compared. Publication and citation practices are so different among fields that citation impact is normalized for cross-field comparisons. In this study, we test the ability of the indicator to discriminate between quality levels of papers as defined by Faculty members at F1000Prime. F1000Prime is a post-publication peer review system for assessing papers in the biomedical area. Thus, we test the convergent validity of I3 (in this study, we test I3/N—the size-independent variant of I3 where I3 is divided by the number of papers) using assessments by peers as baseline and compare its validity with several other (field-normalized) indicators: the mean-normalized citation score, relative-citation ratio, citation score normalized by cited references, characteristic scores and scales, source-normalized citation score, citation percentile, and proportion of papers which belong to the x% most frequently cited papers (PPtop x%). The results show that the PPtop 1% indicator discriminates best among different quality levels. I3 performs similar as (slightly better than) most of the other field-normalized indicators. Thus, the results point out that the indicator could be a valuable alternative to other indicators in bibliometrics.

Normalization of Citation Impact in Economics

This study is intended to facilitate fair research evaluations in economics. Field- and time-normalization of citation impact is the standard method in bibliometrics. Since citation rates for journal papers differ substantially across publication years and Journal of Economic Literature (JEL) classification codes, citation rates should be normalized for the comparison of papers across different time periods and economic subfields. Without normalization, both factors that are independent of research quality bias the results of citation analyses. We introduce two normalized indicators in economics, which are the most important indicators in bibliometrics: (1) the mean normalized citation score (MNCS) compares the citation impact of a focal paper with the mean impact of similar papers published in the same economic subfield and publication year. (2) PPtop 50% is the share of papers that belong to the above-average half in a certain subfield and time period. Since the MNCS is based on arithmetic averages despite skewed citation distributions, we recommend using PPtop 50% for fair comparisons of entities in economics (e.g. researchers, institutions, or countries). In this study, we apply the method to 294 journals (including normalized scores for 192,524 papers) by assigning them to four citation impact classes and identifying 33 outstandingly cited economics journals.

Citation score normalized by cited references (CSNCR): The introduction of a new citation impact indicator

In this paper, a new field-normalized indicator is introduced, which is rooted in early insights in bibliometrics, and is compared with several established field-normalized indicators (e.g. the mean normalized citation score, MNCS, and indicators based on percentile approaches). Garfield (1979) emphasizes that bare citation counts from different fields cannot be compared for evaluative purposes, because the "citation potential" can vary significantly between the fields. Garfield (1979) suggests that "the most accurate measure of citation potential is the average number of references per paper published in a given field". Based on this suggestion, the new indicator is basically defined as follows: the citation count of a focal paper is divided by the mean number of cited references in a field to normalize citations. The new indicator is called citation score normalized by cited references (CSNCR). The theoretical analysis of the CSNCR shows that it has the properties of consistency and homogeneous normalization. The close relation of the new indicator to the MNCS is discussed. The empirical comparison of the CSNCR with other field-normalized indicators shows that it is slightly poorer able to field-normalize citation counts than other cited-side normalized indicators (e.g. the MNCS), but its results are favorable compared to two citing-side indicator variants (SNCS indicators). Taken as a whole, the results of this study confirm the ability of established indicators to field-normalize citations.

The elephant in the room: The problem of quantifying productivity in evaluative scientometrics

In a critical and provocative paper, Abramo and D'Angelo claim that commonly used scientometric indicators such as the mean normalized citation score (MNCS) are completely inappropriate as indicators of scientific performance. Abramo and D'Angelo argue that scientific performance should be quantified using indicators that take into account the productivity of a research unit. We provide a response to Abramo and D'Angelo, indicating where we believe they raise important issues, but also pointing out where we believe their claims to be too extreme.

A farewell to the MNCS and like size-independent indicators

The arguments presented demonstrate that the Mean Normalized Citation Score (MNCS) and other size-independent indicators based on the ratio to publications are not indicators of research performance. The article provides examples of the distortions when rankings by MNCS are compared to those based on indicators of productivity. The authors propose recommendations for the scientometric community to switch to ranking by research efficiency, instead of MNCS and other size-independent indicators.

Normalization of Mendeley reader counts for impact assessment

A different number of citations can be expected for publications appearing in different subject categories and publication years. For this reason, the citation-based normalized indicator Mean Normalized Citation Score (MNCS) is used in bibliometrics. Mendeley is one of the most important sources of altmetrics data. Mendeley reader counts reflect the impact of publications in terms of readership. Since a significant influence of publication year and discipline has also been observed in the case of Mendeley reader counts, reader impact should not be estimated without normalization. In this study, all articles and reviews of the Web of Science core collection with a publication year of 2012 (and a DOI) are used to normalize their Mendeley reader counts. A new indicator that determines the normalized reader impact is obtained –the Mean Normalized Reader Score (MNRS) – and compared with the MNCS. The MNRS enables us to compare the impact a paper has had on Mendeley across subject categories and publication years. Comparisons on the journal and university level show that the MNRS and MNCS correlate larger for 9601 journals than for 76 German universities.

National, disciplinary and temporal variations in the extent to which articles with more authors have more impact: Evidence from a geometric field normalised citation indicator

The importance of collaboration in research is widely accepted, as is the fact that articles with more authors tend to be more cited. Nevertheless, although previous studies have investigated whether the apparent advantage of collaboration varies by country, discipline, and number of co-authors, this study introduces a more fine-grained method to identify differences: the geometric Mean Normalized Citation Score (gMNCS). Based on comparisons between disciplines, years and countries for two million journal articles, the average citation impact of articles increases with the number of authors, even when international collaboration is excluded. This apparent advantage of collaboration varies substantially by discipline and country and changes a little over time. Against the trend, however, in Russia solo articles have more impact. Across the four broad disciplines examined, collaboration had by far the strongest association with impact in the arts and humanities. Although international comparisons are limited by the availability of systematic data for author country affiliations, the new indicator is the most precise yet and can give statistical evidence rather than estimates.

Expected number of citations and the crown indicator

Abstract The mean normalized citation score or crown indicator is a much studied bibliometric indicator that normalizes citation counts across fields. We examine the theoretical basis of the normalization method and, in particular, the determination of the expected number of citations. We observe a theoretical bias that raises the expected number of citations for low citation fields and lowers the expected number of citations for high citation fields when interdisciplinary publications are included.

Multiplicative versus fractional counting methods for co-authored publications. The case of the 500 universities in the Leiden Ranking

This paper studies the assignment of responsibility to the participants in the case of co-authored scientific publications. In the conceptual part, we establish that one shortcoming of the full counting method is its incompatibility with the use of additively decomposable citation impact indicators. In the empirical part of the paper, we study the consequences of adopting the address-line fractional or multiplicative counting methods. For this purpose, we use a Web of Science dataset consisting of 3.6 million articles published in the 2005–2008 period, and classified into 5119 clusters. Our research units are the 500 universities in the 2013 edition of the CWTS Leiden Ranking. Citation impact is measured using the Mean Normalized Citation Score, and the Top 10% indicators. The main findings are the following. Firstly, although a change of counting methods alters co-authorship and citation impact patterns, cardinal differences between co-authorship rates and between citation impact values are generally small. Nevertheless, such small differences generate considerable re-rankings between universities. Secondly, the universities that are more favored by the adoption of a fractional rather than a multiplicative approach are those with a large co-authorship rate for the citation distribution as a whole, a small co-authorship rate in the upper tail of this distribution, a large citation impact performance, and a large number of solo publications.

Ranking institutions within a university based on their scientific performance: A percentile-based approach

Over the recent years, the subject of university rankings has attracted a significant amount of attention and sparked a scientific debate. However, few studies on this topic focus on elaborating the scientific performance of universities’ institutions, such as institutes, schools, and faculties. For this reason, the aim of this study is to design an appropriate framework for evaluating and ranking institutions within a university. The devised methodology ranks institutions based on the number of published papers, mean normalized citation score (MNCS), and four percentile-based indicators using the I-distance method. We applied the proposed framework and scrutinized the University of Belgrade (UB) as the biggest and the best-ranked university in Serbia. Thus, 31 faculties and 11 institutes were compared. Namely, an in-depth percentile-based analysis of the UB papers indexed in the Science Citation Index Expanded (SCIe) and the Social Science Citation Index (SSCI) for the period 2008-2011 is provided. The results clearly show considerable discrepancies in two occasions: first, when it comes to the question of leading author, and second, when it comes to analyzing the percentile rank classes (PRs) of groups of faculties.

List of Reviewers

This paper studies the assignment of responsibility to the participants in the case of co-authored scientific publications. In the conceptual part, we establish that one shortcoming of the full counting method is its incompatibility with the use of additively decomposable citation impact indicators. In the empirical part of the paper, we study the consequences of adopting the address-line fractional or multiplicative counting methods. For this purpose, we use a Web of Science dataset consisting of 3.6 million articles published in the 2005–2008 period, and classified into 5119 clusters. Our research units are the 500 universities in the 2013 edition of the CWTS Leiden Ranking. Citation impact is measured using the Mean Normalized Citation Score, and the Top 10% indicators. The main findings are the following. Firstly, although a change of counting methods alters co-authorship and citation impact patterns, cardinal differences between co-authorship rates and between citation impact values are generally small. Nevertheless, such small differences generate considerable re-rankings between universities. Secondly, the universities that are more favored by the adoption of a fractional rather than a multiplicative approach are those with a large co-authorship rate for the citation distribution as a whole, a small co-authorship rate in the upper tail of this distribution, a large citation impact performance, and a large number of solo publications.

Methods for the generation of normalized citation impact scores in bibliometrics: Which method best reflects the judgements of experts?

Abstract Evaluative bibliometrics compare the citation impact of researchers, research groups and institutions with each other across time scales and disciplines. Both factors, discipline and period – have an influence on the citation count which is independent of the quality of the publication. Normalizing the citation impact of papers for these two factors started in the mid-1980s. Since then, a range of different methods have been presented for producing normalized citation impact scores. The current study uses a data set of over 50,000 records to test which of the methods so far presented correlate better with the assessment of papers by peers. The peer assessments come from F1000Prime – a post-publication peer review system of the biomedical literature. Of the normalized indicators, the current study involves not only cited-side indicators, such as the mean normalized citation score, but also citing-side indicators. As the results show, the correlations of the indicators with the peer assessments all turn out to be very similar. Since F1000 focuses on biomedicine, it is important that the results of this study are validated by other studies based on datasets from other disciplines or (ideally) based on multi-disciplinary datasets.

Sub-field normalization in the multiplicative case: Average-based citation indicators

This paper investigates the citation impact of three large geographical areas – the U.S., the European Union (EU), and the rest of the world (RW) – at different aggregation levels. The difficulty is that 42% of the 3.6 million articles in our Thomson Scientific dataset are assigned to several sub-fields among a set of 219 Web of Science categories. We follow a multiplicative approach in which every article is wholly counted as many times as it appears at each aggregation level. We compute the crown indicator and the Mean Normalized Citation Score (MNCS) using for the first time sub-field normalization procedures for the multiplicative case. We also compute a third indicator that does not correct for differences in citation practices across sub-fields. It is found that: (1) No geographical area is systematically favored (or penalized) by any of the two normalized indicators. (2) According to the MNCS, only in six out of 80 disciplines – but in none of 20 fields – is the EU ahead of the U.S. In contrast, the normalized U.S./EU gap is greater than 20% in 44 disciplines, 13 fields, and for all sciences as a whole. The dominance of the EU over the RW is even greater. (3) The U.S. appears to devote relatively more – and the RW less – publication effort to sub-fields with a high mean citation rate, which explains why the U.S./EU and EU/RW gaps for all sciences as a whole increase by 4.5 and 5.6 percentage points in the un-normalized case. The results with a fractional approach are very similar indeed.

A recursive field-normalized bibliometric performance indicator: an application to the field of library and information science

Two commonly used ideas in the development of citation-based research performance indicators are the idea of normalizing citation counts based on a field classification scheme and the idea of recursive citation weighing (like in PageRank-inspired indicators). We combine these two ideas in a single indicator, referred to as the recursive mean normalized citation score indicator, and we study the validity of this indicator. Our empirical analysis shows that the proposed indicator is highly sensitive to the field classification scheme that is used. The indicator also has a strong tendency to reinforce biases caused by the classification scheme. Based on these observations, we advise against the use of indicators in which the idea of normalization based on a field classification scheme and the idea of recursive citation weighing are combined.