Leiden Ranking Research Articles

An insight into the importance of national university rankings in an international context: the case of the I-UGR rankings of Spanish universities

The great importance international rankings have achieved in the research policy arena warns against many threats consequence of the flaws and shortcomings these tools present. One of them has to do with the inability to accurately represent national university systems as their original purpose is only to rank world-class universities. Another one has to do with the lack of representativeness of universities' disciplinary profiles as they usually provide a unique table. Although some rankings offer a great coverage and others offer league tables by fields, no international ranking does both. In order to surpass such limitation from a research policy viewpoint, this paper analyzes the possibility of using national rankings in order to complement international rankings. For this, we analyze the Spanish university system as a study case presenting the I-UGR Rankings for Spanish universities by fields and subfields. Then, we compare their results with those obtained by the Shanghai Ranking, the QS Ranking, the Leiden Ranking and the NTU Ranking, as they all have basic common grounds which allow such comparison. We conclude that it is advisable to use national rankings in order to complement international rankings, however we observe that this must be done with certain caution as they differ on the methodology employed as well as on the construction of the fields.

Where Are the Global Rankings Leading Us? An Analysis of Recent Methodological Changes and New Developments

This article is based on the analysis of the changes in global university rankings and the new ‘products’ based on rankings data in the period since mid‐2011. It is a summary and continuation of the European University Association (EUA)‐commissioned report ‘Global University Rankings Their Impact, Report II’ which was launched in April 2013. It covers the changes in the ranking methodologies which have been the most visible in the CWTS Leiden Ranking and Webometrics and which have replaced some indicators with newly designed ones. Changes have been made in other rankings as well, but they are less visible. A new U21 ranking was launched in 2012. It is an attempt to rank national higher education systems rather than individual universities. New rankings by conventional ranking providers have demonstrated that in reputation rankings or reputation indicators the scores drop even more sharply than in the most élitist rankings and therefore can be used for even narrower groups of universities.Several ranking providers have started their own data collections and combine ranking data with the data from the newly established data collections and use them for several multi‐indicator classifications or profiling tools. QS has been most productive and has added not only classification and profiling tools, but has also launched a ranking of student cities, and ‘stars’ that universities can obtain.Generally, the rankings’ impact is growing. Let us see where it will bring us.At the same time, some rankings providers have changed language and explain the biases, flaws and misunderstandings created through misuse of rankings or using ranking indicators without proper knowledge.

What proportion of excellent papers makes an institution one of the best worldwide? Specifying thresholds for the interpretation of the results of the SCImago Institutions Ranking and the Leiden Ranking

University rankings generally present users with the problem of placing the results given for an institution in context. Only a comparison with the performance of all other institutions makes it possible to say exactly where an institution stands. In order to interpret the results of the SCImago Institutions Ranking (based on Scopus data) and the Leiden Ranking (based on Web of Science data), in this study we offer thresholds with which it is possible to assess whether an institution belongs to the top 1%, top 5%, top 10%, top 25%, or top 50% of institutions in the world. The thresholds are based on the excellence rate or PPtop 10%. Both indicators measure the proportion of an institution's publications which belong to the 10% most frequently cited publications and are the most important indicators for measuring institutional impact. For example, while an institution must achieve a value of 24.63% in the Leiden Ranking 2013 to be considered one of the top 1% of institutions worldwide, the SCImago Institutions Ranking requires 30.2%.

Informetric explanation of some Leiden Ranking graphs

The S‐shaped functional relation between the mean citation score and the proportion of top 10% publications for the 500 Leiden Ranking universities is explained using results of the shifted Lotka function. Also the concave or convex relation between the proportion of top 100θ% publications, for different fractions θ, is explained using the obtained new informetric model.

Multilevel‐statistical reformulation of citation‐based university rankings: The Leiden ranking 2011/2012

Since the 1990s, with the heightened competition and the strong growth of the international higher education market, an increasing number of rankings have been created that measure the scientific performance of an institution based on data. The Leiden Ranking 2011/2012 (LR) was published early in 2012. Starting from Goldstein and Spiegelhalter's (1996) recommendations for conducting quantitative comparisons among institutions, in this study we undertook a reformulation of the LR by means of multilevel regression models. First, with our models we replicated the ranking results; second, the reanalysis of the LR data showed that only 5% of the PPtop10% total variation is attributable to differences between universities. Beyond that, about 80% of the variation between universities can be explained by differences among countries. If covariates are included in the model the differences among most of the universities become meaningless. Our findings have implications for conducting university rankings in general and for the LR in particular. For example, with Goldstein‐adjusted confidence intervals, it is possible to interpret the significance of differences among universities meaningfully: Rank differences among universities should be interpreted as meaningful only if their confidence intervals do not overlap.

The Leiden ranking 2011/2012: Data collection, indicators, and interpretation

The Leiden Ranking 2011/2012 is a ranking of universities based on bibliometric indicators of publication output, citation impact, and scientific collaboration. The ranking includes 500 major universities from 41 different countries. This paper provides an extensive discussion of the Leiden Ranking 2011/2012. The ranking is compared with other global university rankings, in particular the Academic Ranking of World Universities (commonly known as the Shanghai Ranking) and the Times Higher Education World University Rankings. The comparison focuses on the methodological choices underlying the different rankings. Also, a detailed description is offered of the data collection methodology of the Leiden Ranking 2011/2012 and of the indicators used in the ranking. Various innovations in the Leiden Ranking 2011/2012 are presented. These innovations include (1) an indicator based on counting a university's highly cited publications, (2) indicators based on fractional rather than full counting of collaborative publications, (3) the possibility of excluding non‐English language publications, and (4) the use of stability intervals. Finally, some comments are made on the interpretation of the ranking and a number of limitations of the ranking are pointed out.

Testing differences statistically with the Leiden ranking

The Leiden ranking 2011/2012 provides the Proportion top-10% publications (PP top-10%) as a new indicator. This indicator allows for testing performance differences between two universities for statistical significance.

Severe language effect in university rankings: particularly Germany and France are wronged in citation-based rankings

We applied a set of standard bibliometric indicators to monitor the scientific state-of-arte of 500 universities worldwide and constructed a ranking on the basis of these indicators (Leiden Ranking 2010). We find a dramatic and hitherto largely underestimated language effect in the bibliometric, citation-based measurements of research performance when comparing the ranking based on all Web of Science (WoS) covered publications and on only English WoS covered publications, particularly for Germany and France.

Access, Usage and Citation Metrics: What Function for Digital Libraries and Repositories in Research Evaluation?

The growth and increasing complexity of global science poses a grand challenge to scientists: how to best organise the worldwide evaluation of research programmes and peers? For the 21st century we need not just information on science, but also meta-level scientific information that is delivered to the digital workbench of every researcher. Access, usage and citation metrics will be a major information service that researchers will need on an everyday basis to handle the complexity of science.Scientometrics has been built on centralised commercial databases of high functionality but restricted scope, mainly providing information that may be used for research assessment. Enter digital libraries and repositories: can they collect reliable metadata at the source, ensure universal metric coverage and defray costs? This systematic appraisal of the future role of digital libraries and repositories for metric research evaluation proceeds by investigating the practical inadequacies of current metric evaluation before defining the scope for libraries and repositories as new players. Subsequently, the notion of metrics as research information services is developed. Finally, the future relationship between a) libraries and repositories and b) metrics databases, commercial or non-commercial, is addressed.Services reviewed include Leiden Ranking, Webometrics Ranking of World Universities, Ranking Web of Repositories, COUNTER, MESUR, Harzing Publish or Perish, Hirsch Index, CiteSeer, Citebase, SPIRES, SSRN CiteReader, RePEc LogEc and CitEc, Scopus, Web of Science and Google Scholar.

On artificial stone chimney-pieces

We develop and propose a new counting method at the aggregate level for contributions to scientific publications called modified fractional counting (MFC). We show that, compared to traditional complete-normalized fractional counting, it eliminates the extreme differences in contributions over time that otherwise occur between scientists that mainly publish alone or in small groups and those that publish with large groups of co-authors. As an extra benefit we find that scientists in different areas of research turn out to have comparable average contributions to scientific articles. We test the method on scientists at Norway’s largest universities and find that, at an aggregate level, it indeed supports comparability across different co-authorship practices as well as between areas of research. MFC is thereby useful whenever the research output from institutions with different research profiles are compared, as e.g., in the Leiden Ranking. Finally, as MFC is actually a family of indicators, depending on a sensitivity parameter, it can be adapted to the circumstances.