Berlin Principles Research Articles

The Methodology of Rating Methodologies: The Problem of Constructing Correct University Ratings

The article is devoted to the substantiation of the general (framework) methodology for creating correct methodologies of university ranking, which capable of correctly (truly, in accordance with reality) ranking universities, primarily top ones. The conceptual foundations of such a methodology are formulated. It is based on a mission-based approach to the definition of university and ranking excellence, which involves the determination of the criterion-indicator base of the ranking both by its mission and by the mission of the university, especially a globally leading institution. Essential definitions of relevant missions as key system-creating factors are given. A criterion for the formulation of the true mission is proposed. The principle of mission minimisation of the list of necessary and sufficient ranking criteria / indicators and the principle of their mission synergy for the unambiguous orientation of university development are substantiated. The general methodology prevents the de-specification of the ranking and university missions, the dissynergisation of ranking criteria and indicators, their divergence and fragmentation, and the disorienting action. This methodology covers methodologies both simple (mono) and complex (combined), as well as general (institutional) and sectoral (by academic subjects) rankings. The methodology built on these foundations will contribute to the adoption of rankings as objective and reliable evaluation tools, the effective use of the IREG Berlin principles for the creation of ranking methodologies, the acceleration of the formation of a university ranking culture and its legislative regulation in Ukraine, and the further development of science about universities. This is important for the development of a strategy for the formation of leading universities of the highest excellence (strategy-2), the implementation of the second dimension of the European higher education area in terms of achieving its competitiveness and attractiveness, the approaching victory of Ukraine in the war, the innovative recovery of the country, the establishment of Ukraine as a strong European integrated state.

Defining standards for rankings: An investigation of global university rankings according to the Berlin Principles

AbstractThe role of universities as the engines of knowledge‐based economies drives global internationalisation of higher education. This contributes to a competitive environment where higher education rankings indicate market value. Even though rankings are influential and are used a lot, ranking systems have been heavily critiqued. One of the problems is that there are few if any external checks on how rankings are created. The purpose of the study on which this article reports was to evaluate ranking systems. Within the scope of the study, we have sought to reveal to what extent current ranking systems comply with the Berlin Principles—prepared to create certain rules for rankings, and to ensure that rankings represent quality. A document analysis of publicly available documents online was carried out together with a review of printed and electronic publications on ranking systems. An evaluation form was prepared and used in this study for field experts to fill in. Findings show that ranking systems comply with the Berlin Principles in terms of methodology, transparency, and acceptability at a level that ranges from moderate to low. Overall, rankings do not consider differences in higher education and are not transparent about the processes by which rankings are developed. Rankings should for this reason be interpreted carefully and methodological weaknesses of rankings that can lead to false inferences should be recognised.

Sustainability university rankings: a comparative analysis of UI green metric and the times higher education world university rankings

PurposeThis study aims to answer the research question: How to evaluate the structure of global university sustainability rankings according to the Berlin Principles (BP) framework.Design/methodology/approachThe authors investigated two global sustainability rankings in universities, The UI green metric World University Ranking (WUR) and the Times Higher Education World University Rankings (THE-WUR). The authors performed content analysis regarding their evaluation criteria and assessed both rankings using the BP framework.FindingsResults show that there is still a gap to be filled regarding the specificity of global university sustainability rankings. Although the THE-WUR had a better performance in this research, there are several items for improvement, especially regarding the methodological procedures. There are structural differences, limitations and points for improvement in both rankings. Besides, it may not be possible to have a unique and more appropriate ranking, but one that can be more suitable for a contextual reality.Practical implicationsThis study can be helpful for university managers when deliberating on the most appropriate ranking for their institutions and better preparing their higher education institutions for participating in sustainability-related rankings. Besides, it suggests possible improvements on the rankings’ criteria.Social implicationsThe authors shed light on challenges for improving the existing university sustainability rankings, besides generating insights for developing new ones. In a provocative but constructive perspective, the authors question their bases and understandings of being “the best university” regarding sustainability.Originality/valueThis is the first study that provides an in-depth analysis and comparison between two of the most important global university sustainability rankings.

Kalitenin Standardını Belirlemek: Türk Üniversite Sıralama Sistemlerinin Berlin Prensiplerine Göre İncelenmesi

The increasing competitive environment arising from the commercialization and internationalization of higher education (HE) and the increasing influence of the liberal character of educational policies have led to the emergence of HE ranking systems which have become an inevitable part of HE with the effect they have created. The lack of unambiguous methodological processes is one of the oft-cited critical problems in the HE ranking system literature. These systems vary due to the particular methods they adopt in creating their ranking results. This variation requires a set of principles on how best to perform these rankings. The principles established under Berlin Principles in 2006 aimed to address this need. This study aims to determine to what extent Turkish ranking systems comply with the basic tenets determined in the Berlin Principles. The study was designed as a methodological assessment study and the data were obtained from the websites and printed and electronic publications related to Turkish ranking systems. The data obtained were scored according to an evaluation form consisting of the tenets set out in the Berlin Principles. The findings show that the majority of the Turkish university ranking systems were not designed to be fully compliant with the Berlin Principles and instead they evaluate HE institutions by their research approach and quality definitions.

The Berlin principles on one health – Bridging global health and conservation

For over 15-years, proponents of the One Health approach have worked to consistently interweave components that should never have been separated and now more than ever need to be re-connected: the health of humans, non-human animals, and ecosystems. We have failed to heed the warning signs. A One Health approach is paramount in directing our future health in this acutely and irrevocably changed world. COVID-19 has shown us the exorbitant cost of inaction. The time to act is now.

Evaluation of Environmental Sustainability Higher in Education Ranking Systems: Towards a Flat Intraranking System

This paper reviews different ranking criteria for universities practicing environmental sustainability – named here as “Environmental Sustainability Higher Education ranking systems (ESHERSs)”. The purpose is to evaluate the degree of such criteria in supporting the higher education institutions (HEIs) in achieving environmentally sustainable development goals (SDGs). In this paper, the potential metrics for ranking systems are identified to promote environmentally sustainable development practices within HEIs and support their home countries in achieving SDGs. Three ESHERSs - known in focusing on teaching & research, environment and social scope are evaluated and scored according to their compliance with the Berlin Principles (BPs), where the later are used to propose a new framework for an intra- ranking system that contribute to more effective practice in Environment and Education, i.e. developing University of Bahrain Environmental Sustainability Higher Education ranking system ( UoBESHERS).

Questioning the Shanghai Ranking methodology as a tool for the evaluation of universities: an integrative review

This integrative review reports on methodological questions about the Shanghai Ranking as a tool for the evaluation of universities, questions that are extensible to other rankings. The paper presents a list of methodological problems that are the result of both a review of the literature and the authors’ knowledge, with the aim of improving and refining the ranking in line with the Berlin Principles. The second section makes proposals and provides explanatory notes for improving the evaluation of university institutions. A final inference is that any educational changes undertaken based on conclusions drawn from an institution’s ranking position must be considered highly controversial and questionable.

The Berlin Principles in Russian University Rankings

At present, two national ratings of universities are published annually in Russia, and they should be incorporating the Berlin Principles of university ranking, adopted in 2006 by the International Rankings Expert Group (IREG). There are, however, specific national features involved in ranking Russian universities. What do they consist of? How are the Berlin Principles accounted for in domestic ratings methodologies? The author attempts to answer these questions.

Environmental sustainability of universities: critical analysis of a green ranking

Today, the principles of sustainability and sustainable development represent key points in policy development and activities of the Higher Education Institutions, both for their impact on the environment and the role they play in society. In the wake of the spread of rankings in Higher Education Institutions over the past decades, Universitas Indonesia in 2010 implemented an environmental sustainability ranking, called UI GreenMetric World University Ranking. The aim of this paper is to carry out a constructive analysis of the GreenMetric Ranking in order to improve and strengthen the ranking method. This analysis is based on the review of the scarce literature, the survey of questionnaire and on an evaluation of the guidelines that have been evolving over time. Despite the improvements made over time by the authors, the critical review of the methodology points out that the GreenMetric Ranking still falls short on some issues. Furthermore, the analysis shows the incomplete compliance with the Berlin Principles. The GreenMetric Ranking though, lays a good foundation for the incorporation of the principle of sustainability within the Higher Education Institutions and reflects the need to quantify the efforts towards sustainability. Therefore, there is the need to make this method more scientific and rigorous, suitable to plan sustainability policies in universities.

The Berlin Principles on Ranking Higher Education Institutions: limitations, legitimacy, and value conflict

University rankings have been widely criticized and examined in terms of the environment they create for universities. In this paper, I reverse the question by examining how ranking organizations have responded to criticisms. I contrast ranking values and evaluation with those practiced by academic communities. I argue that the business of ranking higher education institutions is not one that lends itself to isomorphism with scholarly values and evaluation and that this dissonance creates reputational risk for ranking organizations. I argue that such risk caused global ranking organizations to create the Berlin Principles on Ranking Higher Education Institutions, which I also demonstrate are decoupled from actual ranking practices. I argue that the Berlin Principles can be best regarded as a legitimizing practice to institutionalize rankings and symbolically align them with academic values and systems of evaluation in the face of criticism. Finally, I argue that despite dissonance between ranking and academic evaluation, there is still enough similarity that choosing to adopt rankings as a strategy to distinguish one’s institution can be regarded as a legitimate option for universities.

Open scholarship ranking of Chinese research universities

Universities and the members of their faculties, by means of open access, open education, and social media engagement, contribute to many publicly accessible resources of academic values, i.e., open scholarship. To encourage universities to contribute even more to open scholarship, in a more focused and sustainable way, the methodology of Open Scholarship Ranking (OSR) was constructed after a thorough examination and several adjustments based on the Berlin Principles on Ranking of Higher Education Institutions (hereinafter referred to as "the Berlin Principles"). The OSR has met most of the Berlin Principles, and new adjustments helped to improve its quality. A significant correlation has been observed between the OSR results of Chinese research universities and the results from existing comprehensive university rankings. The OSR provides an evaluation framework for universities' performance in open scholarship, and can be regarded as an acceptable way of ranking universities.

Evaluating UI GreenMetric as a tool to support green universities development: assessment of the year 2011 ranking

Over the last two decades, the university ranking has become a global phenomenon. Universitas Indonesia, in 2010 has developed an online world universities ranking, aimed to offer the portrait of the current conditions and policies related to green campus and sustainability in the universities all around the world. The aims of this paper are to introduce the development and improvement of UI GreenMetric and evaluate the implementation and result of the year 2011 ranking. The evaluation method used for the ranking performance was a descriptive and qualitative approach, and the Berlin Principles was employed to assess the conformity and quality of UI GreenMetric. The 2011 ranking results show an increment of the number of participating countries. There are more than 25 universities from the United States participated in this ranking. The main prevailing criterion achieved by many universities is the energy and climate change (about 2500 of 2800 maximum score). Most of the indicators in UI GreenMetric are in compliance with the Berlin Principles. The UI GreenMetric ranking provides opportunities for each university to examine their strength and weakness in promoting green university and sustainable development. Further comments and positive feedback from the participating universities will advance the UI GreenMetric ranking system.

What about excellence in teaching? A benevolent ranking of universities

Existing university rankings apply fixed and exogenous weights based on a theoretical framework, stakeholder or expert opinions. Fixed weights cannot embrace all requirements of a `good ranking' according to the Berlin Principles. As the strengths of universities differ, the weights on the ranking should differ as well. This paper proposes a fully nonparametric methodology to rank universities. The methodology is in line with the Berlin Principles. It assigns to each university the weights that maximize (minimize) the impact of the criteria where university performs relatively well (poor). The method accounts for background characteristics among universities and evaluates which characteristics have an impact on the ranking. In particular, it accounts for the level of tuition fees, an English speaking environment, size, research or teaching orientation. In general, medium sized universities in English speaking countries benefit from the benevolent ranking. On the contrary, we observe that rankings with fixed weighting schemes reward large and research oriented universities. Especially Swiss and German universities significantly improve their position in a more benevolent ranking.

Standards for quality? A critical appraisal of the Berlin Principles for international rankings of universities

This article discusses the principles developed to assure the quality of international ranking practices for higher education, the so-called Berlin Principles, and the role given to them in the higher education community. While the principles are generally regarded as proper quality assurance principles, they are problematic both in their content and form. This study examines the process leading up to the principles as well as their use, thereby highlighting some legitimacy problems with these principles and the potential impact this may have on the further development of the higher education field. The study is based on official documents and published material.

Ranking of rankings: benchmarking twenty-five higher education ranking systems in Europe

The purpose of this study is to evaluate the ranking practices of 25 European higher education ranking systems (HERSs). Ranking practices were assessed with 14 quantitative measures derived from the Berlin Principles on Ranking of Higher Education Institutions (BPs). HERSs were then ranked according to their degree of congruence with the BPs. Additionally, the three domains of methodology, transparency, and consumer-friendliness were proposed to underlie the BPs, and the measures were also applied for assessing HERSs regarding these domains. Results indicate that the cooperating HERSs by CHE (Germany), AQA (Austria) and swissUp (Switzerland) exhibit the highest congruence with the BPs. However, no European HERS demonstrates good overall congruence with the BPs, mainly due to methodological shortcomings. Results further show that HERSs compiled and published by non-profit research entities seem to perform better than the majority of HERSs published by for-profit news outlets. International comparisons indicate that HERSs published in German-speaking countries and the Netherlands tend to exhibit a higher congruence with the BPs. Overall, this study hopes to stimulate the improvement of ranking practices through benchmarking with existing exemplary models. The quantitative assessment tool further promises to be useful in explaining relative stability or change of higher education institutions in HERSs, as well as in helping to explain resource allocation pressures within higher education institutions.

Examining Major Rankings According to the Berlin Principles

While the ranking of higher education institutions (HEIs) has become more and more popular, there are increasing concerns about the quality of such ranking. In response to such legitimate expectations, in May 2006, the International Ranking Expert Group (IREG) developed and endorsed a guideline document – the Berlin Principles on Ranking of Higher Education Institutions (hereafter called the Berlin Principles), which consist of sixteen descriptive and prescriptive principles for ranking covering four aspects: the purpose and goal of ranking; the design and weighting of indicators; the collection and processing of data; and the presentation of ranking results. The purpose of this paper is to examine selected HEIs rankings according to the Berlin Principles as well as to propose concrete ‘Fourteen Criteria’ which, if followed, could enhance the quality of ranking.

The Complex Interplay Between Classification and Ranking of Colleges and Universities: Should the Berlin Principles Apply Equally to Classification?

‘Classification’ and ‘ranking’ are tools for comparing colleges and universities, albeit for different purposes. Although ‘classification’ focuses on identifying similarities rather than ordering institutions, it is closely intertwined with ‘ranking’, whether directly through its content, structure, and procedures, or indirectly through the ways it is used and understood by the wider community of users (e.g., researchers, policymakers, institutional leaders, etc.). For this reason, many of the standards, protocols, and best practices established for ranking systems apply as well to the case of classification. This paper examines the interplay between ‘classification’ and ‘ranking’ in some detail, and applies the Berlin Principles on Ranking of Higher Education Institutions which were adopted in May 2006 to the case of classification, focusing on the Carnegie Classification of Institutions of Higher Education.