Academic quality, league tables, and public policy: A cross-national analysis of university ranking systems

PurposeThe higher education system has been entrusted globally to provide quality education, especially to the youth, and equip them with required skills and capabilities. The visionaries and policymakers of the countries around the world have been working relentlessly to improve the standard of the higher education system by establishing national and global accreditation and ranking bodies and expecting measuring performance through setting up accreditation and ranking parameters. This paper focuses on the review of Indian university accreditation and ranking system and determining its efficacy in improving academic quality for achieving good position in global quality accreditation and ranking.Design/methodology/approachThe study employed exploratory research approach to know about the accreditation and ranking issues of Indian higher education institutions to overcome the challenges for being globally competitive. The accreditation and ranking parameters and score of leading Indian universities was collected from secondary data sources. Similarly, the global ranking parameters and scores of these Indian universities with top global universities was explored. The performance gaps of Indian university in global academic quality parameter is assessed by comparing it with scores of global top universities. Further, each domestic and global accreditation and ranking parameters have been taken up for discussion.FindingsThe study identified teaching and learning, research and industry collaboration as common parameter in the accreditation and ranking by Indian and global accreditation and ranking body. Furthermore, the study revealed that Indian accreditation and ranking body assess leniently on parameters and award high scores as compared to rigorous global accreditation and ranking practice. The study revealed that “research” and “citations” are important parameters for securing prestigious position in global ranking, this is the reason Indian universities are trailing. The study exposed that Indian academic fraternity lack prominence in research, publication and citations as per need of global accreditation and ranking standards.Research limitations/implicationsThe limitation of this study is that it focused only on few Indian and global accreditation and ranking bodies. The future implication of this study will be the use of methodology designed in this study for comparing accreditation and ranking bodies’ parameters of different continents and countries in different economic development stages i.e. emerging and developed economies to know the disparity and shortcomings in their higher education system.Practical implicationsThe article is a review and comparison of national and global accreditation and ranking parameters. The article explored the important criteria and key indicators of accreditation and ranking that would provide an important and meaningful insight to academic institutions of the emerging economies of the world to develop its competitiveness. The study contributed to the literature on identifying benchmark for improving academic and higher education institution quality. This study would be further helpful in fostering new ideas toward setting up of contemporary globally viable and acceptable academic quality standard.Originality/valueThis is possibly the first study conducted with novel methodology of comparing the Indian and global accreditation and ranking parameters to identify the academic quality performance gap and suggesting ways to attain academic benchmark through continuous improvement activity and process for global competitiveness.

Recent interest towards university rankings has led to the development of several ranking systems at national and global levels. Global ranking systems tend to rely on internationally accessible bibliometric databases and reputation surveys to develop league tables at a global level. Given their access and in-depth knowledge about local institutions, national ranking systems tend to include a more comprehensive set of indicators. The purpose of this study is to conduct a systematic comparison of national and global university ranking systems in terms of their indicators, coverage and ranking results. Our findings indicate that national rankings tend to include a larger number of indicators that primarily focus on educational and institutional parameters, whereas global ranking systems tend to have fewer indicators mainly focusing on research performance. Rank similarity analysis between national rankings and global rankings filtered for each country suggest that with the exception of a few instances global rankings do not strongly predict the national rankings.

IntroductionConcerns about reproducibility and impact of research urge improvement initiatives. Current university ranking systems evaluate and compare universities on measures of academic and research performance. Although often useful for marketing purposes, the value of ranking systems when examining quality and outcomes is unclear. The purpose of this study was to evaluate usefulness of ranking systems and identify opportunities to support research quality and performance improvement.MethodsA systematic review of university ranking systems was conducted to investigate research performance and academic quality measures. Eligibility requirements included: inclusion of at least 100 doctoral granting institutions, be currently produced on an ongoing basis and include both global and US universities, publish rank calculation methodology in English and independently calculate ranks. Ranking systems must also include some measures of research outcomes. Indicators were abstracted and contrasted with basic quality improvement requirements. Exploration of aggregation methods, validity of research and academic quality indicators, and suitability for quality improvement within ranking systems were also conducted.ResultsA total of 24 ranking systems were identified and 13 eligible ranking systems were evaluated. Six of the 13 rankings are 100% focused on research performance. For those reporting weighting, 76% of the total ranks are attributed to research indicators, with 24% attributed to academic or teaching quality. Seven systems rely on reputation surveys and/or faculty and alumni awards. Rankings influence academic choice yet research performance measures are the most weighted indicators. There are no generally accepted academic quality indicators in ranking systems.DiscussionNo single ranking system provides a comprehensive evaluation of research and academic quality. Utilizing a combined approach of the Leiden, Thomson Reuters Most Innovative Universities, and the SCImago ranking systems may provide institutions with a more effective feedback for research improvement. Rankings which extensively rely on subjective reputation and “luxury” indicators, such as award winning faculty or alumni who are high ranking executives, are not well suited for academic or research performance improvement initiatives. Future efforts should better explore measurement of the university research performance through comprehensive and standardized indicators. This paper could serve as a general literature citation when one or more of university ranking systems are used in efforts to improve academic prominence and research performance.

In the policy process of higher education, the higher education institutions not only play a role in the policy making process, but also are directly affected by the policies toward building quality culture in their organizations. From this perspective, the paper uses the policy cycle model combined with the Multiple Streams Framework (MSF) of Kingdon (1984) and the quality culture model of the European Union Association (EUA) in order to analyze, clarify and propose groups of solutions to enhance the critical role of higher education institutions in policy process.
 Keywords
 Policy entrepreneurs; quality culture; multiple streams framework
 References
 [1] Viennet, Romane, and Beatriz Pont. "Education Policy Implementation: A Literature Review and Proposed Framework. OECD Education Working Papers, No. 162." OECD Publishing (2017). [2] Howells, J., R. Ramlogan, and S. L. Cheng, The role, context and typology of universities and higher education institutions in innovation systems: A UK perspective, Discussion Papers and Project Reports, Impact of Higher Education Institutions on Regional Economics: A Joint Research Initiative, 2008.[3] Dill, David D, and Maarja Soo, Academic quality, league tables, and public policy: A cross-national analysis of university ranking systems, Higher education 49.4: 495-533, 2005.[4] Kraft, Michael E., and Scott R. Furlong. Public policy: Politics, analysis, and alternatives. Sage, 2012.[5] Kingdon, John W, Agendas, Alternatives and Public Policies, 2nd edition, New York and London, Longman, 2003.[6] Weick, Karl E, The Social Psychology of Organizing, 2nd ed, New York, Random House, 1979.[7] Feldman, Martha S, Order without Design: Information Production and Policy Kaking, Vol. 231, Stanford University Press, 1989.[8] March, James G, Primer on Decision Making: How Decisions Happen, Simon and Schuster, 1994.[9] Wilson, James Q, Bureaucracy: What Government Agencies Do and Why They Do It, Basic Books, 1989.[10] Chow, Anthony, Understanding policy change: multiple streams and national education curriculum policy in Hong Kong, Journal of Public Administration and Governance 4.2 (2014) 49.[12] Zhou, Nan, and Feng Feng, Applying Multiple Streams Theoretical Framework to college matriculation policy reform for children of migrant workers in China, Public Policy and Administration Research 4.10 (2014) 1.[12] Ha, Bui Thi Thu, Tolib Mirzoev, and Maitrayee Mukhopadhyay, Shaping the health policy agenda: the case of safe motherhood policy in Vietnam, International journal of health policy and management 4.11 (2015) 741.[13] Kane, Sumit, The Health Policy Process in Vietnam: Going Beyond Kingdon’s Multiple Streams Theory: Comment on “Shaping the Health Policy Agenda: The Case of Safe Motherhood Policy in Vietnam”, International journal of health policy and management 5.7 (2016) 435.[14] European University Association, Quality Culture in European Countries: A Bottom-Up Approach, EUA Publications, 2006. [15] Liên hiệp các hội khoa học và kỹ thuật Việt Nam, Bản kiến nghị đề xuất một số biện pháp nhằm tiến tới cải cách triệt để và toàn diện nền giáo dục VN, 2005.[16] Bingham, Lisa Blomgren, Tina Nabatchi, and Rosemary O'Leary, The New Governance: Practices and Processes for Stakeholder and Citizen Participation in the Work of Government, Public Administration Review 65.5 (2005) 547.

This chapter intends to explain how global university rankings can be understood as a mechanism holding Taiwan’s interests within the context of the emergence of an international higher education market and the prospect of regionalisation in East Asia. To illustrate Taiwan’s interests in university ranking systems, the chapter argues that league tables can be used to promote Taiwan’s interests in three ways. Firstly, it pointed out that university rankings have been taken by the Taiwanese government as a metric system to indicate the standard of universities, thereby reflecting their distance from the status of world-class university. In this sense, rankings are used as a governing tool to align the architecture of Taiwan’s higher education system, thereby advancing its competitiveness. Secondly, university rankings are seen as a zoning technology promoting the growing trends toward regionalisation of higher education in East Asia. Thirdly, university rankings are considered as a mechanism of agenda setting promoting the discourses of Chineseness in global higher education. These two anticipations are developed based on the context of China’s rise and the emergence of the idea of the Greater China in higher education. They are involved in Taiwan’s interests, as it is believed that the Taiwanese higher education sector can plausibly extend its influences in the process of regionalisation.

Theoretical and methodological approaches to the evaluation of the quality of higher education WITHIN THE CONTEXT OF the world's higher education institutions rating S А. V. Kaydalovа, О. V. Posylkina National pharmaceutical University,Kharkov Summary : In the article we have reviewed the world rankings of the higher education institutions; principles and methodology of ratings, their key indicators, the indices of ranking the higher education institutions and their role in assessing the quality of education. Keywords : quality of education, world rankings of the higher education institutions, indicating indices in determining ratings, criteria for evaluation of the quality level. Іntroduction. The question of the participation of local educational institutions, including medical and pharmaceutical ones, in the world rankings is regarded as a serious problem during the recent years, which is widely discussed inUkraine. At the present stage of development of the higher education the ratings of HEI turn to be seen not only as a means of competitiveness, but also as instruments of assurance of the higher educational quality. To form the national system of ranking of the universities, including medical and pharmaceutical profile, it is important to analyze the international experience of building different ratings. In the system of providing and evaluating the quality of education, the internal and external monitoring of the university plays an important role. If the internal monitoring - is an assessment by the university of its own activity, the external monitoring - is an assessment of the quality of education by the state, society and the educational environment. The aim of this publication is to analyze the methodology of forming global university rankings of the universities, formulation of guidelines and indicators of ranking, the study of the specificity of the external evaluation of the quality of education to strengthen the international domestic medical and pharmaceutical universities that train future specialists in the pharmaceutical field. Results and discussion. Analysis of foreign experience has highlighted that the processes of formation, development and improvement of the world educational systems have been developing in different ways at the international level. In scientific sources it is noted that the European system of the quality of education is based on the standards and recommendations, the principles of which are: the interest of students and employers in the quality of education, autonomy of institutions, internal and external quality of assurance of educational services [3, 5, 11]. It should be noted that the history of the world university ranking was absent up to the eighties. This was due to the lack of competition, both in domestic and foreign educational space. The first step in conducting the external evaluation of the university was the publication by the magazine US News & World Report in 1983 the first in the world ranking of universities, which launched the process of globalization of the higher education. The main purpose of this rating was to provide applicants with information. In aspects of the studing problem it has been found out that for the time being there are more than 50 national and over 10 international ratings for the evaluation of universities [1, 8]. The aim of international ratings is to determine the best universities in the world and evaluation of their activities, but each rating involves the use of its own indices to determine the competitive potential in the universities. During our research we have analyzed the most famous and internationally recognized the global systems of monitoring and ranking of universities and compiled a chronology of the world rankings and summarized their main quantitative characteristics. According to the analysis of the official sources [12-25] it has been found out that the world university rankings have both general trends and significant differences. Common principles of ratings are the following ones: consideration of different indicators with their further grouping due to the validity coefficients, which are determined in each rating individually and also principles of ranking of universities without taking into account their scientific and educational activities. Сonclusions The results of the research methodology and the formation of the international ratings of higher educational institutions showed, first of all, the multiplicity of approaches to assessing the quality of education, variety of criteria for assessing the quality of education, lack of scientifically validated studies of indicators ranking. To ensure quality at the national level and the University level, including pharmaceutical and medical profile, the formation of certain ratings to assess the activities of domestic universities, which requires further research methodology of domestic and national rankings of various countries based on the experience of the world rankings and the use of the most important indicators in the construction of the system of internal monitoring of the activities of the university.

Introduction. In the context of globalisation and internationalisation of higher education, university rankings are becoming an important tool for assessing the quality of education received by students at various higher education institutions around the world. These processes actualise the issues of possibilities for practical use of methodologies for calculation of global and national university rankings.The aim of the study was to develop and apply a methodological approach to multivariate the analysis of Higher Education Institutions (HEIs) classification procedures, to construct and analyse aggregated indicators for global and national rating systems of higher education organisations, and to assess the relationship between them.Methodology and research methods. The current paper presents a system analysis of databases of rating systems and an aggregation of independent evaluations of global and national rankings of HEIs using the methodology of league table analysis based on mathematical apparatus of the voting theory. The dependence of global and national university rankings indicators was investigated using correlation, cluster, factor, regression (linear and polynomial) and dispersion methods of analysis.Results and scientific novelty. A comprehensive comparative analysis of ranking systems and their results was carried out. The authors solved the problem of aggregating multiple heterogeneous studies of global and national ranking systems with their qualitative and quantitative variety of criteria, indicators and methods of assessment. The correlation between the indicators of aggregated global and national rankings was revealed; the regression dependence of the integral national ranking with the results of the leading global rating systems was determined.Practical significance. The developed methodical approach is a convenient and effective mechanism for comprehensive monitoring of the members of educational process.

Purpose – The purpose of this paper is to examine the role of university ranking systems as instruments of university quality assessment. Some controversy surrounds the methodology used to compile such instruments. Accordingly, different compilers have adopted different methods to produce these rankings. This study examines to what extent this diversity in methodology is now converging in the context of Spanish university rankings. Design/methodology/approach – To conduct this research, a two-step approach was adopted. First, the indicators used in four Spanish rankings were examined. Second, empirical analysis was used to identify differences between university rankings. Findings – Results reveal that, despite the vast number and variety of indicators, there is a positive, significant relationship between rankings. Spanish university rankings thus show some degree of convergence. Social implications – Because rankings influence behavior and shape institutional decision making, a better understanding of how these assessment tools are devised is essential. Research on these ranking systems therefore offers an important contribution to improving the quality of higher education institutions. Originality/value – This paper presents the results of a comprehensive survey of Spanish university rankings. It offers a new perspective of the state of the art of the Spanish university ranking system. The paper also presents a set of managerial implications for improving these benchmarking tools.

Quality teaching and learning is critical to producing high caliber University graduates equipped with the knowledge, skills and values to contribute to the knowledge economy and the economic growth and development of countries while ensuring self-efficacy and personal success. Teaching quality measures and indicators have, however, not enjoyed adequate debate and discourse within the higher education sector, and, as such are largely quantitative and measured by proxy in University ranking systems. Proxy teaching indicators used by the Quacquarelli Symonds (QS) World University Rankings and the Times Higher Education (THE) World University Rankings were correlated with U-Multirank indicators applicable to the Faculty of Health Sciences for the period 2007-2011, with the Faculty considered as a microcosm of the University of KwaZulu-Natal. There were no statistically significant differences in the indicators between years. There were just 2 significant correlations, viz., ratio of PhD to Bachelors degrees awarded significantly correlated with throughput from cohort at 95% and 99% while the number of PhDs significantly correlated with graduate employment at 90%. Teaching quality measurement by proxy is thus justifiably contested in University rankings. The challenge for University ranking systems is thus (1) identifying suitable quantitative and qualitative indicators for quality teaching, (2) striking the correct balance between quantitative and qualitative teaching quality indicators, and (3), ensuring that the quantitative/ qualitative indicators address both teaching inputs and teaching impact/learning outcomes.

University Ranking Systems and Proposal of a Theoretical Framework for Ranking of Turkish Universities: A Case of Management Departments

University rankings or ”league tables” a novelty as recently as 15 years ago are today a standard feature in most countries with large higher education systems. We discussed 28 sets of league tables from around the world. In this paper, we update the Usher and Savino results by recording changes in methodology in a few of these rankings, as well as providing data on nine new systems of rankings. All told, twenty-two of these are ”national” league table collected from fifteen countries (Australia, Canada, Chile, China, Hong Kong, Kazakhstan, Italy, the Netherlands, Peru, Poland, Spain, Taiwan, the Ukraine, the United Kingdom and the United States); while four are ”international” or ”cross-national” league tables. Specifically, the paper compares these league tables in terms of their methods of data collection and their selection and weighting of indicators. It also looks at three other systems (the German CHE rankings, the SwissUp rankings and the Canadian University Navigator rankings produced by the Globe and Mail and the Educational Policy Institute) which do not conform to the standard league table ”rules”. Finally, the paper surveys some of the more recent changes in ranking systems around the world and examines the implications of these changes for those parts of the world where ranking is still in its infancy.

Nationally and internationally, universities are ranked in university league tables (ULTs). Sustained academic criticism of the rationale and methodology of compiling ULTs has not stopped these rankings exerting considerable pressure on the decisions of university managers. The compilation of ULTs is an inherently political act, with the choice and weighting of metrics resulting in particular characteristics of individual institutions being rewarded or penalised. One aspect that is currently not considered by league tables is the diversity of the student intake, and the extent to which an institution has been successful in widening participation (WP) in higher education (HE). The need to take action is reflected in target 4.3 of the fourth United Nations Sustainable Development Goal (SDG 4), which aims to “ensure equal access for all women and men to affordable and quality technical, vocational and tertiary education, including university” by 2030. This article explores how current ULT metrics for universities in the United Kingdom (UK) relate to WP. Using publicly available data, the authors found that over 75% of UK league table metrics are negatively related to WP. This has the effect of making institutions with a diverse student body significantly more likely to be lower down in the league tables. The worst relationship with WP is for entry standards. Universities which recruit high-performing students are actively rewarded in the league tables; this fails to recognise that students with high entry grades are more likely to come from privileged backgrounds. The authors developed a ULT which includes a WP score as an explicit league table metric and found that their WP-adjusted table removed the negative relationship between WP and league table rank, resulting in a somewhat fairer comparison between universities. They conclude that ULT compilers have an ethical duty to improve their definition of a “good” university, which in the current HE environment of the UK must include WP. The authors believe this should be an urgent priority for the sector, so that universities with a commitment to widening participation can be recognised and rewarded.

Recent studies related to the influence and impact of the World University Ranking Systems on Higher Education conclude that the league tables have been mostly used for promotional and reputational purposes. The author argues in this paper that the rankings cannot fully impact Higher Education policy making unless the universities go beyond the overall scores and use the various individual indicators behind the rankings. The author proposes a 3-step approach allowing universities to benchmark and compare with peer world universities (at the institutional, field and subfield levels), measuring up against global benchmarks and positioning strategically their institution. Rankings could then be used as powerful diagnosis tools, effective guides for specific goal settings and strategic devices for Global Higher Education.

In recent years the development and use of university rankings, comparisons, and/or league tables has become popular and several methodologies are now frequently used to provide a comparative ranking of universities. These rankings are often based on research and publication activity and also not uncommonly focus on indicators that can be measured rather than those that should be measured. Further, the indicators are generally examined for the university as a whole rather than for university divisions, departments or programs. Implicit also is that placement in the rankings is indicative of quality. This paper provides an overview of the methodologies used for the more popular rankings and summarizes their strengths and weaknesses. It examines the critiques of rankings and league tables to provide appropriate context. The paper then examines the issue of how a university (or a college or program) could be assessed in terms of the quality of its engineering and technology programs. It proposes a set of indicators that could be used to provide relative measures of quality, not so much for individual engineering or technology programs, but rather of the university. Introduction & Methodology Today's world, and by all indicators the world of the future, seems to be increasingly competitive [1] and demanding. Resource scarcity, an increasing imperative for efficiency and effectiveness, manifestly more available information and escalating expectations for quality are but some of the factors that have caused universities, colleges, departments and programs to attend to evaluation, accreditation and invariably rankings and comparisons [2, 3]. Furthermore, increased global and intra-national mobility as well as widespread access to information has created the opportunity for individuals to more carefully research their selection of universities to attend . Perhaps in response to such pressures, there seems to have been an upsurge in the number of agencies, centers, corporations and others concerned with rankings and comparisons (see Appendix A). The International Observatory on Academic Ranking and Excellence (IREG), The Institute for Higher Education Policy (IHEP), The University of Illinois Education and Social Science library has compiled an extensive set of resources on rankings, which are reproduced in the appendices with permission. There have been numerous conferences addressing this topic as well [5, 6]. Notably, many of the most significant players in the ranking/comparison field have agreed upon a formal set of principles that define quality and good practice for rankings and comparisons . These are presented in Appendix B. The authors, in collaboration with their university reference librarians and institutional researchers, conducted an extensive review of the periodical, book, and conference literature. This activity surfaced over 20 different ranking/rating/comparison schemes with significant presence [samples are provided in Appendices C and D] and undoubtedly a multitude of additional ones exist. But, the authors are compelled to ask – What purposes are served by such comparisons [3, 8, 9, 10] and why so many? In terms of methodology, this paper resulted from a Search ➮ Identify ➮ Analyze ➮ Synthesize ➮ Report approach. This began with the co-authors generating a concept map of the key ideas and terms central to their understanding of the problem – i.e., the misunderstandings and misuses of ranking and rating systems. Because the authors operated on both sides of the Atlantic, two significantly different contexts formed the backdrop to this study. The general concept map we used is shown below: These concepts were used to search the large array of databases, currently well over 100, accessible through the Purdue University Library portal. Conventional Boolean logic was employed. Similarly directed searches of contemporary literature occurred in Europe. To begin, it seemed prudent for the authors to begin by asking the prior question, namely to what end do universities exist? Why has society established universities? Here we discovered the root of our problem, i.e., the purposes served by universities are diverse, pluralistic, varied and sometimes contradictory . Among the purposes with critical mass are such purposes as: • Liberal education • Professional education • Knowledge development/research • Public service • Economic development A salient starting point should be an examination of the role and aims of the university. There is great diversity in higher education today, and many universities’ aims are quite different. Thus definition and contextual understanding are important. For example, the American philosopher Robert Paul Wolff speaking from the context of the Vietnam War, addressed the question of the role of the ideal university . He questioned whether the university should serve as a ‘training camp’ for professionals. Wolff directed his criticism against the ideal type of a university of professions towards its lack of intellectual inquiry and critique. He viewed the relationship between professional bodies and academic professionals as being inherently in conflict with the Rankings

University rankings play a crucial role in the higher education landscape, shaping perceptions of academic quality, research productivity, and institutional reputation. This paper provides a comprehensive technical analysis of university ranking methodologies, examining the metrics, methods, and implications of various ranking systems. We explore the diverse approaches used by ranking organisations, the strengths and limitations of different methodologies, and the impact of rankings on higher education institutions and stakeholders. By critically evaluating ranking methodologies and discussing emerging trends and challenges, this paper aims to provide insights into the complex landscape of university rankings and inform discussions on their relevance and utility in the global higher education community.