Pitfalls in aggregating performance measures in higher education

Abstract

National and international rankings of universities are now an accepted part of the higher education landscape. Rankings aggregate different performance measures into a single scale and therefore depend on the methods and weights used to aggregate. The most common method is to scale each variable relative to the highest performing entity prior to aggregating. Other approaches involve transforming the data to allow for the different spread of the variables. This paper evaluates alternative methods and the sensitivity to weights with illustrations from the Times Higher Education and Shanghai Jiao Tong rankings of universities and the U21 rankings of national systems of higher education. The authors conclude that transforming the data clouds interpretation; the choice of included variables is more important than the weights attached to them; and there are limitations in extending ranking to a large number of universities/countries.