Interpreting the outcomes of research assessments: A geometrical approach

Abstract

• Geometric approach for the comparison of frequency distributions of ordinal variables. • Assigning a single score to research assessments given as repartitions in merit classes of scientific publications. • Distance measure defined in the n-simplex following the path toward the best possible scenario in the frequency repartition. • Definition of a geometric-statistical score for building a fair ranking of institutions inhomogeneous in their composition. • Computation of the geometric score using Monte Carlo simulations. • Case study: application to the Italian Quality Research Evaluation. Research evaluations and comparison of the assessments of academic institutions (scientific areas, departments, universities etc.) are among the major issues in recent years in higher education systems. One method, followed by some national evaluation agencies, is to assess the research quality by the evaluation of a limited number of publications in a way that each publication is rated among n classes. This method produces, for each institution, a distribution of the publications in the n classes. In this paper we introduce a natural geometric way to compare these assessments by introducing an ad hoc distance from the performance of an institution to the best possible achievable assessment. Moreover, to avoid the methodological error of comparing non-homogeneous institutions, we introduce a geometric score based on such a distance. The latter represents the probability that an ideal institution, with the same configuration as the one under evaluation, performs worst. We apply our method, based on the geometric score, to rank, in two specific scientific areas, the Italian universities using the results of the research evaluation VQR 2011–2014.