Assessing research excellence: Evaluating the Research Excellence Framework

Mehmet Pinar,Timothy J Horne

doi:10.1093/reseval/rvab042

Mehmet Pinar, Timothy J Horne

Open Access

https://doi.org/10.1093/reseval/rvab042

Copy DOI

Abstract

AbstractPerformance-based research funding systems have been extensively used around the globe to allocate funds across higher education institutes (HEIs), which led to an increased amount of literature examining their use. The UK’s Research Excellence Framework (REF) uses a peer-review process to evaluate the research environment, research outputs and non-academic impact of research produced by HEIs to produce a more accountable distribution of public funds. However, carrying out such a research evaluation is costly. Given the cost and that it is suggested that the evaluation of each component is subject to bias and has received other criticisms, this article uses correlation and principal component analysis to evaluate REF’s usefulness as a composite evaluation index. As the three elements of the evaluation—environment, impact and output—are highly and positively correlated, the effect of the removal of an element from the evaluation leads to relatively small shifts in the allocation of funds and in the rankings of HEIs. As a result, future evaluations may consider the removal of some elements of the REF or reconsider a new way of evaluating different elements to capture organizational achievement rather than individual achievements.

Highlights

Performance-based research funding systems (PRFS) have multiplied since the United Kingdom introduced the first ‘Research Selectivity Exercise’ in 1986
We will examine the effect of the mainstream quality-related research (QR) funding distribution to different panels, units of assessment’ (UOAs), and higher education institutes (HEIs) in England with scenarios 1-4 compared to the official mainstream QR funding allocation
With scenario 1, £249 million and £811 million of mainstream QR funding is distributed based on the Research Excellence Framework (REF) 2014 performances in impact and output elements, indicating that an additional £37 million and £122 million is distributed in the impact and output pots compared to the official scenario, Note that HEIs within inner and outer London area receive 12% and 8% additional QR funding on top of their allocated mainstream QR funding but to examine the effect of the exclusion of alternative scenarios on the allocation of the mainstream QR funding, we do not consider the additional funding allocation that is based on the location of HEI

Summary

Introduction

Performance-based research funding systems (PRFS) have multiplied since the United Kingdom introduced the first ‘Research Selectivity Exercise’ in 1986. Within these relatively narrow boundaries, there is significant variation between both what is assessed in different PRFS, and how the assessment is made. With regards to ‘what’, some focus almost exclusively on research outputs, predominantly journal articles, whereas others, notably the UK’s Research Excellence Framework (REF), assess other aspects of research such as the impact of research and the research environment. With regards to ‘how’, some PRFS use exclusively or predominantly metrics such as citations whereas others use expert peer review, and others still a mix of both methods (Zacharewicz et al, 2019).

Methods

Results

Conclusion