Measuring the Impact of Research: Lessons from the UK's Research Excellence Framework 2014.

Gobinda Chowdhury,Kushwanth Koya,Pete Philipson,Pablo Dorta-González

doi:10.1371/journal.pone.0156978

Abstract

Impactful academic research plays a stellar role in society, pressing to ask the question of how one measures the impact created by different areas of academic research. Measuring the societal, cultural, economic and scientific impact of research is currently the priority of the National Science Foundation, European Commission and several research funding agencies. The recently concluded United Kingdom’s national research quality exercise, the Research Excellence Framework (REF) 2014, which piloted impact assessment as part of the overall evaluation offers a lens to view how impact of research in different disciplines can be measured. Overall research quality was assessed through quality of outputs, ‘impact’ and research environment. We performed two studies using the REF 2014 as a case study. The first study on 363 Impact Case Studies (ICSs) submitted in 5 research areas (UoAs) reveals that, in general, the impact scores were constructed upon a combination of factors i.e. quantity of quartile-one (Q1) publications, quantity and value of grants/income, number of researchers stated in the ICSs, spin-offs created, discoveries/patents and presentation of esteem data, informing researchers/ academics of the factors to consider in order to achieve a better impact score in research impact assessments. However, there were differences among disciplines in terms of the role played by the factors in achieving their overall scores for the ICSs. The outcome of this study is thus a set of impact indicators, and their relationship with the overall score of impact of research in different disciplines as determined in REF2014, which would in the first instance provide some answers to impact measures that would be useful for researchers in different disciplines. The second study extracts the general themes of impact reported by universities by performing a word frequency analysis in all the ICSs submitted in the five chosen research areas, which were substantially varied owing to their fields.

Highlights

The challenge of research evaluationThe National Science Foundation, European Commission which administers Horizon 2020, the UK government’s funding bodies like EPSRC, ESRC etc., and several research funding agencies have called for measuring the impact of research which they support [1,2,3,4]presently, no generalised impact evaluation frameworks exist
The first study on 363 Impact Case Studies (ICSs) submitted in 5 research areas (UoAs) reveals that, in general, the impact scores were constructed upon a combination of factors i.e. quantity of quartile-one (Q1) publications, quantity and value of grants/ income, number of researchers stated in the ICSs, spin-offs created, discoveries/patents and presentation of esteem data, informing researchers/ academics of the factors to consider in order to achieve a better impact score in research impact assessments
The outcome of this study is a set of impact indicators, and their relationship with the overall score of impact of research in different disciplines as determined in REF2014, which would in the first instance provide some answers to impact measures that would be useful for researchers in different disciplines

Summary

Introduction

The challenge of research evaluationThe National Science Foundation, European Commission which administers Horizon 2020, the UK government’s funding bodies like EPSRC, ESRC etc., and several research funding agencies have called for measuring the impact of research which they support [1,2,3,4]presently, no generalised impact evaluation frameworks exist. Research impact evaluation remains a major challenge despite the massive investment in research [5,6,7] Database tools such as Science Citation Index, Web of Science, Scopus, Scholar, InCites, SciVal, h-index and Altmetrics make an effort to construct the quality of research through publication profile and citation profile or both [8,9,10,11]. These measures often remain problematic as a result of inadequate interpretations produced by mere numbers based on citation counts or access and/or download figures [12,13,14,15]. We still do not have an agreed method or measures for assessing the impact of research

Methods

Results

Conclusion