UK Research Excellence Framework Research Articles

The UK Research Excellence Framework and the Matthew effect: Insights from machine learning.

With the high cost of the research assessment exercises in the UK, many have called for simpler and less time-consuming alternatives. In this work, we gathered publicly available REF data, combined them with library-subscribed data, and used machine learning to examine whether the overall result of the Research Excellence Framework 2014 could be replicated. A Bayesian additive regression tree model predicting university grade point average (GPA) from an initial set of 18 candidate explanatory variables was developed. One hundred and nine universities were randomly divided into a training set (n = 79) and test set (n = 30). The model “learned” associations between GPA and the other variables in the training set and was made to predict the GPA of universities in the test set. GPA could be predicted from just three variables: the number of Web of Science documents, entry tariff, and percentage of students coming from state schools (r-squared = .88). Implications of this finding are discussed and proposals are given.

The genre regime of research evaluation: contradictory systems of value around academics’ writing

This article addresses how academics navigate different kinds of prestige and different systems of value around what ‘counts’ in academic writing, focusing particularly on the impact of the genre regime associated with research evaluation in the UK Research Excellence Framework (REF). It draws on data from an Economic and Social Research Council (ESRC)-funded project working with academics across different disciplines and different institutions in England. We interviewed people about their writing practices several times, exploring their practices, life histories, institutional contexts, and the tools and resources they draw on as they write. Academics’ research writing is framed within explicit institutional and departmental strategies around the numbers and publication venues of research outputs, driven by institutions’ need to succeed in the national competitive research evaluation system. Such institutional strategies do not always map well onto other values systems in which academics have been trained and within which they locate themselves. The articles analyses the interviews we carried out, exploring how academics negotiate tensions between these systems of value and considering the implications of this for what is considered to be important in academic work and, therefore, what it means to be an academic.

The bureaucratic distortion of academic work: a transdisciplinary analysis of the UK Research Excellence Framework in the age of neoliberalism

ABSTRACTIn the years since the second Thatcher government (1983–87), and continuing until the present, universities in the UK have been subjected to a series of neoliberal reforms which have had a deleterious effect on academics’ working conditions and on the kind of research they are required to produce. 1986 saw the introduction of regular sector wide audits of research and scholarly activity designed to make academics more ‘productive’ and the institutions in which they worked more ‘competitive’. This article takes as the object of its investigation the Research Excellence Framework (REF), the most recent iteration of the UK government sponsored assessment exercise. It adopts a transdisciplinary approach which draws on political economy, social theory and critical discourse analysis. The analysis exposes the ways in which the Research Excellence Framework constructs an illusion of intellectual excellence and innovation whose true purpose is the neutralization of the university as a centre of independent knowledge creation and learning, and hence as a potential locus of intellectual opposition to the neoliberal hegemony. The article concludes by calling on academics to refuse the narrow model of research valorized by the REF and to reclaim the idea of the university as a public good.

Providing evidence of impact from public engagement with research: A case study from the UK's Research Excellence Framework (REF)

Providing evidence of impact from public engagement with research: A case study from the UK's Research Excellence Framework (REF)

Providing evidence of impact from public engagement with research: A case study from the UK's Research Excellence Framework (REF)

Societal benefits of public engagement were recognized as 'impact' from research in the UK's recent Research Excellence Framework (REF), which determines an allocation of central government funding for universities and shapes the landscape for university researchers undertaking such activities. This paper shares experience from a successful REF Impact Case Study based on a programme of informing/inspiring-type public engagement, illustrating how engagement goals can match definitions of impact for the REF, and summarizing types of evidence used to demonstrate 'reach' and 'significance' of impact in media engagement, face-to-face engagement and online engagement, which represent common activities undertaken by many researchers.

Multiple-criteria performance ranking based on profile distributions: An application to university research evaluations

This article addresses a category of multi-criteria ranking problems in which performance evaluations of the ‘objects’ or ‘alternatives’ to be ranked are not given by unique numbers but by performance profile distributions running over an ordered set of performance score levels. The numerical values to be given to score levels are not specified a priori using cardinal scales. A weighted sum approach is developed based on order statistics to combine the individual profile distributions. In this way, a global ranking indicator is obtained, considering not only mean distribution values but also standard deviations. As a test of feasibility, the resulting Profile Ranking with Order Statistics Evaluations (PROSE) approach has been applied to the performance profile distributions provided by the UK Research Excellence Framework 2014, evaluating the quality of UK research.

Deciphering museums, politics and impact

This paper makes a contribution towards deciphering the relationship between museums, politics and impact. I suggest that this is akin to that between three languages in the early 19th century: Greek, Demotic and Hieroglyphs. I argue that museums should be taken much more seriously by the discipline of politics and international relations. This paper begins with an analysis of the REF 2014 Impact Case Studies submitted under the Politics and International Studies Unit of Assessment. Thereafter, it looks at how museums have been examined in the field of politics and international relations. Finally, it outlines some of the benefits and opportunities of scholars in the field engaging with museums in terms of their research, as potential collaborators, and as partners for knowledge transfer and impactful activities—within and outwith the strictures of the UK Research Excellence Framework (REF).

Examining differentiation in academic responses to research impact policy: mediating factors in the context of educational research

ABSTRACTThis article examines responses to a new impact indicator introduced to the UK Research Excellence Framework (REF). While analysts have pointed to opportunities in impact policy for educational research as a field, the requirement to demonstrate impact beyond the academy poses challenges for research relating to higher education. The study illustrates forms of research used to demonstrate impact in, and on, HE. Through narratives of respondents engaged in HE-related research, the study indicates ways research practices are influenced by the impact indicator. The analysis demonstrates greater diversity in forms of research used to demonstrate impact than portrayed in earlier studies. Formulaic responses to impact are evident in some institutional contexts, which reify particular forms of research. In other cases, respondents portrayed a re-framing impact in ways that resonate with research orientations within the local context. An argument is made for a relational approach to policy interpretation in local organisational contexts.

‘Environment’ Submissions in the UK's Research Excellence Framework 2014

AbstractThere has been much debate about university research assessment exercises. In the UK, a major element of the 2014 Research Excellence Framework (REF2014) has been the research environment. Here we analyse 98 REF2014 ‘environment’ submissions in Business and Management Studies. We explore whether there are distinctive language‐related differences between submissions of high and low ranked universities and conclude that submission writers have a strong incentive to exaggerate strengths and conceal problems. In addition, innate biases such as the ‘halo’ and ‘velcro’ effects may distract the attention of assessors from a submission's strengths and weaknesses, since they are likely to influence their pre‐existing impressions. We propose several changes to improve how environment is evaluated. We also argue that the research environment would be more likely to be enhanced if the number of outputs submitted in future was an average of two and a maximum of four per academic, rather than the maximum of five currently being considered.

Short-Termism in Science: Evidence from the UK Research Excellence Framework

Periodic research evaluations are necessary to allocate funding, but they also create pressure on researchers. Between grants, short contracts and university assessments, researchers often state that evaluations leave them little time to pursue novel research agendas and publish in more challenging outlets. Using data on 495,128 publications from the UK’s governmental university assessment, we show that evaluations affect research output. Publications just before assessment deadlines obtain substantially fewer citations and are published in lower impact-factor outlets. These trends reverse abruptly after the deadlines. Our results highlight the importance of an appropriate length for evaluation periods.

Using Google Scholar institutional level data to evaluate the quality of university research

In recent years, the extent of formal research evaluation, at all levels from the individual to the multiversity has increased dramatically. At the institutional level, there are world university rankings based on an ad hoc combination of different indicators. There are also national exercises, such as those in the UK and Australia that evaluate research outputs and environment through peer review panels. These are extremely costly and time consuming. This paper evaluates the possibility of using Google Scholar (GS) institutional level data to evaluate university research in a relatively automatic way. Several citation-based metrics are collected from GS for all 130 UK universities. These are used to evaluate performance and produce university rankings which are then compared with various rankings based on the 2014 UK Research Excellence Framework (REF). The rankings are shown to be credible and to avoid some of the obvious problems of the REF ranking, as well as being highly efficient and cost effective. We also investigate the possibility of normalizing the results for the university subject mix since science subjects generally produce significantly more citations than social science or humanities.

An Intimate Approach to the Management and Integration of Design Knowledge for Small Firms

In August 2012, I presented a paper entitled ‘‘The Fuzzy Front End of Product Design Projects: How Universities Can Manage Knowledge Transfer and Creation’’ at the International Design Management Research Conference. Via a series of design projects and design support schemes, the paper investigated how a higher education institution with a particular approach to the management and integrating of knowledge on behalf of small firms could bring together manufacturers, subcontractors, design consultancies, market researchers, intellectual property specialists, funding bodies, and other higher education institutions to make for an extremely effective design support network. In particular, the paper dealt with the dynamics of knowledge acquisition during the fuzzy front end of product design projects. It was suggested that, through this novel approach, the management and integration of the different players involved in new product development (NPD), higher education institutions could help small firms in particular achieve effective knowledge transfer, develop new knowledge, and generally reduce and manage uncertainty in the process and therefore utilize design more effectively in generating new products and increasing profitability. This article investigates the work that has taken place in the ensuing years, including material that led to outstanding impact in the 2014 UK Research Excellence Framework evidencing significant economic growth as a result of this approach to the management of the NPD process for small firms. It proposes that the management and integration of those factors leading to successful NPD for small firms require an intimate style of engagement to be effective, demanding a broad knowledge of, and/or an ability to manage, those factors pertinent to the needs of small firms. Based on four case studies, this article attempts to identify the nature of this intimate approach and therefore its potential significance.

Starry-eyed II: the logistics journal ranking debate revisited

PurposeIn a previous paper (McKinnon, 2013), the author questioned the principle and practice of journal ranking and discussed its effects on logistics research. Since then several important developments have occurred prompting a fresh review of the issues. The paper summarises the results of this review with the aim of stimulating further discussion on the subject.Design/methodology/approachNew literature on the journal ranking debate has been reviewed. The validity of the journal ranking as a proxy measure of paper quality is explored using data from the UK Research Excellence Framework (REF) assessment. Changes to the ranking of ten logistics/supply chain management (SCM) journals in four listings are analysed, and possible reasons for the relatively low status of the journals are examined.FindingsThe influence of journal rankings on the academic research process is strengthening while the debate about their legitimacy has intensified. UK REF data cast doubt on the reliability of the journal ranking as an indicator of a paper’s merit. Logistics/SCM journals continue to occupy mid-to-lower tier positions in most listings, though there has been some improvement in their standing.Research limitations/implicationsThe paper aims to alert those managing and undertaking logistics research to the dangers of overreliance on journal rankings in the measurement of research quality and productivity.Practical implicationsThe paper may help logistics/SCM scholars to defend the position of their discipline and resist journal-ranking-induced pressures to marginalise it and devalue its outputs.Social implicationsIn this paper, academic recruitment, promotion and motivation are considered.Originality/valueThe paper sheds new light on the relationship between journal ranking and individual paper quality, on recent changes in the rating of logistics/SCM journals and on the wider debate about the use of bibliometrics in assessing research quality.

Insight workflow: Systematically combining human and computational methods to explore textual data

Analyzing large quantities of real‐world textual data has the potential to provide new insights for researchers. However, such data present challenges for both human and computational methods, requiring a diverse range of specialist skills, often shared across a number of individuals. In this paper we use the analysis of a real‐world data set as our case study, and use this exploration as a demonstration of our “insight workflow,” which we present for use and adaptation by other researchers. The data we use are impact case study documents collected as part of the UK Research Excellence Framework (REF), consisting of 6,679 documents and 6.25 million words; the analysis was commissioned by the Higher Education Funding Council for England (published as report HEFCE 2015). In our exploration and analysis we used a variety of techniques, ranging from keyword in context and frequency information to more sophisticated methods (topic modeling), with these automated techniques providing an empirical point of entry for in‐depth and intensive human analysis. We present the 60 topics to demonstrate the output of our methods, and illustrate how the variety of analysis techniques can be combined to provide insights. We note potential limitations and propose future work.

Ranking Economics Journals Using Data From a National Research Evaluation Exercise

AbstractThis paper describes an algorithm for creating a ranking of economics journals, using data from the 2014 UK Research Excellence Framework (REF) exercise. The ranking generated by the algorithm can be viewed as a measure of the average quality of the papers published in the journal, as judged by the REF Economics and Econometrics sub‐panel, based on the outputs submitted to the REF.

Communicating geomorphology: an empirical evaluation of the discipline's impact and visibility

AbstractConcern among geomorphologists that the discipline's visibility and impact are becoming suppressed are reflected in a series of recent Earth Surface Exchanges (ESEX) commentaries by Tooth et al. This paper from the British Society for Geomorphology (BSG) Communicating Geomorphology Fixed‐Term Working Group (FTWG) reports initial findings from an online survey of BSG members alongside an empirical assessment of the term's prominence in academic output: international peer‐reviewed journals, undergraduate Geoscience degrees in world‐leading institutions and the UK Research Excellence Framework (REF) impact statements. Our observations indicate the scientific standing of the discipline has been retained but the term itself is less widely utilised and we offer a series of suggestions actionable by the geomorphology community. Copyright © 2017 John Wiley & Sons, Ltd.

Towards measuring societal impact of research: Insights from an Irish case study

Academic research is increasingly required to demonstrate economic and policy relevance, with this becoming a key metric by which the success of research projects are being judged. Furthermore, the active, as opposed to passive, participation of citizens in science is now encouraged through dissemination and outreach, using, for example, co-production techniques. These non-traditional academic impacts have become a key component of a number of funding agency calls, most notably the European Union’s research funding programme Horizon 2020. However, exactly how measurable these ‘impacts’ are, particularly social and policy impacts, is unclear as there is not an obvious metric. Additionally, there is no standardised approach to assessing research impact recognised in the social sciences. Using a case study which describes the experience of using public engagement seminars as a means to disseminate academic research to stakeholder communities, this article aims to develop an impact assessment strategy to measure societal impact applicable in the social sciences. Based on recommendations in the UK Research Excellence Framework, amongst other literature, we put forward three steps to better capture research ‘impact’ in a more meaningful way in future research projects: (i) establish the quality of the academic research, (ii) choose appropriate discipline-specific criteria for measuring societal impact and (iii) choose appropriate measurable indicators. Other useful insights include the difficulty of motivating public interest in topics that are no longer high profile or emotive, and hence the necessity to provide access to research findings as early as possible in the research cycle. The article concludes with a discussion of the difficulties of measuring ‘impact’ in a meaningful sense.

Identifying and understanding research impact: A review for occupational scientists

ABSTRACTBackground: Occupational science continues to be a transformative science, providing knowledge that has potential for social and policy change in relation to occupational inequities. However, as researchers become more dependent on funding and funders seek to identify value for their investment, there is a clear need to consider research findings in the occupational science context of impact on participation and health and well-being outcomes.Objectives: The purpose of this review was to identify the approaches that have been used to understand the impacts of health research, to identify ways that research impacts have been defined and measured and to provide recommendations for occupational science.Methods: An umbrella review – a systematic overview of review articles – was conducted to synthesize research relevant to the evaluation of research impact.Findings: Several frameworks have been used to evaluate research impact, and although exploratory methods provide the most meaningful insights, standard approaches have their weaknesses. Therefore, the method chosen should be reflective of the intention of the evaluation. The UK Research Excellence Framework (REF) definition might be best suited for occupational science. Researchers ought to collect data about impact in real time, before a project ends. Occupational science can support increased social and policy impact by conducting research in collaboration with policymakers. Occupational scientists are encouraged to introduce ideas around inequities to the broader research impact community.Conclusion: Considering research impact in occupational science may advance the field by increasing the quality and funding of research. Researchers must consider engaging policymakers and potential users to ensure greater impact.

Revising the Research Excellence Framework: ensuring quality in REF2021, or new challenges ahead?

ABSTRACTThis paper considers the likely shape of the next UK Research Excellence Framework (REF). It explores some of the recommendations from the 2016 Stern Review (those concerned with ‘outputs’) and their envisaged impact on the exercise. Drawing on lessons learnt from the previous round of the REF, and some wider commentary, the paper considers some of the problems emergent from REF2014, whether they can be mitigated, and whether new problems might emerge in REF2021. Prominently, the issues of ‘burden’ and ‘gaming’ are explored. Although there are some grounds for optimism, much more detail will be needed before we can be reasonably confident about achieving a less burdensome and a more accurate evaluation of research quality within universities across the UK.

The "Real" Impact Factor: Reflections on the Impact of the Research Excellence Framework

This paper is an argument for the importance of academics undertaking some (but not only) research that relates to the practical issues faced by practitioners and policy makers and that is geared to achieving impact. It offers a normative argument informed by my experience as a practitioner and an academic and by my experiences in the assessment of impact as part of the UK Research Excellence Framework (REF) in 2014. The paper introduces the nature of the REF and how it was implemented. It also addresses the implications of the performance measurement of impact of REF for Higher Educational Institutions and the individual academics that work within them. In that respect it recognises that performance measures give extrinsic encouragement to particular behaviours. The paper argues that academics should also be intrinsically driven to research that has impact. In order to achieve impact, the paper suggests that we should not see a gap between academics and practitioners, but should instead see practice and academic endeavour as different but complementary elements of the same profession. We should seek to develop better discourses between academics and practitioners and should not attribute greater importance to the views of either party. Instead we should have an engagement that is open to the generation of disagreement as well as agreement but that nevertheless does not see disagreement as the basis for closing down communication.