Research Excellence Framework Research Articles

Can ChatGPT evaluate research quality?

Abstract Purpose Assess whether ChatGPT 4.0 is accurate enough to perform research evaluations on journal articles to automate this time-consuming task. Design/methodology/approach Test the extent to which ChatGPT-4 can assess the quality of journal articles using a case study of the published scoring guidelines of the UK Research Excellence Framework (REF) 2021 to create a research evaluation ChatGPT. This was applied to 51 of my own articles and compared against my own quality judgements. Findings ChatGPT-4 can produce plausible document summaries and quality evaluation rationales that match the REF criteria. Its overall scores have weak correlations with my self-evaluation scores of the same documents (averaging r=0.281 over 15 iterations, with 8 being statistically significantly different from 0). In contrast, the average scores from the 15 iterations produced a statistically significant positive correlation of 0.509. Thus, averaging scores from multiple ChatGPT-4 rounds seems more effective than individual scores. The positive correlation may be due to ChatGPT being able to extract the author’s significance, rigour, and originality claims from inside each paper. If my weakest articles are removed, then the correlation with average scores (r=0.200) falls below statistical significance, suggesting that ChatGPT struggles to make fine-grained evaluations. Research limitations The data is self-evaluations of a convenience sample of articles from one academic in one field. Practical implications Overall, ChatGPT does not yet seem to be accurate enough to be trusted for any formal or informal research quality evaluation tasks. Research evaluators, including journal editors, should therefore take steps to control its use. Originality/value This is the first published attempt at post-publication expert review accuracy testing for ChatGPT.

Measuring research quality in a more inclusive way: Evidence from the UK Research Excellence Framework

Abstract Evidence suggests that common metrics of research quality—e.g. journal publications and citations—are systematically biased against certain groups. But does relying solely on them to evaluate quality lead to lower diversity in academia? In this paper, we start to answer this question by analysing data from the UK’s nationwide research assessment exercise, the Research Excellence Framework. We find that narrowly focussed output-based measures of departmental research quality do indeed negatively correlate with the diversity of departmental staff, while measures of research impact and of the quality of the research environment correlate positively. An aggregate measure that incorporates all three components is therefore likely to better promote staff diversity compared to more narrowly defined output-focused measures. More generally, our results suggest that comprehensive definitions of research quality may be more effective at promoting diversity in academia compared to narrower measures. We further argue that funding decisions informed by broader measures result in more efficient resource allocations across the higher education sector.

Which international co‐authorships produce higher quality journal articles?

AbstractInternational collaboration is sometimes encouraged in the belief that it generates higher quality research or is more capable of addressing societal problems. Nevertheless, while there is evidence that the journal articles of international teams tend to be more cited than average, perhaps from increased international audiences, there is no science‐wide direct academic evidence of a connection between international collaboration and research quality. This article empirically investigates the connection between international collaboration and research quality for the first time, with 148,977 UK‐based journal articles with post publication expert review scores from the 2021 Research Excellence Framework (REF). Using an ordinal regression model controlling for collaboration, international partners increased the odds of higher quality scores in 27 out of 34 Units of Assessment (UoAs) and all Main Panels. The results therefore give the first large scale evidence of the fields in which international co‐authorship for articles is usually apparently beneficial. At the country level, the results suggests that UK collaboration with other high research‐expenditure economies generates higher quality research, even when the countries produce lower citation impact journal articles than the United Kingdom. Worryingly, collaborations with lower research‐expenditure economies tend to be judged lower quality, possibly through misunderstanding Global South research goals.

Graduate prospects explain undergraduate program standing in university league sports science tables.

UK university undergraduate programs are compared by independent subject-specific rankings (Complete, Guardian, and Times), based on data from the National Student Survey, Higher Education Statistics Agency, Universities and Colleges Admissions Service, Research Excellence Framework, and the universities. The sports and exercise science program at the University of Glasgow has steadily risen to currently rank as the top UK program. This investigation aimed to identify the underlying factors that explain this. Therefore, we obtained underlying scores for entry standard, student satisfaction, research foundation, graduate prospects, staff-to-student ratio, expenditure/student, continuation, program support to students, and teaching quality from the ranking providers for years 2010-2024, and statistically modeled which factors significantly affected the rankings. We found that entry standards and graduate prospects strongly correlated significantly with ranking results. Principal component analysis indicated that a model of graduate prospects and entry standards explained 66.2% of the variance in ranking results. Multiple linear regression with all underlying factors included in the model indicated they explained 78% (R2 = 0.78) of the total variance, while stepwise elimination of insignificant factors identified graduate prospects as the sole factor that significantly affected outcome by explaining 71% (R2 = 0.71) of the variance. Therefore, the primary predictor of ranking success in UK university league tables for sports science is graduates' professional success (graduate prospects).NEW & NOTEWORTHY University rankings are used by applicants and stakeholders to judge programs, including undergraduate studies. In the rankings, undergraduate UK programs are compared and contrasted against each other based on how they score for criteria that affect student life and future prospects. Here, we determined the relative influence of those criteria and found that graduate prospects, how students professionally benefit from their study after graduation, is the factor that matters most for the ranking results.

What is a high-quality research environment? Evidence from the UK’s research excellence framework

Abstract As part of the UK university sector’s performance-related research funding model, the ‘REF’ (Research Excellence Framework), each discipline-derived ‘Unit of Assessment’ must submit a statement to provide information about their environment, culture, and strategy for enabling research and impact. Our aim in this paper is to identify the topics on which these statements focus, and how topic variation predicts funding-relevant research environment quality profiles. Using latent Dirichlet allocation topic modelling, we analysed all 1888 disciplinary ‘unit-level’ environment statements from REF2021. Our model identified eight topics which collectively predicted a surprisingly large proportion—58.9%—of the variance in units’ environment scores, indicating that the way in which statements were written contributed substantially to the perceived quality of a unit’s research environment. Assessing research environments will increase in importance in the next REF exercise and the insights found through our analysis may support reflection and discussion about what it means to have a high-quality research environment.

Clinical academics’ experiences of REF2021

The UK Research Excellence Framework (REF) is an assessment of the quality of research carried out in UK Higher Education Institutions (HEIs), performed in 7-year cycles. The outcome impacts the rankings and funding of UK HEIs, which afford the exercise high priority. Much of what REF measures is known to be biased against academics with protected characteristics: for example, women and ethnic minority researchers are less likely to win grants or be published in prestigious journals. Despite changes to REF since 2014, the risk remains that the process might amplify well-recognised existing disparities. The BMA Women in Academic Medicine and Medical Academic Staff Committee carried out a survey of UK clinical academics’ experiences of REF2021. The data indicated the persistence of activities previously characterised as ‘extremely harmful’ in Research England-commissioned work, affecting up to 10% of clinical academics. While acknowledging the limitations of the data, women appeared to be disproportionately affected.

UK Research Excellence Framework (2021) Reform and Implications for China’s Implications for Research Assessment in Higher Education

UK Research Excellence Framework (2021) Reform and Implications for China’s Implications for Research Assessment in Higher Education

Models of knowledge

Art practice has an unusual history inside the university sector. Often originating from specialized academies, as disciplines they usually began life through a focus on practical work. This gave rise to small institutions that have been subsumed into larger ones, becoming colleges or schools within universities rather than continuing their independent lives. The advantages of this are clear: structural, financial and cultural opportunities that come with the status of university subjects. What this article argues is that there has been a considerable price to pay in being so accommodated. Creative practice subjects in a qualifications framework, and to some extent in exercises like the Research Excellence Framework, have been required to adopt intellectual enquiry processes in order to burnish their claims to knowledge. Mostly, these come from the humanities. This article examines the impact of this on creative practice, on acceptance of the research elements of it and on distinguishing the difference between arts-based enquiry and humanities models of interpretation and analysis. That these are increasingly incompatible requires rethinking of the relationship between arts and humanities that is so taken for granted.

The Luck of the Draw: Wellcome's Institutional Fund for Research Culture.

Wellcome's Institutional Fund for Research Culture (IFRC) closed call is an invite-only grant call in 2023. It is a departure from Wellcome's previous methods of institutional funding, providing institutions with up to £1m of grant funding to take on ambitious projects that advance research cultures and research environments that are equitable, diverse and supportive. Recognising the broad range of topics and ideas for advancing positive research cultures, IFRC is the first ever Wellcome-funding scheme to use partial randomisation to allocate funding. Applications were grouped by a funding committee into three categories (Gold, Silver and Bronze), with the applications selected for Gold being directly recommended for funding and all applications in the Silver group being set for funding by the randomiser. Applications grouped into Bronze were not funded. To ensure that this activity Wellcome's principles for openness and transparency, we have included the Python script for the call here. IFRC comes when efforts to fund positive and inclusive research cultures are mainstream. Similar efforts to support research culture activities at scale have come from the Research England Development (RED) Fund, and the next iteration of the Research Excellence Framework (REF 2028) will also mark 25% of the assessment criteria for people, culture and environment. IFRC was not designed with the national picture in mind but is a testament to Wellcome's values as an inclusive funder. The range of projects and geographies that IFRC will fund is exciting.Still, it also threw up several interesting social and philanthropic research questions we want to explore in future community-facing activities following the call. We hope that findings from IFRC projects become a valuable resource for institutions wishing to improve their research cultures and a catalyst for future change and discussion within the sector that makes academic careers more inclusive.

Legal research and the public good: the current landscape

Abstract This paper is a revised version of a plenary lecture delivered at the SLS Annual Conference held at Oxford Brookes University in June 2023. It adopts an auto-ethnographic approach, drawing on the author's long experience of participating in the UK Research Excellence Framework (REF), to assess the current state of legal research in the UK and consider the implications of the increasing importance in the research landscape of engaging in research that serves the public good. The paper explores what falls within the scope of ‘legal research’, particularly for REF purposes, and reflects on how REF-focused research sits within the broader scope of legal scholarly activities. Ideas of the public good are examined and their relation to measures of research impact probed. The paper concludes by painting a broad-brush picture of the current research landscape, identifying key elements and engines of change, and speculating on the direction in which things are going and what should most concern legal scholars going forward. Although primarily focused on legal research in the UK, the paper should be of interest to legal scholars beyond the UK, particularly those in jurisdictions where research assessment exercises are a feature of academic life.

The luck of the draw: Wellcome's Institutional Fund for Research Culture

Wellcome's Institutional Fund for Research Culture (IFRC) closed call is an invite-only grant call in 2023. It is a departure from Wellcome's previous methods of institutional funding, providing institutions with up to £1m of grant funding to take on ambitious projects that advance research cultures and research environments that are equitable, diverse and supportive. Recognising the broad range of topics and ideas for advancing positive research cultures, IFRC is the first ever Wellcome-funding scheme to use partial randomisation to allocate funding. Applications were grouped by a funding committee into three categories (Gold, Silver and Bronze), with the applications selected for Gold being directly recommended for funding and all applications in the Silver group being set for funding by the randomiser. Applications grouped into Bronze were not funded. To ensure that this activity Wellcome's principles for openness and transparency, we have included the Python script for the call here. IFRC comes when efforts to fund positive and inclusive research cultures are mainstream. Similar efforts to support research culture activities at scale have come from the Research England Development (RED) Fund, and the next iteration of the Research Excellence Framework (REF 2028) will also mark 25% of the assessment criteria for people, culture and environment. IFRC was not designed with the national picture in mind but is a testament to Wellcome's values as an inclusive funder. The range of projects and geographies that IFRC will fund is exciting.Still, it also threw up several interesting social and philanthropic research questions we want to explore in future community-facing activities following the call. We hope that findings from IFRC projects become a valuable resource for institutions wishing to improve their research cultures and a catalyst for future change and discussion within the sector that makes academic careers more inclusive.

Where is all the research software? An analysis of software in UK academic repositories

This research examines the prevalence of research software as independent records of output within UK academic institutional repositories (IRs). There has been a steep decline in numbers of research software submissions to the UK’s Research Excellence Framework from 2008 to 2021, but there has been no investigation into whether and how the official academic IRs have affected the low return rates. In what we believe to be the first such census of its kind, we queried the 182 online repositories of 157 UK universities. Our findings show that the prevalence of software within UK Academic IRs is incredibly low. Fewer than 28% contain software as recognised academic output. Of greater concern, we found that over 63% of repositories do not currently record software as a type of research output and that several Universities appeared to have removed software as a defined type from default settings of their repository. We also explored potential correlations, such as being a member of the Russell group, but found no correlation between these metadata and prevalence of records of software. Finally, we discuss the implications of these findings with regards to the lack of recognition of software as a discrete research output in institutions, despite the opposite being mandated by funders, and we make recommendations for changes in policies and operating procedures.

Reflecting on research impact: diversity of religion and belief in primary schools in England and Wales

ABSTRACT The relationship between research, policy and practice in the field of education has long been the focus of much discussion and debate. In the UK context, the growing weight attributed to research impact in the Research Excellence Framework has further intensified interest in how academics can promote the application of their research in non-academic contexts. There are relatively few existing publications that document research impact activities relating to religion and education, highlighting the potential to further explore processes of engagement, knowledge-exchange and research application within this subfield. This article provides a reflective account of a knowledge-exchange and impact project on diversity of religion and belief in primary schools in England and Wales, along with findings from an online survey that aimed to evaluate its impact. The data presented provides evidence for moderate success in meeting the project’s goals, alongside signposts for other researchers wishing to engage in impact activities on comparable topics and in similar educational contexts.

Towards a critical‐conceptual analysis of ‘research culture’

AbstractUniversities and policymakers increasingly use ‘research culture’ and ‘research environment’ to govern as well as describe research. Both terms help frame who is considered a research actor; how researchers interact with the contexts in which they make knowledge; and what is considered malleable when attempting to improve how research is done. There are very few conceptual‐critical analyses of either term, even as each is a complex abstraction with rich and contested histories and usage. I explore both, largely using the example of the United Kingdom (where improving ‘research culture’ is currently prioritised by many funders, and will be assessed by the UK's Research Excellence Framework (REF) in 2028). Research culture has a close relationship with the concept organisational culture, which emerged in the late 1970s and 1980s and prioritised particular – frequently psychological – constructs that focused on the norms, values, and attitudes of an organisation. ‘Research labour’ – the labour relations that underpin how people work together and shape organisational norms, values, and relational dependencies – tends to drop from view. Geographers have much to offer these debates, given how extensively the discipline has contributed to what culture and environment might mean. Institutional, national, and sectoral policies concerning research culture and environment significantly shape how knowledge‐making is understood and intervened on. The processes that ‘research culture’ and ‘research environment’ authorise and foreclose require greater examination.

What works in policing money laundering?

PurposeThis paper aims to provide a comprehensive review of literature on policing money laundering to evaluate the quality of evidence presented in academic and grey literature. In doing so, this paper reflects upon what strategies and mechanisms work in preventing, detecting and disrupting money laundering.Design/methodology/approachTo investigate the quality of evidence presented in academic and grey literature on money laundering, the authors have conducted a structured literature review of over 1,000 sources. The authors have then evaluated the quality of evidence of these sources by using three distinct methods, namely, the Maryland scale, research excellence framework and quality of empirical evidence framework.FindingsThis paper finds a large number of studies on money laundering meeting the highest academic quality standard. However, only a very small number of those studies include empirical evidence that policing measures actually work with no study able to demonstrate a correlation between money laundering measures and risk factors at any point in time.Originality/valueThis paper provides practical and theoretical insights into what works in money laundering and the role of scientific evaluations. It is particularly relevant for the successful implementation of anti-money laundering policy measures, including, for example, the UK’s Economic Crime Plan 2023–2026, which aspires to measure the progress and impact of policing in this area.

Investigating the relationship between research income and research excellence in education: Evidence from the REF2021‐UoA23 data

AbstractThis paper presents and discusses the relationship between the generation of research income and the REF2021 results for universities who submitted a return to Unit of Assessment 23—Education. Research Excellence Framework exercises tend to be time intensive and come with income streams of Government Quality Research funding as well as reputational gains and risks. Based on REF2021 research environment research income data, freely available on the REF2021 website, analyses have been undertaken to investigate the strength of relationships between different types of fundings streams and grade point averages. The results demonstrate strong variation in funding capture across the different universities with Russell Group members tending to generate more income from prestigious funders such as UK Research and Innovation. However, several post‐1992 universities managed to capture significant income from funders like UK and EU governments, without this necessarily translating into higher grade point averages or 4* scores. While generating research income is perceived as an important part of academic life, the paper concludes that achieving research excellence seems more complicated than just following the money.

Do research performances of universities and disciplines in England converge or diverge? An assessment of the progress between research excellence frameworks in 2014 and 2021

Performance-based research funding systems (PBRFSs) have been used in selectively distributing research funding, increasing public money accountability and efficiency. Two recent such evaluations in England were called the Research Excellence Framework (REF), which took place in 2014 and 2021, and the research environment, outputs and impact of the research were evaluated. Even though various aspects of the REF were examined, there has been limited research on how the performance of the universities and disciplines changed between the two evaluation periods. This paper assesses whether there has been convergence or divergence in research quality across universities and subject areas between 2014 and 2021 and found that there was an absolute convergence between universities in all three research elements evaluated, and universities that performed relatively worse in REF in 2014 experienced higher growth in their performance between 2014 and 2021. There was also an absolute convergence in the research environment and impact across different subject areas, but there is no significant convergence in the quality of research outputs across disciplines. Our findings also highlight that there has been an absolute convergence in research quality within the universities (between different disciplines in a given university) and within disciplines (between universities in a given subject).

Exploring the application of machine learning to expert evaluation of research impact.

The objective of this study is to investigate the application of machine learning techniques to the large-scale human expert evaluation of the impact of academic research. Using publicly available impact case study data from the UK's Research Excellence Framework (2014), we trained five machine learning models on a range of qualitative and quantitative features, including institution, discipline, narrative style (explicit and implicit), and bibliometric and policy indicators. Our work makes two key contributions. Based on the accuracy metric in predicting high- and low-scoring impact case studies, it shows that machine learning models are able to process information to make decisions that resemble those of expert evaluators. It also provides insights into the characteristics of impact case studies that would be favoured if a machine learning approach was applied for their automated assessment. The results of the experiments showed strong influence of institutional context, selected metrics of narrative style, as well as the uptake of research by policy and academic audiences. Overall, the study demonstrates promise for a shift from descriptive to predictive analysis, but suggests caution around the use of machine learning for the assessment of impact case studies.

How unpredictable is research impact? Evidence from the UK’s Research Excellence Framework

Abstract Although ex post evaluation of impact is increasingly common, the extent to which research impacts emerge largely as anticipated by researchers, or as the result of serendipitous and unpredictable processes, is not well understood. In this article, we explore whether predictions of impact made at the funding stage align with realized impact, using data from the UK’s Research Excellence Framework (REF). We exploit REF impact cases traced back to research funding applications, as a dataset of 2,194 case–grant pairs, to compare impact topics with funder remits. For 209 of those pairs, we directly compare their descriptions of ex ante and ex post impact. We find that impact claims in these case–grant pairs are often congruent with each other, with 76% showing alignment between anticipated impact at funding stage and the eventual claimed impact in the REF. Co-production of research, often perceived as a model for impactful research, was a feature of just over half of our cases. Our results show that, contrary to other preliminary studies of the REF, impact appears to be broadly predictable, although unpredictability remains important. We suggest that co-production is a reasonably good mechanism for addressing the balance of predictable and unpredictable impact outcomes.

Do bibliometrics introduce gender, institutional or interdisciplinary biases into research evaluations?

Systematic evaluations of publicly funded research sometimes use bibliometrics alone or bibliometric-informed peer review, but it is not known whether bibliometrics introduce biases when supporting or replacing peer review. This article assesses this by comparing three alternative mechanisms for scoring 73,612 UK Research Excellence Framework (REF) journal articles from all 34 field-based Units of Assessment (UoAs) 2014–17: REF peer review scores, field normalised citations, and journal average field normalised citation impact. The results suggest that in almost all academic fields, bibliometric scoring can disadvantage departments publishing high quality research, as judged by peer review, with the main exception of article citation rates in chemistry. Thus, introducing journal or article level citation information into peer review exercises may have a regression to the mean effect. Bibliometric scoring slightly advantaged women compared to men, but this varied between UoAs and was most evident in the physical sciences, engineering, and social sciences. In contrast, interdisciplinary research gained from bibliometric scoring in about half of the UoAs, but relatively substantially in two. In conclusion, out of the three potential sources of bibliometric bias examined, the most serious seems to be the tendency for bibliometric scores to work against high quality departments, assuming that the peer review scores are correct. This is almost a paradox: although high quality departments tend to get the highest bibliometric scores, bibliometrics conceal the full extent of departmental quality advantages, as judged by peer review. This should be considered when using bibliometrics or bibliometric informed peer review.