Expert judgement is inherently noisy, but the consequences of this variability are often underestimated. Here I examine the UK Research Excellence Framework (REF) as a case study illustrating how judgment noise can interact with structural features of a multi-stage evaluation system. Simulations show that noise in both internal departmental reviews and REF panel reviews, in conjunction with the bounded REF rating scale, produces systematic underestimates of the highest-performing departments. Panel noise, rather than internal noise, had a disproportionate impact on department rankings, with even moderate noise producing severe distortions. The case of REF demonstrates that noise embedded in sequential evaluation, for example, in hiring and grant review decisions, does not “cancel out”, but instead can propagate in potentially surprising ways. At a policy level, these results underscore the need for a noise audit of REF panel ratings.

The National Institute for Health and Care Research (NIHR) established nine pilot Collaborations for Leadership in Applied Health Research and Care (CLAHRC) in 2008 which have since grown to the current fifteen Applied Research Collaborations (ARC). The CLAHRC partnership model focuses upon developing and conducting applied health research to improve outcomes for patients. We explored the scale and focus of the CLAHRCs’ contribution to impact case studies submitted to the UK Research Excellence Framework (REF) 2014 and 2021 and how this might be of value for future health research and practice. The study design used a content analysis of the REF 2014 and REF 2021 datasets of impact case studies. We identified impact case studies linked to CLAHRCs from both REF datasets and extracted primarily quantitative data about the institutions involved, subject areas, how they were linked to a CLAHRC and what role the CLAHRC contributed. Qualitative analysis consisted of a purposive sample of eight impact case studies to exemplify the range of subject areas, the scope of involvement of the CLAHRCs and to identify the features associated with the impact. A total of 53 impact case studies related to a CLAHRC were identified from the REF datasets, an increase from 17 in 2014 to 36 in 2021. There is considerable variation in how CLAHRC involvement is described and making a direct attribution of the CLAHRCs is complex and multifaceted. Key pillars of the CLAHRCs such as undertaking applied health research which responds to local and regional needs, knowledge mobilisation and implementing research into practice are exemplified in the sample of impact case studies. There are multiple mechanisms associated with impact with the benefits of collaborative and partnership activities evident. Applied research collaborations such as the CLAHRCs are making a significant contribution to the impact of research in the REF. There is scope to improve the visibility of the CLAHRCs and for CLAHRCs/ARCs to provide a valuable model of partnership capable of strengthening the impact of future impact case studies for a range of purposes.

Purpose We explore how women management scholars navigate “doing” research impact. Drawing on the voices of women leaders of research impact cases in UK business schools, we identify and theorise three distinct autobiographical narratives of coping. To inform systemic changes for more equitable national research policy implementation, we highlight barriers that deter individuals from becoming impactful scholars. Design/methodology/approach This qualitative study comprises semi-structured interviews with 15 mid/late career women research impact case study leaders who were purposively selected to reflect on their lived experiences of impact case study generation. Findings A thematic analysis revealed three mythological archetypes: Sisyphus (a figure in Greek mythology whose eternal repetitive and futile punishment was to roll a boulder uphill, only for it to roll back down), Hestia (the Greek Goddess of the hearth) and Tyche (the Greek Goddess of luck and change). Our empirical study contributes to understanding how individual women management scholars experience and cope with the UK’s research impact policy agenda in the neoliberal university. Research limitations/implications The interviewees’ accounts of coping with REF (Research Excellence Framework) impact case studies are subjective and have influenced the research process and findings. The study has important practical and policy implications. Practical implications The study has important practical and policy implications. Originality/value We call for more inclusive systemic opportunities in business schools in terms of who “does impact” and how they are supported in coping with the demands of research impact work. We advocate rethinking traditional heroic paradigms of academic labour (Harley, 2019), to enable women impact case writers to shift from Sisyphean struggles to greater recognition, inclusion and support for their contributions.

Assessing Psychology in the Research Excellence Framework

Abstract The large language model (LLM) ChatGPT’s quality scores for journal articles correlate more strongly with human judgments than some citation-based indicators in most fields. Averaging multiple ChatGPT scores improves the results, apparently exploiting its internal probability model. To leverage these probabilities, we test two novel strategies: requesting percentage likelihoods for scores and extracting the probabilities of alternative tokens in the responses. These probability estimates were used to calculate weighted average scores. Both strategies were evaluated with five iterations of ChatGPT 4o-mini on 96,800 articles submitted to the U.K. Research Excellence Framework (REF) 2021, using departmental average REF2021 quality scores as a proxy for article quality. The data were analyzed separately for each of the 34 field-based REF Units of Assessment. For the first strategy, explicit requests for tables of score percentage likelihoods substantially decreased the value of the scores (lower correlation with the proxy quality indicator). In contrast, weighed averages of score token probabilities slightly increased the correlation with the quality proxy indicator and these probabilities reasonably accurately reflected ChatGPT’s outputs. The token probability leveraging approach is therefore the most accurate method for ranking articles by research quality as well as being cheaper than comparable ChatGPT strategies.

ABSTRACT This quantitative study of the UK Research Excellence Framework (REF) reveals persistent disciplinary inequities between the 2014 and 2021 cycles. Key findings include: (1) At the macro level, the differences between REF 2014 and REF 2021 show a narrowing trend; however, disparities in high star (4*+3*) outputs persist across the main panels, with the arts and humanities disadvantaged relative to the physical sciences. (2) The apparent correlation between journal article shares and high-star shares is disciplinary rather than merit-based, disappearing once panel effects are controlled, indicating a systematic disadvantage for non-traditional outputs typical of Humanities and Social Sciences; (3) Russell Group universities hold a consistent advantage, most evident in resource-intensive fields, illustrating how concentrated resources sustain success. These results reveal a fundamental misalignment between the REF’s standardized framework and diverse models of knowledge production. Consequently, technical reforms alone cannot ensure equity; achieving it requires flexible evaluation paradigms that genuinely honor disciplinary diversity.

Transforming research culture and environment has been a priority for UK Research and Innovation (UKRI) and has been reflected in recent rounds of the UK Research Excellence Framework (REF). Broadly, research culture encompasses the behaviours, values, expectations, attitudes and norms of research communities and is advanced via areas of academic service in mentoring, skills development, leadership, standards and local development opportunities. There is need to understand how academic units can develop research activity in such a way as to meet criteria of excellence and intensivity within a highly pressured UK Higher Education sector. In this article, we set out the process and outcome of a programme of research culture transformation within an archetypal post-1992 department at Northumbria University in the Northeast of England. Framed around the prospective content of REF People, Culture and Environment (PCE) and Impact narratives, we argue that there are clear means of meeting criteria that protect colleagues by increasing external sources of funding. One consequence of the approach adopted may be a mitigating effect on the prevalence and consequences of passive aggression – a common and debilitating feature of academic units facing pressure. This article serves to provide tangible examples of how a coherent approach to supporting colleagues can deliver change at a time in which opposition to change within universities runs contrary to demands from external sources of funding.

ABSTRACT Drawing on a comprehensive analysis of 105 Impact Case Studies from the Research Excellence Framework (REF) 2021, this paper challenges the prevailing narrative that knowledge valorisation should be predominantly an exercise rooted in commercialisation. We do this by observing the multi-dynamic contributions of arts, humanities and social science (AHSS) research, which contrasts the stereotype in which AHSS disciplines primarily deliver social impact. By examining the multifaceted mechanisms through which AHSS research delivers value, we identify a spectrum of economic impacts that are routinely underestimated or overlooked. Our analysis uncovers that economic impacts frequently arise outside the boundaries of university commercialisation and academic entrepreneurship, highlighting the need for a broader conceptualisation of valorisation activities. To address this, we propose a novel framework that introduces a spectrum of knowledge transfer mechanisms that recognises direct and indirect (enabling) forms of impact through mechanisms that may be more active or passive in generating economic impact in AHSS. This study’s contribution is twofold. From a theoretical perspective, the paper advances a renewed understanding of knowledge valorisation in AHSS. From a policy perspective, the paper offers practical guidance for policymakers and higher education leaders, highlighting that publicly funded AHSS research can deliver significant social and economic impacts beyond conventional commercialisation pathways.

Transforming research culture and environment has been a priority for UK Research and Innovation (UKRI) and has been reflected in recent rounds of the UK Research Excellence Framework (REF). Broadly, research culture encompasses the behaviours, values, expectations, attitudes and norms of research communities and is advanced via areas of academic service in mentoring, skills development, leadership, standards and local development opportunities. There is need to understand how academic units can develop research activity in such a way as to meet criteria of excellence and intensivity within a highly pressured UK Higher Education sector. In this article, we set out the process and outcome of a programme of research culture transformation within an archetypal post-1992 department at Northumbria University in the Northeast of England. Framed around the prospective content of REF People, Culture and Environment (PCE) and Impact narratives, we argue that there are clear means of meeting criteria that protect colleagues by increasing external sources of funding. One consequence of the approach adopted may be a mitigating effect on the prevalence and consequences of passive aggression – a common and debilitating feature of academic units facing pressure. This article serves to provide tangible examples of how a coherent approach to supporting colleagues can deliver change at a time in which opposition to change within universities runs contrary to demands from external sources of funding.

Abstract The purpose of this paper is to evaluate which universities spin out biotechnology SMEs, and why some do this successfully while others do not. The research question is: how do institutional, contextual, and geographic factors shape biotechnology spin-out activity among UK universities? A systematic review was carried out, which examined data from the websites of 26 UK universities in 10 geographic regions and 4 industry bodies. To supplement this review, a questionnaire was used to evaluate the factors which encourage universities to spin out. The results show that biotechnology spin-outs tend to be clustered around certain geographical locations. This clustering activity is not deliberate and exists as a by-product of activity. Interestingly, certain universities with good research backgrounds have been shown to produce little to no biotechnology spin-outs. Six themes were noted as drivers for the creation of biotechnology university spin-outs: increased support from universities; funding opportunities; research excellence framework and knowledge exchange framework metrics and impact; geographical location of universities; revenue and profit; a push on technology transfer and intellectual property. The geographical advantage of being near a biotechnology cluster is clear. However, by building a solid research base with an international reputation, a supportive Technology Transfer Office, education for academic staff in entrepreneurial attributes and behaviours, and engaging with biotechnology accelerators, a university can enhance its spin-out success rate even when that university does not have the benefit of geography and a close regional cluster. Graphical Abstract

Abstract This paper examines the use of hyperbolic and promotional language, or ‘hype’, in the impact case studies submitted to the 2021 Research Excellence Framework (REF), a major research evaluation exercise for UK higher education institutions. With substantial funding and institutional reputations at stake, REF submissions are high-stakes texts that may incentivize rhetorical embellishment to demonstrate research impact. We conducted a corpus-based analysis of 6,361 case studies, focusing on the extent and nature of hype across different academic fields and impact types. The findings reveal a pervasive use of hyping language, with significantly higher frequencies compared to typical academic genres. Disciplines oriented towards ‘pure knowledge’ exhibited the highest levels of hype, especially in attempts to assert certainty and contribution. Cultural and technological impact types were particularly characterized by claims of novelty and potential. The results highlight how the competitive pressures of research assessment foster exaggerated representations of impact, which may compromise the integrity of research communication. We argue for a more measured approach to promoting research impact to preserve the objectivity of assessment processes. This study contributes to the understanding of academic communication under high-stakes evaluation conditions and provides insights for policymakers, assessment panels, and researchers.

What can we learn about psychology research in the UK, and its perceived quality, from examining manuscripts submitted to the psychology, psychiatry and neuroscience subpanel of the 2021 Research Excellence Framework (REF2021)? Using a latent Dirichlet allocation topic modelling approach, we identified 33 topics which collectively summarised the content of the journal articles returned to the subpanel. We found that the composition of submissions to the subpanel, in terms of these topics, explained a large proportion of the variance in the quality assessments they received from the expert peer review subpanel. Our model identified topics which were typically associated with receiving higher and lower unit-level quality assessments. In our discussion we pay particular attention to the fate of qualitative research, and discuss possible accounts for why units who returned a large amount of qualitative work tended to receive lower quality assessments than those who did not.

Abstract Academic librarians often construct bibliometric indicators to support research evaluation. Traditionally, these have been citation-based, but AI alternatives have recently emerged. Although both Google Gemini (1.5 Flash) and ChatGPT (4o and 4o-mini) provide research quality evaluation scores that correlate positively with expert scores in nearly all fields, and more strongly than citations in most, it is not known whether this holds for smaller Large Language Models (LLMs). In response, this article assesses Google’s Gemma-3-27b-it, a downloadable LLM (60 GB). Results for 104,187 articles show that Gemma-3-27b-it scores correlate positively with an expert research quality score proxy for all 34 Units of Assessment (broad fields) from the UK Research Excellence Framework 2021. The Gemma-3-27b-it correlations have 83.8% of the strength of ChatGPT 4o and 94.7% of the strength of ChatGPT 4o-mini correlations. Unlike the two larger LLMs, the Gemma-3-27b-it correlations do not increase substantially when scores are averaged across five repetitions, its scores tend to be lower, and its reports are relatively uniform in style. Overall, the results show that research quality score estimation can be conducted by offline LLMs, so this capability is not an emergent property of only the largest LLMs. Moreover, score improvement through repetition is not a universal feature of LLMs. In conclusion, although the largest LLMs still have the highest research evaluation score estimation capability, smaller ones can also be used for this task, which can be helpful for cost saving or when secure offline processing is required.

Between the first and second International Research Culture Conference (IRCC), there has been increasing advocacy for improved research culture, evidenced through collaborative initiatives, networks and events aimed at addressing various challenges within research environments. This reflection follows the International Research Culture Conference 2024 (IRCC24) and highlights significant developments leading up to it, detailing expanded participation and thematic diversity. Key takeaways from the conference included the influence of the Research Excellence Framework - People Culture and Environment pilot, the importance of continued collaboration and increased need for efficiency in this space. It concludes with reflections on future direction, advocating for continued collaboration, increased evaluation, and learning with and from international perspectives to foster meaningful improvement in research culture.

Abstract This article compares (1) citation analysis with OpenAlex and Scopus, testing their citation counts, document type/coverage, and subject classifications and (2) three citation‐based indicators: raw counts, (field and year) Normalized Citation Scores (NCS), and Normalized Log‐transformed Citation Scores (NLCS). Methods (1&2): The indicators calculated from 28.6 million articles were compared through 8704 correlations on two gold standards for 97,816 UK Research Excellence Framework (REF) 2021 articles. The primary gold standard is ChatGPT scores, and the secondary is the average REF2021 expert review score for the department submitting the article. Results: (1) OpenAlex provides better citation counts than Scopus, and its inclusive document classification/scope does not seem to cause substantial field normalization problems. The broadest OpenAlex classification scheme provides the best indicators. (2) Counterintuitively, raw citation counts are at least as good as nearly all field normalized indicators and better for single years, and NCS is better than NLCS. (1&2) There are substantial field differences. Thus, (1) OpenAlex is suitable for citation analysis in most fields and (2) the major citation‐based indicators seem to work counterintuitively compared to quality judgments. Field normalization seems ineffective because more cited fields tend to produce higher quality work, affecting interdisciplinary research or within‐field topic differences.

ABSTRACT This short communication presents the preliminary outcomes of an ongoing project exploring how initial teacher education research features in the UK’s 5–7 yearly national research assessment exercise, Research Excellence Framework (REF). Most recently, REF has reported in 2014 and 2021. Using a novel sampling method and analytic approach, we found that for REF2014, just 5.5% of outputs submitted by our sample of higher education institutions (HEIs) had a focus on initial teacher education (pre-service). A similar analysis of the same sample of HEIs for the REF2021 showed that, at 5.7%, this proportion was virtually unchanged. The consequences of this for a research-informed profession are discussed.

Abstract This study provides causal evidence on how performance-based research funding affects gender diversity, using the UK’s transition from the Research Assessment Exercise to the Research Excellence Framework in 2009 as a natural experiment. Using difference-in-differences estimation, we compare twenty-four Russell Group UK universities with twenty-three matched US research-intensive universities from 2001 to 2021. Results demonstrate that performance-based funding increased female participation in collaborative research by 10.3 percentage points (0.90 standard deviations). Citation analysis reveals that increased female participation coincided with higher research impact, with treated papers receiving 4.79 more citations on average. Our findings suggest that performance-based research funding effectively promotes gender diversity while maintaining research quality. It is important to note, however, that increased female participation alone does not resolve or address the persistent gender pay disparities in UK academia.

This is a reply to 'Re-considering "impact" for longitudinal social science research: towards more scientific approaches to theorising and measuring the influence of cohort studies' by Charis Bridger Staatz, Evangeline Tabor and Dylan Kneale. Should the impact of longitudinal studies include wider dimensions compared to how research impact is defined by the Research Excellence Framework (REF)? The authors argue that this is because there are some unique challenges faced by longitudinal studies in generating and measuring impact. In particular impact can't just be tracked through physical documents and citations. The temporal nature of the studies means looking beyond individual pieces of research to understand the emerging themes in a body of work, and considering the impact of those themes. They also make the case that not all impact has to be economic or policy driven, and here the capacity building contribution of the studies within academia is vital. Both those findings are welcome. REF needs to better reflect the actual contribution of long-term social science studies, particularly as part of UK's data infrastructure. However, the impact of longitudinal studies is not simply constrained by design but also by short-termism in policy making, and the slow progress made on prevention policy across difference domains of life.

Demonstrating 'impact' has become increasingly important in research and academia alongside growing consideration of the social effects of research and the ethical standards involved. However, there are also concerns about a preoccupation with 'impact' in academia, which may result in a narrow focus on applied research. The most common definitions of impact (for example, UK Research Excellence Framework) emphasise generating measurable change outside of academia. However, this may overlook other important endeavours, such as influencing discourse and development of theory. The implicit assumption that single research projects will trigger measurable policy change is often unrealistic. Data infrastructures are also expected to demonstrate their 'impact', yet existing definitions are levied at the individual researcher or substantive projects. Such definitions do not account for the additional assumptions required for infrastructure to be impactful, and arguably, in their current form are not fit for purpose in demonstrating the full contribution of longitudinal and life course studies to society. We argue that broader definitions of impact should be considered for longitudinal studies and data infrastructure, that account for the importance of 'influence', and recognise the many and multifaceted contributions of longitudinal and life course research. Here we aim to (1) review definitions of impact in the context of longitudinal and life course studies, using UK national cohorts as a case studies; (2) to provide a working definition appropriate for longitudinal research, that incorporates 'influence'; and (3) to consider approaches that can be utilised to track impact.

Abstract This Forum piece highlights the value of collaborations between history academics and schools. It is based on recent work within History UK, an organization that promotes the interests of historians working within UK higher education, and it also highlights recent work within the Historical Association (HA). As school–university collaborations offer multiple opportunities, the article brings together different perspectives, including two case studies. The first case study summarizes experiences from an HA Teacher Fellowship that formed part of a wider project on ‘The People of 1381’, with school‐focused activities complementing other forms of public engagement. The second case study discusses how research and university‐based teaching on ‘Objects of Empire’ generated school sessions aimed at Year 8 students. Following on from these two specific examples, the Forum piece presents insights into how different forms of schools‐related activities developed public impact, as defined by, and captured in, the UK's Research Excellence Framework (REF).