Performance-based Research Funding Research Articles

Unpacking the discourse surrounding the impact agenda in the Hong Kong Research Assessment Exercise 2020

Abstract Research with economic utility and social value has been increasingly valued. Such an emphasis can be evidenced by the newly included assessment element of ‘societal impact’ in performance-based research funding (PBRF) schemes in different higher education systems around the world. This paper investigates how the non-academic impact is constructed and perceived in the Hong Kong Research Assessment Exercise (RAE) 2020, taking into account the local socio-cultural characteristics in the context of Hong Kong. Data sources include 13 impact case studies in the education panel submitted for the Hong Kong RAE 2020 and semi-structured interview with 17 education academics in Hong Kong. Findings revealed that the non-academic impact was constructed through a narrative pattern: (1) problem identification: tensions and synergies between local and international discourse; (2) problem resolution: prioritization of the evidence-based applied education research (with funding); (3) resolution dissemination: strategic employment of promotional genre. The paper discusses how decolonization, academic entrepreneurialism and collectivist culture have characterized the framing and understanding of the non-academic impact in the Hong Kong academia, contributing to the discourse on neoliberalism in higher education by providing a nuanced, local perspective on the impact agenda. Policy implications for a more localized and flexible impact agenda are also provided.

Accounting academic leaders’ perceptions on New Zealand’s performance-based research funding system

Purpose Universities globally have faced the introduction of research performance assessment systems that provide monetary and ranking rewards based on publication outputs. This study aims to seek an understanding of the implementation of performance-based research funding (PBRF) and its impact on the heads of departments (HoDs) and accounting academics in New Zealand (NZ) tertiary institutions. The study explores NZ accounting academics’ experiences and their workload; the relationship between teaching and research in the accounting discipline and any issues and concerns affecting new and emerging accounting researchers because of PBRF. Design/methodology/approach Applying an institutional theoretical lens, this paper explores accounting HoDs’ perceptions concerning the PBRF system’s impact on their academic staff. The research used semi-structured interviews to collect data from NZ’s eight universities. Findings The key findings posit that many institutional processes, some more coercive in nature, whereas others were normative and mimetic, have been put in place to ensure that academics are able to meet the PBRF requirements. HoDs suggest that their staff understand the importance of research, but that PBRF is a challenge to new and emerging researchers and pose threats to their recruitment. New academics must “hit the ground running” as they must demonstrate not only teaching abilities but also already have a track record of research publications; all in all, a daunting experience for new academics to overcome. There is also a teaching and research disconnect. Furthermore, many areas where improvements can be made in the design of this measurement tool remain. Originality/value The PBRF system has significantly impacted on accounting academics. Central university research systems were established that subsequently applied coercive institutional pressures onto line managers to ensure that their staff performed. This finding offers scope for future research to explore a better PBRF that measures and rewards research productivity but without the current system’s unintended negative consequences.

Warnings of declining research productivity: Does Italy buck the trend?

Abstract The paper takes a scientometric approach to measure the change in research productivity of Italian academics before the COVID-19 pandemic outbreak. We propose a composite output/input bibliometric indicator and apply it at the field level, conducting a longitudinal analysis. Although the number of academics in the national academic system has decreased, we register very strong growth in both the number of publications and their scholarly impact. The growth in productivity, with only rare exceptions, crosses almost all fields. However, in areas that are traditionally very internationalized (Biology, Physics, and Chemistry), growth is less sustained than overall average, and also the variability of productivity across fields seems reduced. The main reason for this detail would be the smaller margins for improvement in the fields that had already reached high international standing. What emerges from the analysis goes counter to some alarms of declining scientific productivity at the global level. The Italian case is partly explained by the historic adoption of policies aimed at strengthening competitive mechanisms, in particular through the introduction of systems of performance-based research funding, and bibliometric accreditation for professorship.

Research quality criteria in the Creative Arts

ABSTRACT This paper investigates research quality criteria in the Creative Arts (CA). The CA has been introduced into the higher education and research sector over the last three decades. It is thus a relatively new research field and there is little empirical knowledge on how outputs in this field should be evaluated. Our study applies a mixed-method approach to assess the relevance of quality criteria used in performance-based research funding systems (PRFSs) in 10 countries. The results of a qualitative analysis of interviews with artists-academics (N = 67) and Joint Correspondence Analysis show that when art is evaluated in the context of academic research, both the traditional indicators of artistic quality as well as the cognitive and research-related aspects of the arts are believed to be significant. The JCA analysis also showed that the majority of our respondents found both extrinsic quality criteria (related to reputation and prestige) and intrinsic criteria (related to cognition and development) relevant.

Do research performances of universities and disciplines in England converge or diverge? An assessment of the progress between research excellence frameworks in 2014 and 2021

Performance-based research funding systems (PBRFSs) have been used in selectively distributing research funding, increasing public money accountability and efficiency. Two recent such evaluations in England were called the Research Excellence Framework (REF), which took place in 2014 and 2021, and the research environment, outputs and impact of the research were evaluated. Even though various aspects of the REF were examined, there has been limited research on how the performance of the universities and disciplines changed between the two evaluation periods. This paper assesses whether there has been convergence or divergence in research quality across universities and subject areas between 2014 and 2021 and found that there was an absolute convergence between universities in all three research elements evaluated, and universities that performed relatively worse in REF in 2014 experienced higher growth in their performance between 2014 and 2021. There was also an absolute convergence in the research environment and impact across different subject areas, but there is no significant convergence in the quality of research outputs across disciplines. Our findings also highlight that there has been an absolute convergence in research quality within the universities (between different disciplines in a given university) and within disciplines (between universities in a given subject).

Performance-based research funding: Evidence from the largest natural experiment worldwide

A performance-based research funding system (PRFS) is a nationwide incentive scheme that promotes and rewards university research performance through competition for government funding. The UK’s PRFS, currently the Research Excellence Framework (REF), is considered the oldest, largest and most developed payment-by-results system in academia worldwide. Surprisingly, and despite the strong criticisms, little has been done to quantitatively and casually evaluate the intended and unintended effects of the PRFSs. In this paper, we evaluate the incremental impact of the REF 2014 in the fields of Economics and Business. We use a synthetic control method to compare the performance of UK universities with their artificial counterfactual units constructed using data from US universities. Our analysis shows, on the whole, that the introduction of the REF had a significant and positive impact on the quantity and quality of the scientific research produced at UK universities. However, we do not find a significant effect on the per author measures, suggesting that the REF did not result in an increase in research productivity. We also show that the effects are more heterogeneous across universities than across academic disciplines. We do not find evidence of a shift of research focus from Economics to Business topics, as some feared. But our analysis indicates that the REF 2014 may have contributed to the concentration of research excellence in elite institutions.

Short-term incentives of research evaluations: Evidence from the UK Research Excellence Framework

We document incentive effects of the evaluation deadlines in the UK’s performance-based research funding system. Studying 3,597,272 publications by UK researchers, we find that publications just before assessment deadlines obtain substantially fewer citations and are published in venues with lower impact factors. These trends reverse abruptly after the deadlines. We discuss different factors that contribute to this observation and provide evidence that evaluation deadlines are likely to set incentives against investment in research quality and long-term topics. We conclude that where such shifts in research incentives are not intended, they might require balancing by additional incentives for exploratory, long-term oriented research.

A Bibliometric Exploration of Events Literature In and Outside the Core Journals

There are various positions on what literature represents "core" event management research and to what degree "peripheral" tourism and hospitality (T&H) literature influences and overlaps with this field. Previous literature reviews have made progress but methodological challenges remain. The influences of journal lists, performance-based research funding systems, and selective databases are discussed regarding the journal selection of events researchers. An exploratory approach to charting the core and periphery of events-related research was taken. Firstly, the relevance of a wide range of journals to event management research was tested using a measure of interrater agreement. Twenty-four of the 53 journals reviewed were selected as relevant. Secondly, a bibliometric analysis of highly cited articles was carried out. Two hundred seventy-two of 480 articles from 2008–2018 were selected and further examined using citation analysis. Three, four-star journals accounted for 60% of all references within the sample, despite only accounting for 23% of the total article volume. Quantitative research and articles about festivals or sports events were predominant in the sample, although the research methodology used and types of events chosen by individual articles had limited or no effect on the number of citations received. Most (77%) authors were based in Australia, Europe, New Zealand, and North America. Future reviews are advised to investigate blended approaches, open data, nonselective databases, and international perspectives.

The impact of Italian performance-based research funding systems on the intensity of international research collaboration

Abstract The study of national research assessment exercises serves to evaluate the effectiveness of policies versus their objectives and to improve the formulation of future initiatives. The aim of the current study is to verify whether the introduction of the first performance-based research funding in Italy, based on the 2004–10 VQR assessment, achieved the intended objective of inducing greater international collaboration on the part of researchers. For this, we apply a bibliometric approach based on the observation of coauthorships in Italian and worldwide scientific publications over a 14-year period, beginning in the target years of the VQR assessment. Through an Interrupted Time Series Analysis, we compare the expected and observed patterns of international coauthorship for Italy and the rest of the world. Although the rate of internationalization of Italian research is seen to be increasing, whether this is a consequence of the VQR incentives, or rather part of a global phenomenon of recourse to international collaboration in response to the increasingly complex scientific challenges, it is open to interpretation.

Sources of convergence and divergence in university research quality: evidence from the performance-based research funding system in New Zealand

The introduction of performance-based research funding systems (PBRFS) in many countries has generated new information on their impacts. Recent research has considered whether such systems generate convergence or divergence of research quality across universities and academic disciplines. However, little attention has been given to the processes determining research quality changes. This paper utilises anonymised longitudinal researcher data over 15 years of the New Zealand PBRFS to evaluate whether research quality changes are characterised by convergence or divergence, and the processes determining those dynamics. A unique feature is the use of longitudinal data to decompose changes in researcher quality into contributions arising from the entry, exit and quality transformations of retained researchers, and their impacts on convergence or divergence of research quality across universities and disciplines. The paper also identifies how researcher dynamics vary systematically between universities and disciplines, providing new insights into the effects of these systems.

Creating evaluative homogeneity: Experience of constructing a national journal ranking

Abstract The study investigates the process of creating evaluative homogeneity within a performance-based research funding system. It does so in reference to the experience of creating a national journal ranking in Poland in 2019. It refers to the cases of two disciplines: biology and history. The use of this case is justified by its unique character, that is creating the ranking through means of both bibliometric indicators and expert panels. We therefore asked: What guided the actors who participated in the process of creating the ranking through the use of bibliometric indicators when introducing changes to its initial form? To answer this question, a mixed-methods approach was used. First, we conducted a quantitative analysis of changes made to the ranking during the consecutive steps of its creation. Second, we conducted a qualitative analysis of semi-structured interviews regarding actors’ motivations. By reflecting on the qualitative analysis along with the results of the quantitative part, this study reveals the extent to which actors impacted the form evaluative homogeneity took within the system. It further argues that this is dictated by how actors position themselves in relation to two opposing forces: those favouring the homogenization of research and those maintaining its heterogeneity. As these forces remain imbalanced, the study concludes with a call for further exploration of the interplay between the forces of homogenization and heterogenization and how the tensions between them are mediated within a performance-based research funding system.

The Uptake and Impact of a Label for Peer-Reviewed Books.

This article presents an analysis of the uptake of the GPRC label (Guaranteed Peer Reviewed Content label) since its introduction in 2010 until 2019. GPRC is a label for books that have been peer reviewed introduced by the Flemish publishers association. The GPRC label allows locally published scholarly books to be included in the regional database for the Social Sciences and Humanities which is used in the Flemish performance-based research funding system. Ten years after the start of the GPRC label, this is the first systematic analysis of the uptake of the label. We use a mix of qualitative and quantitative methods. Our two main data sources are the Flemish regional database for the Social Sciences and Humanities, which currently includes 2,580 GPRC-labeled publications, and three interviews with experts on the GPRC label. Firstly, we study the importance of the label in the Flemish performance-based research funding system. Secondly, we analyse the label in terms of its possible effect on multilingualism and the local or international orientation of publications. Thirdly, we analyse to what extent the label has been used by the different disciplines. Lastly, we discuss the potential implications of the label for the peer review process among book publishers. We find that the GPRC label is of limited importance to the Flemish performance-based research funding system. However, we also conclude that the label has a specific use for locally oriented book publications and in particular for the discipline Law. Furthermore, by requiring publishers to adhere to a formalized peer review procedure, the label affects the peer review practices of local publishers because not all book publishers were using a formal system of peer review before the introduction of the label and even at those publishers who already practiced peer review, the label may have required the publishers to make these procedures more uniform.

The different responses of universities to introduction of performance-based research funding

Abstract Governments and organizations design performance-based research funding systems (PBRFS) for strategic aims, such as to selectively allocate scarce resources and stimulate research efficiency. In this work, we analyze the relative change in research productivity of Italian universities after the introduction of such a system, featuring financial and reputational incentives. Using a bibliometric approach, we compare the relative research performance of universities before and after the introduction of PBRFS, at the overall, discipline and field levels. The findings show convergence in the universities’ performance, due above all to the remarkable improvement of the lowest performers. Geographically, the universities of the south (vs central and northern Italy) achieved the greatest improvement in relative performance. The methodology, and results, should be of use to university management and policy-makers.

Disadvantages of Patent Performance Indicators in Performance-Based Research Funding Systems

Although the use of patent indicators in statistics has been under discussion for a long time, there are surprisingly very few studies devoted to explaining the use of patent indicators as performance indicators in performance-based research funding systems. The widespread assumption is based on the perception that patents have some recognizable value as an element of innovation process and this value is undoubtedly transferred to patents that are created in order to fulfill performance indicators. However, there is still no established distinction in the literature between patent indicators used in statistical analysis and patent performance indicators used as goal setting and evaluation tools in the context of performance-based research funding systems. This article is aimed at filling the gap and seeks to clarify the theoretical validity of the use of patent performance indicators in such context. Based on the review of the literature, the article compares the results of seminal works on the nature of patenting and patent indicators to the purposes of its use as patent performance indicators in performance-based research funding systems. The results demonstrate at least five arguments in favor of the statement that the use of patent indicators in the context of performance-based research funding systems requires rethinking.

Performance-Based Research Funding and Its Impacts on Academics’ Publication Patterns in South Korea

Performance-Based Research Funding and Its Impacts on Academics’ Publication Patterns in South Korea

Performance-based publisher ratings and the visibility/impact of books: Small fish in a big pond, or big fish in a small pond?

Abstract This study compares publisher ratings to the visibility and impact of individual books, based on a 2017 data set from three Nordic performance-based research funding systems (PRFS) (Denmark, Norway, and Finland). Although there are Journal Impact Factors (JIFs) for journals, there is no similar indicator for book publishers. National publisher lists are used instead to account for the general “quality” of books, leading to institutional rewards. But, just as the JIF is not recommended as a proxy for the “citedness” of a paper, a publisher rating is also not recommended as a proxy for the impact of an individual book. We introduce a small fish in a big pond versus big fish in a small pond metaphor, where a “fish” is a book and “the pond” represents its publishing house. We investigate how books fit on this metaphorical fish and pond continuum, using WorldCat holdings (visibility) and Google Scholar citations (impact), and test other variables to determine their predictive value with respect to these two indicators. Our statistics show that publisher levels do not have predictive value when other variables are held constant. This has implications for PRFS and book evaluations in general, as well as ongoing developments related to a newly proposed international publisher registry.

Performance-based Institutional Research Funding in Flanders, Belgium

Research and innovation is one of Flanders’ priorities and over the last three decades its public funding has strongly increased. Universities are key actors in this strategy. They have a large autonomy and receive a substantial share of additional R&D expenditures as lump sum funding.The Flemish authorities use quantitative indicators to allocate these lump sums to the universities. The funding formulae take into account each institution’s size, its research performance and, if relevant, its valorization activities. Also significant is the realization of governmental priorities, such as mobility and diversity of the academic staff. This paper describes the development of the Flemish university funding model, analyses its weaknesses and its strengths, and compares it with nine national metrics-based research performance funding systems.Policy highlightsFlanders, like many other regions and nations, has adopted performance-based research funding systems (PRFSs) to improve and provide accountability for its science and innovation system. The Flemish PRFS criteria have evolved considerably over the last three decades, and due to competition between universities and a consensus model of political decision making, the funding formula is comparatively complex.Building on an historical background of lump sum payments to universities, supporting both education and research, special supplementary funds for blue-sky research (BOF) and for strategic applied research, innovation, and outreach activities (IOF) were introduced by the Flemish government in 1994 and 2004, respectively. The “three-legged stool” funding mechanism for research in Flanders is unique within Europe.The introduction of publication and citation metrics in 2003 changed the character of the Flemish PRFS considerably, previously focused on measures of a university’s size in terms of students, degrees granted, and previous funding. Humanities and social sciences (HSS) research metrics were considered only after first adoption of bibliometric methods traditionally deployed for assessment of the natural and life sciences, but this is not unique to Flanders. HSS research has been addressed in Flanders through the creation of a special database to supplement standard citation indexes, the VABB-SHW. This special resource is a strong point of the Flemish PRFS, and other nations should appreciate the value of increased coverage of the HSS literature.The PRFS in Flanders differs from that of nine European nations in several ways: analysis is undertaken annually, it includes a diversity and mobility measure (2006), and one of interdisciplinary will soon be added.Flanders, like other nations using PRFSs, has generally seen increased research output and intellectual property activities after its introduction; however, in some nations the increase began before a PRFS was implemented. In almost all cases, demonstrating a causal link between a PRFS and increased output is difficult since R&D investments and the number of researchers have also increased for a variety of reasons. This is a cautionary note in not over-interpreting the effects or effectiveness of a PRFS.Flanders now exhibits one of the highest levels among European nations of research funding allocations determined by a PRFS: 50% of BOF-funding and 75% of IOF-funding. By employing such a metrics-heavy scheme, Flanders is a good candidate for a detailed study of unintended consequences of a PRFS, at the national, institutional, and research group and individual researcher level. This remains a large gap in our understanding of the use of a PRFS.

Academic research evaluation in artistic disciplines: the case of Poland

This study focuses on the evaluation of artistic disciplines (visual and performing arts) within performance-based university research funding systems. It offers an analysis of the Polish academic research evaluation system and investigates its effects on the scholarly productivity of artists-academics. Poland has adopted a performance-based research funding system that includes evaluation of artistic productions (without considering them as ‘research’) and developed a comprehensive quantification of artistic achievements. We analyse data from the Polish Current Research Information System on 136,894 outputs submitted by 93 art faculties for two evaluation exercises (in 2013 and 2017). Our findings demonstrate a change in scholarly productivity of artists-academics towards publication practices characteristic for science-oriented disciplines. We also find that art departments located in universities exhibit a greater tendency for changing publication practices than departments located in independent art schools. The findings from this study demonstrate that both: (1) the designs of national research evaluation systems and (2) the structures of higher education may influence scholarly outcomes in practice-based disciplines.

Science policy as implicit cultural policy: evaluation of the arts in Polish academia

ABSTRACT This paper is concerned with the evaluation of artistic production within the Polish science policy. It analyses the performance-based research funding system (PRFS) in Polish higher arts education and discusses the assessment techniques and value hierarchies underlying the evaluation model. We analyse data from the Integrated System of Information on Science and Higher Education, to examine the cultural activity of artists-academics in Poland and discuss the impact of science policy on the cultural sector. We also compare the Polish system with alternative solutions introduced in other countries (e.g. UK, Australia, Finland, Sweden) and consider possible consequences of different evaluation paradigms.

Competitive accountability and the dispossession of academic identity: Haunted by an impact phantom

This article discusses the intensification of research performance demands in UK universities in relation to the complex terrain of academic identity formation. It considers whether a demand for academic researchers to produce and evidence economic and societal impact – in the rewards game of the UK’s performance-based research funding system, the Research Excellence Framework (REF) – influences their self-concept as ‘engaged researchers’. While a designation of being REF impactful may be considered constitutive to a researcher’s sense of self-worth and advantageous to their professional and institutional profile, a consultation of researchers included within REF2014 impact case studies challenges these assumptions. Instead, respondents are found to complain of identity dispossession and exploitation by their universities where their public contributions are appropriated for positional gain. Their testimony confirms the prevalence of a culture of ‘competitive accountability’ across UK universities which is with a systemic insatiability for ‘scholarly distinction’, causing the privileging of appearance in rationalisations of publicly funded research. Using the theoretical insights of Guy Debord and Erving Goffman it is argued that REF impact elucidates the UK higher education sector as a ‘society of the Spectacle’ that subjugates ‘authentic’ versions of the academic Self. However, REF-impact is also seen to provide an opportunity for cultural detournément and a means to elicit and concurrently invert ‘simulations’ of research praxis’, thus enabling the assertion or ‘front-staging’ of perceived and idealised academic identities.