Performance-based Funding Systems Research Articles

The game of academic publishing: a review of gamified publication practices in the social sciences

In the last decades, academic publishing in the Social Sciences has experienced a shift toward research management by non-academic stakeholders and performance-based funding systems (PBFSs). The resulting conditions of knowledge production and dissemination are increasingly described with “game” metaphors. This article provides a literature review of research concerning publishing in the Social Sciences and discusses how gamification becomes a key element. Quantifying publication outcomes to assess and financially incentivize research performance results in a highly competitive playing field where access to goods and services is denied to those who play the game poorly. The pressure to publish leads to unethical behavior and predatory publishing which are two side-effects of gamified practices. The reviewed literature also shows unequal starting conditions in terms of gender and language inequalities, as well as the dominance of the Global North. We conclude that the gamification of publication practices in the Social Sciences leads to stressful and dreadful environments.

Authorship regulations in performance-based funding systems and publication behaviour – A case study of German medical faculties

This article investigates whether German medical faculties with different authorship regulations show different publication patterns. In 2004, the German Research Foundation (DFG) suggested a formula consisting of third-party funding, the cumulated JIF of publications and a fractional counting of publications to counteract the increasing inflation of author counts in medical publications. Whereas the third-party funding and the JIF are generally used in research evaluation without variation, the authorship regulation differs among medical faculties. We therefore compare medical faculties using the DFG model - to credit first and last authors with a higher share than middle authors - with those faculties that apply whole counting. We answer the question whether the faculties with the different counting methods also show different authorship and publication behaviour, i.e., authorship and collaboration patterns, the choice of journals (JIF level) and the citation impact (share of highly-cited papers). Findings indicate a clear trend of increasing co-author numbers and of middle-author papers, irrespective of authorship regulation. Publications with DFG model have only a slightly lower average author count and lower shares of middle-author papers than whole-counted publications. Our findings suggest that the DFG regulation has not resulted in a reduction of the number of authors, which was a major aim. Moreover, the results show that the use of whole counting goes together with higher productivity and higher impact, which may be a good reason to select that model.

Accounting academic leaders’ perceptions on New Zealand’s performance-based research funding system

PurposeUniversities globally have faced the introduction of research performance assessment systems that provide monetary and ranking rewards based on publication outputs. This study aims to seek an understanding of the implementation of performance-based research funding (PBRF) and its impact on the heads of departments (HoDs) and accounting academics in New Zealand (NZ) tertiary institutions. The study explores NZ accounting academics’ experiences and their workload; the relationship between teaching and research in the accounting discipline and any issues and concerns affecting new and emerging accounting researchers because of PBRF.Design/methodology/approachApplying an institutional theoretical lens, this paper explores accounting HoDs’ perceptions concerning the PBRF system’s impact on their academic staff. The research used semi-structured interviews to collect data from NZ’s eight universities.FindingsThe key findings posit that many institutional processes, some more coercive in nature, whereas others were normative and mimetic, have been put in place to ensure that academics are able to meet the PBRF requirements. HoDs suggest that their staff understand the importance of research, but that PBRF is a challenge to new and emerging researchers and pose threats to their recruitment. New academics must “hit the ground running” as they must demonstrate not only teaching abilities but also already have a track record of research publications; all in all, a daunting experience for new academics to overcome. There is also a teaching and research disconnect. Furthermore, many areas where improvements can be made in the design of this measurement tool remain.Originality/valueThe PBRF system has significantly impacted on accounting academics. Central university research systems were established that subsequently applied coercive institutional pressures onto line managers to ensure that their staff performed. This finding offers scope for future research to explore a better PBRF that measures and rewards research productivity but without the current system’s unintended negative consequences.

Steering the Direction of Research through Organizational Identity Formation

Public research organizations respond to external pressures from national research evaluation systems, performance-based funding systems and university rankings by translating them into internal goals, rules and regulations and by developing organizational identities, profiles and missions. Organizational responses have primarily been studied at the central organizational level, and research on the steering of research has primarily focused on the impacts of performance-based funding systems. However, research evaluation exercises may also have a formative impact, especially below the central organizational level. This paper uses a case study of a research unit of a biomedical research school in the Netherlands to explore the organizational response to a relatively critical external assessment report. It shows that the participation in the Dutch research evaluation cycle legitimated the formation of a new organizational identity for the research unit, which functions as a frame that suggests to staff members a new interpretation of the type of research that is at the core of what the research unit does. We identify three additional steering mechanisms that support the enactment of the organizational identity: steering by resource allocation, by suggesting and by re-organizing. We, furthermore, explore the epistemic effects – the direction and conduct of research – of the organizational response, through interview data in combination with a bibliometric analysis.

Performance-based research funding: Evidence from the largest natural experiment worldwide

A performance-based research funding system (PRFS) is a nationwide incentive scheme that promotes and rewards university research performance through competition for government funding. The UK’s PRFS, currently the Research Excellence Framework (REF), is considered the oldest, largest and most developed payment-by-results system in academia worldwide. Surprisingly, and despite the strong criticisms, little has been done to quantitatively and casually evaluate the intended and unintended effects of the PRFSs. In this paper, we evaluate the incremental impact of the REF 2014 in the fields of Economics and Business. We use a synthetic control method to compare the performance of UK universities with their artificial counterfactual units constructed using data from US universities. Our analysis shows, on the whole, that the introduction of the REF had a significant and positive impact on the quantity and quality of the scientific research produced at UK universities. However, we do not find a significant effect on the per author measures, suggesting that the REF did not result in an increase in research productivity. We also show that the effects are more heterogeneous across universities than across academic disciplines. We do not find evidence of a shift of research focus from Economics to Business topics, as some feared. But our analysis indicates that the REF 2014 may have contributed to the concentration of research excellence in elite institutions.

Short-term incentives of research evaluations: Evidence from the UK Research Excellence Framework

We document incentive effects of the evaluation deadlines in the UK’s performance-based research funding system. Studying 3,597,272 publications by UK researchers, we find that publications just before assessment deadlines obtain substantially fewer citations and are published in venues with lower impact factors. These trends reverse abruptly after the deadlines. We discuss different factors that contribute to this observation and provide evidence that evaluation deadlines are likely to set incentives against investment in research quality and long-term topics. We conclude that where such shifts in research incentives are not intended, they might require balancing by additional incentives for exploratory, long-term oriented research.

Investing in the Workforce: The Impact of Performance-Based Funding on Student Earnings Outcomes

ABSTRACT A growing number of states use performance-based funding (PBF) systems to tie appropriations to student outcomes. Yet while many studies have examined the effects of PBF on enrollment and completion outcomes, no research has considered whether PBF affects post-college outcomes. This is of particular importance as more states directly incentivize colleges to improve student earnings and have students major in high-demand fields. We used the first detailed longitudinal dataset of state PBF policies to examine the effects of PBF on student earnings and found some modest positive effects concentrated among students who were already enrolled in college when PBF was first implemented. However, these small increases could also reflect other trends occurring prior to the implementation of PBF and analyses show this increase was not sustained among later cohorts.

Sources of convergence and divergence in university research quality: evidence from the performance-based research funding system in New Zealand

The introduction of performance-based research funding systems (PBRFS) in many countries has generated new information on their impacts. Recent research has considered whether such systems generate convergence or divergence of research quality across universities and academic disciplines. However, little attention has been given to the processes determining research quality changes. This paper utilises anonymised longitudinal researcher data over 15 years of the New Zealand PBRFS to evaluate whether research quality changes are characterised by convergence or divergence, and the processes determining those dynamics. A unique feature is the use of longitudinal data to decompose changes in researcher quality into contributions arising from the entry, exit and quality transformations of retained researchers, and their impacts on convergence or divergence of research quality across universities and disciplines. The paper also identifies how researcher dynamics vary systematically between universities and disciplines, providing new insights into the effects of these systems.

Creating evaluative homogeneity: Experience of constructing a national journal ranking

Abstract The study investigates the process of creating evaluative homogeneity within a performance-based research funding system. It does so in reference to the experience of creating a national journal ranking in Poland in 2019. It refers to the cases of two disciplines: biology and history. The use of this case is justified by its unique character, that is creating the ranking through means of both bibliometric indicators and expert panels. We therefore asked: What guided the actors who participated in the process of creating the ranking through the use of bibliometric indicators when introducing changes to its initial form? To answer this question, a mixed-methods approach was used. First, we conducted a quantitative analysis of changes made to the ranking during the consecutive steps of its creation. Second, we conducted a qualitative analysis of semi-structured interviews regarding actors’ motivations. By reflecting on the qualitative analysis along with the results of the quantitative part, this study reveals the extent to which actors impacted the form evaluative homogeneity took within the system. It further argues that this is dictated by how actors position themselves in relation to two opposing forces: those favouring the homogenization of research and those maintaining its heterogeneity. As these forces remain imbalanced, the study concludes with a call for further exploration of the interplay between the forces of homogenization and heterogenization and how the tensions between them are mediated within a performance-based research funding system.

New Evidence on the Evolution and Landscape of Performance Funding for Higher Education

ABSTRACT This study draws upon the first detailed longitudinal dataset on performance-based funding (PBF) to document the evolution and current landscape of PBF in American higher education. We show that while PBF has become increasingly common, states have experimented with adopting, abandoning, and re-adopting PBF over time. We also find a new wave of PBF adoption occurring in the 2010s following the Great Recession. However, PBF remains a relatively weak policy lever with a small share of funds at stake in most states and some states not funding PBF even when it exists in legislation. Equity considerations vary in the student groups they include and less than half of two-year and three-quarters of four-year PBF systems include race when allocating performance funds. Our findings complicate some common characterizations of PBF and offer new insight for researchers examining how PBF policy design shapes student and institutional outcomes.

The Uptake and Impact of a Label for Peer-Reviewed Books.

This article presents an analysis of the uptake of the GPRC label (Guaranteed Peer Reviewed Content label) since its introduction in 2010 until 2019. GPRC is a label for books that have been peer reviewed introduced by the Flemish publishers association. The GPRC label allows locally published scholarly books to be included in the regional database for the Social Sciences and Humanities which is used in the Flemish performance-based research funding system. Ten years after the start of the GPRC label, this is the first systematic analysis of the uptake of the label. We use a mix of qualitative and quantitative methods. Our two main data sources are the Flemish regional database for the Social Sciences and Humanities, which currently includes 2,580 GPRC-labeled publications, and three interviews with experts on the GPRC label. Firstly, we study the importance of the label in the Flemish performance-based research funding system. Secondly, we analyse the label in terms of its possible effect on multilingualism and the local or international orientation of publications. Thirdly, we analyse to what extent the label has been used by the different disciplines. Lastly, we discuss the potential implications of the label for the peer review process among book publishers. We find that the GPRC label is of limited importance to the Flemish performance-based research funding system. However, we also conclude that the label has a specific use for locally oriented book publications and in particular for the discipline Law. Furthermore, by requiring publishers to adhere to a formalized peer review procedure, the label affects the peer review practices of local publishers because not all book publishers were using a formal system of peer review before the introduction of the label and even at those publishers who already practiced peer review, the label may have required the publishers to make these procedures more uniform.

Improvement of Quality of Higher Education Institutions as a Basis for Improvement of Quality of Life

This paper aims to propose a quality assessment model for higher education institutions in the technical–technological field and a system for decision support and optimal management strategies for quality improvement. Obtaining research results is based on surveying stakeholders in higher education and obtaining quantitative data regarding key performance indices. Quantitative data and the genetic algorithm method are applied to determine optimal management strategies for quality improvement. Quality in the higher education sector is among the current issues in the academic community. By monitoring and researching the higher education field and analysing the literature and the current situation in the system of higher education in developing countries, it can be concluded that there is no single way to assess the quality of higher education institutions. This knowledge was a good starting point for the research presented in this paper. Accordingly, the findings include developing a system for quality assessment and the ranking of higher education institutions. Additionally, evaluating the relevance of key performance indicators of higher education institutions differs from different stakeholder perspectives. However, it is possible to develop a system for decision support and the selection of the optimal strategy for improving the performance of study programs and higher education institutions with regard to quality. The practical implications include defining a decision support system that enables the adoption of optimal decisions by the management teams of higher education institutions to improve study programs and the performance of the higher education institutions. The presented system may enable the benchmarking, simulation, and verification of different scenarios for improving the quality and performance of higher education institutions. In this paper, the authors analysed the characteristics, benefits, and drawbacks of different ranking systems to develop and introduce a novel ranking system that suggests weights for the ranking criteria and different perspectives regarding new digital age requirements. The model was tested, and the results are presented to demonstrate the advantages of the developed model. The originality of the research lies in the presented novel model that can be made available to government institutions and serve as a basis for the overall ranking and evaluation of higher education institutions, with the possibility of developing a performance-based funding system. Additionally, other stakeholders can gain an insight into the performance of an institution in relation to their needs and goals.

Academic research evaluation in artistic disciplines: the case of Poland

This study focuses on the evaluation of artistic disciplines (visual and performing arts) within performance-based university research funding systems. It offers an analysis of the Polish academic research evaluation system and investigates its effects on the scholarly productivity of artists-academics. Poland has adopted a performance-based research funding system that includes evaluation of artistic productions (without considering them as ‘research’) and developed a comprehensive quantification of artistic achievements. We analyse data from the Polish Current Research Information System on 136,894 outputs submitted by 93 art faculties for two evaluation exercises (in 2013 and 2017). Our findings demonstrate a change in scholarly productivity of artists-academics towards publication practices characteristic for science-oriented disciplines. We also find that art departments located in universities exhibit a greater tendency for changing publication practices than departments located in independent art schools. The findings from this study demonstrate that both: (1) the designs of national research evaluation systems and (2) the structures of higher education may influence scholarly outcomes in practice-based disciplines.

Science policy as implicit cultural policy: evaluation of the arts in Polish academia

ABSTRACT This paper is concerned with the evaluation of artistic production within the Polish science policy. It analyses the performance-based research funding system (PRFS) in Polish higher arts education and discusses the assessment techniques and value hierarchies underlying the evaluation model. We analyse data from the Integrated System of Information on Science and Higher Education, to examine the cultural activity of artists-academics in Poland and discuss the impact of science policy on the cultural sector. We also compare the Polish system with alternative solutions introduced in other countries (e.g. UK, Australia, Finland, Sweden) and consider possible consequences of different evaluation paradigms.

Competitive accountability and the dispossession of academic identity: Haunted by an impact phantom

This article discusses the intensification of research performance demands in UK universities in relation to the complex terrain of academic identity formation. It considers whether a demand for academic researchers to produce and evidence economic and societal impact – in the rewards game of the UK’s performance-based research funding system, the Research Excellence Framework (REF) – influences their self-concept as ‘engaged researchers’. While a designation of being REF impactful may be considered constitutive to a researcher’s sense of self-worth and advantageous to their professional and institutional profile, a consultation of researchers included within REF2014 impact case studies challenges these assumptions. Instead, respondents are found to complain of identity dispossession and exploitation by their universities where their public contributions are appropriated for positional gain. Their testimony confirms the prevalence of a culture of ‘competitive accountability’ across UK universities which is with a systemic insatiability for ‘scholarly distinction’, causing the privileging of appearance in rationalisations of publicly funded research. Using the theoretical insights of Guy Debord and Erving Goffman it is argued that REF impact elucidates the UK higher education sector as a ‘society of the Spectacle’ that subjugates ‘authentic’ versions of the academic Self. However, REF-impact is also seen to provide an opportunity for cultural detournément and a means to elicit and concurrently invert ‘simulations’ of research praxis’, thus enabling the assertion or ‘front-staging’ of perceived and idealised academic identities.

Just a Matter of Time? Women’s Career Advancement in Neo-Liberal Academia. An Analysis of Recruitment Trends in Italian Universities

Recently, the Italian higher education system has experienced two profound changes: the strong feminization of its academic staff and the implementation of market-based reforms aimed at fostering cost efficiency and economic productivity. Such reforms include the reshaping of the academic career ladder envisaged by the last university reform, the so called Gelmini reform (law 240/2010), and the adoption of a performance-based funding system. Both elements occurred in parallel with a strong cut in turnover. By accessing unique data on recruitment covering the last two decades, which were provided by the Italian Ministry of Education, University, and Research’s statistical office, this study aims at investigating these changes from a gendered perspective. More specifically, it firstly aims at analyzing if the feminization of the academic staff is due to an effective improvement of gender equality in recruitment or, rather, to demographic dynamics; secondly, it investigates to what extent the recent neo-liberal transformations, and more specifically the reshaping of the career structure combined with the limitations on hiring, has had any implications in terms of women’s recruitment and advancement. The results suggest that the road to gender equality is extremely slow and non-linear. The introduction, with the Gelmini reform, of the new fixed-term assistant professor has tightened female access to the tenure track. Moreover, female recruitment remained substantially unchanged over the period among associate and full professors, thus suggesting that the feminization of the academic staff is not due to an effective improvement of gender equality in recruitment, but also to demographic dynamics, such as the retirement of men who are concentrated in the older cohorts.

Performance-Based Funding in American Higher Education: A Systematic Synthesis of the Intended and Unintended Consequences

This systematic synthesis examines the intended and unintended consequences of performance-based funding (PBF) policies in higher education. Within this synthesis, we focus particularly on evidence from research studies with strong causal inference designs in an effort to understand the impacts of these policies. PBF adoption is generally associated with null or modest positive effects on the intended outcomes of retention and graduation, but there is also compelling evidence that PBF policies lead to unintended outcomes related to restricting access, gaming of the PBF system, and disadvantages for underserved student groups and under-resourced institution types. PBF policies including equity provisions for colleges that enroll or graduate underserved student groups have been found to offset some of these unintended consequences. Based on this synthesis, we offer policy recommendations and directions for future research on PBF policies, including the need to consider variation in policy design and the broader state policy context.

Fast growth of the number of proceedings papers in atypical fields in the Czech Republic is a likely consequence of the national performance-based research funding system

Abstract This article compares the growth rates of Web of Science indexed research publications in 11 European countries from 2000 to 2015. The growth of publication output in the Czech Republic was very fast: the annual production increased more than threefold. However, the number of articles increased only 2.6-fold, whereas the number of proceedings papers increased eightfold. During 2013–15 almost one-third of the papers published by researchers based in the Czech Republic were proceedings papers, a much higher share than in any other benchmark country. Such a high share is undesirable, because proceedings papers are generally much less often cited than journal articles. The growth of the number of proceedings papers is fastest in 17 fields belonging to the social sciences which usually do not hold proceedings papers in high regard. Our analysis shows that this undesirable development started after the introduction of national performance-based research funding system (PRFS) in the Czech Republic. Hence, the effort made to publish proceedings papers seems to reflect an optimization strategy in the light of the PRFS. In the Czech PRFS, proceedings papers have been rewarded point values similar to articles in refereed journals and a large portion of the institutional funding is allocated according to the evaluation results. As a consequence of very fast growth of proceedings papers in the social sciences, the university institutional funding in these fields has grown faster than in other fields. In conclusion, the fast growth of proceedings papers represents an adaptive behavior to the context of the PRFS.

The increasing significance of impact within the Research Excellence Framework (REF)

IntroductionThe UK's Research Excellence Framework (REF) is a performance-based research funding system that was introduced in 2014. REF is a process of expert review in universities which assesses the quality and reach of research across higher education. It highlights areas of excellence, benchmarks the quality of previously funded research and provides accountability for ongoing public investment in research. In addition to assessing the quality of research outputs and the environment in which research is undertaken, the REF became the first exercise to evaluate the impact of research outside of academia. DiscussionThis article specifically addresses the impact component of the REF, explaining what is understood by the term ‘impact’ and considering how impact is gaining significance within the REF framework as a measure of research excellence. It also reflects on the complexities of measuring impact and ongoing resistance from academics to the impact agenda, which they believe threatens academic freedom and appears only to value research that generates specific outcomes. In order to address these and other issues arising from the first-ever assessment of impact in REF 2014, institutions were consulted on the proposed guidance for the REF 2021 submission. ConclusionThe final REF 2021 guidance has sought to provide clearer detail on interpretations of impact and its underpinning research in order to reassure researchers that their academic freedom will not come under threat. However, this process still fails to capture other impactful research being undertaken across institutions, which falls outside of the strict REF criteria.

What Brings International Students to Norway?

What Brings International Students to Norway?