Performance-based Funding Systems Research Articles

Italy’s national research assessment: some unpleasant effects

ABSTRACTHigher education institutions are in a period of profound transformation. In particular, many governments have implemented mechanisms that attempt to relate funding to performance. In this article, we reflect about the relevance of those changes from a regional point of view, by analysing the recent introduction of a performance-based research funding system in Italy. The findings lead to policy indications, questioning the effectiveness of selective funding of universities based on national research assessment exercises when there is heterogeneity across regions in universities scientific performance and, at the same time, there is a huge dispersion of research performance within the individual universities.

Identifying publications in questionable journals in the context of performance-based research funding.

In this article we discuss the five yearly screenings for publications in questionable journals which have been carried out in the context of the performance-based research funding model in Flanders, Belgium. The Flemish funding model expanded from 2010 onwards, with a comprehensive bibliographic database for research output in the social sciences and humanities. Along with an overview of the procedures followed during the screenings for articles in questionable journals submitted for inclusion in this database, we present a bibliographic analysis of the publications identified. First, we show how the yearly number of publications in questionable journals has evolved over the period 2003-2016. Second, we present a disciplinary classification of the identified journals. In the third part of the results section, three authorship characteristics are discussed: multi-authorship, the seniority-or experience level-of authors in general and of the first author in particular, and the relation of the disciplinary scope of the journal (cognitive classification) with the departmental affiliation of the authors (organizational classification). Our results regarding yearly rates of publications in questionable journals indicate that awareness of the risks of questionable journals does not lead to a turn away from open access in general. The number of publications in open access journals rises every year, while the number of publications in questionable journals decreases from 2012 onwards. We find further that both early career and more senior researchers publish in questionable journals. We show that the average proportion of senior authors contributing to publications in questionable journals is somewhat higher than that for publications in open access journals. In addition, this paper yields insight into the extent to which publications in questionable journals pose a threat to the public and political legitimacy of a performance-based research funding system of a western European region. We include concrete suggestions for those tasked with maintaining bibliographic databases and screening for publications in questionable journals.

Exploring the Relationship Between Performance-Based Funding Design and Underrepresented Student Enrollment at Community Colleges

Objective: An increasing number of states are adopting performance-based funding (PBF) systems for their public colleges, but there are concerns that PBF dissuades colleges from recruiting and enrolling students with a lower likelihood of success. Some states have attempted to address this concern by providing additional funds for successfully serving low-income, underrepresented minority, or adult students, but the effectiveness of these particular provisions has yet to be examined among 2-year colleges. I explore whether these provisions have affected historically underrepresented student enrollments at community colleges. Method: I use generalized difference-in-difference panel regression techniques combined with data from the Integrated Postsecondary Education Data System and information from states on their performance funding policies from the 2004-2005 to 2014-2015 academic years to address the research questions. I classified colleges based on whether they had a PBF system with equity provisions, a PBF system without equity provisions, or no PBF in a given year and compared these three groups of institutions. Results: I find little evidence that PBF policies, regardless of their design characteristics, have a relationship with traditionally underrepresented student enrollment levels. Contributions: Although there are concerns that PBF systems induce community colleges to selectively recruit students with a higher probability of success, I find no systemic evidence of that practice. However, as PBF systems become higher stakes and more ingrained in state higher education funding, this finding deserves further study.

Funding for Results in Higher Education

The Ontario government’s plan to make 60 percent of its transfers to colleges and universities based on results could transform the way higher education is funded, even if the details remain sketchy, says a new report from the C.D. Howe Institute. In “Funding for Results in Higher Education”, author Alex Usher looks at other Performance Based Funding (PBF) plans around the world, and notes that Canada is playing catch-up with many American states as well as European countries such as Denmark, the United Kingdom, France, Finland, Norway, and the Netherlands, who already use these funding models extensively. He then goes on to assess the details of the Ontario proposal – which would create the largest PBF system in North America.

The missing link: How university managers mediate the impact of a performance-based research funding system

The impact of Performance-based Research Funding System (PRFS) has received increasing attention in recent years. However, the literature has focused on individual-level or country-level effects, mostly ignoring the ‘missing link’ between PRFSs and their effects: university managers. Drawing upon the sociology of numbers and theories of organizational translation, this article uses the concepts of actionability and legitimacy to analyse how and why managers at two Danish universities undertook local translations of a new Bibliometric Research Indicator (BRI). While scholars have emphasized how research evaluation systems can be either strong or weak by design, this study demonstrates how local managers to some extent make national systems strong or weak. The study also finds though that managers’ translations are conditioned by a range of background factors. These factors are identified as financial incentives, problem definitions, indicator competition, and identity and culture.

The Flemish Performance-based Research Funding System: A Unique Variant of the Norwegian Model

Abstract The BOF-key is the performance-based research funding system that is used in Flanders, Belgium. In this paper we describe the historical background of the system, its current design and organization, as well as its effects on the Flemish higher education landscape. The BOF-key in its current form relies on three bibliometric parameters: publications in Web of Science, citations in Web of Science, and publications in a comprehensive regional database for SSH publications. Taken together, the BOF-key forms a unique variant of the Norwegian model: while the system to a large extent relies on a commercial database, it avoids the problem of inadequate coverage of the SSH. Because the bibliometric parameters of the BOF-key are reused in other funding allocation schemes, their overall importance to the Flemish universities is substantial.

Redesigning the Model of Book Evaluation in the Polish Performance-based Research Funding System

Abstract Purpose This study aims to present the key systemic changes in the Polish book evaluation model to focus on the publisher list, as inspired by the Norwegian Model. Design/methodology/approach In this study we reconstruct the framework of the 2010 and 2018 models of book evaluation in Poland within the performance-based research funding system. Findings For almost 20 years the book evaluation system in Poland has been based on the verification of various technical criteria (e.g. length of the book). The new 2018 model is based on the principle of prestige inheritance (a book is worth as much as its publisher is) and is inspired by the publisher list used in the Norwegian Model. In this paper, we argue that this solution may be a more balanced policy instrument than the previous 2010 model in which neither the quality of the publisher nor the quality of the book played any role in the evaluation. Research limitations We work from the framework of the 2018 model of book evaluation specified in the law on higher education and science from 20 July 2018, as implementation acts are not available yet. Practical implications This study may provide a valuable point of reference on how structural reforms in the research evaluation model were implemented on a country level. The results of this study may be interesting to policy makers, stakeholders and researchers focused on science policy. Originality/value This is the very first study that presents the new framework of the Polish research evaluation model and policy instruments for scholarly book evaluation. We describe what motivated policy makers to change the book evaluation model, and what arguments were explicitly raised to argue for the new solution.

How the use of business intelligence tools in the fistula sector can improve surgical performance

Aims: The aim of Operation Fistula’s ongoing work is to incentivize our partnering clinical teams by providing performance-based funding. We do this using a coordinated chain of business intelligence tools, and by sharing detailed customized visual dashboards so they can track both patient outcomes and surgical performance. We have repurposed a number of leading business intelligence tools to create a patient tracking and performance management system. This platform combines several technologies to collect, store, and analyze data, which we communicate with our partners.
 Methods: In order for data to be collected into our system, our partner surgeons and nurses input patient data directly into customized, standardized, electronic forms on a tablet or phone. Results are uploaded in real time, and feed into the data visualization software. Our London-based team is then able to create data visualization dashboards, based on the collected data, which are shared with stakeholders in our partnering countries.
 Results: We are tracking surgical outcomes using our mobile data collection and visualization programmes. We are also measuring the impact of the digital performance-based funding model, currently being used in the SAVA region in Madagascar. Our partnering clinical teams have responded positively to the introduction of our system, which we have found to ensure quality standards of care.
 Conclusions: The initial phase of this work has yielded positive results. We have seen increased engagement and motivation from our partnering clinical staff, which exemplifies how both a performance-based funding system, and access to digestible data, can improve surgical performance.

Performance-based funding: A case study in Madagascar

Aims: To successfully introduce a digital performance-based funding system into Sambava Hospital in the SAVA region of Madagascar in order that surgeons and clinical care workers would be incentivized to provide high-quality care to a greater number of patients.
 Methods: We used a combination of monetary and non-monetary incentives, the first is that we provide a cash transfer when Dr. Fidèle and his team perform in line with our minimum standards of care; the quality of surgery and the entire continuum of care is measured by the use of Operation Fistula’s Global Obstetric Fistula Electronic Registry (GOFER) application. The non-monetary incentives include visualisations of Dr. Fidele’s surgical progress so that he can more effectively monitor the practices of the team.
 Results: This pilot in Sambava Hospital revealed that the data collection tool serves three purposes: It ensures that the surgeries performed and nursing care maintained meet a minimum standard of care; It acts as a verification mechanism that triggers payment; and The data collected enables Dr. Fidèle to track and measure his own surgical progress, with our surgical dashboards.
 Conclusions: The initial outcome of this pilot program has yielded positive results. From March to September, the treatment of 40 patients has been accounted for and transfers have been made, on receipt of corresponding data.
 Keywords: Funding, high quality care, patients

Performance-based university funding and the drive towards ‘institutional meritocracy’ in Italy

Many countries, including Italy, are increasingly managing their public higher education systems in accordance with the New Public Management principle that private-sector management practices improve efficiency and quality. A key mechanism has been the introduction of performance-based funding systems designed to reward ‘high-performing’ institutions and incentivise ‘lesser-performing’ institutions to improve. Instead of improving efficiency and quality across the board, however, we argue that performance-based funding systems naturalise longstanding structurally determined inequalities between institutions by recasting national higher education systems as competitive institutional meritocracies in which institutional inequalities are redefined as objective indicators of intrinsic ‘merit’ or worth. We illustrate how performance-based university funding systems naturalise pre-existing inequalities between universities drawing on the case of Italy, a country characterised by longstanding inequalities between its northern and southern regions which demonstrably impact on the apparent ‘performance’ of universities. The concept of institutional meritocracy captures the illusory nature of this performance game.

How arbitrary are the weights assigned to books in performance-based research funding? An empirical assessment of the weight and size of monographs in Flanders

PurposeThe purpose of this paper is to present an empirical assessment of the weight assigned to monographs in the publication indicator of the performance-based research funding system (PRFS) in Flanders, Belgium. By relating publication weight to publication size the authors offer an alternative perspective on the production of scholars who publish monographs. This perspective on weights is linked to the aggregation level at which PRFS indicators are used: the national/regional one as opposed to the local one. In Flanders as elsewhere the publication indicator designed for funding distribution between universities has sometimes trickled down to institutions, their faculties and departments.Design/methodology/approachAs an alternative indicator of scholarly production the authors propose the median number of pages of a publication type. Measuring the size of publications allows to compare the weight ratio between monographs and journal articles in the publication indicator to their size ratio in the VABB-SHW database. The authors compare two levels, one of four universities and one of 16 disciplines.FindingsMedian publication size differences between disciplines are much larger than those between universities. This indicates that an increase of monographs’ weight in the publication indicator would hardly affect funding distribution at the regional level. Disciplines with a relatively large share of monographs, however, would contribute more to the publication indicator. Hence an increase of monographs’ weight might provide a better balance between fields and between publication types.Originality/valueThis paper presents a thought experiment regarding the weight assigned to different publication types in the publication indicator of the Flemish PRFS: what would happen if this weight were replaced by the median number of pages of a publication type? In doing so, we highlight that such weighting schemes play an important role in finding a balance between fields of research. The sizeable differences between weight and size ratios offer a new and critical perspective on the weighting schemes currently used in PRFS, also in other countries.

Do performance‐based research funding systems affect research production and impact?

AbstractWe study the potential impact of introducing performance‐based funding systems (PBFS) on national research systems, using information on the number of publications and their scientific impact (citations or publications in top‐ranked journals) for 31 countries over the period 1996–2016. The analysis is performed both at the aggregate level and looking separately at each of the six main scientific areas identified according to the Organisation for Economic Co‐operation and Development (OECD) classification. On average, PBFS are found to increase the number of publications, though the effect is only temporary and fades after a few years. Looking at the scientific impact, PBFS are found to have a negligible effect on excellence as measured by the share of articles published in the top journals, irrespective of the type of assessment adopted. On the contrary, PBFS have some influence on average research quality, as measured by the number of citations per paper normalised with respect to the field.

Evaluating ‘impact’ in the UK’s Research Excellence Framework (REF): liminality, looseness and new modalities of scholarly distinction

ABSTRACTLittle is known about the process of evaluating the economic and societal impact of research undertaken in university settings. In this paper, we explore the accounts of senior academics and user-assessors, populating disciplinary sub-panels spanning the humanities and social sciences, convened to judge and ‘score’ the impact claims of researchers from UK universities as a new component of research evaluation within the specific context of the UK’s performance-based research funding system (PBRFS), the Research Excellence Framework (REF). We perceive from their accounts the emergence of a new and liminal space in the production of scholarly ‘distinction’ that is unlike archetypal modalities of academic excellence. Analogously, we identify an emotional and intellectual vulnerability in the review process and the loosening of the structures reviewers traditionally call upon in making value-determinations that simultaneously facilitate their role as impact evaluators and create new modalities in scholarly distinction.

Do Performance-Based Funding Policies Affect Underrepresented Student Enrollment?

ABSTRACTMore states are using performance-based funding (PBF) systems in an effort to incentivize public colleges to operate more effectively. Responding to concerns about equity, states are also adopting provisions that encourage colleges to serve more students who at risk of not completing college. In this paper, I examine whether PBF policies in general—and more specifically policies that have provisions designed to incentivize colleges to enroll underrepresented students—are associated with the number of lower-income, underrepresented minority, and adult students enrolled at four-year public colleges. Using a generalized difference-in-difference framework, I find little evidence that PBF systems as a whole meaningfully affect underrepresented student enrollment. However, the presence of bonuses for serving at-risk students appears to help mitigate any efforts to enroll a more advantaged student body that may be present in other PBF systems.

Analysis of writing programmes for academics: application of a transactional and systems approach

ABSTRACTIn an international culture of performance-based funding systems for higher education institutions (HEIs), research productivity and output are essential for the ability of HEIs to continue. While many institutions have performance targets for the next performance assessment, few institutions have an operational strategy for how staff are to produce research publications. A transactional and systems approach to academic writing may be helpful in identifying how academics’ structures and beliefs influence their writing productivity. This paper aims to provide an overview and analysis, using a transactional and systems approach, of a selection of theory-based writing programmes available for academics. The authors summarise the strengths and weaknesses of each these programmes and demonstrate the gap in academic writing programmes which currently exists, based on the transactional and systems approach. The paper provides recommendations in terms of strategic development to support academic to increase their research and scholarly output.

ОПТИМІЗАЦІЯ ФІНАНСУВАННЯ ДЕРЖАВНИХ ВИЩИХ НАВЧАЛЬНИХ ЗАКЛАДІВ УКРАЇНИ

The paper considers the ways to optimize financing public higher education institutions of Ukraine. Public higher education institutions acutely feel the lack of financial resources. The study describes that it is necessary to reform the models, methods and forms of financing higher education institutions. The paper explains the impact of autonomy of higher education institutions on their development. The autonomy level of university determines the possibilities for diversification from additional sources. The results found that more autonomy of higher education institutions will allow them effectively generate and use financial resources. The review outlines the diversification of financial resources public universities. One of the key factors of the university success is to implement the diversification strategy into the overall academic strategy and mission of the higher education institution. The analysis recommends the performance-based funding system and public higher education institutions achieve certain performance indicators. The performance-based funding system will promote higher competitiveness of education institutions and improve the quality of higher education in general. The conclusions suggest the development trends of financing public higher education institutions of Ukraine.

Shifting from Enrollment- to Performance-Based Funding in Higher Education: What Can We Learn from Washington's Experience?

Performance-based funding (PBF) in higher education has grown in recent years as a means of institutional accountability and incentive for improving student success. Although most states have successfully implemented their respective systems, research on early funding models suggests a difficult fiscal environment can introduce tension between theory and practice of the concept. This policy brief uses the case of Washington State's redesign of the Student Achievement Initiative to describe new implications around this tension. The revision focused on using a base reallocation as the funding source in the context of diminished state resources and the importance of college buy-in. Regression analyses tested the alignment between the principles and metrics for awarding funds, which resulted in a funding model that awarded the maximum dollars related to performance versus college characteristics. Policy makers considering new or revised PBF systems can benefit from critical lessons learned from Washington State's comprehensive process and final product.

'Excellence' and Exclusion: The Individual Costs of Institutional Competitiveness.

A performance-based funding system like the United Kingdom’s ‘Research Excellence Framework’ (REF) symbolizes the re-rationalization of higher education according to neoliberal ideology and New Public Management technologies. The REF is also significant for disclosing the kinds of behaviour that characterize universities’ response to government demands for research auditability. In this paper, we consider the casualties of what Henry Giroux (2014) calls “neoliberalism’s war on higher education” or more precisely the deleterious consequences of non-participation in the REF. We also discuss the ways with which higher education’s competition fetish, embodied within the REF, affects the instrumentalization of academic research and the diminution of academic freedom, autonomy and criticality.

Managing civic activities by performance: impacts of the government’s performance-based funding system and the domain structure in Finnish sports policy

ABSTRACTThis article focuses on a turning point in Finnish sports policy at which the new performance-based governmental steering model replaced the old corporatist operational model. This shift is approached through a long-term analysis of Finnish sports policy and structures. The government’s power increased when the old hierarchical and representation-based pyramid structure of the sports movement crumbled and was replaced by a sports community that was based on looser collaboration. The article reviews the performance-based funding system applied by the Finnish government to sports organisations and the structural reform of the sports movement. The aim is to estimate the impacts of these changes on large national sport federations, in particular. The article analyses the effects of the new steering model and the development process of the various domains in the areas of youth sports, elite sports, adapted physical activities and adult sports. The results of the analysis show that the structural reform was never fully completed and that the impact of performance-based steering has waned in the activities of the large sport federations in the 2000s.

Counting quality? The Czech performance-based research funding system

After the fall of the Iron Curtain and a subsequent period of restructuring the research and innovation system, the Czech Republic has introduced a performance-based research funding system, commonly known as the Evaluation Methodology. The Evaluation Methodology is purely quantitative and focused solely on research outputs (publications, patents, prototypes, etc.) to determine the amount of institutional funding for research organizations. While aiming to depersonalize and depoliticize the allocation of institutional funding in the research system, improve research productivity, and safeguard accountability, we argue that the Evaluation Methodology has in fact become a negative example of a performance-based research funding system. Our analysis of the Evaluation Methodology shows that it has introduced considerable instability and unpredictability in the Czech research system, making strategic planning for research organizations difficult. The article contributes to a growing body of literature on research evaluation and performancebased research funding systems, discussing the impacts of introducing such systems in countries including the UK, Spain, Slovakia, Hong Kong, Australia, Poland, Italy, New Zealand, Flanders, Norway, Denmark, and Finland. The Czech case provides new insights in the interactions between politico-economic regimes and research policy, while also directing the attention of research policy scholars to significant developments in Central and Eastern European countries.