Performance-based Research Funding Systems Research Articles

Miksi suomalaisen tutkimuksen tuottavuus kasvoi, mutta laatu heikentyi?

Performance-based research funding systems are often assumed to create an increasing pressure to publish among academics. Presumption is that national level incentives put pressure on researchers to increase quantity of publishing at the expense of quality (publishing pressure hypothesis). Janne Pölönen and Otto Auranen argued in their article in Information Studies 2 (36) that the hypothesis does not apply to Finland. The purpose of this article is to continue the discussion on the hypothesis. I repeat the empirical study by Pölönen and Auranen, and I will end up with somewhat opposite results. Secondly, I discuss the factors affecting the productivity and quality of research. Changes in productivity and quality of research are not just related to the introduction of publications as a criterion for research funding, but the range of explanatory factors is much wider. I will highlight the make-up of researchers, the amount of funding and competitive funding, and look at their explanatory power in respect to the changes of the productivity and the impact of Finnish scientific research.

Unique, but still best practice? The Research Excellence Framework (REF) from an international perspective

Inspired by The Metric Tide report (2015) on the role of metrics in research assessment and management, and Lord Nicholas Stern’s report Building on Success and Learning from Experience (2016), which deals with criticisms of REF2014 and gives advice for a redesign of REF2021, this article discusses the possible implications for other countries. It also contributes to the discussion of the future of the REF by taking an international perspective. The article offers a framework for understanding differences in the motivations and designs of performance-based research funding systems (PRFS) across countries. It also shows that a basis for mutual learning among countries is more needed than a formulation of best practice, thereby both contributing to and correcting the international outlook in The Metric Tide report and its supplementary Literature Review.

Special section on performance-based research funding systems

Special section on performance-based research funding systems

Some considerations about causes and effects in studies of performance-based research funding systems

Some considerations about causes and effects in studies of performance-based research funding systems

Julkaisupaine suomalaisessa tiedeyhteisössä

Performance-based research funding systems are often assumed to create an increasing pressure to publish among academics. Presumption is that national level incentives put pressure on researchers to increase quantity of publishing at the expense of quality. In Finland publications have been a criterion in the government’s basic funding model for universities since 2007. We analysed national personnel and publication data on Finnish universities, and found that the average publication productivity among Finnish university staff has not increased from 1998 to 2016. We also looked at the Web of Science publications in 1998-2014, and found that publication productivity of academics is not exceptionally high in Finland, compared to Sweden, Denmark and Norway. Citation impact has not remarkably decreased in Finland compared to the other Nordic countries. In our view, the experience of pressure to publish is also, and perhaps primarily, due to competition for research funding and academic positions partaken by an increasing number of researchers.

Selling ‘impact’: peer reviewer projections of what is needed and what counts in REF impact case studies. A retrospective analysis

The intensification of an audit culture in higher education is made no more apparent than with the growing prevalence of performance-based research funding systems like the UK’s Research Excellence Framework (REF) and the introduction of new measures of assessment like ‘impact’ or more specifically, the economic and societal impacts of research. Detractors of this regulatory intervention, however, question the legitimacy and credibility of such a system for, and focus within, the evaluation of research performance. Within this study, we specifically sought to understand the process of evaluating the impact of research by gaining unique access as observers of a simulated impact evaluation exercise populated by aproximately n = 90 senior academic peer reviewers and user assessors, undertaken within one UK research-intensive university prior to and in preparation of its submission to REF2014. Over an intensive two-day period, we observed how peer reviewers and user assessors grouped into four overarching disciplinary panels went about deliberating and scoring impact, presented in the form of narrative-based case studies. Among other findings, our observations revealed that in their efforts to evaluate impact, peer reviewers were indirectly promoting a kind of impact mercantilism, where case studies that best sold impact were those rewarded with the highest evaluative scores.

Impacts of performance-based research funding systems: The case of the Norwegian Publication Indicator

There has been a growing use of performance-based research funding systems (PRFS) as a policy tool. With the introduction of the Publication Indicator in 2004, Norway joined this international trend in which the allocation of basic funds is increasingly linked to performance indicators. The purpose of this article is to present and discuss the main results of a recent evaluation of the Norwegian Publication Indicator, which examines the Indicator’s impact on publishing patterns, its properties, and how it has functioned in practice. This includes both a broad range of potential effects such as the Indicator’s impact on the quantity and the quality of publications, Norwegian language publishing, and length of articles and monographs. It also includes an examination of properties such as the Indicator’s legitimacy and transparency, how it functions as a measure of research performance across different fields, its use as a management tool, and how the system is organized and administrated in practice. In examining these questions, the article draws on a number of different data sources, including large-scale surveys of both researchers and research managers, multilevel case studies, and bibliometric analysis. The article concludes with a discussion of the implications of the analysis both for further development of the Norwegian Model and for PRFS in general.

Counting quality? The Czech performance-based research funding system

After the fall of the Iron Curtain and a subsequent period of restructuring the research and innovation system, the Czech Republic has introduced a performance-based research funding system, commonly known as the Evaluation Methodology. The Evaluation Methodology is purely quantitative and focused solely on research outputs (publications, patents, prototypes, etc.) to determine the amount of institutional funding for research organizations. While aiming to depersonalize and depoliticize the allocation of institutional funding in the research system, improve research productivity, and safeguard accountability, we argue that the Evaluation Methodology has in fact become a negative example of a performance-based research funding system. Our analysis of the Evaluation Methodology shows that it has introduced considerable instability and unpredictability in the Czech research system, making strategic planning for research organizations difficult. The article contributes to a growing body of literature on research evaluation and performancebased research funding systems, discussing the impacts of introducing such systems in countries including the UK, Spain, Slovakia, Hong Kong, Australia, Poland, Italy, New Zealand, Flanders, Norway, Denmark, and Finland. The Czech case provides new insights in the interactions between politico-economic regimes and research policy, while also directing the attention of research policy scholars to significant developments in Central and Eastern European countries.

How incentives trickle down: Local use of a national bibliometric indicator system

The question of how the incentives of national performance-based research funding systems affect local management practices within the higher education sector has high empirical and theoretical importance, but has so far received limited attention. From a traditional organization theory perspective the Norwegian system represents a puzzle: it is of marginal economic importance and the external pressure to fully implement the system at local levels is weak. A loose coupling between the national system and the local implementation would therefore be expected. Yet, in many instances we document a quite tight coupling between system-level incentives and local practices. Based on a recent evaluation of the Norwegian model, which identifies a number of indirect mechanisms that contribute to the trickling down of incentives, this paper shows that traditional coupling perspectives are oversimplistic. A large variation across institutions, fields and departments is, however, also observed and possible explanations for this are discussed.

Performance-based university research funding systems

The university research environment has been undergoing profound change in recent decades and performance-based research funding systems (PRFSs) are one of the many novelties introduced. This paper seeks to find general lessons in the accumulated experience with PRFSs that can serve to enrich our understanding of how research policy and innovation systems are evolving. The paper also links the PRFS experience with the public management literature, particularly new public management, and understanding of public sector performance evaluation systems. PRFSs were found to be complex, dynamic systems, balancing peer review and metrics, accommodating differences between fields, and involving lengthy consultation with the academic community and transparency in data and results. Although the importance of PRFSs seems based on their distribution of universities’ research funding, this is something of an illusion, and the literature agrees that it is the competition for prestige created by a PRSF that creates powerful incentives within university systems. The literature suggests that under the right circumstances a PRFS will enhance control by professional elites. PRFSs since they aim for excellence, may compromise other important values such as equity or diversity. They will not serve the goal of enhancing the economic relevance of research.