Performance-based Research Funding Systems Research Articles

Research quality criteria in the Creative Arts

ABSTRACT This paper investigates research quality criteria in the Creative Arts (CA). The CA has been introduced into the higher education and research sector over the last three decades. It is thus a relatively new research field and there is little empirical knowledge on how outputs in this field should be evaluated. Our study applies a mixed-method approach to assess the relevance of quality criteria used in performance-based research funding systems (PRFSs) in 10 countries. The results of a qualitative analysis of interviews with artists-academics (N = 67) and Joint Correspondence Analysis show that when art is evaluated in the context of academic research, both the traditional indicators of artistic quality as well as the cognitive and research-related aspects of the arts are believed to be significant. The JCA analysis also showed that the majority of our respondents found both extrinsic quality criteria (related to reputation and prestige) and intrinsic criteria (related to cognition and development) relevant.

Do research performances of universities and disciplines in England converge or diverge? An assessment of the progress between research excellence frameworks in 2014 and 2021

Performance-based research funding systems (PBRFSs) have been used in selectively distributing research funding, increasing public money accountability and efficiency. Two recent such evaluations in England were called the Research Excellence Framework (REF), which took place in 2014 and 2021, and the research environment, outputs and impact of the research were evaluated. Even though various aspects of the REF were examined, there has been limited research on how the performance of the universities and disciplines changed between the two evaluation periods. This paper assesses whether there has been convergence or divergence in research quality across universities and subject areas between 2014 and 2021 and found that there was an absolute convergence between universities in all three research elements evaluated, and universities that performed relatively worse in REF in 2014 experienced higher growth in their performance between 2014 and 2021. There was also an absolute convergence in the research environment and impact across different subject areas, but there is no significant convergence in the quality of research outputs across disciplines. Our findings also highlight that there has been an absolute convergence in research quality within the universities (between different disciplines in a given university) and within disciplines (between universities in a given subject).

A Bibliometric Exploration of Events Literature In and Outside the Core Journals

There are various positions on what literature represents "core" event management research and to what degree "peripheral" tourism and hospitality (T&H) literature influences and overlaps with this field. Previous literature reviews have made progress but methodological challenges remain. The influences of journal lists, performance-based research funding systems, and selective databases are discussed regarding the journal selection of events researchers. An exploratory approach to charting the core and periphery of events-related research was taken. Firstly, the relevance of a wide range of journals to event management research was tested using a measure of interrater agreement. Twenty-four of the 53 journals reviewed were selected as relevant. Secondly, a bibliometric analysis of highly cited articles was carried out. Two hundred seventy-two of 480 articles from 2008–2018 were selected and further examined using citation analysis. Three, four-star journals accounted for 60% of all references within the sample, despite only accounting for 23% of the total article volume. Quantitative research and articles about festivals or sports events were predominant in the sample, although the research methodology used and types of events chosen by individual articles had limited or no effect on the number of citations received. Most (77%) authors were based in Australia, Europe, New Zealand, and North America. Future reviews are advised to investigate blended approaches, open data, nonselective databases, and international perspectives.

The impact of Italian performance-based research funding systems on the intensity of international research collaboration

Abstract The study of national research assessment exercises serves to evaluate the effectiveness of policies versus their objectives and to improve the formulation of future initiatives. The aim of the current study is to verify whether the introduction of the first performance-based research funding in Italy, based on the 2004–10 VQR assessment, achieved the intended objective of inducing greater international collaboration on the part of researchers. For this, we apply a bibliometric approach based on the observation of coauthorships in Italian and worldwide scientific publications over a 14-year period, beginning in the target years of the VQR assessment. Through an Interrupted Time Series Analysis, we compare the expected and observed patterns of international coauthorship for Italy and the rest of the world. Although the rate of internationalization of Italian research is seen to be increasing, whether this is a consequence of the VQR incentives, or rather part of a global phenomenon of recourse to international collaboration in response to the increasingly complex scientific challenges, it is open to interpretation.

Sources of convergence and divergence in university research quality: evidence from the performance-based research funding system in New Zealand

The introduction of performance-based research funding systems (PBRFS) in many countries has generated new information on their impacts. Recent research has considered whether such systems generate convergence or divergence of research quality across universities and academic disciplines. However, little attention has been given to the processes determining research quality changes. This paper utilises anonymised longitudinal researcher data over 15 years of the New Zealand PBRFS to evaluate whether research quality changes are characterised by convergence or divergence, and the processes determining those dynamics. A unique feature is the use of longitudinal data to decompose changes in researcher quality into contributions arising from the entry, exit and quality transformations of retained researchers, and their impacts on convergence or divergence of research quality across universities and disciplines. The paper also identifies how researcher dynamics vary systematically between universities and disciplines, providing new insights into the effects of these systems.

The different responses of universities to introduction of performance-based research funding

Abstract Governments and organizations design performance-based research funding systems (PBRFS) for strategic aims, such as to selectively allocate scarce resources and stimulate research efficiency. In this work, we analyze the relative change in research productivity of Italian universities after the introduction of such a system, featuring financial and reputational incentives. Using a bibliometric approach, we compare the relative research performance of universities before and after the introduction of PBRFS, at the overall, discipline and field levels. The findings show convergence in the universities’ performance, due above all to the remarkable improvement of the lowest performers. Geographically, the universities of the south (vs central and northern Italy) achieved the greatest improvement in relative performance. The methodology, and results, should be of use to university management and policy-makers.

Disadvantages of Patent Performance Indicators in Performance-Based Research Funding Systems

Although the use of patent indicators in statistics has been under discussion for a long time, there are surprisingly very few studies devoted to explaining the use of patent indicators as performance indicators in performance-based research funding systems. The widespread assumption is based on the perception that patents have some recognizable value as an element of innovation process and this value is undoubtedly transferred to patents that are created in order to fulfill performance indicators. However, there is still no established distinction in the literature between patent indicators used in statistical analysis and patent performance indicators used as goal setting and evaluation tools in the context of performance-based research funding systems. This article is aimed at filling the gap and seeks to clarify the theoretical validity of the use of patent performance indicators in such context. Based on the review of the literature, the article compares the results of seminal works on the nature of patenting and patent indicators to the purposes of its use as patent performance indicators in performance-based research funding systems. The results demonstrate at least five arguments in favor of the statement that the use of patent indicators in the context of performance-based research funding systems requires rethinking.

Performance-based publisher ratings and the visibility/impact of books: Small fish in a big pond, or big fish in a small pond?

Abstract This study compares publisher ratings to the visibility and impact of individual books, based on a 2017 data set from three Nordic performance-based research funding systems (PRFS) (Denmark, Norway, and Finland). Although there are Journal Impact Factors (JIFs) for journals, there is no similar indicator for book publishers. National publisher lists are used instead to account for the general “quality” of books, leading to institutional rewards. But, just as the JIF is not recommended as a proxy for the “citedness” of a paper, a publisher rating is also not recommended as a proxy for the impact of an individual book. We introduce a small fish in a big pond versus big fish in a small pond metaphor, where a “fish” is a book and “the pond” represents its publishing house. We investigate how books fit on this metaphorical fish and pond continuum, using WorldCat holdings (visibility) and Google Scholar citations (impact), and test other variables to determine their predictive value with respect to these two indicators. Our statistics show that publisher levels do not have predictive value when other variables are held constant. This has implications for PRFS and book evaluations in general, as well as ongoing developments related to a newly proposed international publisher registry.

Performance-based Institutional Research Funding in Flanders, Belgium

Research and innovation is one of Flanders’ priorities and over the last three decades its public funding has strongly increased. Universities are key actors in this strategy. They have a large autonomy and receive a substantial share of additional R&D expenditures as lump sum funding.The Flemish authorities use quantitative indicators to allocate these lump sums to the universities. The funding formulae take into account each institution’s size, its research performance and, if relevant, its valorization activities. Also significant is the realization of governmental priorities, such as mobility and diversity of the academic staff. This paper describes the development of the Flemish university funding model, analyses its weaknesses and its strengths, and compares it with nine national metrics-based research performance funding systems.Policy highlightsFlanders, like many other regions and nations, has adopted performance-based research funding systems (PRFSs) to improve and provide accountability for its science and innovation system. The Flemish PRFS criteria have evolved considerably over the last three decades, and due to competition between universities and a consensus model of political decision making, the funding formula is comparatively complex.Building on an historical background of lump sum payments to universities, supporting both education and research, special supplementary funds for blue-sky research (BOF) and for strategic applied research, innovation, and outreach activities (IOF) were introduced by the Flemish government in 1994 and 2004, respectively. The “three-legged stool” funding mechanism for research in Flanders is unique within Europe.The introduction of publication and citation metrics in 2003 changed the character of the Flemish PRFS considerably, previously focused on measures of a university’s size in terms of students, degrees granted, and previous funding. Humanities and social sciences (HSS) research metrics were considered only after first adoption of bibliometric methods traditionally deployed for assessment of the natural and life sciences, but this is not unique to Flanders. HSS research has been addressed in Flanders through the creation of a special database to supplement standard citation indexes, the VABB-SHW. This special resource is a strong point of the Flemish PRFS, and other nations should appreciate the value of increased coverage of the HSS literature.The PRFS in Flanders differs from that of nine European nations in several ways: analysis is undertaken annually, it includes a diversity and mobility measure (2006), and one of interdisciplinary will soon be added.Flanders, like other nations using PRFSs, has generally seen increased research output and intellectual property activities after its introduction; however, in some nations the increase began before a PRFS was implemented. In almost all cases, demonstrating a causal link between a PRFS and increased output is difficult since R&D investments and the number of researchers have also increased for a variety of reasons. This is a cautionary note in not over-interpreting the effects or effectiveness of a PRFS.Flanders now exhibits one of the highest levels among European nations of research funding allocations determined by a PRFS: 50% of BOF-funding and 75% of IOF-funding. By employing such a metrics-heavy scheme, Flanders is a good candidate for a detailed study of unintended consequences of a PRFS, at the national, institutional, and research group and individual researcher level. This remains a large gap in our understanding of the use of a PRFS.

Excellence seekers, pragmatists, or sceptics: Ways of applying performance‐based research funding systems at new universities and university colleges in Sweden

AbstractThe Swedish system of research funding has undergone major changes, while competitive funding schemes gradually have gained popularity, at the expense of institutional block grants. In recent years, there has been a strong drive to improve research performance in universities as a result of governmental desires, resulting in performance‐based research funding systems (PRFS) being introduced to distribute block grants. The study on which this article reports sought to understand the development of increased government intent to steer resources for research, by investigating the behaviour of universities and university colleges in Sweden in terms of internal organisation and distribution of funding. The article investigates the structure of and motives for applying PRFS at seven new universities and university colleges through a document and interview study. We identified three categories of higher education institutions: the excellence seekers, the pragmatists and the sceptics. Universities rhetorically signal compliance with governmental propositions to distribute funding according to performance measures, but their actions are limited. Thus, national incentives, models and measures influence decisions at lower institutional levels in a rather modest way.

Understanding Indigenous Exploitation Through Performance Based Research Funding Reviews in Colonial States.

Countries with significant indigenous populations, such as Australia, New Zealand and the Nordic countries, are providing increased support for improvements in the number of indigenous academics represented in higher education and engaged in research. Such developments have occurred at the same time as the implementation of performance-based research funding systems. However, despite the significance of such systems for academic careers and knowledge diffusion there has been relatively little consideration of the way within which they meet the needs of indigenous academics and knowledges. Drawing primarily on the New Zealand context, this perspective paper questions the positioning of Māori researchers and Māori research epistemologies (Kaupapa Maori) within the Performance Based Research Fund and the contemporary neoliberal higher education system. It is argued that the present system, rather than being genuinely inclusive, serves to reinforce the othering of Māori episteme and therefore perpetuates the hegemony of Western and colonial epistemologies and research structures. As such, there is a need to raise fundamental questions about the present ecologies of knowledge that performance based research systems create not only in the New Zealand higher education research context but also within other countries that seek to advance indigenous research.

National Lists of Scholarly Publication Channels: An Overview and Recommendations for Their Construction and Maintenance

Abstract Purpose This paper presents an overview of different kinds of lists of scholarly publication channels and of experiences related to the construction and maintenance of national lists supporting performance-based research funding systems. It also contributes with a set of recommendations for the construction and maintenance of national lists of journals and book publishers. Design/methodology/approach The study is based on analysis of previously published studies, policy papers, and reported experiences related to the construction and use of lists of scholarly publication channels. Findings Several countries have systems for research funding and/or evaluation, that involve the use of national lists of scholarly publication channels (mainly journals and publishers). Typically, such lists are selective (do not include all scholarly or non-scholarly channels) and differentiated (distinguish between channels of different levels and quality). At the same time, most lists are embedded in a system that encompasses multiple or all disciplines. This raises the question how such lists can be organized and maintained to ensure that all relevant disciplines and all types of research are adequately represented. Research limitation The conclusions and recommendations of the study are based on the authors’ interpretation of a complex and sometimes controversial process with many different stakeholders involved. Practical implications The recommendations and the related background information provided in this paper enable mutual learning that may feed into improvements in the construction and maintenance of national and other lists of scholarly publication channels in any geographical context. This may foster a development of responsible evaluation practices. Originality/value This paper presents the first general overview and typology of different kinds of publication channel lists, provides insights on expert-based versus metrics-based evaluation, and formulates a set of recommendations for the responsible construction and maintenance of publication channel lists.

Does monetary support increase citation impact of scholarly papers?

One of the main indicators of scientific development of a given country is the number of papers published in high impact scholarly journals. Many countries introduced performance-based research funding systems (PRFSs) to create a more competitive environment where prolific researchers get rewarded with subsidies to increase both the quantity and quality of papers. Yet, subsidies do not always function as a leverage to improve the citation impact of scholarly papers. This paper investigates the effect of the publication support system of Turkey (TR) on the citation impact of papers authored by Turkish researchers. Based on a stratified probabilistic sample of 4,521 TR-addressed papers, it compares the number of citations to determine if supported papers were cited more often than those of not supported ones, and if they were published in journals with relatively higher citation impact in terms of journal impact factors, article influence scores and quartiles. Both supported and not supported papers received comparable number of citations per paper, and were published in journals with similar citation impact values. Findings suggest that subsidies do not seem to be an effective incentive to improve the quality of scholarly papers. Such support programs should therefore be reconsidered.

What constitutes high quality higher education pedagogical research?

Over the last 20 years there has been significant growth in the volume of higher education pedagogical research across disciplines and national contexts, but inherent tensions in defining quality remain. In this paper we present a framework to support understanding of what constitutes internationally excellent research, drawing on a range of conceptual frameworks, international and national performance-based research funding systems, discipline/professional body frameworks, and research council guidance. While acknowledging the contested nature of excellence in pedagogical research, we provide criteria to guide discussion and to support individual and organisational learning. A key premise is that if learning and teaching in higher education are to be enhanced, considerable investment is required in supporting the development of integrated academics where emphasis is on both research and practice to inform pedagogy. Research and evaluation are essential aspects of teaching and need to be embedded within it. The framework is designed to enable colleagues to develop the necessary tools and approaches to support understanding of educational research and adapt these within their disciplinary context.

Does faculty disciplinary background play a role in the publication pattern of an interdisciplinary research area? The case of science education in Brazil

Research in the Social Sciences and Humanities is published in a variety of formats and types of documents. Local journals are relevant for these fields, particularly in non-English-speaking countries, but coverage of large bibliographic databases is still limited, leading to a large percentage of non-source items. This is an acknowledged challenge for Performance-based Research Funding Systems and also for national evaluation exercises, such as the evaluation of the graduate degree programs performed by CAPES in Brazil. Our assumption is that the evaluation of interdisciplinary research may face similar challenge, particularly when faculty´s disciplinary background is predominantly from the Social Sciences and Humanities. The purpose of this article is to analyse an interdisciplinary research area in Brazil to calculate the percentage of non-source items in its scholarly output, in comparison to the 3 areas in which the majority of its faculty graduated. We extracted 11,000 publications reported by 140 degree programs from the Teaching area -also known as Science Education, and found 77.5% of non-source titles. We also analysed another 50,000 publications from the Education, Astronomy/Physics and Chemistry areas. Comparison showed that the Teaching Area had a similar publication pattern to the Education Area, and typical of the Humanities, which is the predominant Major Area of its faculty graduation. Our conclusion is that faculty disciplinary background may influence publication patterns and stakeholders involved in research evaluation should consider using alternative sources of data when assessing the scholarly output of interdisciplinary research, besides the main international bibliographic databases.

Problems and Considerations in the Design of Bibliometric Indicators for National Performance-Based Research Funding Systems

Problems and Considerations in the Design of Bibliometric Indicators for National Performance-Based Research Funding Systems

Ambiguity in identification of peer-reviewed publications in the Finnish and Flemish performance-based research funding systems

Abstract In performance-based research funding systems evidence of peer review is often considered a requirement for publications to be included. Originating from the sciences, pre-publication peer review is very common in the publishing process, also in the social sciences and humanities. Sometimes, however, it is ambiguous whether a publication is peer-reviewed or not. In this contribution, we analyse the ambiguity in identifying a journal’s or publication’s peer-review status by comparing the classification of journals in Finland and Flanders, and by taking stock of Finnish authors’ reporting of peer-review status of publications. We find that ambiguity in terms of peer review status is rather common, especially in the humanities. Indeed, we find differences in peer review status in about 10 per cent of all cases, both when comparing Finland and Flanders, and when comparing author-reported and centralised identification of peer-review status.

Measuring changes in publication patterns in a context of performance-based research funding systems: the case of educational research in the University of Gothenburg (2005–2014)

We present a novel way to frame a discussion of changes in publication patterns that occur in a context of performance-based research funding systems (PRFSs). Adopting an approach derived from social epistemology, we foreground the dialectical nature of knowledge. This allows us to relate changes in publication patterns to PRFS and show the tensions that emerge between relatively diverse research fields with diverse publication practices and bibliometric indicators within PRFSs that reinforce a singular view of research goals. Specifically, we highlight that the employment of bibliometric indicators result in a fixed hierarchy among communication media that may be at odds with the goals within research fields subjected to PRFSs. These ideas are illustrated with an empirical analysis of changes in publication patterns within the field of educational research at the University of Gothenburg (GU; 2005–2014) in Sweden. We contrast the observed changes with implicit priorities in the national and institutional PRFS that operate in this context since 2009. Findings from bibliometric analysis indicate a move away from publication traditions that used to be characteristic of educational research: the growth in the number of journal articles is greater than that in the number of book chapters, while the number of reports is on a declining slope. In relation to PRFSs, we show that conclusive judgement on the desirability of the observed changes is hardly achievable. If one adopts the aims implicit in PRFS, research performance appears to be enhanced. If one sides with the views of many educational researchers, then some of the trends might be an indication of undesirable changes.

Are book publications disappearing from scholarly communication in the social sciences and humanities?

PurposeThe purpose of this paper is to analyze the evolution in terms of shares of scholarly book publications in the social sciences and humanities (SSH) in five European countries, i.e. Flanders (Belgium), Finland, Norway, Poland and Slovenia. In addition to aggregate results for the whole of the social sciences and the humanities, the authors focus on two well-established fields, namely, economics & business and history.Design/methodology/approachComprehensive coverage databases of SSH scholarly output have been set up in Flanders (VABB-SHW), Finland (VIRTA), Norway (NSI), Poland (PBN) and Slovenia (COBISS). These systems allow to trace the shares of monographs and book chapters among the total volume of scholarly publications in each of these countries.FindingsAs expected, the shares of scholarly monographs and book chapters in the humanities and in the social sciences differ considerably between fields of science and between the five countries studied. In economics & business and in history, the results show similar field-based variations as well as country variations. Most year-to-year and overall variation is rather limited. The data presented illustrate that book publishing is not disappearing from an SSH.Research limitations/implicationsThe results presented in this paper illustrate that the polish scholarly evaluation system has influenced scholarly publication patterns considerably, while in the other countries the variations are manifested only slightly. The authors conclude that generalizations like “performance-based research funding systems (PRFS) are bad for book publishing” are flawed. Research evaluation systems need to take book publishing fully into account because of the crucial epistemic and social roles it serves in an SSH.Originality/valueThe authors present data on monographs and book chapters from five comprehensive coverage databases in Europe and analyze the data in view of the debates regarding the perceived detrimental effects of research evaluation systems on scholarly book publishing. The authors show that there is little reason to suspect a dramatic decline of scholarly book publishing in an SSH.

Follow the money? How Australian universities replicate national performance-based funding mechanisms

Recent studies exploring universities’ internal adaptations to national performance-based research funding (PBRF) systems have found evidence for isomorphism as well as for variation in institutional response. Yet there remains a lack of investigation focusing on national PBRF settings which appear to be particularly conducive to isomorphism. Addressing the resulting lacuna, this article investigates the extent to which Australian universities replicate national PBRF indicators in their individual-level performance management frameworks for academic staff. Drawing on data from 33 Australian universities (of 39 eligible institutions), and taking into consideration the differences in these universities’ level of research intensity, this article finds that universities overwhelmingly replicate the major national PBRF indicators internally. If variation was evident, then mostly in the form of minor modifications to these indicators, not in the choice of indicators per se. Analysis of the Australian case thus demonstrates strong vertical alignment between national and institutional research governance mechanisms as well as considerable convergence in the formal organization and governance of research activities at Australian universities.