Performance-Based Research Fund Research Articles

New Zealand’s bibliometric record in research and development: 1990–2008

Over the last two decades, the New Zealand government has sought to achieve improved outcomes in science and technology through structural reform, rather than increased levels of investment in the sector. How successful has this strategy been? Here we look at New Zealand’s bibliometric output over the last two decades, using the Thompson Reuters Web of Science database. The database reveals that bibliometric productivity has remained static over this time period, with changes in publication rate driven by increasing levels of full–time equivalent staff members, principally through growth in the tertiary sector. Although the citation impact of university publications has increased, the citation impact of the Crown research institutes has grown at a similar, if not faster, rate, suggesting that the performance-based research fund is not responsible for this increase.

The performance based research fund in New Zealand: taking stock and looking forward

This paper reviews changes in New Zealand universities since the introduction of the Performance Based Research Fund (PBRF) in 2003, and evaluates changes in relation to the stated objectives. This stocktake of research findings is in part a response to the official report of the Review Panel, which made no attempt to review evidence of performance. A key objective was to achieve an improvement in research quality. It is suggested that improvements have been related closely to the incentives created by the scheme, and achieved by considerable staff turnover. The present stocktake of the changed nature of universities and the details of the evaluation process suggests that substantial simplifications could usefully be made while maintaining incentives that are at the heart of any PBRF.

Research productivity during PhD enrolment for New Zealand sociology 2010–19: latest decade output patterns by gender and university

ABSTRACT Sociology has come late to the practice of publishing during PhD enrolment. This contrasts to fields such as science disciplines (STEM) and psychology where publishing during PhD is part of the disciplinary culture. In New Zealand sociology, over the last decade, the traditional PhD monograph continues to be the main format. Closer measurement of academic research outputs by the New Zealand Performance-Based Research Fund (PBRF) creates new pressures in the oversupplied academic-research labour market. This study has three main findings. First, within the decade since the PBRF was introduced, the publishing patterns of sociology PhDs have changed very little on the surface. It should be noted, however, that more than half of these students successfully published at least one refereed output, suggesting a possible disciplinary shift in students attempting to produce publications during candidacy. Second, overall in this time period, men and women published during their PhD at equal rates. This, however, masks variation by the institution. Women outperformed men at all institutions except at the largest sociology programme, Auckland University, where they published at the same rate as men. Third, gender differences by quality, sociology versus non-sociology publications and by Australasian versus international publications showed no significant differences.

GLASS CEILINGS IN NEW ZEALAND UNIVERSITIES: Inequities in Māori and Pacific promotions and earnings

Māori and Pacific academics make up less than 4% and 1% respectively of New Zealand professors. We investigated ethnic inequities in promotions and earnings in New Zealand universities. Using New Zealand’s Performance-Based Research Fund (PBRF) data (2003, 2012, 2018) we found that Māori and Pacific men and also women academics, compared with non-Māori non-Pacific men academics, had significantly lower odds of being an associate professor or professor (professoriate) or of being promoted, and had lower earnings. These inequities were not explained by research performance (measured by PBRF scores), age or field, and remained over time, particularly for women. Māori and Pacific women academics earned on average $7,713 less in 2018 than non-Māori non-Pacific men academics and had 65% lower odds of being promoted into the professoriate from 2003 to 2018. Our findings suggest that current inequities for Māori and Pacific academics will persist without systemic change in New Zealand universities.

Understanding Indigenous Exploitation Through Performance Based Research Funding Reviews in Colonial States.

Countries with significant indigenous populations, such as Australia, New Zealand and the Nordic countries, are providing increased support for improvements in the number of indigenous academics represented in higher education and engaged in research. Such developments have occurred at the same time as the implementation of performance-based research funding systems. However, despite the significance of such systems for academic careers and knowledge diffusion there has been relatively little consideration of the way within which they meet the needs of indigenous academics and knowledges. Drawing primarily on the New Zealand context, this perspective paper questions the positioning of Māori researchers and Māori research epistemologies (Kaupapa Maori) within the Performance Based Research Fund and the contemporary neoliberal higher education system. It is argued that the present system, rather than being genuinely inclusive, serves to reinforce the othering of Māori episteme and therefore perpetuates the hegemony of Western and colonial epistemologies and research structures. As such, there is a need to raise fundamental questions about the present ecologies of knowledge that performance based research systems create not only in the New Zealand higher education research context but also within other countries that seek to advance indigenous research.

The emergence of creative practice as research

The term ‘Creative Practice as Research’ is now in common usage in the tertiary sector, although it is relatively new in its inception. This article traces the rise of the term (and its variations), which emerged about the same time as the tertiary auditing processes, such as Aotearoa New Zealand’s Performance-Based Research Fund (PBRF). But creative practitioners had already been sneaking production into the traditional university, at times facing resistance and even derision from scholars undertaking more conventional research within the arts, humanities and social science departments. The author argues that the term Creative Practice as Research, and the many practices under its umbrella such as journalism, is now widely accepted, in part because it has been convenient, fulfilling particular needs within a changing tertiary landscape. Its greater acceptance allows traditional universities to respond to student demand for skills-based learning without losing their reputation for research excellence. But the term also suits the former polytechnics, or ‘new universities’, that are eager to imbue their craft and technical teaching history and practice with richer research content. Drawing on a new wave of ‘production studies’, the article also explores how a specific instance of Creative Practice, the documentary, does indeed fulfil the requirement of research as articulated through other academic disciplines such as the social sciences. Furthermore, documentary and other creative practices can contribute to ‘impact’, an increasingly important metric deployed in the assessment of research within the tertiary sector.

Effects of the Performance-Based Research Fund and Other Factors on the Efficiency of New Zealand Universities: A Malmquist Productivity Approach

Universities and academic institutions play a very crucial role in nation development through the production of highly competent manpower. The eight universities in New Zealand have been recognized as some of the top academic institutions in the world. However, the rankings from different international organizations are declining and are hardly likely to rise. Government policies in funding allocation are being blamed for the universities’ regress as they operate with an insufficient amount of funds. This study uses the Malmquist Productivity Index model to examine the technical efficiency, technological change, and productivity performance of the eight universities. This model uses a variety of inputs (number of academic and non-academic staff and total enrolment) and outputs (number of degree and postgraduate graduates, total graduates, and total operating revenue from the Equivalent Full-Time Student (EFTS) funding system and the Performance-Based Research Fund (PBRF), etc.) obtained for the period 2013–2018. The overall results show that the average catch-up and frontier-shift efficiencies of the universities are roughly in a “no-change” scenario, meaning that the universities did not make any progress over these years. The Malmquist Productivity Index (MPI) also shows a stable result, with average final values slightly higher than 1, wherein only five universities reached an actual productivity score of 1. It is recommended that the universities improve their internal factors, including personnel, equipment, facilities, and student services, while taking accounts of external aspects, such as rapid growth in technological environments and innovations, to achieve sustainable organizational progress and improved productivity. The re-assessment of government policies for funding allocation is also suggested. This research paper offers insights into the New Zealand universities’ productivity performances for the past few years. This could be used as a reference for other purposes.

Is external research assessment associated with convergence or divergence of research quality across universities and disciplines? Evidence from the PBRF process in New Zealand

ABSTRACT Performance-based research evaluations have been adopted in several countries both to measure research quality in higher education institutions and as a basis for the allocation of funding across institutions. Much attention has been given to evaluating whether such schemes have increased the quality and quantity of research. This paper examines whether the introduction of the New Zealand Performance-Based Research Fund process produced convergence or divergence in measured research quality across universities and disciplines between the 2003 and 2012 assessments. Two convergence measures are obtained. One, referred to as β-convergence, relates to the relationship between changes in average quality and the initial quality level. The second concept, referred to as σ-convergence, relates to the changes in the dispersion in average research quality over time. Average quality scores by discipline and university were obtained from individual researcher data, revealing substantial β- and σ-convergence in research quality over the period. The hypothesis of uniform rates of convergence across almost all universities and disciplines is supported. The results provide insights into the incentives created by performance-based funding schemes.

The ‘disciplinary effect’ of the performance-based research fund process in New Zealand

This paper examines how the research quality of academic disciplines within New Zealand universities has evolved since the Performance-based Research Fund (PBRF) began in 2003. It uses a database consisting of an anonymous ‘quality category’ (QC) for each person in the 2003 and 2012 assessment rounds. Individuals are assigned to nine discipline groups and the paper measures the distribution of researchers across disciplines within universities. There has been little change in the distribution and their concentration within and across universities. Exceptions are increases in the shares of medicine and agriculture, and a reduction in the share of education. Average Quality Scores are derived for each discipline. All groups substantially increased their scores. Transition matrices show that there are significant differences in the dynamics of disciplines during the PBRF process. Changes in the discipline composition of universities explains little of the proportional improvement of research quality among New Zealand universities.

An evaluation of metrics used by the Performance-based Research Fund process in New Zealand

ABSTRACTThe New Zealand Performance-based Research Fund applies a set of metrics to assess researchers and rank disciplines and universities. The process involves giving a total weighted score (based on three components) to individuals and then assigning them to one of four Quality Categories (QCs), used to derive Average Quality Scores (AQS). This paper evaluates the properties of these metrics and argues that QC thresholds influence the final distribution of scores. The paper also demonstrates that the derivation of AQSs depends on the weights assigned to each QC and the distribution of portfolios. The method used to determine raw scores also has an independent effect on the distribution of scores. The paper compares how research rankings of New Zealand universities would vary if alternative summary measures, based on the total weighted scores rather than QCs, were used to evaluate performance.

The Impact on Research Quality of Performance-Based Funding: The Case of New Zealand’s PBRF Scheme

This paper discusses the impact on research quality of New Zealand universities of the Performance-Based Research Fund from 2003 to 2012. This is a peer-review process involving assessment of individual researchers. The contribution to improvement in research quality of transitions among research quality categories and entrants and exits of individuals are identified. A substantial component of change has been the removal of non-research active staff. There has been population ageing due to retention of older and higher-quality researchers and a large reduction in the number of younger researchers. Significant differences among universities are evident in the patterns of transformation. The paper also critically considers the PBRF assessment process and characteristics of the metrics used, suggesting scope for improvement in the assessment of researchers and the way in which universities are ranked. (This abstract was borrowed from another version of this item.)

The evolution of research quality in New Zealand universities as measured by the performance-based research fund process

ABSTRACTThis paper examines how research quality of New Zealand university staff has evolved since introduction in 2003 of the Performance-based Research Fund (PBRF). The analysis uses a database consisting of an anonymous ‘quality category’ (QC) for each individual assessed in each of the three PBRF rounds. Emphasis is on evaluation of organisational changes. The paper examines the extent to which each university's average quality score (AQS) changed as a result of changes in QCs of existing staff over time and from staff exit and entry. The data also include information about the age of staff evaluated in PBRF. This is used to assess changes in the age distribution of researchers across universities, and ages of those making transitions within universities and between grades. A number of hypotheses regarding organisation change in response to the introduction of PBRF are discussed and tested by comparing universities with different patterns of change.

Rationale for the Performance-Based Research Fund at Malaysian Higher Education Institutions

Rationale for the Performance-Based Research Fund at Malaysian Higher Education Institutions

Will Scholars Trump Teachers In New Zealand Teacher Education?

The Performance-Based Research Fund (PBRF) assessment process in 2003 highlighted the research imperative for academic staff in New Zealand teacher education. This imperative was not new: it was implicit in the tertiary education changes of 1990, which ended the university monopoly over degree granting and gave autonomy to colleges of education and polytechnics. Previous assumptions about the roles of university and college academics were challenged. Few teacher educators had engaged in research before 1990; staff were recruited from the profession on the basis of their professional expertise. Developing a research culture alongside the demands of teaching and professional involvement in schools leads to tensions that few institutions worldwide have been able to solve. This paper examines the experience of two New Zealand teacher education institutions in responding to the new research imperative, and then considers the impact of the PBRF process and reporting on policy and practice. It identifies significant issues for resourcing and developing capacity but concludes that research is an imperative of professional practice that has the capacity to enrich our teaching and inform policy. However, maintaining balance and equilibrium among the contradictory demands and pressures of research and teaching is still an essential goal if we are to serve education well.

Navigating research currents – emerging Pasifika researcher?

The title of this work speaks to my experience as an emerging researcher in a tertiary environment. Seventeen years ago, during my introduction to research as a required topic of study, I recall making the comment ‘... up until now I have measured things with a ruler’. More recently I have come to understand the research processes as premised on the economic imperatives of an information age. Implicit in this is the acquisition of knowledge based on intellectual property, where research for the sake of research vies with sites seeking to create purposeful research as conduits to informing policy and improved circumstances. In truth, the drive and thrust of Performance Based Research Fund (PBRF) imperatives in my place of work has changed the original content of my work in bridging education and social work training to where the teaching component is no longer the primary focus. Certainly external factors have played a contributing role in structural and systems changes of the institution (for example, amalgamation with a leading university), however the PBRF environment across campuses is now the norm. Research outputs increasingly help to fund tertiary institutions. This article draws attention to some reflections of the writer with regard to notions of an emerging Pasifika researcher.

Advocating Journalism Practice-as-research

New Zealand’s Performance-Based Research Fund regime makes no explicit provision for journalism practice-as-research, although it does not exclude it either. The Tertiary Education Commission (TEC) has hinted at applied research changes that may open the door to investigative journalism and other long-form modes for the next audit in 2018. As a growing global debate responds to international pressure on journalism practitioner-academics to publish scholarly outputs and seek external research income, there has been a parallel paradigm shift in defining methodologies for some journalism as research. This article advocates a bolder approach by journalism educators in New Zealand to push the boundaries for greater acceptance of journalism research methodologies and to claim an enhanced academic space as a ‘critic and conscience of society’.

The Performance-based Research Fund, gender and a cultural cringe

This article explores ways the Performance-based Research Fund (PBRF) produces gendered results and expresses a cultural cringe. It is argued that the research evaluation is fixated with being ‘world-class’ at the expense of academic practice that focuses on New Zealand. In this context, disadvantage faced by female academics under the PBRF can be re-imagined as an exemplar of a broader experience faced by all New Zealand-trained and focused academics. At the same time, the PBRF has produced some embarrassing results for neoliberal policy-makers and somewhat empowers academics as arbiters of excellence by reifying elements of peer review.

Rethinking the geography PhD in New Zealand: navigating through contexts, circumstances and challenges

We situate the contemporary PhD experience in the New Zealand context of robust international networks and scholarship, but one in which aspiring doctoral candidates have, until recently, been encouraged to study overseas. Of late, however, an increase in doctoral registrations can be linked to a series of drivers in place within the universities since the new millennium. With the impetus of the Performance-Based Research Fund, domestic scholarships, and the waiving of fees for international students the PhD has changed from a largely open-ended individual pursuit to one embedded within, and potentially, contributing to a wider research culture. Questions remain, however, about the experience of doctoral study itself, and we pose these with reference to the need for further consideration of the PhD’s role in academic identity, disciplinary reproduction and knowledge production.

Q&A Jane Harding: individual approach.

Jane Harding is deputy vice-chancellor for research and professor of neonatology at the University of Auckland, which is New Zealand's most well-funded university under the Performance-Based Research Fund. She discusses the country's approach to assessing science and measuring impact, and describes why she prefers a model that grades the individual not the research group.

Unpredictable feelings: academic women under research audit

Academic research is subject to audit in many national settings. In Aotearoa/New Zealand, the government regulates the flow of publicly funded research income into tertiary institutions through the Performance‐Based Research Fund (PBRF). This article enquires into the effects of the PBRF by exploring data collected from 16 academic women of varying rank in the arts, humanities and social sciences. References to the women's emotions were extracted from the transcripts of semi‐structured interviews, in which they were asked about experiences of being researchers and doing research in the early years of the PBRF. The analysis presented here undertakes a politicised reading of those emotions—as capacities subject to governmentality—to engage with two issues currently explored in the international higher education literature: (1) the gendered effects of audit, and (2) the notable absence of collective political resistance on the part of academics. We argue that, while the PBRF does produce emotions implicated in reshaping researcher practices, identities, and relationships with colleagues, those effects are fractured and far from predictably gendered. Moreover, the dominant flavour of the feelings described is unlikely to support collective political resistance and more likely to be implicated in increased individualism, pressure to perform and judgmentalism towards others. At the same time, some emotions prompted and sustained forms of active resistance towards research audit's threat to academic imaginations and collegial culture.