Research Assessment Exercise Research Articles

Research in UK universities: a tale of two subjects – Economics and Econometrics; and Business and Management Studies

ABSTRACT This paper explores the comparative performance of two subject areas (Units of Assessment – UOAs) in the UK national Research Assessment Exercises of 1992, 1996, 2001, 2008 and the Research Excellence Framework of 2014. The Units of Assessment are Economics and Econometrics; and Business and Management Studies. Given the wide range of disciplines in the Business and Management Studies UOA universities have the choice of submitting their economics and econometrics research either to it or to the Economics and Econometrics UOA. Over the period 1992–2014 the number of universities entering the Economics and Econometrics UOA declined significantly in contrast to the growth in the number entering the Business and Management Studies UOA. The reasons for this include the finding that university managers were more likely to tolerate poor performance (measured by internal and external gaps) in Business and Management Studies than in Economics and Econometrics. Considerable differences were found between old (pre-1992) and new (post-1992) universities, with Economics and Econometrics now solely the domain of old universities as new ones have withdrawn. In sharp contrast, Business and Management Studies has a good balance of old and new universities.

On the Shapley value and its application to the Italian VQR research assessment exercise

Abstract Research assessment exercises have now become common evaluation tools in a number of countries. These exercises have the goal of guiding merit-based public funds allocation, stimulating improvement of research productivity through competition and assessing the impact of adopted research support policies. One case in point is Italy's most recent research assessment effort, VQR 2011–2014 (Research Quality Evaluation), which, in addition to research institutions, also evaluated university departments, and individuals in some cases (i.e., recently hired research staff and members of PhD committees). However, the way an institution's score was divided, according to VQR rules, between its constituent departments or its staff members does not enjoy many desirable properties well known from coalitional game theory (e.g., budget balance, fairness, marginality). We propose, instead, an alternative score division rule that is based on the notion of Shapley value, a well known solution concept in coalitional game theory, which enjoys the desirable properties mentioned above. For a significant test case (namely, Sapienza University of Rome, the largest university in Italy), we present a detailed comparison of the scores obtained, for substructures and individuals, by applying the official VQR rules, with those resulting from Shapley value computations. We show that there are significant differences in the resulting scores, making room for improvements in the allocation rules used in research assessment exercises.

The UK Research Excellence Framework and the Matthew effect: Insights from machine learning.

With the high cost of the research assessment exercises in the UK, many have called for simpler and less time-consuming alternatives. In this work, we gathered publicly available REF data, combined them with library-subscribed data, and used machine learning to examine whether the overall result of the Research Excellence Framework 2014 could be replicated. A Bayesian additive regression tree model predicting university grade point average (GPA) from an initial set of 18 candidate explanatory variables was developed. One hundred and nine universities were randomly divided into a training set (n = 79) and test set (n = 30). The model “learned” associations between GPA and the other variables in the training set and was made to predict the GPA of universities in the test set. GPA could be predicted from just three variables: the number of Web of Science documents, entry tariff, and percentage of students coming from state schools (r-squared = .88). Implications of this finding are discussed and proposals are given.

Academic misconduct, misrepresentation and gaming: A reassessment

The motivation for this Special Issue is increasing concern not only with academic misconduct but also with less easily defined forms of misrepresentation and gaming. In an era of intense emphasis on measuring academic performance, there has been a proliferation of scandals, questionable behaviors and devious stratagems involving not just individuals but also organizations, including universities, editors and reviewers, journal publishers, and conference organizers. This introduction first reviews the literature on the prevalence of academic misconduct, misrepresentation and gaming (MMG). The core of the article is organized around a life-cycle model of the production and dissemination of research results. We synthesize the findings in the MMG literature at the level of the investigator or research team, emphasizing that misbehavior extends well beyond fabrication and falsification to include behaviors designed to exaggerate or to mislead readers as to the significance of research findings. MMG is next explored in the post-research review, publication, and post-publication realms. Moving from the individual researcher to the organizational level, we examine how MMG can be engaged in by either journals or organizations employing or funding the researchers. The changing institutional environment including the growth of research assessment exercises, increased quantitative output measurement and greater pressure to publish may all encourage MMG. In the final section, we summarize the main conclusions and offer suggestions both on how we might best address the problems and on topics for future research.

A ‘power law’ based method to reduce size-related bias in indicators of knowledge performance: An application to university research assessment

Abstract The knowledge production provided by universities is essential to sustaining a country’s long-term economic growth and international competitiveness. Many nations are thus driving to create sustainable and effective funding environments. The evaluation of university knowledge, productivity and research quality becomes critical, with ever increasing share of public funding allocated on the basis of research assessment exercises. Nevertheless, the existing methods to assess the universities’ knowledge production are often affected by limits and biases, extensively discussed in the scientific literature. In this paper we study how to reduce the effect of size-related bias due to university size on the indicators of knowledge performance used in evaluation exercises. We propose an innovative utilization of the scale-free property of the power laws as a scaling relationship, to normalize research productivity indicators, and provide results independent by the university size. Our method has evident policy implications and gives a contribution for the future design of assessment exercises. We apply our findings in a recent Italian research assessment exercise.

Revisiting the scientometric conceptualization of impact and its measurement

The development of scientometric indicators and methods for evaluative purposes, requires a multitude of assumptions, conventions, limitations, and caveats. Given this, we cannot permit ambiguities in the key concepts forming the basis of scientometric science itself, or research assessment exercises would rest on quicksand. This conceptual work attempts to spell out some principles leading to a clear definition of "impact" of research, and above all, of the appropriate scientometric indicator to measure it. The aim is to stimulate a discussion aimed at a definitive convergence on the meaning and measurement of a fundamental concept of the scientometric science.

Rise of Authoritarianism in Higher Education: A Critical Analysis of the Research Assessment Exercise in British Universities

This paper presents a critical analysis of the Research Assessment Exercise (RAE) in terms of theory, practice, and a Kierkegaardian metaphor related to issues of authoritarianism, the individual, objectivity, subjectivity, and maieutic communication. The assumptions that underlie the principles of the RAE are problematic in that conflict and contradictions are an inevitable part of these assumptions, particularly as they relate to issues of power and resistance. Governments should embrace risk - a leap of faith - and trust the university to take care of itself. To do otherwise can only end in failure of the university and its ability to conduct research in good faith.

Publishing in and about English: challenges and opportunities of Chinese multilingual scholars’ language practices in academic publishing

This paper reports on a qualitative inquiry into Chinese multilingual scholars’ language practices in academic publishing as the Chinese government develops higher education competitiveness and expands multilingual foreign language education. Fifteen Chinese multilingual scholars from five non-English language divisions were interviewed. Bourdieu’s linguistic market theory was adopted as the guiding framework to understand the scholars’ multilingual practices in academic publishing. The findings suggest that institutional research assessment based on key index lists functioned as a market unification mechanism that devalued the scholars’ multilingual capital. Meanwhile, the implicit language policy of publishing in and about English perpetuated and reproduced structural inequality in the international and national academic publishing marketplace. Despite this perceived disadvantaged position, the scholars generated new language practices to counteract the structural constraints. In light of these findings, policy makers should be aware of the implicit language policies in index-based research assessment exercises, and should take into consideration the specific circumstances of different language majors when developing research assessment policies. Measures are suggested to empower multilingual scholars with capacities to participate equally in knowledge construction, regardless of the languages that they work with.

Human capital loss in an academic performance measurement system

PurposeThe purpose of this paper is to examine the perceptions of academic human capital (HC) towards a university’s research performance measurement system (PMS), in response to a national research assessment exercise (RAE).Design/methodology/approachThis paper draws on a subset of the data from a large mixed method case study research project about the impact of ERA on an Australian public sector university.FindingsThe findings reveal that the research PMSs were designed, implemented and used as a tool to measure and manage the research performance of HC within the university. The case study university performed well in the RAE. However, this also led to several unintended consequences in the form of fear and anxiety, gaming and strategic initiatives, a focus on quantity and not the quality of research, and increased workload, which led to a loss in the stock of HC.Practical implicationsThis empirical evidence can inform governments and policy makers of the unintended consequences of government research evaluations on academic HC. University managers could improve the design of HC management systems by not only measuring academic HC performance, but also providing training and resources to enhance, support and maintain the overall well-being of academics.Originality/valueThis study provides insights regarding the connection between a university’s PMS and academic HC and contributes to the academic literature on intellectual capital and PMSs.

Academic Misconduct, Misrepresentation and Gaming: A Reassessment

The motivation for this Special Issue is increasing concern not only with academic misconduct but also with less easily defined forms of misrepresentation and gaming. In an era of intense emphasis on measuring academic performance, there has been a proliferation of scandals, questionable behaviors and devious stratagems involving not just individuals but also organizations, including universities, editors and reviewers, journal publishers, and conference organizers. This introduction first reviews the literature on the prevalence of academic misconduct, misrepresentation and gaming (MMG). The core of the article is organized around a life-cycle model of the production and dissemination of research results. We synthesize the findings in the MMG literature at the level of the investigator or research team, emphasizing that misbehavior extends well beyond fabrication and falsification to include behaviors designed to exaggerate or to mislead readers as to the significance of research findings. MMG is next explored in the post-research review, publication, and post-publication realms. Moving from the individual researcher to the organizational level, we examine how MMG can be engaged in by either journals or organizations employing or funding the researchers. The changing institutional environment including the growth of research assessment exercises, increased quantitative output measurement and greater pressure to publish may all encourage MMG. In the final section, we summarize the main conclusions and offer suggestions both on how we might best address the problems and on topics for future research.

‘Environment’ Submissions in the UK's Research Excellence Framework 2014

AbstractThere has been much debate about university research assessment exercises. In the UK, a major element of the 2014 Research Excellence Framework (REF2014) has been the research environment. Here we analyse 98 REF2014 ‘environment’ submissions in Business and Management Studies. We explore whether there are distinctive language‐related differences between submissions of high and low ranked universities and conclude that submission writers have a strong incentive to exaggerate strengths and conceal problems. In addition, innate biases such as the ‘halo’ and ‘velcro’ effects may distract the attention of assessors from a submission's strengths and weaknesses, since they are likely to influence their pre‐existing impressions. We propose several changes to improve how environment is evaluated. We also argue that the research environment would be more likely to be enhanced if the number of outputs submitted in future was an average of two and a maximum of four per academic, rather than the maximum of five currently being considered.

Universities as celebrities? How the media select information from a large research assessment exercise

The article enters the international debate on university rankings, observing the visibility premium that institutions gain from their publication. Despite the deep academic skepticism towards the composite indicators used at this aim, rankings are an adaptive solution for individuals to treat information uncertain in nature. Rankings summarize many university quality dimensions into one single number and they fit well the need of media to package information. We have used data from the press review of the Italian VQR 2004–10 to analyze the factors that lead universities to be more frequently cited, under the hypothesis that the visibility of a university is a function of its intrinsic characteristics, such as number of students, prestige, age, or size or density of population in its location. Through a set of regression models, we find that the only variable that matters is the presence in the top positions.

Measuring impact in the humanities: Learning from accountability and economics in a contemporary history of cultural value

This article addresses the future of research assessment within higher education in the UK from a humanities perspective. Recent changes to policy (such as The Browne Report 2010 and the 2014 REF) indicates that humanities research is increasingly required to provide quantifiable or commercial results in order to attain value. Although research assessment exercises have been a formal part of UK higher education since the first Research Assessment Exercise (RAE) in 1986, the last 6 years have seen a significant change in how research is valued within the academy. Specifically, this paper responds to the increasing prioritisation of 'impact' measurement in research assessment criteria. The article situates recent changes in higher education within a historical context of cultural policymaking in the UK from the 1980s to the present day. Such an undertaking highlights the specific challenges and nuances within the shift towards 'impact'. Firstly, this paper details how public cultural institutions (such as museums and art galleries) became subject to practises of economisation and social accountability as a result of 1980s cultural policy. A rich field of literature from museology and arts management provides valuable sources and testimonies that should be considered in the future of the academic humanities. Secondly, this paper considers the implications of the creative industries upon the perception of knowledge production since the 1990s. Following this specific history of cultural assessment mechanisms in the UK, this article concludes by demonstrating that neither the adoption of a purely economic approach nor a refusal of accountability will serve the humanities. Whilst there is a wealth of social science research that explores valuation methods and assessment culture there is a lack of humanities research within this vital debate. This article presents a response from a humanities perspective. As a result, this contribution raises awareness of the urgent need for humanities scholars to engage in these emerging and significant debates concerning the future of research assessment in the UK.

How does research evaluation impact educational research? Exploring intended and unintended consequences of research assessment in the United Kingdom, 1986–2014

Research evaluation systems in many countries aim to improve the quality of higher education. Among the first of such systems, the UK’s Research Assessment Exercise (RAE) dating from 1986 is now the Research Excellence Framework (REF). Highly institutionalised, it transforms research to be more accountable. While numerous studies describe the system’s effects at different levels, this longitudinal analysis examines the gradual institutionalisation and (un)intended consequences of the system from 1986 to 2014. First, we analyse historically RAE/REF’s rationale, formalisation, standardisation, and transparency, framing it as a strong research evaluation system. Second, we locate the multidisciplinary field of education, analysing the submission behaviour (staff, outputs, funding) of departments of education over time to find decreases in the number of academic staff whose research was submitted for peer review assessment; the research article as the preferred publication format; the rise of quantitative analysis; and a high and stable concentration of funding among a small number of departments. Policy instruments invoke varied responses, with such reactivity demonstrated by (1) the increasing submission selectivity in the number of staff whose publications were submitted for peer review as a form of reverse engineering, and (2) the rise of the research article as the preferred output as a self-fulfilling prophecy. The funding concentration demonstrates a largely intended consequence that exacerbates disparities between departments of education. These findings emphasise how research assessment impacts the structural organisation and cognitive development of educational research in the UK.

Some observations on the current state of book reviewing in the social sciences

Key points The number of printed book reviews in the social sciences is currently declining. The number of online book reviews is currently increasing. There are ways of measuring the impact of book reviews so that they can be included in research assessment exercises.

Reference Publication Year Spectroscopy (RPYS) with publications in the area of academic efficiency studies: what are the historical roots of this research topic?

ABSTRACTIn this study, we explore the historical roots of the relatively new topic in scientometrics of academic efficiency assessments. We are interested in the contributions of researchers from the past which have been revealed as important for the topic in the long run. The technique of Reference Publication Year Spectroscopy (RPYS) has been recently introduced which is based on the analysis of the frequency with which references are cited in the publications of a specific research field (here: academic efficiency assessments). The study is based on papers conducted for a systematic review of empirical articles on technical efficiency in academic research production: 60 papers (published between 1992 and 2012) and 1314 cited references. Results indicated that 5 peaks are clearly identifiable until 2000. They correspond, respectively, to the years 1957 (The founding article of Farrell), 1978 (Proposition of a new promising approach: the Data Envelopment Analysis (DEA) by Charnes et al.), 1988 (Research-teaching multi-output model and integration of quality indicators), 1990 (DEA in the service of Research Assessment Exercise) and 1997 (Introduction of weight restrictions in DEA). The peaks are described with the underlying publications and recent developments (since 2000) in the area of academic efficiency studies are outlined.

Learning from the UK’s research impact assessment exercise: a case study of a retrospective impact assessment exercise and questions for the future

National governments spend significant amounts of money supporting public research. However, in an era where the international economic climate has led to budget cuts, policymakers increasingly are looking to justify the returns from public investments, including in science and innovation. The so-called ‘impact agenda’ which has emerged in many countries around the world is part of this response; an attempt to understand and articulate for the public what benefits arise from the research that is funded. The United Kingdom is the most progressed in implementing this agenda and in 2014 the national research assessment exercise, the Research Excellence Framework, for the first time included the assessment of research impact as a component. For the first time within a dual funding system, funding would be awarded not only on the basis of the academic quality of research, but also on the wider impacts of that research. In this paper we outline the context and approach taken by the UK government, along with some of the core challenges that exist in implementing such an exercise. We then synthesise, together for the first time, the results of the only two national evaluations of the exercise and offer reflections for future exercises both in the UK and internationally.

Comments on the correspondence “On tit for tat: Franceschini and Maisano versus ANVUR regarding the Italian research assessment exercise VQR 2011–2014”, J. Informetr., 11 (2017), 783–787

Comments on the correspondence “On tit for tat: Franceschini and Maisano versus ANVUR regarding the Italian research assessment exercise VQR 2011–2014”, J. Informetr., 11 (2017), 783–787

On tit for tat: Franceschini and Maisano versus ANVUR regarding the Italian research assessment exercise VQR 2011–2014

This study inserts in the stream of research on the perverse effects that PBRF systems can induce in the subjects evaluated. The authors’ opinion is that more often than not, it is the doubtful scientific basis of the evaluation criteria that leave room for opportunistic behaviors. The work examines the 2004–2010 Italian national research assessment (VQR) to test the lack of possible opportunistic behavior by universities in order to limit the penalization of their performance (and funding) due to the presence of scientifically unproductive professors in faculty. In particular, institutions may have favored “gift authorship” practices. The analysis thus focuses on the output of professors who were unproductive in the VQR publication window, but became productive (“new productives”) in the following five years. A number of universities show a higher than average share of publications by new productives that are in co-authorship exclusively with colleagues from the same university. Although this might be thought to reflect opportunistic behavior by universities, the empirical evidence does not support this assumption.

A rejoinder to the comments of Benedetto et al. on the paper “Critical remarks on the Italian research assessment exercise VQR 2011–2014” (Journal of Informetrics, 11(2): 337–357)

The paper Critical remarks on the Italian research assessment exercise VQR 2011-2014 (Franceschini & Maisano, 2017) analyzed some vulnerabilities of the recently concluded Italian assessment exercise. Some apical (former and current)members of ANVUR promptly commented on our criticisms through a letter to the editor (Benedetto, Checchi, Graziosi, & Malgarini, 2017). We believe that this letter is not very convincing. In the following, we provide a rejoinder to the comments directed to our paper.