UK Research Assessment Exercise Research Articles

How do policy instruments generate new ones? Analysing policy instruments feedback and interaction in educational research in England, 1986-2014

The study of research funding arrangements and the production of scientific knowledge has been marked by a lack of understanding about how research funding instruments interact and how these instruments shape policy-making and research fields. To fill this research gap, this study is theoretically supported by policy feedback and policy instruments’ interaction studies. It investigates the effects of the UK's research assessment exercise in the creation of the most emblematic national thematic research program for the field of educational research in the country – the Teaching and Learning Research Programme (TLRP). Based on qualitative analysis of policy documents and semi-structured interviews with policy-makers and boundary-spanners, this paper shows how the research assessment exercise contributed to the creation of the TLRP and how the interaction between the two policy instruments shaped the field of educational research in England. In particular, the results show a) how the institutionalisation of the research assessment led to frame a “quality problem” in educational research that legitimated several policy initiatives, including the creation of the national thematic research programme (interpretative effects) and a shift in resources allocation (resource/incentive effects); and b) how the interaction between the two policy instruments contributed to methodological and epistemic drifts in the field of educational research.

Evaluating journal quality: A review of journal citation indicators and ranking in business and management

Evaluating the quality of academic journals is becoming increasing important within the context of research performance evaluation. Traditionally, journals have been ranked by peer review lists such as that of the Association of Business Schools (UK) or though their journal impact factor (JIF). However, several new indicators have been developed, such as the h-index, SJR, SNIP and the Eigenfactor which take into account different factors and therefore have their own particular biases. In this paper we evaluate these metrics both theoretically and also through an empirical study of a large set of business and management journals. We show that even though the indicators appear highly correlated in fact they lead to large differences in journal rankings. We contextualise our results in terms of the UK's large scale research assessment exercise (the RAE/REF) and particularly the ABS journal ranking list. We conclude that no one indicator is superior but that the h-index (which includes the productivity of a journal) and SNIP (which aims to normalise for field effects) may be the most effective at the moment.

Statistical modelling of citation exchange between statistics journals.

SummaryRankings of scholarly journals based on citation data are often met with scepticism by the scientific community. Part of the scepticism is due to disparity between the common perception of journals’ prestige and their ranking based on citation counts. A more serious concern is the inappropriate use of journal rankings to evaluate the scientific influence of researchers. The paper focuses on analysis of the table of cross‐citations among a selection of statistics journals. Data are collected from the Web of Science database published by Thomson Reuters. Our results suggest that modelling the exchange of citations between journals is useful to highlight the most prestigious journals, but also that journal citation data are characterized by considerable heterogeneity, which needs to be properly summarized. Inferential conclusions require care to avoid potential overinterpretation of insignificant differences between journal ratings. Comparison with published ratings of institutions from the UK's research assessment exercise shows strong correlation at aggregate level between assessed research quality and journal citation ‘export scores’ within the discipline of statistics.

The Impact of the UK Research Assessment Exercise

International Higher Education (IHE) publishes insightful, informed, and high-quality commentary and analysis on trends and issues of importance to higher education systems, institutions, and stakeholders around the world.

Use of the h-index to measure the quality of the output of health services researchers

To assess the use of the h-index to measure the quality of the output of health services researchers. Online survey, with bibliometric analysis of a convenience volunteer sample of researchers mainly in the UK, North America and Australasia. Self-reported from Google Scholar: h-index; number of papers; number of citations; number of papers with ≥ 10 citations. There were complete responses from 532 health services researchers of whom 371 (70%) were from the UK. Of the bibliometric measures, the h-index appeared to be the best discriminator between other measures of quality (e.g. seniority; entry into the last UK Research Assessment Exercise). The median h-index was 12, with 90th and 95th quantiles of 40 and 52, respectively. Statisticians had the highest h-index with qualitative researchers the lowest (median 16 and 7, respectively). The h-index was predicted to increase by approximately 1 point annually with the biggest increase in statisticians and smallest in qualitative researchers when estimated by quantile regression. The h-index is a useful summary measure of output and quality of health services researchers. However, any accurate interpretation of bibliometric measures needs to take into account a person's research discipline.

The UK Research Assessment Exercise and the narrowing of UK economics

The aim of this paper is to delineate an empirically grounded, structure-causal going concern recursive model of UK economics that, in the context of the RAE and local department decision-making, explains the progressive elimination of heterodox economics, the progressive homogenization of mainstream economics from 1992 to the present, and the continued rise to dominance of a select group of departments, and indicates whether these ‘regularities’ will continue under the Research Excellence Framework selectivity exercise in 2014.

The Effect of Department Size on Quality of Research in Science: Evidence from the UK Research Assessment Exercise

We examine whether the organisation of individual researchers into departments has an impact on the average research quality, beyond the impact of the individuals themselves. Using data on science disciplines generated from the UK Research Assessment Exercises in 1996, 2001 and 2008, we find that department size has an independent positive effect on research quality.

The pursuit of academic excellence and business engagement: is it irreconcilable?

Universities currently need to satisfy the demands of different audiences. In light of the increasing policy emphasis on "third mission" activities, universities are attempting to incorporate these into their traditional missions of teaching and research. University strategies to accomplishing its traditional missions are well-honed and routinized, but the incorporation of the third mission is posing important strategic and managerial challenges for universities. This study explores the relationship between university---business collaborations and academic excellence in order to examine the extent to which academic institutions can balance these objectives. Based on data from the UK Research Assessment Exercise 2001 at the level of the university department, we find no systematic positive or negative relationship between scientific excellence and engagement with industry. Across the disciplinary fields reported in the 2001 Research Assessment Exercise (i.e. engineering, hard sciences, biomedicine, social sciences and the humanities) the relationship between academic excellence and engagement with business is largely contingent on the institutional context of the university department. This paper adds to the growing body of literature on university engagement with business by examining this activity for the social sciences and the humanities. Our findings have important implications for the strategic management of university departments and for higher education policy related to measuring the performance of higher education research institutions.

Metrics-Based Assessments of Research: Incentives for ‘Institutional Plagiarism’?

The issue of plagiarism--claiming credit for work that is not one's own, rightly, continues to cause concern in the academic community. An analysis is presented that shows the effects that may arise from metrics-based assessments of research, when credit for an author's outputs (chiefly publications) is given to an institution that did not support the research but which subsequently employs the author. The incentives for what is termed here "institutional plagiarism" are demonstrated with reference to the UK Research Assessment Exercise in which submitting units of assessment are shown in some instances to derive around twice the credit for papers produced elsewhere by new recruits, compared to papers produced 'in-house'.

STATISTICS OF STATISTICIANS: CRITICAL MASS OF STATISTICS AND OPERATIONAL RESEARCH GROUPS

Using a recently developed model, inspired by mean field theory in statistical physics, and data from the UK's Research Assessment Exercise, we analyse the relationship between the qualities of statistics and operational research groups and the quantities of researchers in them. Similar to other academic disciplines, we provide evidence for a linear dependency of quality on quantity up to an upper critical mass, which is interpreted as the average maximum number of colleagues with whom a researcher can communicate meaningfully within a research group. The model also predicts a lower critical mass, which research groups should strive to achieve to avoid extinction. For statistics and operational research, the lower critical mass is estimated to be 9 ± 3. The upper critical mass, beyond which research quality does not significantly depend on group size, is 17 ± 6.

Contributions for Innovative Institutional Research Quality Assessment Practices and Processes

The urge for the achievement of excellence patterns in research has led higher education institutions to become more and more aware of their role in the process by putting their strategic emphasis on the development of assessment tools towards the monitoring and evaluation of research quality. We intend to review the lessons learned from current research assessment experiences and collect good practices and recommendations using semi-directed interviews with evaluation panel members, institutions’ representatives and researchers involved in the last UK's Research Assessment Exercise. Our major aim is to present and discuss a set of guidelines as a first input for the design of an evaluation framework to evaluate and monitor the quality of research at an institutional level. We seek to add significantly to our understanding of what can be actually done using strategic planning to look forward in the areas of research engagement/resources and institutional culture, productivity/performance and innovation, quality/merit and impact and sustainability/support. Overall, the challenges of promoting research are discussed and strategies for its enhancement are included.

Fair and Open Approach to Academic Publishing

The Journal Systemic Practice and Action Research (SPAR) aims to encourage into print authors and practitioners of systemic thinking and practice from all kinds of background. In this note we describe both the publishing world into which SPAR has emerged and the systemic and inclusive thinking behind the journal’s publishing policy. We set out our manifesto for a fair and open approach to academic publishing. “A rich and diverse set of potential bibliometric and scientometric predictors of research performance quality and importance are emerging today—from the classic metrics (publication counts, journal impact factors and individual article/author citation counts) to promising new online metrics such as download counts, hub/authority scores and growth/decay chronometrics. In and of themselves, however, metrics are circular: They need to be jointly tested and validated against what it is that they purport to measure and predict, with each metric weighted according to its contribution to their joint predictive power. The natural criterion against which to validate metrics is expert evaluation by peers; a unique opportunity to do this is offered by the 2008 UK Research Assessment Exercise, in which a full spectrum of metrics can be jointly tested, field by field, against peer rankings.” (Harnard 2008)

Bernard Gilmartin

To mark the retirement of Professor Bernard Gilmartin from his personal chair in optometry, Aston University’s Ophthalmic Research Group organised a 2-day conference celebrating Bernard’s many contributions to optometric education and research, and his tireless commitment to advancing and developing the profession of optometry. This Festschrift special issue of Ophthalmic and Physiological Optics includes a selection of papers based on the material presented at the event in September 2011, reflecting many of Bernard’s personal research interests. Moreover, it seems very apposite to record this occasion in The College of Optometrists’ journal that Bernard so skilfully edited between 1987 and 2000, and for which he now acts as an Associate Editor. Bernard Gilmartin was born and educated in Manchester, and commenced his studies in optometry at City University, London, in 1965. He was awarded a First Class Honours degree in optometry in 1968 and following General Optical Council registration, returned to City to complete a PhD. A further year was spent at City as a lecturer, followed by 2 years in full-time optometric practice. In 1974 he joined Aston University, Birmingham, as a lecturer, and following promotion to senior lectureship (1988) and readership (1994), was awarded a personal chair in optometry in 1998. He completed a 5 year term as Director of Research for the School of Life and Health Sciences at Aston in 2006 and was founder and convener of the Ophthalmic Research Group between 1984 and 2010. During his time at Aston, Bernard published over 130 research papers together with four book chapters and two edited books principally on the topics of accommodation, autonomic nervous system function and myopia. He was a senior examiner for The College of Optometrists Professional Qualifying Examinations for 30 years and acts as an educationalist for General Optical Council institutional visits, and more recently was a member of the Allied Health Professions and Studies Panel for the 2008 UK Research Assessment Exercise and chair of the College of Optometrists Research Committee. Bernard was elected to a Fellowship of the American Academy of Optometry in 1990, to a Life Fellowship of The College of Optometrists in 2000 ‘in recognition of his outstanding contribution to the College and the profession over many years’ and was awarded the Chancellor’s Medal from Aston University in July 2009 for ‘outstanding and sustained contributions to the University’. Importantly, in addition to Bernard’s many academic achievements, he remains universally respected by both students and peers alike. His trusted and considered council will be sorely missed by colleagues and students at Aston. On behalf of all who have known and worked with Bernard, we would like to wish him a very happy retirement.

Politics and Practice in Economic Geography edited by Adam Tickell, Eric Sheppard, Jamie Peck and Trevor Barnes

Politics and Practice in Economic Geography edited by Adam Tickell, Eric Sheppard, Jamie Peck and Trevor Barnes

Evaluating University Research Performance Using Metrics

Evaluations of research quality in universities are now widely used in the advanced economies. The UK's Research Assessment Exercise (RAE), which began in 1986, is the most highly developed of these research evaluations. Based on peer review and involving some sixty-nine panels evaluating the research work of more than 50,000 academic staff, the exercise is expensive and time consuming. In this article, we examine the possibility that a quantitative, metrics-based approach can provide a low-cost alternative to expensive, qualitative peer review. To do this, we build on our previous work on political science by extending a metrics-based model to chemistry, using the results of the 2001 RAE. Our results show that no single model will apply across science and non-science disciplines. Any metrics approach to performance evaluation has to use a discipline-specific suite of indicators.

The first Italian research assessment exercise: A bibliometric perspective

In December 2003, seventeen years after the first UK research assessment exercise, Italy started up its first-ever national research evaluation, with the aim to evaluate, using the peer review method, the excellence of the national research production. The evaluation involved 20 disciplinary areas, 102 research structures, 18,500 research products and 6661 peer reviewers (1465 from abroad); it had a direct cost of 3.55 millions Euros and a time length spanning over 18 months. The introduction of ratings based on ex post quality of output and not on ex ante respect for parameters and compliance is an important leap forward of the national research evaluation system toward meritocracy. From the bibliometric perspective, the national assessment offered the unprecedented opportunity to perform a large-scale comparison of peer review and bibliometric indicators for an important share of the Italian research production. The present investigation takes full advantage of this opportunity to test whether peer review judgements and (article and journal) bibliometric indicators are independent variables and, in the negative case, to measure the sign and strength of the association. Outcomes allow us to advocate the use of bibliometric evaluation, suitably integrated with expert review, for the forthcoming national assessment exercises, with the goal of shifting from the assessment of research excellence to the evaluation of average research performance without significant increase of expenses.

The RCCM 2009 Survey: Mapping Doctoral and Postdoctoral CAM Research in the United Kingdom

Complementary and Alternative Medicine (CAM) is widely available in the UK and used frequently by the public, but there is little high quality research to sustain its continued use and potential integration into the NHS. There is, therefore, a need to develop rigorous research in this area. One essential way forward is to train and develop more CAM researchers so that we can enhance academic capacity and provide the evidence upon which to base strategic healthcare decisions. This UK survey identified 80 research active postgraduates registered for MPhils/PhDs in 21 universities and were either current students or had completed their postgraduate degree during the recent UK Research Assessment Exercise (RAE) 2001–2008. The single largest postgraduate degree funder was the university where the students registered (26/80). Thirty-two projects involved randomized controlled trials and 33 used qualitative research methods. The UK RAE also indicates a significant growth of postdoctoral and tenured research activity over this period (in 2001 there were three full time equivalents; in 2008 there were 15.5) with a considerable improvement in research quality. This mapping exercise suggests that considerable effort is currently being invested in developing UK CAM research capacity and thus inform decision making in this area. However, in comparative international terms UK funding is very limited. As in the USA and Australia, a centralized and strategic approach by the National Institute of Health Research to this currently uncoordinated and underfunded activity may benefit CAM research in the UK.

The UK Research Assessment Exercise (RAE 2008) and retail research output

The 2008 Research Assessment Exercise in the United Kingdom provides a further opportunity to consider changing trends and patterns in retail research, following on a previous commentary in this journal. This comparison shows that pressures continue to mount and are impacting retail knowledge creation and dissemination practices, not least in terms of those engaged in retail research, the topics and approaches utilised, the publication tactics and strategies and thus the standing of retail research in the United Kingdom and its reputation internationally.

Can national research assessment exercises be used locally to inform research strategy development? The description of a methodological approach to the UK RAE 2008 results with a focus on one institution

National mechanisms for comparing the research profiles of higher education institutions (HEIs) have become increasingly common. Probably the best known of these is the Research Assessment Exercise (RAE) conducted in the United Kingdom, and used as the basis for the allocation of research funding. Such exercises are expensive. They would have additional value if the data could be used by HEIs to inform the development of their research strategies. In this paper we use publicly available RAE outcome data to demonstrate this potential. We contrast the two units’ research profiles with other units of assessment within the HEI, with other like-units nationally, and finally we examine the relative performance of all the HEI’s units of assessment against their national counter-parts. Finally we discuss the kinds of insights these data may offer in the development of research strategy at the level of the institution, and at the level of the School or Department.

Editorial: Rating research performance

Editorial: Rating research performance