UK Research Assessment Exercise Research Articles

Research quality assessment in education: Impossible science, possible art?

For better or for worse, the assessment of research quality is one of the primary drivers of the behaviour of the academic community with all sorts of potential for distorting that behaviour. So, if you are going to assess research quality, how do you do it? This article explores some of the problems and possibilities, with particular reference to the UK Research Assessment Exercise and discussion around the proposed new Research Excellence Framework and the ongoing work of the Framework 7 European Education Research Quality Indicators project (EERQI). It begins by asking whether there are any meaningful generic criteria of quality which can be applied to research, and tension between such criteria and the diverse and sometimes contradictory requirements of educational research. It then looks at attempts to identify measurable indicators of quality, including consideration of the location of the publication, citation and download counts, and approaches based semantic analysis of machine readable text—but finds all these quasi ‘scientific’ attempts at quality assessment wanting (hence the ‘impossible science’). This is all the more the case because of their attachment to extrinsic correlates of quality rather than intrinsic characteristics of quality and hence the probability that the measures will induce behaviours not conducive to quality enhancement. Instead the article turns to a different approach. This is better expressed perhaps as quality ‘appreciation’, ‘discernment’ or even ‘connoisseurship’ and is rooted in the arts and humanities rather than in (quasi) science. It considers whether this might offer a better approximation to the kind of judgement involved in quality assessment of a piece of research writing than the sort of metrics approaches favoured in current discussion.

Comment: Citation Statistics

We welcome this critique of simplistic one-dimen sional measures of academic performance, in particular the naive use of impact factors and the h-index, and we can only extend sympathy to colleagues who are be ing judged using some of the techniques described in the paper. In particular we welcome the report's em phasis on the need for careful modeling of citation data rather than relying on simple summary statistics. Our own work on league tables adopts a modeling ap proach that seeks to understand the factors associated with institutional performance and at the same time to quantify the statistical uncertainty that surrounds insti tutional rankings or future predictions of performance. In the present commentary we extend this approach to an analysis of the 2008 UK Research Assessment Ex ercise (RAE) for Universities. Before we describe our analysis it is important to comment on an important modeling problem that arises in the analysis of citation data, alluded to but not dis cussed in detail in the report, nor, as far as we know, elsewhere. A principal difficulty with indices such as the h-index or simple citation counts is that there are inevitable dependencies between individual scientists' values. This is because a citation is to a paper with, in general, several authors, rather than to each specific au thor. Thus, for example, if two authors nearly always write all their papers together, they will tend to have very similar values. If they belong to the same uni versity department then their scores do not supply in dependent bits of information in compiling an overall score or rank for that department. Currently this issue is in the RAE, albeit imperfectly, by the re quirement that the same paper cannot be entered more than once by different authors for a given university department. In a citation based system this would also need to be recognized. In addition, if our two authors were in different, competing departments, we would also need to recog nize this since the dependency would affect the accu racy of any comparisons we make. We also note that this will, to some extent, affect our own analyses that we present below, and it will be expected to overesti mate the accuracy of our rankings. Unfortunately we have no data that would allow us to estimate, even ap proximately, how important this is. To deal with this problem satisfactorily would involve a model that in corporated effects for each author and the detailed information about the authorship of each paper that was cited. Goldstein (2003, Chapter 12.5) describes a multilevel multiple membership model that can be used for this purpose, where individual authors become level 2 and papers are level 1 units. The UK Research Assessment Exercise was pub lished on 18th December 2008, covering the years 2001-2008. 52,409 staff from 159 institutions were grouped into 67 units of assessment (UOA): up to 4 publications for each individual were considered as well as other activities and markers of esteem. Pan els drawn from around 1000 peer reviewers then pro duced a profile for each group, summariz ing in blocks of 5% the proportion of each submission judged by the panels to have met each of the follow ing quality levels: world-leading (4*), internation ally excellent (3*), internationally recognized (2*), nationally recognized (1*), and unclassified. This procedure is notable in terms of its use of peer judg ment rather than simple metrics, and allowing a dis tribution of performance rather than a single measure. All the data is available for downloading (Research As sessment Exercise, 2008). Figure 1 shows the results relevant for most statisti cians: the 30 groups entered under UOA22: Statistics and Operational Research. These have been ordered into a league table using the average number of stars which we shall term the mean score, which is the procedure adopted by the media. Also reported is the number of full-time equivalent staff in the submission. Controversy surrounds this number as it is unknown how selective institutions were in submitting staff? David Spiegelhalter is Winton Professor of Public Understanding of Risk, Statistical Laboratory, Centre for Mathematical Sciences, Wilberforce Road, Cambridge CB3 OWB, UK. Harvey Goldstein is Professor of Social Statistics, University of Bristol, 35 Berkeley Square, Bristol BS8 l JA, UK.

Comment: Bibliometrics in the Context of the UK Research Assessment Exercise

Research funding and reputation in the UK have, for over two decades, been increasingly dependent on a regular peer-review of all UK departments. This is to move to a system more based on bibliometrics. Assessment exercises of this kind influence the behavior of institutions, departments and individuals, and therefore bibliometrics will have effects beyond simple measurement.

Tourism Geographies and the South African National Research Foundation's Researcher Rating System: International Connections and Local Disjunctures

The function, form and outcomes of research assessment exercises have received considerable attention in the international journal press. However, as is generally the case with ‘international’ human geographical discourse, the foci of these debates are heavily tainted with the concerns of Anglo-American academics, with the UK's much-discussed Research Assessment Exercise (RAE), in particular, being under near-perpetual scrutiny. The idea of assessing research output is, however, not unique to the institutions of higher learning in the north. Systems and institutions of research assessment are also present in countries of the south. As an example, this paper outlines a rating scheme that has been functioning in South Africa for more than two decades. It is the contention of this investigation that this rating scheme has a range of consequences echoing ‘the international experience’. Perhaps most disturbing is the potential crushing of a post-colonial ‘South/South African’ tourism geography intellectual voice.

Metrics or Peer Review? Evaluating the 2001 UK Research Assessment Exercise in Political Science

Evaluations of research quality in universities are now widely used in the advanced economies. The UK's Research Assessment Exercise (RAE) is the most highly developed of these research evaluations. This article uses the results from the 2001 RAE in political science to assess the utility of citations as a measure of outcome, relative to other possible indicators. The data come from the 4,400 submissions to the RAE political science panel. The 28,128 citations analysed relate not only to journal articles, but to all submitted publications – including authored and edited books and book chapters. The results show that citations are the most important predictor of the RAE outcome, followed by whether or not a department had a representative on the RAE panel. The results highlight the need to develop robust quantitative indicators to evaluate research quality which would obviate the need for a peer evaluation based on a large committee. Bibliometrics should form the main component of such a portfolio of quantitative indicators.

Open access scientometrics and the UK Research Assessment Exercise

Scientometric predictors of research performance need to be validated by showing that they have a high correlation with the external criterion they are trying to predict. The UK Research Assessment Exercise (RAE) — together with the growing movement toward making the full-texts of research articles freely available on the web — offer a unique opportunity to test and validate a wealth of old and new scientometric predictors, through multiple regression analysis: Publications, journal impact factors, citations, co-citations, citation chronometrics (age, growth, latency to peak, decay rate), hub/authority scores, h-index, prior funding, student counts, co-authorship scores, endogamy/exogamy, textual proximity, download/co-downloads and their chronometrics, etc. can all be tested and validated jointly, discipline by discipline, against their RAE panel rankings in the forthcoming parallel panel-based and metric RAE in 2008. The weights of each predictor can be calibrated to maximize the joint correlation with the rankings. Open Access Scientometrics will provide powerful new means of navigating, evaluating, predicting and analyzing the growing Open Access database, as well as powerful incentives for making it grow faster.

ASSESSING THE ASSESSMENT OR, THE RAE AND THE OPTIMAL ORGANIZATION OF UNIVERSITY RESEARCH

ABSTRACTThe UK Research Assessment Exercise (RAE) is assessed as an incentive scheme affecting the allocation of research talent of varying ‘quality’ across departments. The ‘centres of excellence’ policy implicitly pursued through the RAE is an optimal allocation strategy only if all departments in all disciplines are of the generalist variety, i.e. each pursues a research path through all its stages. Conversely, the RAE‐induced research allocation minimizes efficiency if applied to specialist departments, when resources are concentrated on one specific research obstacle. It is argued that the RAE should not take the organization of University research as exogenous, but rather should encourage specialization. All results are obtained by applying to University research concepts and solutions borrowed from the mathematical theory of systems reliability.

Validating research performance metrics against peer rankings

A rich and diverse set of potential bibliometric and scientometric predictors of research performance quality and importance are emerging today, from the classic metrics (publication counts, journal impact factors and individual article/author citation counts) to promising new online metrics such as download counts, hub/authority scores and growth/decay chronometrics. In and of themselves, however, metrics are circular: They need to be jointly tested and validated against what it is that they purport to measure and predict, with each metric weighted according to its contribution to their joint predictive power. The natural criterion against which to validate metrics is expert evaluation by peers, and a unique opportunity to do this is offered by the 2008 UK Research Assessment Exercise, in which a full spectrum of metrics can be jointly tested, field by field, against peer rankings.

Research Quality Assessment and the Metrication of the Social Sciences

The British system of quality assessment of research in universities, known as the Research Assessment Exercise (RAE), has recently been the subject of major public policy review and debate. The system of research quality or performance assessment has been running for over twenty years, although many of its facets have changed as has the increasingly marketised political economy. Nevertheless, the UK RAE has been the prototype for the growth and development of such systems internationally, although how different countries have conceived of such forms of review has varied greatly. The question of the relationship between research quality in higher education and the public funding of research lies at the heart of what has become a contentious and acrimonious debate in the UK. While these issues can be seen as fundamentally about social and economic matters, in fact the social sciences as an organised group of subjects or interests have not played a key role in the public arena. This article outlines the contours of the recent debates in the UK, by comparison and contrast with the ways in which such systems of performance and quality assessment have been debated inter alia in Australia, New Zealand, France and the Netherlands. In essence, the issues have centred upon questions of measurement of performance known as metrication, and bibliometrics versus social judgments about research quality.

The future of research evaluation rests with an intelligent combination of advanced metrics and transparent peer review

The paper discusses the strengths and limitations of ‘metrics’ and peer review in large-scale evaluations of scholarly research performance. A real challenge is to combine the two methodologies in such a way that the strength of the first compensates for the limitations of the second, and vice versa. It underlines the need to systematically take into account the unintended effects of the use of metrics. It proposes a set of general criteria for the proper use of bibliometric indicators within peer-review processes, and applies these to a particular case: the UK Research Assessment Exercise (RAE). Copyright , Beech Tree Publishing.

Benchmarking research development in nursing: Curran's competitive advantage as a framework for excellence

This paper offers a developmental benchmark against which to assess nursing research achievements and excellence. Internationally, the academy of nursing, like other disciplines, has able people fully capable of delivering high-quality research. This raises the question of why, during systematic, peer-review research assessment exercises nursing appears at the bottom of the academic units assessed both in the UK RAE (Research Assessment Exercise, 1992, 1996, 2001) and the New Zealand PBRF (Performance Based Research Fund, 2003). To examine this issue, after tracing the background of the international drive for research excellence and the UK's Research Assessment Exercise (RAE), Curran's analysis of competitive advantage in higher education is introduced as a means of analysing nursing's entry into higher education and the RAE. Together with data comparing the development of nursing with other disciplines, analysis suggests that UK nursing's performance in RAEs was as a new, hard-working, emerging, applied discipline which demonstrated a predictable research trajectory with strong trends of growth in quality and quantity. Recognising that research assessment is both a politically driven, socially constructed competition for scarce resources, and an analysis of the discipline, the conclusion argues that excellence and integrity should form the foundation of a disciplinary identity which incorporates the diversity of the subject. It also notes that without secure research income streams the growth in critical mass of the `knowledge community' and nursing's capacity to contribute to health improvement will remain vulnerable despite strong positive trends.

The UK Research Assessment Exercise: the evolution of a national research evaluation system

The Research Assessment Exercise (RAE) represents one of the most institutionalised forms of research evaluation in the OECD economies. It has become a primary means of concentrating resources for research in a relatively small number of universities. Its main purpose is to inform funding decisions, and the indirect effects come from the public signalling of quality. The RAE seems to have reinforced the traditional ‘high science’ ideals of British universities, encouraging greater coordination of research around traditional disciplinary concerns and inhibiting applied research. Debates among universities and policy-makers have led to a reshaping of the exercise. Its evolution into a highly elaborate procedure combined with its failure as a strategic policy tool to steer university research towards socio-economic impacts have led to its abandonment in its present form. Copyright , Beech Tree Publishing.

The Infrastructure of Educational Research in Scotland

This article offers a descriptive and analytical account of the current state of educational research in Scotland, viewed against the background of constitutional change within the United Kingdom and debates about the nature, function and quality of research activity and output. It starts by considering how to undertake the task in hand and argues that it is necessary to draw on a combination of institutional, developmental and interactive approaches. Thereafter the relatively disappointing Scottish results in the 2001 UK Research Assessment Exercise are reported to set the scene for subsequent efforts to improve research capacity. The role of the Scottish Executive in determining research priorities, commissioning projects and using the findings of research in helping to inform policy is outlined, taking account of important structural changes following devolution. A key development has been the Applied Educational Research Scheme (AERS), a collaborative university-led initiative, funded jointly by the Scottish Executive and the Scottish Funding Council (previously the Scottish Higher Education Funding Council). This consists both of substantive research networks and a capacity-building programme of research training for beginning and developing researchers. Established research centres, with different histories and emphases, are described in order to raise issues about the balance between applied research, linked to policy priorities, and fundamental research which seeks to develop new theories and models. The role of the Scottish Educational Research Association, which has links with the British and European Educational Research Associations, is considered. Various attempts by different bodies to promote teacher engagement in research are also described. In the final section an attempt is made to assess what progress has been made. It is concluded that while the overall climate for the conduct of educational research has improved, providing real opportunities to demonstrate the value of research, there remain a number of challenging problems. These include: issues of research quality; inadequate opportunity to undertake fundamental (as distinct from applied) research; tensions between the demands of teacher education programmes and universities' research aspirations; the risk of ‘collaboration’ becoming ‘consensus’; and the absence of a developed international perspective.

ASSESSING THE QUALITY OF RESEARCH: A CHALLENGE FOR NURSING

In research terms nursing has come a long way in a relatively short period of time. The focus within the discipline has been to encourage the expansion of research as a project-based activity, with insufficient emphasis placed on programmatic work or rigorous research training. Internationally, interest in the concept of quality and relevance of research has been increasing, especially in countries where universities are mainly state funded. In many western countries state-funded services have come under close scrutiny. It is perhaps easier to evaluate the funds spent by universities on education, as the output is graduates who populate the labour market. However, it is not as straightforward to locate the benefits of the tax dollar spent on research. In a context where demand for funds for research outstrips supply, it is reasonable to be selective in where and how the research dollar is spent. One way of approaching selectivity is to concentrate on recognising and rewarding good quality research and distributing funds accordingly. Since 1986 in the UK there has been a national Research Assessment Exercise (RAE). The principal drivers for national research assessment are to improve the quality of research, to ensure value for money, to encourage collaboration to avoid duplication and to build on excellence. The assessment is based on peer review by discipline-specific panels of experts. In 2003 New Zealand introduced the Performance-based Review Fund (PBRF) and the Australian government is in the process of introducing the Research Quality Framework (RQF). Both the PBRF and RQF are based predominantly on the UK RAE framework. There are differences between the three approaches. In the RAE and RQF the unit of assessment is a subject area; in the PBRF it is the individual. The RQF and RAE use a similar quality scale for output, with the highest grade indicating world leading (then internationally excellent, international standard, original and sound, falls below standard), whereas the PBRF compresses the top three grades into one grade ‘A’. In addition, the RQF has an impact scale, so research which is judged at least to be methodologically rigorous and of high originality will also be rated for impact on a scale depicting social, economic, cultural environmental or other benefits regionally, nationally or internationally. Discussions with nurses indicate a positive response to the notion of an impact scale. Caution will need to be exercised with respect to the differentiation between research (the purpose of the RQF) and development work, which is a distinctive feature of Australian nursing research. The RAE is notoriously unpopular amongst academics. RAE dissenters do not subscribe to the notion of an elite and argue that the creation of centres of excellence militates against the development of the discipline more broadly. The RAE is also criticised for relegating teaching to second place and wasting resources. In nursing some benefits have arisen out of the RAE. In 1992 only 21 schools of nursing completed the assessment; by 2001 the number had risen to 43 indicating an increase in research activity. In terms of overall performance, nursing as a discipline was ranked bottom with only four schools receiving a score of 5 (top score being 5 star). The outcome of the RAE was beneficial because additional resources were invested in the mid-range schools to enhance research capacity; we hope to see the benefits in terms of improved scores in the 2008 RAE. Similarly, in New Zealand additional funds have been made available to develop the research potential of nursing. Embedded within the process of a research selectivity exercise is a judgement about the quality of a single output, usually a publication in a journal. Journal impact factors and citations are widely used as proxies for excellence in science, but because much nursing research is published in journals without impact factors, in the RAE and PBRF papers have been read and panel consensus judgement made on quality. The judgement process is peer based and has the usual imperfections of peer review, but for nursing there is no real alternative. What is clear is that schools that do well in research have a certain critical mass and understand the concept of excellence. It is important for the future of nursing in the university sector that we raise our gaze and aspire towards excellence. The nursing profession needs to publicly endorse quality research and change the focus from individual research projects towards a more programmatic approach. The challenge for the future is to identify research leaders and to harness their energies to build sustainable programmes of research that meet the standards of international peer review, and impact on policy and practice.

Editorial: What makes a JCN paper international?

Editorial: What makes a <i>JCN</i> paper international?

Planning Educational Research and the UK Research Assessment Exercise

The contemporary relationship between research and teaching in higher education is a complex and contested one. In the UK context, for example, academics and university managers are currently confr...

IT IS ABOUT PEOPLE

Griffith Edwards’ essay (Edwards 2006) on the story of the Journal so far gives us a glimpse into the reasons for its rise from obscurity to one of the top specialist journals in the field. One of those reasons is not stated in the piece, but it is surely apparent to anyone to who has taken an interest in the journal over the past 26 years—it is the vision of the man himself. I will not dwell on this, but I raise it because I believe it is important to recognize that any enterprise, whether it be a journal or a government, is dependent ultimately on the vision and enterprise of key individuals and the hard work, intelligence and commitment of many more. Bad systems can be made to work by competent, conscientious and good people and good systems can be made to fail by incompetent, careless or ill-motivated people. We live in a time when it seems that this rather obvious point is being overlooked and when there seems to be an unmerited faith in the ability of ever more cumbersome and intrusive procedures to make up for human failings. I have little doubt that in good time there will be a backlash against this and perhaps we will move on to a more enlightened era, where the emphasis will be on trying to make our human capital as good as it can be. In the meantime, we must do the best we can in a world where red tape is seen as the panacea for human failings. Why do I make such a big issue of this in a commentary about journal publishing? It is because I think it is relevant to the future of high quality specialist journals such as Addiction. On the face of it, things could not be rosier. The Journal receives ever more high-quality submissions, and sales and readership figures are better than one might have dared to hope for in years gone by. But there are pressures in the system. The increasing number of submissions means that there is pressure on the precious resource of people: people to contribute to the editing and refereeing process, people to take the vision forward. At the same time, many countries such as the United Kingdom are engaging in ‘research assessment exercises’ which, alas, devalue journal editing and refereeing as an activity in favour of getting a small number of papers in non-specialist journals that have a high impact factor. In the case of the UK's Research Assessment Exercise (RAE), being editor of a journal can at most count 5% towards the ‘score’ one obtains, whereas publishing papers in non-specialist journals counts 75%. So why do people like myself and Addiction's dedicated team of regional, deputy and assistant editors give up so much time to the journal? Why do we sacrifice RAE ‘brownie points’ in service of the Journal? Not for the money, because none of us is paid personally for the work. Certainly not to make friends, because turning down a valued colleague's papers is no way to cement a friendship. We do it because we care about the field—I personally care about the legacy handed to me by Griffith Edwards and his forebears. We have been handed the torch and we do not want to drop it or let it go out. We are fallible and we make mistakes and we try to develop systems that minimize these. As Griffith Edwards’ history of the journal shows, making a successful journal is all about people—people who see themselves as part of an invisible college working with a shared vision. As long as we have people who care enough about making a contribution through service with a journal, we will find a way to make whatever publishing model we end up with work.

Editorial: UK research assessment exercise

Editorial: UK research assessment exercise

The UK research assessment exercise

This article examines and critiques the UK research assessment exercise (RAE) and its implications for research and writing in HRD. It describes the process and identifies weaknesses in some of the concepts and related criteria applied in assessing and grading research outputs and in particular the notions of ‘international’ and ‘world class’ as standards against which to judge those outputs. The article argues that the process has detrimental effects for emerging subjects such as HRD, for those engaged in research in such subjects and for journals such as HRDI which are relatively new to business and management. It concludes that UK academics working in the field of HRD are mistaken if they allow the RAE to drive their research and choice of publications and outlets, and that it is important to support the journals which are playing a critical role in establishing the subject.

The Evolution of the UK’s Research Assessment Exercise: Publications, Performance and Perceptions

This article examines how the UK Research Assessment Exercise (RAE), along with the development of performance indicators, fits into the production of research within a changing policy context. The historical evolution of the RAE from 1985–2001 is considered, before looking specifically at how research output (in the form of journal publications) was, and is, assessed. Changes in process and output measurements necessitated a move from quantitative to quality evaluation, which affected both academic publishing and the role of journals within the RAE. The issue of quality – in terms of perceptions and performance – as applied to both is explored.