Future Of Research Assessment Research Articles

The future of societal impact assessment using peer review: pre-evaluation training, consensus building and inter-reviewer reliability

There are strong political reasons underpinning the desire to achieve a high level of inter-reviewer reliability (IRR) within peer review panels. Achieving a high level of IRR is synonymous with an efficient review system, and the wider perception of a fair evaluation process. Therefore, there is an arguable role for a more structured approach to the peer review process during a time when evaluators are effectively novices in practice with the criterion, such as with societal impact. This article explores the consequences of a structured peer review process that aimed to increase inter-reviewer reliability within panels charged with assessing societal impact. Using a series of interviews from evaluators from the UK’s Research Excellence Framework conducted before (pre-evaluation) and then again after the completion of the process (post-evaluation), it explores evaluators’ perceptions about how one tool of a structured evaluation process, pre-evaluation training, influenced their approaches to achieving a consensus within the peer review panel. Building on lessons learnt from studies on achieving inter-reviewer reliability and from consensus building with peer review groups, this article debates the benefits of structured peer review processes in cases when the evaluators are unsure of the criterion (as was the case with the Impact criterion), and therefore the risks of a low IRR are increased. In particular, this article explores how individual approaches to assessing Impact were normalized during group deliberation around Impact and how these relate to evaluators’ perceptions of the advice given during the pre-evaluation training. This article is published as part of a collection on the future of research assessment.

Evaluating policy-relevant research: lessons from a series of theory-based outcomes assessments

The increasing external demand from research funders and research managers to assess, evaluate and demonstrate the quality and the effectiveness of research is well known. Less discussed, but equally important, is the evolving interest and use of research evaluation to support learning and adaptive management within research programmes. This is especially true in a research-for-development context where research competes with other worthy alternatives for overseas development assistance funding and where highly complex social, economic and ecological environments add to evaluation challenges. Researchers and research managers need to know whether and how their interventions are working to be able to adapt and improve their programmes as well as to be able to satisfy their funders. This paper presents a theory-based research evaluation approach that was developed and tested on four policy-relevant research activities: a long-term forest management research programme in the Congo Basin; a large research programme on forests and climate change; a multi-country research project on sustainable wetlands management, and; a research project of the furniture value chain in one district in Indonesia. The first used Contribution Analysis and the others used purpose-built outcome evaluation approaches that combined concepts and methods from several approaches. Each research evaluation began with documentation of a theory of change (ToC) that identified key actors, processes and results. Data collected through document reviews, key informant interviews and focus group discussions were analysed to test the ToCs against evidence of outcomes in the form of discourse, policy formulation and practice change. The approach proved valuable as a learning tool for researchers and research managers and it has facilitated communication with funders about actual and reasonable research contributions to change. Evaluations that employed a participatory approach with project scientists and partners noticeably supported team learning about past work and about possible adaptations for the future. In all four cases, the retrospective ToC development proved challenging and resulted in overly-simplistic ToCs. Further work is needed to draw on social scientific theories of knowledge translation and policy processes to develop and further test more sophisticated theories of change. This theory-based approach to research evaluation provides a valuable means of assessing research effectiveness (summative value) and supports learning and adaptation (formative value) at the project or programme scale. The approach is well suited to the research-for-development projects represented by the case studies, but it should be applicable to any research that aspires to have a societal impact. This article is published as part of a collection on the future of research assessment.

The future of research assessment in the humanities: bottom-up assessment procedures

Research assessment in the social sciences and humanities (SSH) is delicate. Assessment procedures meet strong criticisms from SSH scholars and bibliometric research shows that the methods that are usually applied are ill-adapted to SSH research. While until recently research on assessment in the SSH disciplines focused on the deficiencies of the current assessment methods, we present some European initiatives that take a bottom-up approach. They focus on research practices in SSH and reflect on how to assess SSH research with its own approaches instead of applying and adjusting the methods developed for and in the natural and life sciences. This is an important development because we can learn from previous evaluation exercises that whenever scholars felt that assessment procedures were imposed in a top-down manner without proper adjustments to SSH research, it resulted in boycotts or resistance. Applying adequate evaluation methods not only helps foster a better valorization of SSH research within the research community, among policymakers and colleagues from the natural sciences, but it will also help society to better understand SSH’s contributions to solving major societal challenges. Therefore, taking the time to encourage bottom-up evaluation initiatives should result in being able to better confront the main challenges facing modern society. This article is published as part of a collection on the future of research assessment.

The impact agenda and the search for a good life

Plato and Aristotle are plumbed for insights concerning the underlying assumptions of the impact agenda, and more generally for how philosophy can offer greater assistance on policy questions. Aristotle reveals the limitations of ‘impact’ as a way of framing discussions about the relevance of academic knowledge. Plato offers more general counsel on the challenges facing philosophers who seek to be relevant to the concerns of policymakers. Together, their work suggests that policy-sensitive philosophers can help decision makers be more self-conscious about the assumptions underlying their work. This article is published as part of a collection on the future of research assessment.

\u201cExcellence R Us\u201d: university research and the fetishisation of excellence

The rhetoric of “excellence” is pervasive across the academy. It is used to refer to research outputs as well as researchers, theory and education, individuals and organizations, from art history to zoology. But does “excellence” actually mean anything? Does this pervasive narrative of “excellence” do any good? Drawing on a range of sources we interrogate “excellence” as a concept and find that it has no intrinsic meaning in academia. Rather it functions as a linguistic interchange mechanism. To investigate whether this linguistic function is useful we examine how the rhetoric of excellence combines with narratives of scarcity and competition to show that the hyper-competition that arises from the performance of “excellence” is completely at odds with the qualities of good research. We trace the roots of issues in reproducibility, fraud, and homophily to this rhetoric. But we also show that this rhetoric is an internal, and not primarily an external, imposition. We conclude by proposing an alternative rhetoric based on soundness and capacity-building. In the final analysis, it turns out that that “excellence” is not excellent. Used in its current unqualified form it is a pernicious and dangerous rhetoric that undermines the very foundations of good research and scholarship. This article is published as part of a collection on the future of research assessment.

Exploring the effectiveness, efficiency and equity (3e’s) of research and research impact assessment

AbstractThe allocation of research funding can benefit greatly from robust analysis of what has worked in research. In turn, these analyses can help advocacy initiatives and demonstrate accountability to taxpayers and donors. Capturing and mapping data on the inputs, processes, outputs, outcome and impact of research is crucial for these analyses. In this article we argue that the research community as a whole—including funders, researchers and administrators—is potentially in a position where it can assess or evaluate research not just according to academic outputs (production of knowledge), but also its outcomes and/or impact (effects on society). Using an exploratory framework that assesses effectiveness, efficiency and equity (3e’s) of research and research assessment both in terms of academic outputs and non-academic impact, we also argue that most assessments are primarily examining the effectiveness of research, as tools are not yet available to systematically assess research for its efficiency and equity. This article is published as part of a special issue on the future of research assessment.

The changing role of metrics in research institute evaluations undertaken by the Chinese Academy of Sciences (CAS)

AbstractAs an efficient S&T management tool, the Chinese Academy of Sciences (CAS) institute evaluations have experienced four major systems in which quantitative indicators have played a great role. To better understand the roles of quantitative indicators in CAS institute evaluations and institute development, this article aims to discuss the history and development of the CAS institute evaluation systems. It is concluded that institute evaluation has played a significant role not only in promoting scientific progress but also in guiding the direction and focus of the institutes’ development, thereby aligning it with the CAS’s strategy at different phases. Our intention is that this article offers useful insights for a wide scientific community into CAS’ attitudes on the application of metrics in research institute evaluations. This article is published as part of a collection on the future of research assessment.

Assessing (for) impact: future assessment of the societal impact of research

AbstractNational research assessments play a role in providing accountability—to funders, government and civil society—for the activities of largely autonomous research systems. Increasingly, an element of such assessments is the assessment of societal impact. In this article, I review the development of impact assessment, with a focus on the UK’s Research Excellence Framework, and consider implications and challenges for the future. Notwithstanding these challenges, I further argue that the assessment of societal impact needs to become a more central aspect of research evaluation. This article is published as part of a collection on the future of research assessment.

Altmetrics: diversifying the understanding of influential scholarship

AbstractThe increase in the availability of data about how research is discussed, used, rated, recommend, saved and read online has allowed researchers to reconsider the mechanisms by which scholarship is evaluated. It is now possible to better track the influence of research beyond academia, though the measures by which we can do so are not yet mature enough to stand on their own. In this article, we examine a new class of data (commonly called “altmetrics”) and describe its benefits, limitations and recommendations for its use and interpretation in the context of research assessment. This article is published as part of a collection on the future of research assessment.

Citation Counts: Are They Good Predictors of Rae Scores? A Bibliometric Analysis of RAE 2001

The question of how best to assess research performance is clearly of great concern. In December 2007, HEFCE launched a national consultation on the future of Research Assessment, proposing that a Research Excellence Framework replaces the current Research Assessment Exercise. Fundamentally the Research Excellence Framework involves a shift to metrics. Views on the effectiveness of metrics for assessing research are mixed, so in this report we seek to explore empirically the question of whether metrics based on citation counts are strongly correlated with peer review assessments of research quality. We use data from RAE 2001, covering all departments and all universities in the UK. At a more disaggregated level the data used in this study includes: [i] the individual submissions made to RAE 2001 - a database of 203,743 research output records, one for each output submitted in RAE 2001; [ii] the citation counts for each of the submitted items when they are journal articles (141,789 of the original 203,743 items). We have interrogated the ISA Web of Science and produced citation counts for every article included in RAE 2001 where data is available. This gives us citation counts for 112,201 publications (55.1% of the original 203,743 submitted items); [iii] the actual RAE 2001 scores produced and published by HEFCE. We contrast these scores - which are based on peer review - with those that would have been produced had bibliometrics, based on citation counts, been used in RAE 2001. The findings of this analysis are: [i] citation counts are a reasonable proxy for peer assessment in some subjects, such as Biological Sciences, Clinical Sciences, Chemistry and Psychology; [ii] however, citation counts are a weak proxy for a large number of disciplines, including fields within Biomedical related subjects and Engineering-related subjects. This is despite the fact that these subjects have good coverage in the Web of Science, in terms of the outputs submitted to RAE 2001; [iii] as others have reported, the Web of Science offers only partial coverage in some subjects, and hence the use of bibliometrics becomes increasingly less valuable as we move from Biomedical, Physical Sciences and Engineering, to Socials Sciences, Literature and Arts and Humanities.

The Research Assessment Exercise has lost support: Supporting Excellence in the Science Base

An online membership consultation that took place over the summer on the future of research assessment and funding found that 82% of respondents consider that the credibility of the Research Assessment Exercise (RAE) has been seriously undermined by the inability of Higher Education Funding Councils to reward appropriately improved performance following RAE 2001. Only 21% of respondents consider that the RAE is the fairest and most rigorous system available for assessing research quality.