How do organisations implement research impact assessment (RIA) principles and good practice? A narrative review and exploratory study of four international research funding and administrative organisations

Adam Kamenetzky,Saba Hinrichs-Krapels

doi:10.1186/s12961-019-0515-1

Abstract

BackgroundPublic research funding agencies and research organisations are increasingly accountable for the wider impacts of the research they support. While research impact assessment (RIA) frameworks and tools exist, little is known and shared of how these organisations implement RIA activities in practice.MethodsWe conducted a review of academic literature to search for research organisations’ published experiences of RIAs. We followed this with semi-structured interviews from a convenience sample (n = 7) of representatives of four research organisations deploying strategies to support and assess research impact.ResultsWe found only five studies reporting empirical evidence on how research organisations put RIA principles into practice. From our interviews, we observed a disconnect between published RIA frameworks and tools, and the realities of organisational practices, which tended not to be reported.We observed varying maturity and readiness with respect to organisations’ structural set ups for conducting RIAs, particularly relating to leadership, skills for evaluation and automating RIA data collection. Key processes for RIA included efforts to engage researcher communities to articulate and plan for impact, using a diversity of methods, frameworks and indicators, and supporting a learning approach. We observed outcomes of RIAs as having supported a dialogue to orient research to impact, underpinned shared learning from analyses of research, and provided evidence of the value of research in different domains and to different audiences.ConclusionsPutting RIA principles and frameworks into practice is still in early stages for research organisations. We recommend that organisations (1) get set up by considering upfront the resources, time and leadership required to embed impact strategies throughout the organisation and wider research ‘ecosystem’, and develop methodical approaches to assessing impact; (2) work together by engaging researcher communities and wider stakeholders as a core part of impact pathway planning and subsequent assessment; and (3) recognise the benefits that RIA can bring about as a means to improve mutual understanding of the research process between different actors with an interest in research.

Highlights

Public research funding agencies and research organisations are increasingly accountable for the wider impacts of the research they support
We recommend that organisations (1) get set up by considering upfront the resources, time and leadership required to embed impact strategies throughout the organisation and wider research ‘ecosystem’, and develop methodical approaches to assessing impact; (2) work together by engaging researcher communities and wider stakeholders as a core part of impact pathway planning and subsequent assessment; and (3) recognise the benefits that research impact assessment (RIA) can bring about as a means to improve mutual understanding of the research process between different actors with an interest in research
The emerging practice of research impact assessment (RIA) is an area where there have been a number of developments – be these analytical tools to help conceptualise impact, such as the Payback framework [2], the inclusion of impact as a criteria to determine the allocation of public funds to higher educational institutions or methods to determine the wider ‘spill-over’ effects from government and charitable investments in research, as a means to advocate for the value of the combined contribution of these sectors to national research and development efforts [3]

Summary

Introduction

Public research funding agencies and research organisations are increasingly accountable for the wider impacts of the research they support. There is an increasing drive for organisations that fund, support and/or administer research (hereafter referred to as ‘research organisations’) to be held accountable for various administrative and research governance functions and for the longer-term impacts of the research that their activities and funding support This is evidenced by the proliferation of approaches to assess research processes, policies and productivity [1]. Drawing on insights from over 400 scholars and practitioners attending five ISRIA schools – with stakeholders from research funding agencies, and the health sector as the most highly represented among these – ISRIA developed a series of best practice guidelines [5] These guidelines aim to distil down ‘what works’ in RIA, and how to situate such practices in a broader organisational context. The ISRIA guidelines encourage organisations to (1) analyse the research context, (2) reflect on the purpose of RIA, (3) identify stakeholders’ needs, (4) engage with the research community, (5) choose appropriate conceptual frameworks, (6) choose appropriate evaluation methods and data sources, (7) choose indicators and metrics responsibly, (8) consider ethics and conflicts of interest, (9) communicate results, and (10) share learning

Objectives

Methods

Results

Discussion

Conclusion