A common standard for the evaluation of public engagement with research

Mark S Reed,Lucy Armitage,Sophie Duncan,Bryony Frost,Charlotte Thorley,Lynn Frewer,Diana Pound,Paul Manners

doi:10.18546/rfa.02.1.13

Abstract

Despite growing interest in public engagement with research, there are many challenges to evaluating engagement. Evaluation findings are rarely shared or lead to demonstrable improvements in engagement practice. This has led to calls for a common 'evaluation standard' to provide tools and guidance for evaluating public engagement and driving good practice. This paper proposes just such a standard. A conceptual framework summarizes the three main ways in which evaluation can provide judgements about, and enhance the effectiveness of, public engagement with research. A methodological framework is then proposed to operationalize the conceptual framework. The standard is developed via a literature review, semi-structured interviews at Queen Mary University of London and an online survey. It is tested and refined in situ in a large public engagement event and applied post hoc to a range of public engagement impact case studies from the Research Excellence Framework. The goal is to standardize good practice in the evaluation of public engagement, rather than to use standard evaluation methods and indicators, given concerns from interviewees and the literature about the validity of using standard methods or indicators to cover such a wide range of engagement methods, designs, purposes and contexts. Adoption of the proposed standard by funders of public engagement activities could promote more widespread, high-quality evaluation, and facilitate longitudinal studies to draw our lessons for the funding and practice of public engagement across the higher education sector.

Highlights

Interest in public engagement with research has never been higher
A new typology of engagement published by Reed et al (2017), argues that types of engagement can in theory be distinguished by their mode and agency, leading to combinations of top-down and bottom-up approaches for informing, consulting and collaborating with publics in more or less co-productive ways:
They were asked a range of questions under each of the topics used in the semi-structured interviews, including: the most valuable resources they drew upon to inform their evaluation of public engagement, methods and approaches for evaluating public engagement, key challenges for evaluating public engagement; they were asked to identify indicators of successful public engagement

Summary

Introduction

Interest in public engagement with research has never been higher. The Higher Education Funding Council for England (HEFCE) defined public engagement with research as: Specialists in higher education listening to, developing their understanding of, and interacting with non-specialists. The European Commission (2015) identifies public engagement as one of the six ‘keys’ for responsible research and innovation, and is considering ways of better evaluating the impact of its research in the successor to Horizon 2020. The Department for Business, Innovation and Skills (BIS, 2010) reframed these in their ‘public engagement triangle’ as transmitting, receiving and collaborating. Widely used, these typologies of engagement have limited theoretical basis. A new typology of engagement published by Reed et al (2017), argues that types of engagement can in theory be distinguished by their mode and agency, leading to combinations of top-down and bottom-up approaches for informing, consulting and collaborating with publics in more or less co-productive ways (in line with NCCPE, 2017b):

Objectives

Methods

Results

Discussion

Conclusion