How do policy evaluators understand complexity?

Pete Barbrook-Johnson,Jeremy Phillipson,Sara Giorgi,Amy Proctor

doi:10.1177/1356389020930053

Pete Barbrook-Johnson, Jeremy Phillipson + Show 2 more

Open Access

https://doi.org/10.1177/1356389020930053

Copy DOI

Abstract

There is a well-documented interest in how insights from the study of complexity can be applied to policy evaluation. However, important questions remain as to how complexity is understood and used by policy evaluators. We present findings from semi-structured interviews with 30 UK policy evaluators working in food, energy, water and environment policy domains. We explore how they understand, use and approach complexity, and consider the implications for evaluation research and practice. Findings reveal understandings of complexity arising from contextual factors, scale-related issues and perceptions of unpredictability. The evidence indicates terminological and analogical use of complexity and its concepts by policy evaluators, but limited evidence of its literal use. Priorities for the future include framing complexity more pragmatically and as an opportunity not a cost. Communicating this up the policy hierarchy is the key to progressing complexity-appropriate evaluation – this can be enabled by strengthening links between policy evaluation and academic communities.

Highlights

Over the last 20 years, there has been a growing interest in applying insights from the study of complex adaptive systems to public policy settings (Anzola et al, 2017; Byrne and Callaghan, 2014; Byrne and Uprichard, 2012)
We present findings from qualitative research with UK policy evaluators, focusing mainly on those who are working within government who commission and conduct evaluations
We first consider the ways in which respondents understand complexity, and explore how evaluators are approaching evaluation in response, or not, to these understandings

Summary

Introduction

Over the last 20 years, there has been a growing interest in applying insights from the study of complex adaptive systems to public policy settings (Anzola et al, 2017; Byrne and Callaghan, 2014; Byrne and Uprichard, 2012). Key characteristics associated with complex systems include their adaptive and dynamic nature, feedback loops, multiple scales, thresholds for change, areas of high and low stability, and open or ill-defined boundaries that can span (socio-technical) domains or areas of expertise and responsibility. Such features result in systems characterised by tipping points, non-linearity, emergent new properties, and unpredictability (Centre for the Evaluation of Complexity Across the Nexus (CECAN), 2018). These properties of complex adaptive systems are often evident in the contexts, systems and behaviours that are the focus of a particular policy intervention or suite of interventions, and in the characteristics of the interventions themselves, and policy evaluation

Objectives

Methods

Discussion

Conclusion