Using surveys in realist evaluation

Abstract

Realist evaluation is a theory-based approach to evaluation which examines what works in what respects and to what extent, for whom, in what contexts and how. It focuses on identifying underlying mechanisms which cause outcomes and the contextual factors necessary for their operation. Despite realist evaluation being ‘agnostic’ about methods, qualitative approaches have been heavily favoured in practice. This paper argues that the use of surveys can considerably strengthen standard approaches to realist analysis while still being consistent with the philosophical underpinnings of realist evaluation. It provides examples drawn from various research and evaluation projects. The paper demonstrates that when program theory is well-developed, quantitative data collected through surveys can be used to examine all three elements of realist hypothesis testing: contexts, mechanisms and outcomes. The paper concludes with five principles for the design and analysis of survey data in realist evaluations.