Value and roles of qualitative methods in natural experimental evaluations

Abstract

Abstract Background For natural experimental evaluations that estimate effect sizes of population health interventions, qualitative evidence contributes multiple benefits. Incorporating qualitative methods strengthens natural experimental evaluations, particularly when studying interventions in complex systems, and contributes to minimising evaluation design weaknesses. Methods Guidance was developed from three international workshops and an online consultation with researchers, journal editors and end users of natural experimental evaluations. The project team comprised researchers with expertise in natural experimental evaluations. The project included an oversight group and an advisory group of independent experts, including researchers, a journal editor, and scientific and public health advisors on central and local government. Results The guidance recommends that most evaluations of natural experiments will require the inclusion of qualitative evidence. Qualitative methods are valuable when describing the natural experiment intervention, the system and context it exists within, and theories of change. Qualitative work can help appropriately define exposed and unexposed groups, investigate causal mechanisms, facilitate meaningful selection of outcomes, and understand stakeholder perspectives on the intervention and its effects. Conclusions Gaining an understanding of the value and application of qualitative evidence will help producers and users of natural experimental evaluations conduct and identify robust evaluations. To fully benefit the evaluation, qualitative methods should be integrated throughout.