Evidence synthesis in international development: a critique of systematic reviews and a pragmatist alternative

Abstract

Systematic reviews are an instrument of Evidence-Based Policy designed to produce comprehensive, unbiased, transparent and clear assessments of interventions’ effectiveness. From their origins in medical fields, systematic reviews have recently been promoted as offering important advances in a range of applied social science fields, including international development. Drawing on a case study of a systematic review of the effectiveness of community mobilisation as an intervention to tackle HIV/AIDS, this article problematises the use of systematic reviews to summarise complex and context-specific bodies of evidence. Social development interventions, such as ‘community mobilisation’ often take different forms in different interventions; are made successful by their situation in particular contexts, rather than being successful or unsuccessful universally; and have a rhetorical value that leads to the over-application of positively valued terms (e.g. ‘community mobilisation’), invalidating the keyword search process of a systematic review. The article suggests that the policy interest in definitive summary statements of ‘the evidence’ is at odds with academic assessments that evidence takes multiple, contradictory and complex forms, and with practitioner experience of the variability of practice in context. A pragmatist philosophy of evidence is explored as an alternative. Taking this approach implies expanding the definition of forms of research considered to be ‘useful evidence’ for evidence-based policy-making; decentralising decisions about ‘what works’ to allow for the use of local practical wisdom; and prioritising the establishment of good processes for the critical use of evidence, rather than producing context-insensitive summaries of ‘the evidence’.