Evidence, Evaluation and the ‘Tyranny of Effect Size’: a proposal to more accurately measure programme impacts in adult and family literacy

Abstract

As literacy grows in importance, policymakers’ demands for programme quality grow, too. Evidence on the effectiveness of adult and family literacy programmes is limited at best: research gaps abound, and programme evaluations are more often than not based on flawed theories of programme impact. In the absence of robust evidence on the full range of short‐ and long‐term programme impacts, it is difficult to accurately measure intervention effectiveness. Too frequently, researchers and policymakers focus only on short‐term, easily measured outcomes, creating a ‘tyranny of effect size’ that may systematically underestimate impact while simultaneously distorting practice. However, the answer does not lie in turning away from quantitative research. Doing so will consign adult and family literacy to the margins of public policy, when they should be in the mainstream. Longitudinal research from Turkey and the US suggests a need for revised, more subtle theories of how adult literacy and family literacy programmes work, and the diverse ways they benefit participants. By working together more closely and intelligently, researchers, policymakers and practitioners can develop evaluation strategies that more accurately measure programme effects. The key is combining methodological rigour with fully fleshed out theories of literacy development and programme impact.