How accurate are retrospective survey data? Evidence from rural Ethiopia

Abstract

ABSTRACT Development actors are always seeking reliable and cost-effective methods to assess the impact of their programmes. In particular, there are frequently calls to evaluate programmes for which no pre-intervention (or ‘baseline’) data are available. In these cases, evaluators often rely on retrospective survey questions to reconstruct the baseline situation. This article explores the accuracy of such retrospective survey data, using data from two surveys carried out nearly six years apart among women in rural Ethiopia. We find that the proportion of survey items for which baseline data and retrospective data do not agree is 22%. Responses to the retrospective questions are more closely associated with respondents’ situation at the time of the survey than with their situation at the time they were being asked to recall. Consequently, 72% of respondents were allocated to different quintiles of household wealth, depending on whether the true baseline or the retrospective baseline data were used. We show that controlling for retrospective baseline data can considerably underestimate the impact of the intervention being evaluated. This suggests that there is a need for caution in interpreting the findings of evaluations based on such data and in drawing policy conclusions from them.