Assessing recall bias and measurement error in high-frequency social data collection for human-environment research

Andrew Bell,Patrick Ward,Mary Killilea,Md Ehsanul Haque Tamal

doi:10.1007/s11111-019-0314-1

Abstract

A major impediment to understanding human-environment interactions is that data on social systems are not collected in a way that is easily comparable to natural systems data. While many environmental variables are collected with high frequency, gridded in time and space, social data is typically conducted irregularly, in waves that are far apart in time. These efforts typically engage respondents for hours at a time, and suffer from decay in participants’ ability to recall their experiences over long periods of time. Systematic use of mobile and smartphones has the potential to transcend these challenges, with a critical first step being an evaluation of where survey respondents experience the greatest recall decay. We present results from, to our knowledge, the first systematic evaluation of recall bias in components of a household survey, using the Open Data Kit (ODK) platform on Android smartphones. We tasked approximately 500 farmers in rural Bangladesh with responding regularly to components of a large household survey, randomizing the frequency of each task to be received weekly, monthly, or seasonally. We find respondents’ recall of consumption and experience (such as sick days) to suffer much more greatly than their recall of the use of their households’ time for labor and farm activities. Further, we demonstrate a feasible and cost-effective means of engaging respondents in rural areas to create and maintain a true socio-economic “baseline” to mirror similar efforts in the natural sciences.

Highlights

A major impediment to understanding human-environment interactions is that data on social systems are not collected in a way that is comparable to natural systems data
Regular engagement via short tasks can provide a high-frequency, representative, socio-economic baseline in a cost-effective manner. All this does not come free of charge, so it is important to develop an understanding of what streams of data are worth knowing with high frequency, and what gains are possible relative to a conventional, much more infrequent survey approach, taking into consideration data losses due to missed intra-period variation and difficulty in recall
Farmers responded to short survey tasks administered in the Open Data Kit (ODK) platform (Brunette et al 2013), launched via a custom, userfriendly app, on a regular basis

Summary

Introduction

A major impediment to understanding human-environment interactions is that data on social systems are not collected in a way that is comparable to natural systems data. While many environmental variables are collected with high frequency, gridded in time and space, social data is typically conducted irregularly, in waves that are far apart in time These efforts typically engage respondents for hours at a time, and suffer from decay in participants’ ability to recall their experiences over long periods of time. Regular engagement via short tasks (e.g., survey items, economic games, or various forms of micro-experiments) can provide a high-frequency, representative, socio-economic baseline in a cost-effective manner All this does not come free of charge, so it is important to develop an understanding of what streams of data are worth knowing with high frequency, and what gains are possible relative to a conventional, much more infrequent survey approach, taking into consideration data losses due to missed intra-period variation and difficulty in recall. As an illustrative example of geographically representative data collection efforts, consider integrated household survey efforts (e.g., Malawi 2012; Ahmed 2013) or demographic censuses (e.g., NSO 2010; Zambia 2013)

Objectives

Methods

Results

Conclusion