Can learning be measured by phone? Evidence from Kenya

Abstract

School closures induced by COVID-19 placed heightened emphasis on alternative ways to measure learning besides in-person exams. We leverage the administration of phone-based assessments (PBAs) measuring numeracy and literacy for primary school children in Kenya, along with in-person standardized tests administered to the same students prior to school shutdowns, to assess the validity of PBAs. Compared to repeated in-person assessments, PBAs did not severely misclassify students’ relative performance, but PBA scores did tend to be further from baseline in-person scores than repeated in-person assessments from each other. As such, PBAs performed well at measuring aggregate but not individual learning levels. Administrators can therefore use these tools for aggregate measurement, such as in the context of impact evaluation, but should be wary of PBAs for individual-level tracking or high-stakes decisions. Results also reveal the importance of making deliberate efforts to reach a representative sample and selecting items that provide discriminating power. Finally, in cases when collecting in-person data is possible, this study illustrates that, depending on the logistical capabilities of the data collecting agency, PBAs do not always have a cost-effectiveness advantage over in-person data collection.