Inter- and intra-rater reliability of handpump functionality field tests

Abstract

Sustaining the functionality of drinking water supplies in low- and middle-income countries is a longstanding challenge. Growing awareness of this problem has motivated increased attention to validly and reliably measuring water point functionality, including among handpumps, which serve approximately 9 % of the global population. Yet the most widely used indicator of functionality, whether a water point provides water, has limited validity, reliability, and utility. We tested the inter-rater (agreement among measurements taken by different people) and intra-rater (agreement among repeated measurements taken by the same person) reliability of three handpump functionality field tests in Uganda: pump capacity, 10-minute leakage rate, and flowrate. One person equipped with a stopwatch and a 20-liter container can complete the tests for one handpump in 15 min. The same three to four raters each conducted the tests three times on 28 handpumps. Different sets of four to five raters each conducted the tests once on 32 handpumps. Intraclass correlation coefficients were estimated to indicate inter- and intra-rater reliability. Ten-minute leakage rate had the highest inter-rater reliability, followed by pump capacity. Flowrate, which is commonly measured manually as part of handpump functionality assessments, had poor inter-rater reliability. Indicators derived from all three tests had high intra-rater reliability. Drawing on our inter-rater reliability results, we propose a fully quantitative procedure and validate an ordinal scale of physical handpump functionality based on the 10-minute leakage rate and pump capacity tests. This measurement procedure can be usefully incorporated into service delivery monitoring and research to enhance the objectivity, utility, and comparability of global handpump functionality data. Future studies can test the reliability of these indicators in other contexts and their value for predicting handpump breakdown.