Factors Influencing Roll-Out of an m-Health Intervention in India: Results from a Dynamic and Responsive Process Evaluation

Abstract

Abstract Objectives Implementation research can support iterative program improvement, yet limited examples of such dynamic experiences exist. We conducted a multi-stage and multi-method process evaluation (PE) of a large-scale m-Health intervention in India that currently reaches 581,282 frontline workers (FLWs). The intervention (Common Application Software (CAS)) includes a smartphone application that digitizes FLW service-tracking registers and acts as a job aid and multiple dashboards for real-time monitoring. We examined factors affecting the roll-out of CAS in two Indian states. Methods Program impact pathways were developed with implementers to inform research questions and methods. Data were collected across many time points between 2016 and 2018 to temporally align with program roll-out. In total, we did in-person surveys with FLWs (n = 100), supervisors (n = 55), other staff (n = 18), two phone surveys with FLWs (n = 404), interviews with national and state-level stakeholders (n = 21), and routine meetings with implementers. Short phone surveys were conducted between in-person surveys to give rapid feedback to implementers. Results Impediments to roll-out included state readiness, delays in device procurement and set-up, dashboard readiness, and low data storage space. Over time, FLWs satisfaction with training remained consistent (100–99%), use of CAS (82–100%), and preference to use it (74–80%) increased. Challenges declined over time [e.g., slow internet (79– >58%), battery heating (62– >45%), and network issues (68– >52%). Most FLWs were knowledgeable of CAS but were confused with symbols/colors in certain domains of CAS. Supervisors were satisfied with training, were knowledgeable of CAS, but hardware issues precluded some from using it. Use of the dashboard, a critical feedback mechanism, was limited due to gaps in staff training and version modifications. Conclusions A dynamic and engaged PE implemented with inputs from multiple stakeholders helped provide timely and meaningful feedback to a large-scale program, enabling corrective actions and increasing potential for impact of the program. Implementing a dynamic PE is time and resource intensive and required flexibility in timing and modalities of data collection. Funding Sources Bill & Melinda Gates Foundation, via grants to IFPRI and UCSF.