An Evaluation Service for Digital Public Health Interventions: User-Centered Design Approach.

Kassandra Karpathakis,Simon Dixon,Henry W W Potts,Gene Libow,Elizabeth Murray,Felix Greaves

doi:10.2196/28356

Abstract

BackgroundDigital health interventions (DHIs) have the potential to improve public health by combining effective interventions and population reach. However, what biomedical researchers and digital developers consider an effective intervention differs, thereby creating an ongoing challenge to integrating their respective approaches when evaluating DHIs.ObjectiveThis study aims to report on the Public Health England (PHE) initiative set out to operationalize an evaluation framework that combines biomedical and digital approaches and demonstrates the impact, cost-effectiveness, and benefit of DHIs on public health.MethodsWe comprised a multidisciplinary project team including service designers, academics, and public health professionals and used user-centered design methods, such as qualitative research, engagement with end users and stakeholders, and iterative learning. The iterative approach enabled the team to sequentially define the problem, understand user needs, identify opportunity areas, develop concepts, test prototypes, and plan service implementation. Stakeholders, senior leaders from PHE, and a working group critiqued the outputs.ResultsWe identified 26 themes and 82 user needs from semistructured interviews (N=15), expressed as 46 Jobs To Be Done, which were then validated across the journey of evaluation design for a DHI. We identified seven essential concepts for evaluating DHIs: evaluation thinking, evaluation canvas, contract assistant, testing toolkit, development history, data hub, and publish health outcomes. Of these, three concepts were prioritized for further testing and development, and subsequently refined into the proposed PHE Evaluation Service for public health DHIs. Testing with PHE’s Couch-to-5K app digital team confirmed the viability, desirability, and feasibility of both the evaluation approach and the Evaluation Service.ConclusionsAn iterative, user-centered design approach enabled PHE to combine the strengths of academic and biomedical disciplines with the expertise of nonacademic and digital developers for evaluating DHIs. Design-led methodologies can add value to public health settings. The subsequent service, now known as Evaluating Digital Health Products, is currently in use by health bodies in the United Kingdom and is available to others for tackling the problem of evaluating DHIs pragmatically and responsively.

Highlights

For public health interventions to significantly impact healthy life expectancy in a population, reach has to be combined with efficacy [1]
The barriers identified were organizational and disciplinary silos associated with public health commissioning, design and development of digital services, and academic evaluation; unclear expectations and roles; different disciplinary expectations on what tools and methods to use for evaluation and how to measure success or failure; limited time, capacity, and funding; lack of evaluation experience; and rapidly evolving digital health intervention Jobs To Be Done (JTBD) (DHI)
It will continue to be unrealized if how we evaluate and use evaluations to inform the iterative design and development of DHIs do not use the perspectives of both biomedical research and digital development

Summary

Introduction

For public health interventions to significantly impact healthy life expectancy in a population, reach has to be combined with efficacy [1]. There is considerable interest in the potential of digital health interventions (DHIs) to improve public health. DHIs are delivered on digital platforms, such as the web or smartphone apps intended to deliver health care or health promotion [2]. Such DHIs are expected to combine the reach of large-scale population initiatives, such as media advertising, with the efficacy of individual treatments. Digital health interventions (DHIs) have the potential to improve public health by combining effective interventions and population reach. What biomedical researchers and digital developers consider an effective intervention differs, thereby creating an ongoing challenge to integrating their respective approaches when evaluating DHIs

Methods

Results

Discussion

Conclusion