Assessing fitness-for-purpose and comparing the suitability of COVID-19 multi-country models for local contexts and users

Hannah E Clapham ,Mohamed Gad,Itamar Megiddo,Adrian Gheorghe,Wilfred Ndifon,Raymond Hutubessy,Paula M Luz,Jane Labadin,Thomas Wilkinson,Nejma Cheikh,Marc Brisson,Jodie Mcvernon,Akhona Tshangela,Francis Ruiz,Shankar Prinja,N J Gay ,Marelize Görgens ,Chris Painter ,Joshua A Salomon ,Brooke E Nichols

doi:10.12688/gatesopenres.13224.1

Abstract

Background: Mathematical models have been used throughout the COVID-19 pandemic to inform policymaking decisions. The COVID-19 Multi-Model Comparison Collaboration (CMCC) was established to provide country governments, particularly low- and middle-income countries (LMICs), and other model users with an overview of the aims, capabilities and limits of the main multi-country COVID-19 models to optimise their usefulness in the COVID-19 response. Methods: Seven models were identified that satisfied the inclusion criteria for the model comparison and had creators that were willing to participate in this analysis. A questionnaire, extraction tables and interview structure were developed to be used for each model, these tools had the aim of capturing the model characteristics deemed of greatest importance based on discussions with the Policy Group. The questionnaires were first completed by the CMCC Technical group using publicly available information, before further clarification and verification was obtained during interviews with the model developers. The fitness-for-purpose flow chart for assessing the appropriateness for use of different COVID-19 models was developed jointly by the CMCC Technical Group and Policy Group. Results: A flow chart of key questions to assess the fitness-for-purpose of commonly used COVID-19 epidemiological models was developed, with focus placed on their use in LMICs. Furthermore, each model was summarised with a description of the main characteristics, as well as the level of engagement and expertise required to use or adapt these models to LMIC settings. Conclusions: This work formalises a process for engagement with models, which is often done on an ad-hoc basis, with recommendations for both policymakers and model developers and should improve modelling use in policy decision making.

Highlights

Throughout the COVID-19 pandemic, mathematical models that mimic the natural history and dynamics of COVID-19 have been used in countries across the globe to inform policymaking decisions, such as the allocation of scarce healthcare resources and the implementation of policy measures to curb the spread of COVID-19
The primary objectives of the first phase of the project were to develop: 1) A flow chart of key questions to assess the fitness-forpurpose of a set of widely used COVID-19 epidemiological models, with additional focus placed on their appropriateness for use in low- and middle-income countries (LMICs)
These questions include a consideration of the aim of the model, how it has been adapted to each setting, how it could be used by analysts and policymakers in LMICs and the level of interaction with modellers they would need

Summary

Introduction

Throughout the COVID-19 pandemic, mathematical models that mimic the natural history and dynamics of COVID-19 have been used in countries across the globe to inform policymaking decisions, such as the allocation of scarce healthcare resources and the implementation of policy measures to curb the spread of COVID-19. A comprehensive understanding of these characteristics often requires some knowledge of mathematical modelling and concepts related to natural history of disease, infectious disease epidemiology and health economics These prerequisites can make it difficult for policymakers to assess the fitnessfor-purpose of models and to understand which of the many models available are the most appropriate for given policy questions and in the local context, given the time constraints they often work within. As the purpose of COVID-19 models varies, model users may need to use different models to address different policy questions, and choosing which model to use is not as simple as just attempting to understand which model is the ‘best’ model This lack of transparency and accessibility of mathematical modelling carries risks: policymakers may end up misunderstanding, misusing or even ignoring the results of models, which in the worst case could result in harmful policy responses. Conclusions: This work formalises a process for engagement with models, which is often done on an ad-hoc basis, with recommendations for both policymakers and model developers and should improve modelling use in policy decision making

Objectives

Methods

Results

Conclusion