Predictive performance of international COVID-19 mortality forecasting models

Joseph Friedman,James Collins,Theo Vos,Robert C Reiner,Christopher E Troeger,Ryan M Barber,Stephen S Lim,Austin Carter,Emmanuela Gakidou,Simon I Hay,Christopher J L Murray,Patrick Liu,David M Pigott

doi:10.1038/s41467-021-22457-w

Abstract

Forecasts and alternative scenarios of COVID-19 mortality have been critical inputs for pandemic response efforts, and decision-makers need information about predictive performance. We screen n = 386 public COVID-19 forecasting models, identifying n = 7 that are global in scope and provide public, date-versioned forecasts. We examine their predictive performance for mortality by weeks of extrapolation, world region, and estimation month. We additionally assess prediction of the timing of peak daily mortality. Globally, models released in October show a median absolute percent error (MAPE) of 7 to 13% at six weeks, reflecting surprisingly good performance despite the complexities of modelling human behavioural responses and government interventions. Median absolute error for peak timing increased from 8 days at one week of forecasting to 29 days at eight weeks and is similar for first and subsequent peaks. The framework and public codebase (https://github.com/pyliu47/covidcompare) can be used to compare predictions and evaluate predictive performance going forward.

Highlights

Forecasts and alternative scenarios of COVID-19 mortality have been critical inputs for pandemic response efforts, and decision-makers need information about predictive performance
Some comparisons have been conducted for models describing the epidemic in the United States[10,11,12,13], to our knowledge, similar analyses have not been undertaken for models covering multiple countries, despite the growing global impact of COVID-19
Seven models, which fit all inclusion criteria, were evaluated (Table 1). These included those modelled by DELPHI-MIT (Delphi)[14,15], Youyang Gu (YYG)[10], the Los Alamos National Laboratory (LANL)[16], Imperial College London (Imperial)[17], the SIKJ-Alpha model from the USC Data Science Lab (SIKJalpha)[18] and the Institute for Health Metrics and Evaluation (IHME)[19]

Summary

Introduction

Forecasts and alternative scenarios of COVID-19 mortality have been critical inputs for pandemic response efforts, and decision-makers need information about predictive performance. We screen n = 386 public COVID-19 forecasting models, identifying n = 7 that are global in scope and provide public, date-versioned forecasts We examine their predictive performance for mortality by weeks of extrapolation, world region, and estimation month. Some comparisons have been conducted for models describing the epidemic in the United States[10,11,12,13], to our knowledge, similar analyses have not been undertaken for models covering multiple countries, despite the growing global impact of COVID-19 This analysis, conducted by members of the Institute for Health Metrics and Evaluation (IHME) COVID-19 Forecasting team, introduces a publicly available evaluation framework, including full access to all data and code (https://github.com/ pyliu47/covidcompare), for assessing the predictive validity of COVID-19 mortality forecasts. This will, over time, serve as a reference for decisionmakers on historical model performance and provide insight into which models should be considered for critical decisions in the future

Methods

Results

Conclusion