Influenza Forecasting Research Articles

Integrating information from historical data into mechanistic models for influenza forecasting.

Seasonal influenza causes significant annual morbidity and mortality worldwide. In France, it is estimated that, on average, 2 million individuals consult their GP for influenza-like-illness (ILI) every year. Traditionally, mathematical models used for epidemic forecasting can either include parameters capturing the infection process (mechanistic or compartmental models) or rely on time series analysis approaches that do not make mechanistic assumptions (statistical or phenomenological models). While the latter make extensive use of past epidemic data, mechanistic models are usually independently initialized in each season. As a result, forecasts from such models can contain trajectories that are vastly different from past epidemics. We developed a mechanistic model that takes into account epidemic data from training seasons when producing forecasts. The parameters of the model are estimated via a first particle filter running on the observed data. A second particle filter is then used to produce forecasts compatible with epidemic trajectories from the training set. The model was calibrated and tested on 35 years' worth of surveillance data from the French Sentinelles Network, representing the weekly number of patients consulting for ILI over the period 1985-2019. Our results show that the new method improves upon standard mechanistic approaches. In particular, when retrospectively tested on the available data, our model provides increased accuracy for short-term forecasts (from one to four weeks into the future) and peak timing and intensity. Our new approach for epidemic forecasting allows the integration of key strengths of the statistical approach into the mechanistic modelling framework and represents an attempt to provide accurate forecasts by making full use of the rich surveillance dataset collected in France since 1985.

Optimizing Disease Outbreak Forecast Ensembles.

On the basis of historical influenza and COVID-19 forecasts, we found that more than 3 forecast models are needed to ensure robust ensemble accuracy. Additional models can improve ensemble performance, but with diminishing accuracy returns. This understanding will assist with the design of current and future collaborative infectious disease forecasting efforts.

Forecasting influenza epidemics in China using transmission dynamic model with absolute humidity

BackgroundAn influenza forecasting system is critical to influenza epidemic preparedness. Low temperature has long been recognized as a condition favoring influenza epidemic, yet it fails to justify the summer influenza peak in tropics/subtropics. Recent studies have suggested that absolute humidity (AH) had a U-shape relationship with influenza survival and transmission across climate zones, indicating that a unified influenza forecasting system could be established for China with various climate conditions. MethodsOur study has generated weekly influenza forecasts by season and type/subtype in northern and southern China from 2011 to 2021, using a forecasting system combining an AH-driven susceptible-infected-recovered-susceptible (SIRS) model and the ensemble adjustment Kalman filter (EAKF). Model performance was assessed by sensitivity and specificity in predicting epidemics, and by accuracies in predicting peak timing and magnitude. ResultsOur forecast system can generally well predict seasonal influenza epidemics (mean sensitivity>87.5%; mean specificity >80%). The average forecast accuracies were 82% and 60% for peak timing and magnitude at 3–6 weeks ahead for northern China, higher than those of 42% and 20% for southern China. The accuracy was generally better when the forecast was made closer to the actual peak time. DiscussionThe established AH-driven forecasting system can generally well predict the occurrence of seasonal influenza epidemics in China.

Evaluation of FluSight influenza forecasting in the 2021–22 and 2022–23 seasons with a new target laboratory-confirmed influenza hospitalizations

Accurate forecasts can enable more effective public health responses during seasonal influenza epidemics. For the 2021–22 and 2022–23 influenza seasons, 26 forecasting teams provided national and jurisdiction-specific probabilistic predictions of weekly confirmed influenza hospital admissions for one-to-four weeks ahead. Forecast skill is evaluated using the Weighted Interval Score (WIS), relative WIS, and coverage. Six out of 23 models outperform the baseline model across forecast weeks and locations in 2021–22 and 12 out of 18 models in 2022–23. Averaging across all forecast targets, the FluSight ensemble is the 2nd most accurate model measured by WIS in 2021–22 and the 5th most accurate in the 2022–23 season. Forecast skill and 95% coverage for the FluSight ensemble and most component models degrade over longer forecast horizons. In this work we demonstrate that while the FluSight ensemble was a robust predictor, even ensembles face challenges during periods of rapid change.

The prediction of influenza-like illness using national influenza surveillance data and Baidu query data

BackgroundSeasonal influenza and other respiratory tract infections are serious public health problems that need to be further addressed and investigated. Internet search data are recognized as a valuable source for forecasting influenza or other respiratory tract infection epidemics. However, the selection of internet search data and the application of forecasting methods are important for improving forecasting accuracy. The aim of the present study was to forecast influenza epidemics based on the long short-term memory neural network (LSTM) method, Baidu search index data, and the influenza-like-illness (ILI) rate.MethodsThe official weekly ILI% data for northern and southern mainland China were obtained from the Chinese Influenza Center from 2018 to 2021. Based on the Baidu Index, search indices related to influenza infection over the corresponding time period were obtained. Pearson correlation analysis was performed to explore the association between influenza-related search queries and the ILI% of southern and northern mainland China. The LSTM model was used to forecast the influenza epidemic within the same week and at lags of 1–4 weeks. The model performance was assessed by evaluation metrics, including the mean square error (MSE), root mean square error (RMSE) and mean absolute error (MAE).ResultsIn total, 24 search queries in northern mainland China and 7 search queries in southern mainland China were found to be correlated and were used to construct the LSTM model, which included the same week and a lag of 1–4 weeks. The LSTM model showed that ILI% + mask with one lag week and ILI% + influenza name were good prediction modules, with reduced RMSE predictions of 16.75% and 4.20%, respectively, compared with the estimated ILI% for northern and southern mainland China.ConclusionsThe results illuminate the feasibility of using an internet search index as a complementary data source for influenza forecasting and the efficiency of using the LSTM model to forecast influenza epidemics.

Forecasting influenza incidence as an ordinal variable using machine learning

Background: Many mechanisms contribute to the variation in the incidence of influenza disease, such as strain evolution, the waning of immunity and changes in social mixing. Although machine learning methods have been developed for forecasting, these methods are used less commonly in influenza forecasts than statistical and mechanistic models. In this study, we applied a relatively new machine learning method, Extreme Gradient Boosting (XGBoost), to ordinal country-level influenza disease data. Methods: We developed a machine learning forecasting framework by adopting the XGBoost algorithm and training it with surveillance data for over 32 countries between 2010 and 2018 from the World Health Organisation’s FluID platform. We then used the model to predict incidence 1- to 4-week ahead. We evaluated the performance of XGBoost forecast models by comparing them with a null model and a historical average model using mean-zero error (MZE) and macro-averaged mean absolute error (mMAE). Results: The XGBoost models were consistently more accurate than the null and historical models for all forecast time horizons. For 1-week ahead predictions across test sets, the mMAE of the XGBoost model with an extending training window was reduced by 78% on average compared to the null model. Although the mMAE increased with longer prediction horizons, XGBoost models showed a 62% reduction in mMAE compared to the null model for 4-week ahead predictions. Our results highlight the potential utility of machine learning methods in forecasting infectious disease incidence when that incidence is defined as an ordinal variable. In particular, the XGBoost model can be easily extended to include more features, thus capturing complex patterns and improving forecast accuracy. Conclusion: Given that many natural extreme phenomena are often described on an ordinal scale when informing planning and response, these results motivate further investigation of using similar scales for communicating risk from infectious diseases.

Sustainable and intelligent time-series models for epidemic disease forecasting and analysis

There is an increasing risk of outbreaks escalating into epidemics, despite huge advances in medical science. Epidemics like COVID-19, Monkeypox, Influenza and HIV have been affecting people and public health infrastructure at an alarming rate around the world. COVID-19 alone has infected more than 500 million people out of which 6 million have died over 100 countries. HIV is also a major global public health issue and has claimed 85.6 million lives till 2023. Forecasting the trends of these epidemics is important in order to efficiently manage national and global health risks by improving early warning systems. Therefore an intelligent framework to forecast epidemic diseases is proposed and a detailed comparative analysis is conducted using different time-series models. This study contributes to (Sustainable Development Goal) SDG-3 by predicting epidemics disease trends precisely using ARIMA, Polynomial Regression, SARIMA, Holt’s, Fb-Prophet time-series models, which can decrease the burden on healthcare systems. Using the best-suited models, the Mean Absolute Percentage Error (MAPE) values for Monkeypox, HIV, COVID-19 and Influenza forecasting were 0.0129, 0.0035, 0.0011, and 0.024 respectively.

Application progress of ensemble forecast technology in influenza forecast based on infectious disease model.

To comprehensively understand the application progress of ensemble forecast technology in influenza forecast based on infectious disease model, so as to provide scientific references for further research. In this study, two keywords of "influenza" and "ensemble forecast" are selected to search and select the relevant literatures, which are then outlined and summarized. It is found that: In recent years, some studies about ensemble forecast technology for influenza have been reported in the literature, and some well-performed influenza ensemble forecast systems have already been operationally implemented and provide references for scientific prevention and control. In general, ensemble forecast can well represent various uncertainties in forecasting influenza cases based on infectious disease models, and can achieve more accurate forecasts and more valuable information than single deterministic forecast. However, there are still some shortcomings in the current studies, it is suggested that scientists engaged in influenza forecast based on infectious disease models strengthen cooperation with scholars in the field of numerical weather forecast, which is expected to further improve the skills and application level of ensemble forecast for influenza.

Visual displays for communicating scientific uncertainty in influenza forecasts

We offer a general method for testing the usability of visual displays communicating scientific uncertainty, illustrated with publicly available results from CDC's influenza forecasts. The heavy toll of seasonal influenza has prompted major investments in improving these forecasts, making them a focus of machine learning research. However, little research has been devoted to how well users can understand and use these forecasts to inform decisions under uncertainty. Our approach extends psychological theory to experimental tasks posing hypothetical, but realistic decisions using alternative displays based on actual forecasts. Based on Tversky's theory of conceptual-spatial congruence, we predicted actual and perceived usability of four displays (bar chart, tree map, PDF, and 90% confidence interval). Participants (N = 301, recruited on Amazon MTurk) were randomly assigned to use one of four displays for four decision tasks, created to reflect our extension of the theory. We evaluated participants' comprehension, confidence, and judgments of perceived helpfulness, when the display and the decision were congruent or non-congruent. Participants had better comprehension with the most familiar display (bar chart), for all four decisions. However, they did not perceive that display as more helpful or have greater confidence in their responses to it. Participants who reported greater familiarity with a display performed more poorly, despite expressing greater confidence and rating it as more helpful. We discuss the need to evaluate performance, as well as ratings, and the opportunities to extend theoretical frameworks to specific contexts.

Comparison of combination methods to create calibrated ensemble forecasts for seasonal influenza in the U.S.

The characteristics of influenza seasons vary substantially from year to year, posing challenges for public health preparation and response. Influenza forecasting is used to inform seasonal outbreak response, which can in turn potentially reduce the impact of an epidemic. The United States Centers for Disease Control and Prevention, in collaboration with external researchers, has run an annual prospective influenza forecasting exercise, known as the FluSight challenge. Uniting theoretical results from the forecasting literature with domain-specific forecasts from influenza outbreaks, we applied parametric forecast combination methods that simultaneously optimize model weights and calibrate the ensemble via a beta transformation and made adjustments to the methods to reduce their complexity. We used the beta-transformed linear pool, the finite beta mixture model, and their equal weight adaptations to produce ensemble forecasts retrospectively for the 2016/2017, 2017/2018, and 2018/2019 influenza seasons in the U.S. We compared their performance to methods that were used in the FluSight challenge to produce the FluSight Network ensemble, namely the equally weighted linear pool and the linear pool. Ensemble forecasts produced from methods with a beta transformation were shown to outperform those from the equally weighted linear pool and the linear pool for all week-ahead targets across in the test seasons based on average log scores. We observed improvements in overall accuracy despite the beta-transformed linear pool or beta mixture methods' modest under-prediction across all targets and seasons. Combination techniques that explicitly adjust for known calibration issues in linear pooling should be considered to improve probabilistic scores in outbreak settings.

Neural network models for influenza forecasting with associated uncertainty using Web search activity trends.

Influenza affects millions of people every year. It causes a considerable amount of medical visits and hospitalisations as well as hundreds of thousands of deaths. Forecasting influenza prevalence with good accuracy can significantly help public health agencies to timely react to seasonal or novel strain epidemics. Although significant progress has been made, influenza forecasting remains a challenging modelling task. In this paper, we propose a methodological framework that improves over the state-of-the-art forecasting accuracy of influenza-like illness (ILI) rates in the United States. We achieve this by using Web search activity time series in conjunction with historical ILI rates as observations for training neural network (NN) architectures. The proposed models incorporate Bayesian layers to produce associated uncertainty intervals to their forecast estimates, positioning themselves as legitimate complementary solutions to more conventional approaches. The best performing NN, referred to as the iterative recurrent neural network (IRNN) architecture, reduces mean absolute error by 10.3% and improves skill by 17.1% on average in nowcasting and forecasting tasks across 4 consecutive flu seasons.

Predicting Seasonal Influenza Hospitalizations Using an Ensemble Super Learner: A Simulation Study.

Accurate forecasts can inform response to outbreaks. Most efforts in influenza forecasting have focused on predicting influenza-like activity, with fewer on influenza-related hospitalizations. We conducted a simulation study to evaluate a super learner's predictions of 3 seasonal measures of influenza hospitalizations in the United States: peak hospitalization rate, peak hospitalization week, and cumulative hospitalization rate. We trained an ensemble machine learning algorithm on 15,000 simulated hospitalization curves and generated weekly predictions. We compared the performance of the ensemble (weighted combination of predictions from multiple prediction algorithms), the best-performing individual prediction algorithm, and a naive prediction (median of a simulated outcome distribution). Ensemble predictions performed similarly to the naive predictions early in the season but consistently improved as the season progressed for all prediction targets. The best-performing prediction algorithm in each week typically had similar predictive accuracy compared with the ensemble, but the specific prediction algorithm selected varied by week. An ensemble super learner improved predictions of influenza-related hospitalizations, relative to a naive prediction. Future work should examine the super learner's performance using additional empirical data on influenza-related predictors (e.g., influenza-like illness). The algorithm should also be tailored to produce prospective probabilistic forecasts of selected prediction targets.

RESEAT: Recurrent Self-Attention Network for Multi-Regional Influenza Forecasting.

Early forecasting of influenza is an important task for public health to reduce losses due to influenza. Various deep learning-based models for multi-regional influenza forecasting have been proposed to forecast future influenza occurrences in multiple regions. While they only use historical data for forecasting, temporal and regional patterns need to be jointly considered for better accuracy. Basic deep learning models such as recurrent neural networks and graph neural networks have limited ability to model both patterns together. A more recent approach uses an attention mechanism or its variant, self-attention. Although these mechanisms can model regional interrelationships, in state-of-the-art models, they consider accumulated regional interrelationships based on attention values that are calculated only once for all of the input data. This limitation makes it difficult to effectively model the regional interrelationships that change dynamically during that period. Therefore, in this article, we propose a recurrent self-attention network (RESEAT) for various multi-regional forecasting tasks such as influenza and electrical load forecasting. The model can learn regional interrelationships over the entire period of the input data using self-attention, and it recurrently connects the attention weights using message passing. We demonstrate through extensive experiments that the proposed model outperforms other state-of-the-art forecasting models in terms of the forecasting accuracy for influenza and COVID-19. We also describe how to visualize regional interrelationships and analyze the sensitivity of hyperparameters to forecasting accuracy.

Model-agnostic meta-learning-based region-adaptive parameter adjustment scheme for influenza forecasting

Deep learning models perform well when there is enough data available for training, but otherwise the performance deteriorates rapidly owing to the so-called data shortage problem. Recently, model-agnostic meta-learning (MAML) was proposed to alleviate this problem by embedding common prior knowledge from different tasks into the initial parameters of the target model. Data shortages are very common in regional influenza predictions, and MAML also often struggles with regional influenza forecasting, especially when region-specific knowledge, such as peak timing or intensity, varies. In this paper, we propose a novel MAML-based parameter adjustment scheme for influenza forecasting, called MARAPAS. The fundamental idea of our scheme is to adjust the initial parameters obtained from common knowledge to a target region by using adjustment variables. We experimentally show that MARAPAS outperforms other MAML-based methods, in terms of root mean square error and Pearson correlation coefficient. Particularly, this scheme improves the forecasting performance by up to 34 % compared with that of the state-of-the-art schemes. We also show the robust forecasting accuracy of our scheme and demonstrate its applicability by performing zero-shot COVID-19 forecasting.

Leveraging Google Search Data and Artificial Intelligence Methods for Provincial-level Influenza Forecasting: A South African Case Study

This paper investigates the usefulness of Google search patterns with Artificial Intelligence (AI) techniques for timely influenza-like illness (ILI) forecasting for each of the nine South African provinces. Traditional surveillance methods are limited by delays in reporting. Existing digital disease surveillance studies that employ alternative online data have scarcely explored sub-Saharan African countries. In South Africa, Google search data has only been recently studied for ILI surveillance at the national level. Meanwhile, the differences in socio-economic and technological conditions across provinces call for a finer spatial investigation. We perform correlation analysis between Google trends (GT) data for 21 ILI-related terms and real-life ILI surveillance data for each province. Next, we develop models to assess the predictive performance of these GT data for forecasting ILI rates, using time series, machine learning, and deep learning methods. We observe sufficient correlation for only two of the nine provinces: Gauteng and Western Cape. Thus, GT data could only be used to forecast ILI in these two provinces. Interestingly, these two provinces are regarded as the most economically developed. In the other seven provinces, LSTM, a deep learning technique, gives more accurate predictions than a baseline autoregressive model when only past ILI data are used for forecasting future ILI trends. The results reveal that, for provinces for which GT data is sufficiently available, it is not only free and fast, but is an effective predictor on its own as well as when added to past ILI data for forecasting future ILI infection rates. The correlation analysis suggests an association between provincial socio-economic development and the use of digital platforms for disease surveillance. Overall, the study established the need for finer scale ILI forecasting which will inform targeted planning for disease surveillance and interventions.

Self-Attention-Based Deep Learning Network for Regional Influenza Forecasting.

Early prediction of influenza plays an important role in minimizing the damage caused, as it provides the resources and time needed to formulate preventive measures. Compared to traditional mechanistic approach, deep/machine learning-based models have demonstrated excellent forecasting performance by efficiently handling various data such as weather and internet data. However, due to the limited availability and reliability of such data, many forecasting models use only historical occurrence data and formulate the influenza forecasting as a multivariate time-series task. Recently, attention mechanisms have been exploited to deal with this issue by selecting valuable data in the input data and giving them high weights. Particularly, self-attention has shown its potential in various forecasting tasks by utilizing the predictive relationship between objects from the input data describing target objects. Hence, in this study, we propose a forecasting model based on self-attention for regional influenza forecasting, called SAIFlu-Net. The model exploits a long short-term memory network for extracting time-series patterns of each region and the self-attention mechanism to find the similarities between the occurrence patterns. To evaluate its performance, we conducted extensive experiments with existing forecasting models using weekly regional influenza datasets. The results show that the proposed model outperforms other models in terms of root mean square error and Pearson correlation coefficient.

Fast and accurate influenza forecasting in the United States with Inferno.

Infectious disease forecasting is an emerging field and has the potential to improve public health through anticipatory resource allocation, situational awareness, and mitigation planning. By way of exploring and operationalizing disease forecasting, the U.S. Centers for Disease Control and Prevention (CDC) has hosted FluSight since the 2013/14 flu season, an annual flu forecasting challenge. Since FluSight’s onset, forecasters have developed and improved forecasting models in an effort to provide more timely, reliable, and accurate information about the likely progression of the outbreak. While improving the predictive performance of these forecasting models is often the primary objective, it is also important for a forecasting model to run quickly, facilitating further model development and improvement while providing flexibility when deployed in a real-time setting. In this vein I introduce Inferno, a fast and accurate flu forecasting model inspired by Dante, the top performing model in the 2018/19 FluSight challenge. When pseudoprospectively compared to all models that participated in FluSight 2018/19, Inferno would have placed 2nd in the national and regional challenge as well as the state challenge, behind only Dante. Inferno, however, runs in minutes and is trivially parallelizable, while Dante takes hours to run, representing a significant operational improvement with minimal impact to performance. Forecasting challenges like FluSight should continue to monitor and evaluate how they can be modified and expanded to incentivize the development of forecasting models that benefit public health.

Epidemic tracking and forecasting: Lessons learned from a tumultuous year

Epidemic forecasting has garnered increasing interest in the last decade, nurtured and scaffolded by various forecasting challenges organized by groups within the US federal government, including the Centers for Disease Control and Prevention (CDC) (1⇓–3), Office of Science and Technology Policy (OSTP) (4), and Defense Advanced Research Projects Agency (DARPA) (5), and elsewhere (6, 7). In 2017, after several years of experimentation with flu forecasting in academic groups, the CDC decided to incorporate influenza forecasting into its normal operations, including weekly public communications (8) and briefing to higher-ups. To provide more reliable infrastructure and support for its forecasting needs, the CDC in 2019 designated two national Centers of Excellence for Influenza Forecasting, one at the University of Massachusetts at Amherst (https://reichlab.io/people) and one at Carnegie Mellon University (https://delphi.cmu.edu/about/center-of-excellence/). Not unrelatedly, the last decade has also seen a rise in the importance of digital surveillance streams in public health, with improving epidemic tracking and forecasting models being a key application of these data. Digital streams, such as search and social media trends, have constituted a large part of the focus (9⇓⇓⇓⇓–14); however, even more broadly, data from auxiliary streams that operate outside of traditional public health reporting, such as online surveys, medical devices, or electronic medical records (EMRs), have received considerable attention as well (15⇓⇓⇓⇓⇓⇓⇓⇓⇓–25). The Carnegie Mellon Delphi group, which the two of us colead, has worked in both of these emerging disciplines—epidemic forecasting and building relevant auxiliary signals to aid such forecasting models—since 2012. In 2020, as the pandemic broke out, we struggled like many other groups to find ways to contribute to the national efforts to respond to the pandemic. We ended up shifting our … [↵][1]1To whom correspondence may be addressed. Email: ryantibs{at}cmu.edu. [1]: #xref-corresp-1-1

EPIsembleVis: A geo-visual analysis and comparison of the prediction ensembles of multiple COVID-19 models

We present EPIsembleVis, a web-based comparative visual analysis tool for evaluating the consistency of multiple COVID-19 prediction models. Our approach analyzes a collection of COVID-19 predictions from different epidemiological models as an ensemble and utilizes two metrics to quantify model performance. These metrics include (a) prediction uncertainty (represented as the dispersion of predictions in each ensemble) and (b) prediction error (calculated by comparing individual model predictions with the recorded data). Through an interactive visual interface, our approach provides a data-driven workflow for (a) selecting and constructing the COVID-19 model prediction ensemble based on the spatiotemporal overlap of available predictions of multiple epidemiological models, (b) quantifying the model performance using both the uncertainty of each model prediction ensemble, and the error of each ensemble member that represents individual model predictions, and (c) visualizing the spatiotemporal variability in the projection performance of individual models using a suite of novel ensemble visualization techniques, such as the data availability map, a spatiotemporal textured-tile calendar, multivariate rose chart, and time-series leaflet glyph. We demonstrate the capability of our ensemble visual interface through a case study that investigates the performance of weekly COVID-19 predictions, which are provided through the COVID-19 Forecast Hub UMass-Amherst Influenza Forecasting Center of Excellence [47] for the United States and United States Territories. The EPIsembleVis tool is implemented using open-source web technologies and adaptive system design, rendering it interoperable with Elasticsearch and Kibana for automatically ingesting COVID-19 predictions from online repositories, and it is generalizable for analyzing worldwide projections from more epidemiological models.

Adaptively stacking ensembles for influenza forecasting.

Seasonal influenza infects between 10 and 50 million people in the United States every year. Accurate forecasts of influenza and influenza-like illness (ILI) have been named by the CDC as an important tool to fight the damaging effects of these epidemics. Multi-model ensembles make accurate forecasts of seasonal influenza, but current operational ensemble forecasts are static: they require an abundance of past ILI data and assign fixed weights to component models at the beginning of a season, but do not update weights as new data on component model performance is collected. We propose an adaptive ensemble that (i) does not initially need data to combine forecasts and (ii) finds optimal weights which are updated week-by-week throughout the influenza season. We take a regularized likelihood approach and investigate this regularizer's ability to impact adaptive ensemble performance. After finding an optimal regularization value, we compare our adaptive ensemble to an equal-weighted and static ensemble. Applied to forecasts of short-term ILI incidence at the regional and national level, our adaptive model outperforms an equal-weighted ensemble and has similar performance to the static ensemble using only a fraction of the data available to the static ensemble. Needing no data at the beginning of an epidemic, an adaptive ensemble can quickly train and forecast an outbreak, providing a practical tool to public health officials looking for a forecast to conform to unique features of a specific season.