Forecast Evaluation Research Articles

Evaluation of FluSight influenza forecasting in the 2021–22 and 2022–23 seasons with a new target laboratory-confirmed influenza hospitalizations

Accurate forecasts can enable more effective public health responses during seasonal influenza epidemics. For the 2021–22 and 2022–23 influenza seasons, 26 forecasting teams provided national and jurisdiction-specific probabilistic predictions of weekly confirmed influenza hospital admissions for one-to-four weeks ahead. Forecast skill is evaluated using the Weighted Interval Score (WIS), relative WIS, and coverage. Six out of 23 models outperform the baseline model across forecast weeks and locations in 2021–22 and 12 out of 18 models in 2022–23. Averaging across all forecast targets, the FluSight ensemble is the 2nd most accurate model measured by WIS in 2021–22 and the 5th most accurate in the 2022–23 season. Forecast skill and 95% coverage for the FluSight ensemble and most component models degrade over longer forecast horizons. In this work we demonstrate that while the FluSight ensemble was a robust predictor, even ensembles face challenges during periods of rapid change.

The Year of Polar Prediction site Model Intercomparison Project (YOPPsiteMIP) phase 1: project overview and Arctic winter forecast evaluation

Abstract. Although the quality of weather forecasts in the polar regions is improving, forecast skill there still lags behind lower latitudes. So far there have been relatively few efforts to evaluate processes in numerical weather prediction systems using in situ and remote sensing datasets from meteorological observatories in the terrestrial Arctic and Antarctic compared to the mid-latitudes. Progress has been limited both by the heterogeneous nature of observatory and forecast data and by limited availability of the parameters needed to perform process-oriented evaluation in multi-model forecast archives. The Year of Polar Prediction (YOPP) site Model Inter-comparison Project (YOPPsiteMIP) is addressing this gap by producing merged observatory data files (MODFs) and merged model data files (MMDFs), bringing together observations and forecast data at polar meteorological observatories in a format designed to facilitate process-oriented evaluation. An evaluation of forecast performance was performed at seven Arctic sites, focussing on the first YOPP Special Observing Period in the Northern Hemisphere (NH-SOP1) in February and March 2018. It demonstrated that although the characteristics of forecast skill vary between the different sites and systems, an underestimation in boundary layer temperature variability across models, which goes hand in hand with an inability to capture cold extremes, is a common issue at several sites. It is found that many models tend to underestimate the sensitivity of the 2 m air temperature (T2m) and the surface skin temperature to variations in radiative forcing, and the reasons for this are discussed.

Evaluation of forecasts by a global data-driven weather model with and without probabilistic post-processing at Norwegian stations

Abstract. During the last 2 years, tremendous progress has been made in global data-driven weather models trained on numerical weather prediction (NWP) reanalysis data. The most recent models trained on the ERA5 reanalysis at 0.25° resolution demonstrate forecast quality on par with ECMWF's high-resolution model with respect to a wide selection of verification metrics. In this study, one of these models, Pangu-Weather, is compared to several NWP models with and without probabilistic post-processing for 2 m temperature and 10 m wind speed forecasting at 183 Norwegian SYNOP (surface synoptic observation) stations up to +60 h ahead. The NWP models included are the ECMWF HRES, ECMWF ENS and the HARMONIE-AROME ensemble model MEPS with 2.5 km spatial resolution. Results show that the performances of the global models are on the same level, with Pangu-Weather being slightly better than the ECMWF models for temperature and slightly worse for wind speed. The MEPS model clearly provided the best forecasts for both parameters. The post-processing improved the forecast quality considerably for all models but to a larger extent for the coarse-resolution global models due to stronger systematic deficiencies in these. Apart from this, the main characteristics in the scores were more or less the same with and without post-processing. Our results thus confirm the conclusions from other studies that global data-driven models are promising for operational weather forecasting.

Probabilistic forecasting with a hybrid Factor-QRA approach: Application to electricity trading

This paper presents a novel hybrid approach for constricting probabilistic forecasts that combines both the Quantile Regression Averaging (QRA) method and the factor-based averaging scheme. The performance of the approach is evaluated on data sets from two European energy markets — the German EPEX SPOT and the Polish Power Exchange (TGE). The results show that the newly proposed method outperforms literature benchmarks in terms of statistical measures: the empirical coverage and the Christoffersen test for conditional coverage. Moreover, in line with recent literature trends, the economic value of forecasts is evaluated based on the trading strategy using probabilistic price predictions to optimize the operation of an energy storage system. The results suggest that apart from the use of statistical measures, there is a need for the economic evaluation of forecasts.

Conducting volcanic ash cloud exercises: practising forecast evaluation procedures and the pull-through of scientific advice to the London VAAC

The London Volcanic Ash Advisory Centre (VAAC) provides forecasts on the expected presence of volcanic ash in the atmosphere to mitigate the risk to aviation. It is fundamentally important that operational capability is regularly tested through exercises, to guarantee an effective response to an event. We have developed exercises which practise the pull-through of scientific advice into the London VAAC, the forecast evaluation process, and the decision-making procedures and discussions needed for generating the best possible forecasts under real-time conditions. London VAAC dispersion model forecasts are evaluated against observations. To test this capability in an exercise, we must create observation data for a hypothetical event. We have developed new methodologies for generating and using simulated satellite and lidar retrievals. These simulated observations enable us to practise our ability to interpret, compare, and evaluate model output and observation data under real-time conditions. Forecast evaluation can benefit from an understanding of how different choices of model setup (input parameters), model physics, and driving meteorological data impact the predicted extent and concentration of ash. Through our exercises, we have practised comparing output from model simulations generated using different models, model setups, and meteorological data, supplied by different institutions. Our exercises also practise the communication and interaction between Met Office (UK) scientists supporting the London VAAC and external experts, enabling knowledge exchange and discussions on the interpretation of model output and observations, as we strive to deliver the best response capability for the aviation industry and stakeholders. In this paper, we outline our exercise methodology, including the use of simulated satellite and lidar observations, and the development of the strategy to compare output generated from different modelling systems. We outline the lessons learnt, including the benefits and challenges of conducting exercises which practise our ability to provide scientific advice for an operational response at the London VAAC.

Distribution‐Based Model Evaluation and Diagnostics: Elicitability, Propriety, and Scoring Rules for Hydrograph Functionals

AbstractDistribution forecasts P over future quantities or events are routinely made in hydrology but usually traded for a (likelihood‐weighted) mean or median prediction to accommodate error measures or scoring functions such as the mean absolute error or mean squared error. Case in point is the so‐called KG efficiency (KGE) of Gupta et al. (2009, https://doi.org/10.1016/j.jhydrol.2009.08.003) and improvements thereof (Lamontagne et al., 2020, https://doi.org/10.1029/2020wr027101), which have rapidly gained popularity among hydrologists as alternative scoring functions to the commonly used Nash and Sutcliffe (1970, https://doi.org/10.1016/0022‐1694(70)90255‐6) efficiency, but are equally exclusive in how they quantify model performance using only single‐valued output of the quantities of interest. This point‐valued mapping necessarily implies a loss of information about model performance. This paper advocates the use of probabilistic watershed model training, evaluation and diagnostics. Distribution evaluation opens a mature literature on scoring rules whose strong statistical underpinning provides, as we will demonstrate, the theory, context and guidelines necessary for the development of robust information‐theoretically principled metrics for watershed signatures. These so‐called hydrograph functionals are scalar‐valued mappings of major behavioral watershed functions embodied in a strictly proper scoring rule. We discuss past developments that led to the current state‐of‐the‐art of distribution evaluation in hydrology and review scoring rules for dichotomous and categorical events, quantiles (intervals) and density forecasts. We are particularly concerned with elicitable functionals and scoring rule propriety, discuss the decomposition of scoring rules into a sharpness, reliability and entropy term and present diagnostically appealing strictly proper divergence scores of hydrograph functionals for flood frequency analysis, flow duration and recession curves. The usefulness and power of distribution‐based model evaluation and diagnostics by means of scoring rules is demonstrated on simple illustrative problems and discharge distributions simulated with watershed models using random sampling and Bayesian model averaging. The presented theory (a) enables a more complete evaluation of distribution forecasts, (b) offers a statistically principled means for watershed model training, evaluation, diagnostics and selection using hydrograph functionals and/or extreme events and (c) provides a universal framework for metric development of watershed signatures, promoting metric standardization and reproducibility.

Evaluation of Precipitation Forecast by the Operational China Meteorological Administration Mesoscale Model During the 2020 Meiyu Period

AbstractThis study evaluated the precipitation forecast produced by the operational China Meteorological Administration Mesoscale model (CMA‐MESO) during the “super violent” Meiyu season of 2020. Generally, CMA‐MESO, which runs with ∼3‐km‐grid resolution, is able to reproduce the distribution and diurnal variation of precipitation. However, the precipitation amount is greatly overestimated, especially in eastern coastal areas of China. Precipitation in that region usually occurs with two peaks: one in the morning that mostly reflects organized precipitation systems, and the other in the afternoon generated mostly by local convection. Analyses showed that overestimation of low‐level wind speed is the main reason for the overestimation of precipitation. CMA‐MESO produces low‐level winds that are overly strong, which greatly enhance the predicted convergence at night, leading to overestimation of precipitation. Additionally, the stronger wind speed increases the estimated transport of water vapor to the eastern coastal area, producing fake convection near the coastal mountains as the perturbed wind direction turns toward the mountain area in the afternoon. In comparison with ERA5, CMA‐MESO tends to overestimate (underestimate) the temperature in the northwest (southeast), and the larger temperature gradient increases the pressure gradient, resulting in the stronger low‐level wind speed.

Hypothesis testing for performance evaluation of probabilistic seasonal rainfall forecasts

Hypothesis testing for performance evaluation of probabilistic seasonal rainfall forecasts

Real-time forecasting of COVID-19-related hospital strain in France using a non-Markovian mechanistic model.

Projects such as the European Covid-19 Forecast Hub publish forecasts on the national level for new deaths, new cases, and hospital admissions, but not direct measurements of hospital strain like critical care bed occupancy at the sub-national level, which is of particular interest to health professionals for planning purposes. We present a sub-national French framework for forecasting hospital strain based on a non-Markovian compartmental model, its associated online visualisation tool and a retrospective evaluation of the real-time forecasts it provided from January to December 2021 by comparing to three baselines derived from standard statistical forecasting methods (a naive model, auto-regression, and an ensemble of exponential smoothing and ARIMA). In terms of median absolute error for forecasting critical care unit occupancy at the two-week horizon, our model only outperformed the naive baseline for 4 out of 14 geographical units and underperformed compared to the ensemble baseline for 5 of them at the 90% confidence level (n = 38). However, for the same level at the 4 week horizon, our model was never statistically outperformed for any unit despite outperforming the baselines 10 times spanning 7 out of 14 geographical units. This implies modest forecasting utility for longer horizons which may justify the application of non-Markovian compartmental models in the context of hospital-strain surveillance for future pandemics.

Event-based evaluation of operational ENSO forecasting models in 2002–2020: Implications for seasonal water resources management

Given the practical implications of El Niño-Southern Oscillation (ENSO) forecasting for developing decision support systems and its importance in global climate prediction on seasonal to interannual scales, significant efforts have been made to enhance ENSO forecasts. This study presents an event-based evaluation of ENSO forecasts and discusses the implications of such forecasts to water resources management. Using ENSO forecasts for the period 2002–2020, this study evaluates and compares the results of two types of forecasting models, i.e., statistical and dynamical models, in predicting Sea Surface Temperatures (SSTs). Forecasting skills are evaluated for distinct target seasons via two metrics, spearman correlation and mean squared error. In addition, the performance of the two types of models at different lead times are also evaluated. Results reveal that the forecasting skills of these models are comparable, both of which exhibit higher forecasting skills for the boreal fall-winter season and lower skills for the boreal spring season. Event-based analyses show that dynamical and statistical models under-forecast SST anomaly at the onset month for El Niño events, although the forecasting error diminishes with a reduced forecasting lead for most occasions. For La Niña events, SST anomaly forecasting errors could be positive or negative. It is also difficult for the models to accurately predict the quick shift from one ENSO phase to another. Factors that contribute to such challenges are discussed. The implication of ENSO forecasts for water resources management, mainly streamflow forecasts, for a regional water supply utility in the southeastern United States is also discussed. Results from this study provide insights into ENSO forecasting skills and their practical implications.

Evaluation of Probabilistic Forecasts of Binary Events with the Neighborhood Brier Divergence Skill Score

Abstract A procedure for evaluating the quality of probabilistic forecasts of binary events has been developed. This is based on a two-step procedure: pooling of forecasts on the one hand and observations on the other hand, on all the points of a neighborhood in order to obtain frequencies at the neighborhood length scale and then to calculate the Brier divergence for these neighborhood frequencies. This score allows the comparison of a probabilistic forecast and observations at the neighborhood length scale, and therefore, the rewarding of event forecasts shifted from the location of the observed event by a distance smaller than the neighborhood size. A new decomposition of this score generalizes that of the Brier score and allows the separation of the generalized resolution, reliability, and uncertainty terms. The neighborhood Brier divergence skill score (BDnSS) measures the performance of the probabilistic forecast against the sample climatology. BDnSS and its decomposition have been used for idealized and real cases in order to show the utility of neighborhoods when comparing at different scales the performances of ensemble forecasts between themselves or with deterministic forecasts or of deterministic forecasts between themselves. Significance Statement A pooling of forecasts on the one hand and observations on the other hand, on all the points of a neighborhood, is performed in order to obtain frequencies at the neighborhood scale. The Brier divergence is then calculated for these neighborhood frequencies to compare a probabilistic forecast and observations at the neighborhood scale. A new decomposition of this score generalizes that of the Brier score and allows the separation of the generalized resolution, reliability, and uncertainty terms. This uncertainty term is used to define the neighborhood Brier divergence skill score which is an alternative to the popular fractions skill score, with a more appropriate denominator.

Decoding the Atmosphere: Optimising Probabilistic Forecasts with Information Gain

Probabilistic prediction models exist to reduce surprise about future events. This paper explores the evaluation of such forecasts when the event of interest is rare. We review how the family of Brier-type scores may be ill-suited to evaluate predictions of rare events, and we offer an alternative to information-theoretical scores such as Ignorance. The reduction in surprise provided by a set of forecasts is represented as information gain, a frequent loss function in machine learning training, meaning the reduction in ignorance over a baseline having received a new forecast. We evaluate predictions of a synthetic dataset of rare events and demonstrate the differences in interpretation of the same datasets depending on whether the Brier or Ignorance score is used. While the two types of scores are broadly similar, there are substantial differences in interpretation at extreme probabilities. Information gain is measured in units of bits, an irreducible unit of information, that allows forecasts of different variables to be comparatively evaluated fairly. Further insight from information-based scores is gained via a similar reliability–discrimination decomposition as found in Brier-type scores. We conclude by crystallising multiple concepts to better equip forecast-system developers and decision-makers with tools to navigate complex trade-offs and uncertainties that characterise meteorological forecasting. To this end, we also provide computer code to reproduce data and figures herein.

Evaluation of debris-flow building damage forecasts

Abstract. Reliable forecasts of building damage due to debris flows may provide situational awareness and guide land and emergency management decisions. Application of debris-flow runout models to generate such forecasts requires combining hazard intensity predictions with fragility functions that link hazard intensity with building damage. In this study, we evaluated the performance of building damage forecasts for the 9 January 2018 Montecito postfire debris-flow runout event, in which over 500 buildings were damaged. We constructed forecasts using either peak debris-flow depth or momentum flux as the hazard intensity measure and applied each approach using three debris-flow runout models (RAMMS, FLO-2D, and D-Claw). Generated forecasts were based on averaging multiple simulations that sampled a range of debris-flow volume and mobility, reflecting typical sources and magnitude of pre-event uncertainty. We found that only forecasts made with momentum flux and the D-Claw model could correctly predict the observed number of damaged buildings and the spatial patterns of building damage. However, the best forecast only predicted 50 % of the observed damaged buildings correctly and had coherent spatial patterns of incorrectly predicted building damage (i.e., false positives and false negatives). These results indicate that forecasts made at the building level reliably reflect the spatial pattern of damage but do not support interpretation at the individual building level. We found the event size strongly influences the number of damaged buildings and the spatial pattern of debris-flow depth and velocity. Consequently, future research on the link between precipitation and the volume of sediment mobilized may have the greatest effect on reducing uncertainty in building damage forecasts. Finally, because we found that both depth and velocity are needed to predict building damage, comparing debris-flow models against spatially distributed observations of building damage is a more stringent test for model fidelity than comparison against the extent of debris-flow runout.

FinOps-driven optimization of cloud resource usage for high-performance computing using machine learning

Cloud computing is gaining popularity in high-performance computing applications. Its utilization enables advanced simulations when local computing resources are limited. However, cloud usage may increase costs and entail resource unavailability risks. This article presents an original approach that employs machine learning to predict long-term cloud resource usage. This enables optimizing resource utilization through appropriate reservation plans, reducing the associated costs. The solution developed utilizes statistical models, XGBoost, neural networks and the Temporal Fusion Transformer. Long-term prediction of cloud resource consumption, especially the Cloud Resource Usage Optimization System that is critical for prolonged simulations, involves using prediction results to dynamically create resource reservation plans across various virtual machine types for HPC on the Google Cloud Platform. Experiments using real-life production data demonstrate that the TFT prediction model improved prediction quality (by 31.4%) compared to the best baseline method, particularly in adapting to chaotic changes in resource consumption. However, it should be noted that the best prediction model in terms of error magnitude might not be the most suitable for resource reservation planning. This was validated by the neural network-based method, introducing an FR metric for forecast evaluation. Resource reservation plans were assessed both qualitatively and quantitatively, focusing on various aspects like a service-level agreement compliance and potential downtime. This paper is an extension of work originally presented during the International Conference on Computational Science — ICCS 2023, entitled “Long-Term Prediction of Cloud Resource Usage in High-Performance Computing”.

Trend-Fourier Time Series Regression Model for Secular-Cyclical Datasets

The study proposed a Trend-Fourier Regression (TFR) model to handle time series datasets with simultaneous trend and cyclical variations. The model steps involve identification, estimation, diagnosis and forecasting. The Nigerian monthly Crude Oil Price (NMCOP) was used to implement the model and NMCOP was identified as trend-cyclical. The model estimation using Ordinary Least Squares method indicates that an increase in time will result in changes in NMCOP. Durbin-Watson statistics, histogram and autocorrelation function of residual plots were used to diagnose and specify the model to be stable. The coefficient of determination (R^2) indicates that over 80% of dependent variable variations were explained, with an adjusted (R^2) indicating a predictive ability exceeding 80%. The model efficiency was confirmed through out-sample and forecast evaluations, revealing superiority due to its smaller MAE, RMSE, and MAPE values, indicating minimal error. Conclusively, the TFR model is suitable for datasets that exhibit trend-cyclical variations simultaneously.

A Rose by Any Other Name: On Basic Scores from the 2 × 2 Table and the Plethora of Names Attached to Them

Abstract Forecast evaluation metrics have been discovered and rediscovered in a variety of contexts, leading to confusion. We look at measures from the 2 × 2 contingency table and the history of their development and illustrate how different fields working on similar problems has led to different approaches and perspectives of the same mathematical concepts. For example, probability of detection (POD) is a quantity in meteorology that was also called prefigurance in the field, while the same thing is named recall in information science and machine learning, and sensitivity and true positive rate in the medical literature. Many of the scores that combine three elements of the 2 × 2 table can be seen as either coming from a perspective of Venn diagrams or from the Pythagorean means, possibly weighted, of two ratios of performance measures. Although there are algebraic relationships between the two perspectives, the approaches taken by authors led them in different directions, making it unlikely that they would discover scores that naturally arose from the other approach. We close by discussing the importance of understanding the implicit or explicit values expressed by the choice of scores. In addition, we make some simple recommendations about the appropriate nomenclature to use when publishing interdisciplinary work.

Challenges of Operational Weather Forecast Verification and Evaluation

Abstract Operational agencies face significant challenges related to the verification and evaluation of weather forecasts. These challenges were investigated in a series of online workshops and polls engaging operational personnel from six countries. Five key themes emerged: inadequate verification approaches for both existing and emerging products; incomplete and uncertain observations; difficulties in accurately capturing users’ real-world experiences using simplified metrics; poor communication and understanding of forecasts and complex verification information; and institutional factors such as limited resources, evolving meteorologist roles, and concerns over reputational damage. We identify nearly 50 operationally relevant scientific questions and suggest calls to action. Addressing these needs includes designing forecast systems with verification as a central consideration, enhancing the availability of observations, and developing and adopting community software systems. Additionally, we propose the establishment of an international community comprising environmental and social science researchers, statisticians, verification practitioners, and users to provide sustained support for this collective endeavor.

Evaluation of Wintertime Precipitation Estimates and Forecasts in the Mountains of Colorado

Abstract Wintertime precipitation poses many observational and forecasting challenges, especially in the complex topography of the western United States where radar beam blockage and difficulty siting in situ observations yields more sparse observations than in the eastern United States. Uncertainty in western U.S. winter precipitation is known to be high, so much so that some studies have found model simulated precipitation to produce similar or better large-scale estimates of annual precipitation than gridded observational products during climatologically anomalous years. This study evaluates high-resolution gridded precipitation estimates from Multi-Radar Multi-Sensor (MRMS) and Stage IV as well as forecasts from NOAA’s High-Resolution Rapid Refresh (HRRR) model in the Colorado Rocky Mountains. Gridded precipitation estimates and forecasts are compared with in situ SNOTEL measurements for two seasons of wintertime precipitation. The influence of forecast length, lead time, and model elevation on seasonal precipitation predictions from the HRRR are investigated. Additional comparisons are made with the relatively dense network of observations deployed in Colorado’s East River Watershed during the Study of Precipitation, the Lower Atmosphere and Surface for Hydrometeorology (SPLASH) campaign. Gridded products and forecasts are found to underestimate cold-season precipitation by 25%–65% relative to in situ and aircraft measurements, with longer forecast periods and lead times (6–24 h) having smaller biases (25%–30%) than shorter forecast periods and lead times (55%–65%). The assessment of multiple years of observations indicates that these biases are related more to the data and methods used to create the gridded products and forecasts than to precipitation characteristics. Significance Statement In the mountainous western United States, it is very challenging to both observe and forecast wintertime precipitation, yet snowfall plays an important role in providing the region’s annual water supply. This study aims to increase our understanding of the biases in observations and forecasts of snowfall in the Colorado Rocky Mountains, which can in turn impact forecasts of water availability for the ensuing warm season. In this study we find high-resolution gridded precipitation estimates and forecasts to underestimate cold-season precipitation when compared with in situ observing stations, with longer-range forecasts (e.g., daily) being the least biased. These findings were consistent over two years of study and have broad implications for the hydrologic modeling and water management communities.

Forecasting the Stock Market Returns Using nonlinear hybrid GARCH-SETAR model

Forecasting stock market returns is a valuable tool for investors seeking to enhance their gains in stock trading. Predicting stock prices proves to be a formidable endeavor due to its substantial volatility, non-linear characteristics trends, and responsiveness to multifaceted variables, including economic conditions, market trends, seasonality, and sentiment. Despite these complexities, non-linear methodologies like threshold time series and conditional heteroscedasticity models are underutilized. This study aims to assess the predictive capabilities of a hybrid GARCH-SETAR model in the context of stock market returns, encompassing both Islamic and conventional stocks listed on the Pakistan Stock Exchange. The Islamic and conventional stock markets in Pakistan represented by KMI-30, KSE-30 and KSE-100 selected contain daily data from January, 2012 till June, 2023. After the confirmation of stationarity, ARCH effect and non-linearity by ADF, PP unit root, ARCH-LM and BDS test respectively the best estimated linear traditional Box-Jenkins ARIMA, non-linear threshold SETAR and ARIMA-GARCH models are selected based on AIC and BIC information criteria’s. The best proposed hybrid GARCH-SETAR model is also selected based on AIC and BIC information criteria’s. It is revealed that the one-step-ahead recursive forecast performance proposed hybrid GARCH-SETAR model outperforms all other selected linear and non-linear models for both Islamic and conventional stock markets based on RMSE, MAE, MAPE and SMAPE forecast evaluation criteria’s.

Overview of potential customer prediction for products based on machine learning

With the development of the Internet, e-commerce, as a new way of online shopping, has provided great convenience to people's lives. Due to the complex potential relationship between consumers and products, it is difficult to recommend products according to consumers' needs, which increases the difficulty of online shopping. Therefore, how to predict the potential consumers of commodities has attracted the wide attention of researchers. Fortunately, machine learning-based approaches have made text and images that can model complex underlying relationships between data. Therefore, researchers have introduced machine learning into the product potential consumer prediction field and achieved good results. This paper first introduces the relevant data set and the product potential users' forecast evaluation index. Then, it summarizes the relevant product potential consumer prediction methods based on machine learning. Finally, the paper summarizes the whole article and looks forward to future research methods.