Forecast Evaluation Research Articles

Global Evaluation of Seasonal Precipitation and Temperature Forecasts from NMME

AbstractWe present a comprehensive global evaluation of monthly precipitation and temperature forecasts from 16 seasonal forecasting models within the NMME Phase-1 system, using Multi-Source Weighted-Ensemble Precipitation version 2 (MSWEP-V2; precipitation) and Climate Research Unit TS4.01 (CRU-TS4.01; temperature) data as reference. We first assessed the forecast skill for lead times of 1–8 months using Kling–Gupta efficiency (KGE), an objective performance metric combining correlation, bias, and variability. Next, we carried out an empirical orthogonal function (EOF) analysis to compare the spatiotemporal variability structures of the forecasts. We found that, in most cases, precipitation skill was highest during the first lead time (i.e., forecast in the month of initialization) and rapidly dropped thereafter, while temperature skill was much higher overall and better retained at higher lead times, which is indicative of stronger temporal persistence. Based on a comprehensive assessment over 21 regions and four seasons, we found that the skill showed strong regional and seasonal dependencies. Some tropical regions, such as the Amazon and Southeast Asia, showed high skill even at longer lead times for both precipitation and temperature. Rainy seasons were generally associated with high precipitation skill, while during winter, temperature skill was low. Overall, precipitation forecast skill was highest for the NASA, NCEP, CMC, and GFDL models, and for temperature, the NASA, CFSv2, COLA, and CMC models performed the best. The spatiotemporal variability structures were better captured for precipitation than temperature. The simple forecast averaging did not produce noticeably better results, emphasizing the need for more advanced weight-based averaging schemes.

State‐dependent evaluation of predictive ability

AbstractThis study systematically broadens the relevance of possible model performance asymmetries across business cycles in the spirit of the recent state‐dependent forecast evaluation literature (e.g. Chauvet & Potter, 2013) to hundreds of macroeconomic indicators and deepens the forecast evaluation of the recent factor model literature on hundreds of target variables (e.g. Stock & Watson, 2012b) in a state‐dependent manner. Our results are consistent with both strands of the literature and generalize the former to over 200 macroeconomic indicators and differentiate the latter across three levels of temporal granularity: We document systematic model performance differences in both absolute and relative terms across business cycles (longitudinal) as well as across variable groups (cross‐sectional) and find these performance differences to be robust across several alternative specifications. The cross‐sectional prevalence and robustness of state dependency shown in this article encourages economic forecasters to complement model performance assessments with a state‐dependent evaluation of predictive ability.

Evaluating quantile-bounded and expectile-bounded interval forecasts

In many different contexts, decision-making is improved by the availability of probabilistic predictions. The accuracy of probabilistic forecasting methods can be compared using scoring functions and insight provided by calibration tests. These tests evaluate the consistency of predictions with the observations. Our main agenda in this paper is interval forecasts and their evaluation. Such forecasts are usually bounded by two quantile forecasts. However, a limitation of quantiles is that they convey no information regarding the size of potential exceedances. By contrast, the location of an expectile is dictated by the whole distribution. This prompts us to propose expectile-bounded intervals. We provide interpretation, a consistent scoring function and a calibration test. Before doing this, we reflect on the evaluation of forecasts of quantile-bounded intervals and expectiles, and suggest extensions of previously proposed calibration tests in order to guard against strategic forecasting. We illustrate ideas using day-ahead electricity price forecasting.

Macroeconomic forecasting for Pakistan in a data-rich environment

ABSTRACT This article forecasts the CPI inflation, GDP growth and the weighted average overnight repurchase rate in Pakistan using 161 predictors covering a period from July 2007 to July 2017. We use the naïve mean model and the autoregressive model as benchmark models and compare their forecasting performance against the dynamic factor model (DFM) and sophisticated machine learning methods such as the Ridge regression, the LASSO, the Elastic net and a few variants of Bagging. The main purpose of the article is to determine, how well the commonly used DFM which has been used for time series forecasting for a long time, performs against the recently developed penalized regression methods in forecasting key macroeconomic variables in Pakistan. We forecast the variables of interest over 12 months forecast horizon. The forecast evaluation criteria used to compare the forecast performance of these models is the RMSE and MASE. For each variable of interest, we find that, for majority of the cases considered, one of the competing approaches outperform the benchmark models and other competing approaches at majority of forecast horizons. Our results show that, on the balance, the machine learning approaches perform better than the benchmark, the autoregressive and the DFM.

Choices in the Verification of S2S Forecasts and Their Implications for Climate Services

AbstractSubseasonal predictions bridge the gap between medium-range weather forecasts and seasonal climate predictions. This time scale is crucial for operations and planning in many sectors such as energy and agriculture. For users to trust these predictions and efficiently make use of them in decision-making, the quality of predicted near-surface parameters needs to be systematically assessed. However, the method to follow in a probabilistic evaluation of subseasonal predictions is not trivial. This study aims to offer an illustration of the impact that the verification setup might have on the calculation of the skill scores, thus providing some guidelines for subseasonal forecast evaluation. For this, several forecast verification setups to calculate the fair ranked probability skill score for tercile categories have been designed. These setups use different number of samples to compute the fair RPSS as well as different ways to define the climatology, characterized by different time periods to average (week or month). These setups have been tested by evaluating 2-m temperature in ECMWF-Ext-ENS 20-yr hindcasts for all of the initializations in 2016 against the ERA-Interim reanalysis. Then, the implications on skill score values of each of the setups are analyzed. Results show that to obtain a robust skill score several start dates need to be employed. It is also shown that a constant monthly climatology over each calendar month may introduce spurious skill score associated with the seasonal cycle. A weekly climatology bears similar results to a monthly running-window climatology; however, the latter provides a better reference climatology when bias adjustment is applied.

Big Data Analytics in the Fight against Major Public Health Incidents (Including COVID-19): A Conceptual Framework.

Major public health incidents such as COVID-19 typically have characteristics of being sudden, uncertain, and hazardous. If a government can effectively accumulate big data from various sources and use appropriate analytical methods, it may quickly respond to achieve optimal public health decisions, thereby ameliorating negative impacts from a public health incident and more quickly restoring normality. Although there are many reports and studies examining how to use big data for epidemic prevention, there is still a lack of an effective review and framework of the application of big data in the fight against major public health incidents such as COVID-19, which would be a helpful reference for governments. This paper provides clear information on the characteristics of COVID-19, as well as key big data resources, big data for the visualization of pandemic prevention and control, close contact screening, online public opinion monitoring, virus host analysis, and pandemic forecast evaluation. A framework is provided as a multidimensional reference for the effective use of big data analytics technology to prevent and control epidemics (or pandemics). The challenges and suggestions with respect to applying big data for fighting COVID-19 are also discussed.

Oil prices and experts’ forecasts of seven exchange rates

ABSTRACT Research indicates that survey-based forecasts of exchange rates fail to be rational. In addition, the literature points to oil prices as a key factor in determining/predicting exchange rates. We investigate these two research topics by focusing on Blue Chip multi-month forecasts of seven exchange rates for 1999–2015. Our investigation utilizes the recent forecast evaluation method that allows for the possibility of asymmetric loss. We show that Blue Chip forecasts, while free of systematic bias, cannot beat the random walk benchmark, and, at some forecast horizons, the errors fail to be orthogonal to changes in oil prices. We further show that, for four out of seven exchange rates, the change in oil prices explains the information content difference between the Blue Chip and random walk forecasts. Put together, our findings suggest that the inaccuracy of Blue Chip forecasts may be in part due to an inefficient use of oil price information available at the time of the forecast and, thus, forecasters should pay special attention to oil price movements when forecasting exchange rates.

A moving average heterogeneous autoregressive model for forecasting the realized volatility of the US stock market: Evidence from over a century of data

AbstractThis study forecasts the monthly realized volatility of the US stock market covering the period of February 1885 to September 2019 using a recently developed novel approach – a moving average heterogeneous autoregressive (MAT‐HAR) model, which treats threshold as a moving average generated time‐varying parameter rather than as a fixed or unknown parameter. The significance of asymmetric information in realized volatility of stock market forecasting is also considered by examining the case of good and bad realized volatility. The Clark and West, Journal of Econometrics, 2007, 138, 291–311 forecast evaluation approach is employed to evaluate the forecast performance of the proposed predictive model vis‐à‐vis the conventional HAR and threshold HAR (T‐HAR) models. We find evidence in favour of the MAT‐HAR model relative to the HAR and T‐HAR models. Also observed is the significant role of asymmetry in modelling the realized volatility as good realized volatility and bad realized volatility yield dissimilar predictability results. Our results are not sensitive to the choice of sample periods and realized volatility measures.

Does high-frequency crude oil futures data contain useful information for predicting volatility in the US stock market? New evidence

Abstract This study examines whether high-frequency crude oil futures data contain useful information to forecast the realized volatility (RV) of the US stock market from both in- and out-of-sample perspectives. There are several significant findings. First, from the in-sample analysis, crude oil futures RV exhibits a significant positive impact on the future S&P 500 volatility. Second, the out-of-sample results reveal that the prediction models, including crude oil futures RV, outperform the related competing models, implying that crude oil RV is an important predictive factor for the US stock market. Third, we further find that the primary forecasting ability of crude oil RV is reflected in high-frequency information, negative crude oil RV, and high volatility level. Finally, the out-of-sample empirical results based on different forecasting windows, alternative forecast evaluation approaches, subsample analysis, different prediction models, alternative MIDAS lags, and controlling the leverage effect are robust to our conclusions.

A Flexible Mixed-Frequency Vector Autoregression with a Steady-State Prior

Abstract We propose a Bayesian vector autoregressive (VAR) model for mixed-frequency data. Our model is based on the mean-adjusted parametrization of the VAR and allows for an explicit prior on the “steady states” (unconditional means) of the included variables. Based on recent developments in the literature, we discuss extensions of the model that improve the flexibility of the modeling approach. These extensions include a hierarchical shrinkage prior for the steady-state parameters, and the use of stochastic volatility to model heteroskedasticity. We put the proposed model to use in a forecast evaluation using US data consisting of 10 monthly and three quarterly variables. The results show that the predictive ability typically benefits from using mixed-frequency data, and that improvement can be obtained for both monthly and quarterly variables. We also find that the steady-state prior generally enhances the accuracy of the forecasts, and that accounting for heteroskedasticity by means of stochastic volatility usually provides additional improvements, although not for all variables.

Evaluation of Precipitation Forecast of System: Numerical Tools for Hurricane Forecast

Heavy rainfall events, typically associated with tropical cyclones (TCs), provoke intense flooding, consequently causing severe losses to life and property. Therefore, the amount and distribution of rain associated with TCs must be forecasted precisely within a reasonable time to guarantee the protection of lives and goods. In this study, the skill of the Numerical Tool for Hurricane Forecast (NTHF) for determining rainfall pattern, average rainfall, rainfall volume, and extreme amounts of rain observed during TCs is evaluated against Tropical Rainfall Measuring Mission (TRMM) data. A sample comprising nine systems formed in the North Atlantic basin from 2016 to 2018 is used, where the analysis begins 24 h before landfall. Several statistical indices characterising the abilities of the NTHF and climatology and persistence model for rainfalls (R-CLIPER) for forecasting rain as measured by the TRMM are calculated at 24, 48, and 72 h forecasts for each TC and averaged. The model under consideration presents better forecasting skills than the R-CLIPER for all the attributes evaluated and demonstrates similar performances compared with models reported in the literature. The proposed model predicts the average rainfall well and presents a good description of the rain pattern. However, its forecast of extreme rain is only applicable for 24 h.

Pseudoprospective Evaluation of UCERF3-ETAS Forecasts during the 2019 Ridgecrest Sequence

ABSTRACTThe 2019 Ridgecrest sequence provides the first opportunity to evaluate Uniform California Earthquake Rupture Forecast v.3 with epidemic-type aftershock sequences (UCERF3-ETAS) in a pseudoprospective sense. For comparison, we include a version of the model without explicit faults more closely mimicking traditional ETAS models (UCERF3-NoFaults). We evaluate the forecasts with new metrics developed within the Collaboratory for the Study of Earthquake Predictability (CSEP). The metrics consider synthetic catalogs simulated by the models rather than synoptic probability maps, thereby relaxing the Poisson assumption of previous CSEP tests. Our approach compares statistics from the synthetic catalogs directly against observations, providing a flexible approach that can account for dependencies and uncertainties encoded in the models. We find that, to the first order, both UCERF3-ETAS and UCERF3-NoFaults approximately capture the spatiotemporal evolution of the Ridgecrest sequence, adding to the growing body of evidence that ETAS models can be informative forecasting tools. However, we also find that both models mildly overpredict the seismicity rate, on average, aggregated over the evaluation period. More severe testing indicates the overpredictions occur too often for observations to be statistically indistinguishable from the model. Magnitude tests indicate that the models do not include enough variability in forecasted magnitude-number distributions to match the data. Spatial tests highlight discrepancies between the forecasts and observations, but the greatest differences between the two models appear when aftershocks occur on modeled UCERF3-ETAS faults. Therefore, any predictability associated with embedding earthquake triggering on the (modeled) fault network may only crystalize during the presumably rare sequences with aftershocks on these faults. Accounting for uncertainty in the model parameters could improve test results during future experiments.

Predictability of China’s Stock Market Returns Based on Combination of Distribution Forecasting Models

No consensus exists in the literature on whether stock prices can be predicted, with most existing studies employing point forecasting to predict returns. By contrast, this study adopts the new perspective of distribution forecasting to investigate the predictability of the stock market using the model combination strategy. Specifically, the Shanghai Composite Index and the Shenzhen Component Index are selected as research objects. Seven models – GARCH-norm, GARCH-sstd, EGARCH-sstd, EGARCH-sstd-M, one-component Beta-t-EGARCH, two-component Beta-t-EGARCH, and the EWMA-based nonparametric model – are employed to perform distribution forecasting of the returns. The results of out-of-sample forecasting evaluation show that none of the individual models is “qualified” in terms of predictive power. Therefore, three combinations of individual models were constructed: equal weight combination, log-likelihood score combination, and continuous ranked probability score combination. The latter two combinations were found to always have significant directional predictability and excess profitability, which indicates that the two combined models may be closer to the real data generation process; from the perspective of economic evaluation, they may have a predictive effect on the conditional return distribution in China’s stock market.

Path Properties of Atmospheric Transitions: Illustration with a Low-Order Sudden Stratospheric Warming Model

AbstractMany rare weather events, including hurricanes, droughts, and floods, dramatically impact human life. To accurately forecast these events and characterize their climatology requires specialized mathematical techniques to fully leverage the limited data that are available. Here we describe transition path theory (TPT), a framework originally developed for molecular simulation, and argue that it is a useful paradigm for developing mechanistic understanding of rare climate events. TPT provides a method to calculate statistical properties of the paths into the event. As an initial demonstration of the utility of TPT, we analyze a low-order model of sudden stratospheric warming (SSW), a dramatic disturbance to the polar vortex that can induce extreme cold spells at the surface in the midlatitudes. SSW events pose a major challenge for seasonal weather prediction because of their rapid, complex onset and development. Climate models struggle to capture the long-term statistics of SSW, owing to their diversity and intermittent nature. We use a stochastically forced Holton–Mass-type model with two stable states, corresponding to radiative equilibrium and a vacillating SSW-like regime. In this stochastic bistable setting, from certain probabilistic forecasts TPT facilitates estimation of dominant transition pathways and return times of transitions. These “dynamical statistics” are obtained by solving partial differential equations in the model’s phase space. With future application to more complex models, TPT and its constituent quantities promise to improve the predictability of extreme weather events through both generation and principled evaluation of forecasts.

A Study on the Development of Rainfall Risk Information Considering Local Characteristics

Natural disasters caused by climate change frequently occur around the world. In Korea, flood warnings and storm warnings are used to prepare for flood damage. However, flood warnings are specifically issued for rivers while storm warnings use the same criteria throughout the country. Accordingly, there is a limit to effective flood preparation which reflects the regional characteristics of each administrative district. In this study, the flood risk matrix for each administrative district is developed in order to provide flood risk forecast information that reflects upon local characteristics. Moreover, the applicability is evaluated by using dichotomous forecasting. As a result of the forecast's evaluation, the probability of detection was determined to be over 80%. Therefore, it is possible to prepare for a flood through the forecast information of the administrative district using the flood risk matrix. In the future, it will be necessary to study how to improve the utilization of flood risk forecasting information through the continuous investigation of past damage phenomenon along with a review of flood-influencing factors.

The Year of Polar Prediction in the Southern Hemisphere (YOPP-SH)

AbstractThe Year of Polar Prediction in the Southern Hemisphere (YOPP-SH) had a special observing period (SOP) that ran from 16 November 2018 to 15 February 2019, a period chosen to span the austral warm season months of greatest operational activity in the Antarctic. Some 2,200 additional radiosondes were launched during the 3-month SOP, roughly doubling the routine program, and the network of drifting buoys in the Southern Ocean was enhanced. An evaluation of global model forecasts during the SOP and using its data has confirmed that extratropical Southern Hemisphere forecast skill lags behind that in the Northern Hemisphere with the contrast being greatest between the southern and northern polar regions. Reflecting the application of the SOP data, early results from observing system experiments show that the additional radiosondes yield the greatest forecast improvement for deep cyclones near the Antarctic coast. The SOP data have been applied to provide insights on an atmospheric river event during the YOPP-SH SOP that presented a challenging forecast and that impacted southern South America and the Antarctic Peninsula. YOPP-SH data have also been applied in determinations that seasonal predictions by coupled atmosphere–ocean–sea ice models struggle to capture the spatial and temporal characteristics of the Antarctic sea ice minimum. Education, outreach, and communication activities have supported the YOPP-SH SOP efforts. Based on the success of this Antarctic summer YOPP-SH SOP, a winter YOPP-SH SOP is being organized to support explorations of Antarctic atmospheric predictability in the austral cold season when the southern sea ice cover is rapidly expanding.

Monitoring atmospheric composition and air quality using commercial aircraft - IAGOS

IAGOS (In-service Aircraft for a Global Observing System) is a European Research Infrastructure for global observations of atmospheric composition using commercial aircraft. Commercial aircraft are ideal platforms for providing long-term in-situ measurements with high vertical and temporal resolution, particularly at cruise altitude (between 9 and 13 km) in the climate-sensitive region of the atmosphere known as the upper troposphere-lower stratosphere (UTLS). IAGOS also provides landing and take-off profiles at almost 300 airports throughout the world which are of major interest for air quality issues. Fully automated instruments are permanently installed on Airbus A330 aircraft operated by different airlines. Data are collected on about 500 flights per aircraft per year. All the aircraft measure the abundances of many essential climate variables, chiefly ozone and the precursor carbon monoxide, water vapour, clouds and meteorological parameters. Additional instruments can be installed to measure either total odd nitrogen, nitrogen oxides, aerosols, or the greenhouse gases carbon dioxide and methane. The data are transmitted in near to real real time to weather services and are freely available for the scientific community, national air quality prediction centres and the Copernicus Atmosphere Monitoring Service (CAMS). We describe the importance of these measurements in the monitoring of global atmospheric composition and air quality. In particular, we show examples from the Copernicus Atmosphere Monitoring Service (CAMS) where IAGOS data are used in the evaluation and improvement of forecasts of air quality over Europe.

A Similarity-Based Approach for Macroeconomic Forecasting

SummaryIn the aftermath of the recent financial crisis there has been considerable focus on methods for predicting macroeconomic variables when their behaviour is subject to abrupt changes, associated for example with crisis periods. We propose similarity-based approaches as a way to handle parameter instability and apply them to macroeconomic forecasting. The rationale is that clusters of past data that match the current economic conditions can be more informative for forecasting than the entire past behaviour of the variable of interest. We apply our methods to predict both simulated data in a set of Monte Carlo experiments, and a broad set of key US macroeconomic indicators. The forecast evaluation exercises indicate that similarity-based approaches perform well, in general, in comparison with other common time-varying forecasting methods, and particularly well during crisis episodes.

Multimodel Ensemble Forecasts of Precipitation Based on an Object-Based Diagnostic Evaluation

Abstract We analyzed 24-h accumulated precipitation forecasts over the 4-month period from 1 May to 31 August 2013 over an area located in East Asia covering the region 15.05°–58.95°N, 70.15°–139.95°E generated with the ensemble prediction systems (EPS) from ECMWF, NCEP, UKMO, JMA, and CMA contained in the TIGGE dataset. The forecasts are first evaluated with the Method for Object-Based Diagnostic Evaluation (MODE). Then a multimodel ensemble (MME) forecast technique that is based on weights derived from object-based scores is investigated and compared with the equally weighted MME and the traditional gridpoint-based MME forecast using weights derived from the point-to-point metric, mean absolute error (MAE). The object-based evaluation revealed that attributes of objects derived from the ensemble members of the five individual EPS forecasts and the observations differ consistently. For instance, their predicted centroid location is more southwestward, their shape is more circular, and their orientation is more meridional than in the observations. The sensitivity of the number of objects and their attributes to methodological parameters is also investigated. An MME prediction technique that is based on weights computed from the object-based scores, median of maximum interest, and object-based threat score is explored and the results are compared with the ensemble forecasts of the individual EPS, the equally weighted MME forecast, and the traditional superensemble forecast. When using MODE statistics for the forecast evaluation, the object-based MME prediction outperforms all other predictions. This is mainly because of a better prediction of the objects’ centroid locations. When using the precipitation-based fractions skill score, which is not used in either of the weighted MME forecasts, the object-based MME forecasts are slightly better than the equally weighted MME forecasts but are inferior to the traditional superensemble forecast that is based on weights derived from the point-to-point metric MAE.

The Performance of a Revised Simplified Arakawa–Schubert (SAS) Convection Scheme in the Medium-Range Forecasts of the Korean Integrated Model (KIM)

Abstract The Korea Institute of Atmospheric Prediction Systems (KIAPS) has developed a new global numerical weather prediction model, named the Korean Integrated Model (KIM). This paper presents the cumulus parameterization scheme (CPS) used in KIM, which originates from the simplified Arakawa–Schubert (SAS) convection scheme in the National Centers for Environmental Prediction (NCEP) Global Forecast System (GFS) and has undergone numerous modifications in an effort to improve the medium-range forecast skill for precipitation and large-scale fields. The modifications include the following: 1) the threshold of the trigger condition is updated to consider the dependency on the environmental relative humidity (RH) averaged over the subcloud layer in order to suppress the trigger of convection in dry low-level environments; 2) the entrainment rate is modified to increase the sensitivity to environmental humidity, so that enhanced entrainment under lower RH conditions leads to a greater decrease in the strength of the convection that develops in drier environments; 3) the autoconversion parameter from cloud condensate to convective precipitation is changed to have a temperature dependency above the freezing level; 4) the closure is modified to consider rapidly varying boundary layer forcing; 5) the effect of the convection-induced pressure gradient force in convective momentum transport is enhanced in the upper part of the convective updrafts; and 6) scale awareness that enables a mass-flux CPS to work seamlessly at various grid sizes across gray-zone resolutions is addressed. The evaluation of medium-range forecasts with the KIM CPS reveals higher forecast skill, especially over the tropics, in comparison with its original version.