Forecast Performance Research Articles

Step by Step—A Quarterly Evaluation of EU Commission's GDP Forecasts

ABSTRACTThe European Commission's growth forecasts play a crucial role in shaping policies and provide a benchmark for many (national) forecasters. The annual forecasts are built on quarterly estimates, which do not receive much attention and are hardly known. Therefore, this paper provides a comprehensive analysis of multiperiod ahead quarterly GDP growth forecasts for the European Union (EU), euro area, and several EU member states with respect to first‐release and current‐release data. Forecast revisions and forecast errors are analyzed, and the results show that the forecasts are not systematically biased. However, GDP forecasts for several member states tend to be overestimated at short‐time horizons. Furthermore, the final forecast revision in the current quarter is generally downward biased for almost all countries. Overall, the differences in mean forecast errors are minor when using real‐time data or pseudo–real‐time data and these differences do not significantly impact the overall assessment of the forecasts' quality. Additionally, the forecast performance varies across countries, with smaller countries and Central and Eastern European countries (CEECs) experiencing larger forecast errors. The paper provides evidence that there is still potential for improvement in forecasting techniques both for nowcasts but also forecasts up to eight quarters ahead. In the latter case, the performance of the mean forecast tends to be superior for many countries.

Probabilistic weather forecasting with machine learning.

Weather forecasts are fundamentally uncertain, so predicting the range of probable weather scenarios is crucial for important decisions, from warning the public about hazardous weather to planning renewable energy use. Traditionally, weather forecasts have been based on numerical weather prediction (NWP)1, which relies on physics-based simulations of the atmosphere. Recent advances in machine learning (ML)-based weather prediction (MLWP) have produced ML-based models with less forecast error than single NWP simulations2,3. However, these advances have focused primarily on single, deterministic forecasts that fail to represent uncertainty and estimate risk. Overall, MLWP has remained less accurate and reliable than state-of-the-art NWP ensemble forecasts. Here we introduce GenCast, a probabilistic weather model with greater skill and speed than the top operational medium-range weather forecast in the world, ENS, the ensemble forecast of the European Centre for Medium-Range WeatherForecasts4. GenCast is an ML weather prediction method, trained on decades of reanalysis data. GenCast generates an ensemble of stochastic 15-day global forecasts, at 12-h steps and 0.25° latitude-longitude resolution, for more than 80 surface and atmospheric variables, in 8 min. It has greater skill than ENS on 97.2% of 1,320 targets we evaluated and better predicts extreme weather, tropical cyclone tracks and wind power production. This work helps open the next chapter in operational weather forecasting, in which crucial weather-dependent decisions are made more accurately and efficiently.

Diel temperature patterns unveiled: High-frequency monitoring and deep learning in lake Kasumigaura

The diel variation of water temperatures is crucial information for lakes. This study investigated such variation in Lake Kasumigaura, the second largest lake in Japan, situated 60 km from the Tokyo metropolitan area, utilizing high-frequency monitoring and deep learning techniques. Data from high-frequency monitoring of vertical water temperatures at seven depths were employed. A convolutional neural network (CNN) based deep learning model was developed and assessed across three input scenarios. The impact of forecast horizons ranging from one to 48 h ahead on model performance was examined. Results indicate a degradation in model performance with increasing forecast horizons, irrespective of input scenario or water depth. Notably, the CNN model demonstrates superior performance in near-term and medium-term forecasts compared to long-term predictions, underscoring the need for enhanced efforts in long-term forecasting. Overall, the CNN model effectively reproduces vertical water temperature profiles and captures diel variations in lake water temperatures. Incorporating additional input variables does not necessarily improve model performance; however, using surface water temperatures and air temperatures as inputs produces acceptable results for modeling vertical temperature profiles. These findings have implications for lake management in Lake Kasumigaura (e.g., controlling phytoplankton and zooplankton communities) and offer insights into water temperature modeling for other lakes.

Dissipation Scaled Internal Wave Drag in a Global Heterogeneously Coupled Internal/External Mode Total Water Level Model

AbstractThis study showcases a global, heterogeneously coupled total water level system wherein salinity and temperature outputs from a coarser‐resolution (12 km) ocean general circulation model are used to calculate density‐driven terms within a global, higher‐resolution (2.5 km) depth‐averaged total water level model. We demonstrate that the inclusion of baroclinic forcing in the barotropic model requires modification of the internal wave drag term to prevent excess degradation of tidal results compared to the barotropic model. By scaling the internal tide dissipation by an easy to calculate dissipation ratio, the resulting heterogeneously coupled model has complex root mean square errors (RMSE) of 2.27 cm in the deep ocean and 12.16 cm in shallow waters for the tidal constituent. While this represents a 10%–20% deterioration as compared to the barotropic model, the improvements in total water level prediction more than offset this degradation. Global median RMSE compared to observations of total water levels, 30‐day sea levels, and non‐tidal residuals improve by 1.86 (18.5%), 2.55 (42.5%), and 0.36 (5.3%) cm respectively. The drastic improvement in model performance highlights the importance of including density‐driven effects within global hydrodynamic models and will help to improve the results of both hindcasts and forecasts in modeling extreme and nuisance flooding. With only an 11% increase in model run time compared to the fully barotropic total water level model, this approach paves the way for high resolution coastal water level and flood models to be used alongside climate models, improving operational forecasting of total water levels.

Using Simulated Radiances to Understand the Limitations of Satellite‐Retrieved Volcanic Ash Data and the Implications for Volcanic Ash Cloud Forecasting

AbstractVolcanic Ash Advisory Centers (VAACs) have generated volcanic ash forecasts for the aviation industry since the mid‐1990s. The excellent spatial and temporal coverage of satellite data makes them critical to the validation of ash dispersion model forecasts. This study investigates the limitations of satellite‐retrieved volcanic ash data through the production of simulated radiances for a range of ash cloud properties encompassing the satellite retrieval's sensitivity. We run a detection and retrieval algorithm (Francis et al., 2012, https://doi.org/10.1029/2011JD016788) on these simulated ash clouds and assess the sensitivity and performance of the algorithms. Expected limitations are highlighted, including a lack of sensitivity to particles larger than ∼10 μm in radius and challenges in accurately retrieving heights in the stratosphere. However, other previously poorly defined limitations are also constrained, such as the reduction in sensitivity as ash column loading increases in optically thick ash clouds and increasingly underestimated column loading when column loadings are >∼7 g m−2. We consider the implications of the identified limitations when using satellite‐retrieved ash column loadings to verify dispersion model output. We show that, accounting for the limitations of the satellite retrieval, a significant proportion of mass in the model output can lie outside the sensitivity range of the satellite detection and retrieval. This demonstrates the importance of understanding observations' limitations when comparing to model output. This knowledge should be used when verifying operational volcanic ash cloud forecasts.

Lightning-Fast Convective Outlooks: Predicting Severe Convective Environments With Global AI-Based Weather Models.

Severe convective storms are among the most dangerous weather phenomena and accurate forecasts mitigate their impacts. The recently released suite of AI-based weather models produces medium-range forecasts within seconds, with a skill similar to state-of-the-art operational forecasts for variables on single levels. However, predicting severe thunderstorm environments requires accurate combinations of dynamic and thermodynamic variables and the vertical structure of the atmosphere. Advancing the assessment of AI-models toward process-based evaluations lays the foundation for hazard-driven applications. We assess the forecast skill of the top-performing AI-models GraphCast, Pangu-Weather and FourCastNet for convective parameters at lead-times up to 10days against reanalysis and ECMWF's operational numerical weather prediction model IFS. In a case study and seasonal analyses, we see the best performance by GraphCast and Pangu-Weather: these models match or even exceed the performance of IFS for instability and shear. This opens opportunities for fast and inexpensive predictions of severe weather environments.

Bayesian Inference for Long Memory Stochastic Volatility Models

We explore the application of integrated nested Laplace approximations for the Bayesian estimation of stochastic volatility models characterized by long memory. The logarithmic variance persistence in these models is represented by a Fractional Gaussian Noise process, which we approximate as a linear combination of independent first-order autoregressive processes, lending itself to a Gaussian Markov Random Field representation. Our results from Monte Carlo experiments indicate that this approach exhibits small sample properties akin to those of Markov Chain Monte Carlo estimators. Additionally, it offers the advantages of reduced computational complexity and the mitigation of posterior convergence issues. We employ this methodology to estimate volatility dependency patterns for both the SP&500 index and major cryptocurrencies. We thoroughly assess the in-sample fit and extend our analysis to the construction of out-of-sample forecasts. Furthermore, we propose multi-factor extensions and apply this method to estimate volatility measurements from high-frequency data, underscoring its exceptional computational efficiency. Our simulation results demonstrate that the INLA methodology achieves comparable accuracy to traditional MCMC methods for estimating latent parameters and volatilities in LMSV models. The proposed model extensions show strong in-sample fit and out-of-sample forecast performance, highlighting the versatility of the INLA approach. This method is particularly advantageous in high-frequency contexts, where the computational demands of traditional posterior simulations are often prohibitive.

Prediction of Pollutant Emissions from a Low-Speed Marine Engine Based on Harris Hawks Optimization and Lightgbm

With the rapid development of data science, machine learning has been widely applied to research on pollutant emission prediction in internal combustion engines due to its excellent responsiveness and generalization ability. This article introduces Lightgbm (LGB), which belongs to ensemble learning, to predict the pollutant emissions from a low-speed two-stroke marine engine. The dataset used to train LGB was derived from a one-dimensional performance simulation model of the engine, which was rigorously verified for its reliability by experimental data. To further improve the forecast performance of the LGB model, we used Harris Hawks Optimization (HHO) to automatically optimize the hyperparameters of the model, and finally, we analyzed the importance of the model features. The results show that changes in engine control parameters have significant influences on NOx and soot emissions from the engine, which can serve as the basis for the selection of the LGB model features; the LGB model was able to accurately predict pollutant concentrations from the engine with much higher accuracy than a single decision tree (DT) model; combining with HHO, the predictive ability of the LGB model was significantly improved, such as for the validation set prediction results, the mean absolute error (MAE) was reduced by about 20%, the mean squared error (MSE) was reduced by about 30%, and the coefficient of determination (R2) was increased by about 0.005; and the importance analysis of the model features indicated that the combustion condition of the fuel was highly correlated with the generation of the pollutants, and the fuel injection phases can be adjusted in practice to achieve highly efficient and low-emission processes of combustion. The results of this study can provide references for the development of a new generation of highly efficient and low-pollution marine engines.

Regime‐Switching Density Forecasts Using Economists' Scenarios

ABSTRACTWe propose an approach for generating macroeconomic density forecasts that incorporate information on multiple scenarios defined by experts. We adopt a regime‐switching framework in which sets of scenarios (“views”) are used as Bayesian priors on economic regimes. Predictive densities coming from different views are then combined by optimizing objective functions of density forecasting. We illustrate the approach with an empirical application to quarterly real‐time forecasts of the US GDP growth rate, in which we exploit the Fed's macroeconomic scenarios used for bank stress tests. We show that the approach achieves good accuracy in terms of average predictive scores and good calibration of forecast distributions. Moreover, it can be used to evaluate the contribution of economists' scenarios to density forecast performance.

An overview of the effects of algorithm use on judgmental biases affecting forecasting

In the realm of forecasting, judgmental biases often hinder efficiency and accuracy. Algorithms present a promising avenue for decision makers to enhance their forecasting performance. In this overview, we scrutinized the occurrence of the most relevant judgmental biases affecting forecasting across 162 papers, drawing from four recent reviews and papers published in forecasting journals, specifically focusing on the use of algorithms. Thirty-three of the 162 papers (20.4%) at least briefly mentioned one of twelve judgmental biases affecting forecasting. Our comprehensive analysis suggests that algorithms can potentially mitigate the adverse impacts of biases inherent in human judgment related to forecasting. Furthermore, these algorithms can leverage biases as an advantage, enhancing forecast accuracy. Intriguing revelations have surfaced, focusing mainly on four biases. By providing timely, relevant, well-performing, and consistent algorithmic advice, people can be effectively influenced to improve their forecasts, considering anchoring, availability, inconsistency, and confirmation bias. The findings highlight the gaps in the current research landscape and provide recommendations for practitioners. They also lay the groundwork for future studies on utilizing algorithms (e.g., large language models) and overcoming judgmental biases to improve forecasting performance.

A novel U-LSTM-AFT model for hourly solar irradiance forecasting

Countries emphasize the development of renewable energy to combat climate change, ensure energy security, and promote sustainable economic growth. Grid-connected photovoltaic (PV) power generation, a key to energy transition, can significantly reduce power generation costs. Still, its output power is unstable due to the weather, so an accurate solar irradiance forecast is crucial. The current hybrid model mainly suffers from inadequate feature extraction and insufficient time-dependent processing when dealing with complex time series data, which affects prediction accuracy. Therefore, a new solar irradiance forecast model, U-Shaped LSTM-AFT (U-LSTM-AFT) is proposed in this paper, which draws on the architecture and ideas of U-Net. The model improves the efficiency of feature extraction through the up-sampling and down-sampling modules while combining with smaller pooling kernels to optimize feature processing further. In addition, the model introduces the Long Short-Term Memory Network (LSTM) and Attention-Free Transformer (AFT) to enhance prediction accuracy and achieve more efficient forecast performance. The method was validated using three real-world irradiance datasets from diverse US locations representing different climate types. The experimental results demonstrate that the proposed U-LSTM-AFT model achieves higher forecasting accuracy in predicting PV power generation than traditional models like LSTM and U-Net, with significant performance improvements. Mean Absolute Error (MAE), Root Mean Square Error (RMSE), Normalized Root Mean Square Error (nRMSE), R-squared (R2), and Forecast Skill (FSnRMSE) are the performance evaluation metrics of this forecasting scheme. The three datasets reached mean values of 23.076 W/m2, 50.142 W/m2, 0.046, 98.680 %, and 0.238, respectively.

Valid time shifting ensemble Kalman filter (VTS-EnKF) for dust storm forecasting

Abstract. Dust storms pose significant risks to health and property, necessitating accurate forecasting for preventive measures. Despite advancements, dust models grapple with uncertainties arising from emission and transport processes. Data assimilation addresses these by integrating observations to rectify model error, enhancing forecast precision. The ensemble Kalman filter (EnKF) is a widely used assimilation algorithm that effectively optimize model states, particularly in terms of intensity adjustment. However, the EnKF's efficacy is challenged by position errors between modeled and observed dust features, especially under substantial position errors. This study introduces the valid time shifting ensemble Kalman filter (VTS-EnKF), which combines stochastic EnKF with a valid time shifting mechanism. By recruiting additional ensemble members from neighboring valid times, this method not only accommodates variations in dust load but also explicitly accounts for positional uncertainties. Consequently, the enlarged ensemble better represents both the intensity and positional errors, thereby optimizing the utilization of observational data. The proposed VTS-EnKF was evaluated against two severe dust storm cases from spring 2021, demonstrating that position errors notably deteriorated forecast performance in terms of root mean square error (RMSE) and normalized mean bias (NMB), impeding the EnKF's effective assimilation. Conversely, the VTS-EnKF improved both the analysis and forecast accuracy compared to the conventional EnKF. Additionally, to provide a more rigorous assessment of its performance, experiments were conducted using fewer ensemble members and different time intervals.

Enhancing flood forecasting and warning precision through multi-task deep learning approaches

ABSTRACT This study proposes a multi-task deep learning model for simultaneous prediction of time-series water levels and flood risk thresholds, aiming to enhance flood forecasting precision. Using AutoKeras, single-task and multi-task models were optimised to predict water levels 10–360 min ahead based on 720 min of prior data. The multi-task model consistently outperformed the single-task model across multiple evaluation metrics, including correlation coefficients, RMSE, Nash–Sutcliffe efficiency, and Kling–Gupta efficiency scores. Real-time prediction tests on actual rainfall events further validated the multi-task model's improved accuracy and applicability in operational flood forecasting. The study demonstrates significant progress in flood prediction methodologies, offering a more comprehensive approach to forecasting and categorising flood incidents.

Tropical Interbasin Interaction as Effective Predictors of Late-Spring Precipitation Variability in the Southern Great Plains

Abstract The southern Great Plains experience fluctuating precipitation extremes that significantly impact agriculture and water management. Despite ongoing efforts to enhance forecast accuracy, the underlying causes of these climatic phenomena remain inadequately understood. This study elucidates the relative influence of the tropical Pacific and Atlantic basins on April–May–June precipitation variability in this region. Our partial ocean assimilation experiments using the Community Earth System Model unveil the prominent role of interbasin interaction, with the Pacific and Atlantic contributing approximately 70% and 30%, respectively, to these interbasin contrasts. Our statistical analyses suggest that these tropical interbasin contrasts could serve as a more reliable indicator for late-spring precipitation anomalies than El Niño–Southern Oscillation. The conclusions are reinforced by analyses of seven climate forecasting systems within the North American Multi-Model Ensemble, offering an optimistic outlook for enhancing real-time forecasting of late-spring precipitation in the southern plains. However, the current predictive skills of the interbasin contrasts across the prediction systems are hindered by the lower predictability of the tropical Atlantic Ocean, pointing to the need for future research to refine climate prediction models further. Significance Statement Agriculture and infrastructure in the southern plains face challenges from severe late-spring precipitation extremes. Traditional predictors like El Niño–Southern Oscillation (ENSO) lose effectiveness during the critical spring-to-summer transition, creating a forecasting gap. This study introduces the concept of tropical interbasin interactions, known to enhance seasonal predictability for late-spring precipitation in the southern plains. Novel climate model experiments highlight contributions from the tropical Pacific and Atlantic, offering a promising predictability that potentially surpasses the limitations of ENSO-based predictions. These outcomes hold the potential for developing operational forecasts of late-spring precipitation anomalies in the southern plains, enabling proactive risk management.

Automated operational forecasting of monsoon low pressure systems

Automated operational forecasting of monsoon low pressure systems

Temporal cross‐validation in forecasting: A case study of COVID‐19 incidence using wastewater data

AbstractTwo predominant methodologies in forecasting temporal processes include traditional time series models and machine learning methods. This paper investigates the impact of time series cross‐validation (TSCV) on both approaches in the context of a case study predicting the incidence of COVID‐19 based on wastewater data. The TSCV framework outlined in the paper begins by engineering interpretable features hypothesized as potential predictors of COVID‐19 incidence. Feature selection and hyperparameter tuning are then utilized with TSCV to identify the best features and hyperparameters for optimal model performance given a specific forecast horizon. While evidence supporting the utility of TSCV for auto‐regressive integrated moving average model with exogenous variables (TS‐ARIMAX) forecasts is lacking in this study, such an approach proves advantageous for gradient boosting machine forecasts (TS‐GBM). In Wyoming, for instance, TS‐GBM had a 34.9% improvement compared to naïve predictions, whereas GBM without TSCV only had a 15.6% improvement. However, TSCV also enhances interpretability for both TS‐ARIMAX and TS‐GBM models as this approach selects specific features, such as lagged values of COVID‐19 cases, based on forecast performance and forecast length. Future research should work to explore the influence of stationarity and model averaging on the performance of TSCV in forecasting applications.

Loan Approval Prediction Improved by XGBoost Model Based on Four-Vector Optimization Algorithm

Abstract. This paper discusses an improved XGBoost model based on four-vector optimization algorithm to improve the accuracy of loan approval prediction. Through the analysis of correlation heat maps, we found that there were significant positive and negative correlations among some variables, which laid the foundation for subsequent machine learning analysis. Based on this, we compare the traditional machine learning algorithm with the improved model to evaluate its performance in loan approval forecasting. In the confusion matrix analysis of the training set, the improved XGBoost model demonstrated excellent performance, with all loan approval predictions being correct with 100% accuracy. However, the performance in the test set was slightly different, with 1,182 projects receiving correct loan approval predictions and 99 projects forecasting errors. Of these, 26 projects that should have been predicted to be "unapproved" were incorrectly labeled as "approved," while 73 projects that should have been predicted to be "approved" were incorrectly labeled as "unapproved." These results suggest that we still need to pay attention to the misjudgment of the model in practical application. By synthesizing all model evaluation indicators, we found that the improved XGBoost model based on the four-vector optimization algorithm has a higher accuracy in loan approval prediction than the traditional XGBoost model, with an increase of 1.5%. In addition, the other evaluation indicators also show a trend of significantly better than the traditional model. This study shows that the four-vector optimization algorithm can effectively improve the application effect of XGBoost model in the field of loan approval, and provide more accurate data support and decision-making basis for the financial industry. In the future, we will continue to explore the potential and application prospects of this algorithm in other fields.

Comparative Study on Forecast Performance from Decomposition, Winter’s and Sarima Models

This study investigates the forecasting accuracy of three univariate time series models—Decomposition, Winter’s method, and Seasonal Autoregressive Integrated Moving Average (SARIMA) to predict the Agricultural GDP of Nigeria. Quarterly data on Nigeria's Agricultural GDP from 2010 to the first quarter of 2023, obtained from the National Bureau of Statistics, were analyzed. The study applied Box-Jenkins SARIMA modeling, time series decomposition, and Winter’s method to compare their forecasting accuracy using Root Mean Square Error (RMSE) as the selection criterion. The results revealed that the SARIMA (0, 0, 2)(2, 1, 0) model outperformed the other methods, with the lowest RMSE, indicating its superior accuracy in forecasting Agricultural GDP. Winter’s method and the Decomposition method. The forecast from the SARIMA model indicated a positive trend in Nigeria’s Agricultural GDP over the study period, reinforcing the sector’s critical role in economic growth.

Subseasonal-to-seasonal (S2S) prediction of atmospheric rivers in the Northern Winter

Atmospheric rivers (ARs) are characterized by intense lower tropospheric plumes of moisture transport that are frequently responsible for midlatitude wind and precipitation extremes. The prediction of ARs at subseasonal-to-seasonal (S2S) timescales is currently at a low level of skill, reflecting a need to improve our understanding of their underlying sources of predictability. Based on 20 year hindcast experiments from the Geophysical Fluid Dynamics Laboratory’s SPEAR S2S forecast system, we evaluate the S2S prediction skill of AR activities in the northern winter. Higher forecast skill is detected for high-frequency AR activities (3–7 days/week) compared to low-frequency AR activities (1–2 days/week), even though the occurrence rate of high-frequency ARs exceeds that of low-frequency ARs. For the first time, we have applied the Average Predictability Time technique to the SPEAR system to identify the three most predictable modes of AR in the North Pacific sector. These modes can be attributed to the influences of the El Niño–Southern Oscillation, the Pacific North American pattern, and the Arctic Oscillation. S2S AR forecast skill in western United States is modulated by various phases of large-scale variability. This study highlights potential windows of opportunity for operational S2S AR forecasting.

Confidence interval estimation for reliability of a two parameter Weibull distribution under complete and censored samples

Estimating the reliability function of the two parameter Weibull distribution is a critical task in various fields, providing essential insights into product longevity and optimizing maintenance strategies. Due to its adaptable parameters namely scale parameter α and shape parameter β, the Weibull distribution models diverse failure behaviours, from early-life defects to wear-out periods. By evaluating reliability at key time phases namely early life (t < 0.1α), useful life (0.1α ≤ t ≤ α), and end-of-life (t > α), this estimation process supports quality control, operational forecasting, and end-of-life planning. Such reliability assessments help minimize operational disruptions, improve cost management, and support strategic planning across industries where dependability and lifecycle management are essential, such as in manufacturing, healthcare, and technology sectors. Although it holds practical significance, the estimation of confidence interval (CI) for the reliability function of two parameter Weibull distribution has been relatively underexplored in the literature. This paper presents a new approach for constructing CI for the reliability function of the Weibull distribution using the generalized variable (GV) technique, applicable to both complete samples and type II singly right-censored samples. The empirical evaluation of this method indicates that it provides coverage probabilities that are closely aligned with the nominal level, even when dealing with small uncensored samples (as small as 5) and censored samples where the proportion of censored observations can reach 70%. In comparison, traditional methods for the Weibull distribution tend to yield less reliable or widely varied coverage probabilities for complete samples. The findings are demonstrated through practical examples. Keywords: Reliability, Confidence interval, Generalized Variable approach, Censoring.