Forecasting Competition Research Articles

An adaptive volatility method for probabilistic forecasting and its application to the M6 financial forecasting competition

In this paper, we address the problem of probabilistic forecasting using an adaptive volatility method rooted in classical time-varying volatility models and leveraging online stochastic optimization algorithms. These principles were successfully applied in the M6 forecasting competition under the team named AdaGaussMC. Our approach takes a unique path by embracing the Efficient Market Hypothesis (EMH) instead of trying to beat the market directly. We focus on evaluating the efficient market and emphasize the importance of online forecasting in adapting to the dynamic nature of financial markets. The three key points of our approach are: (a) apply the univariate time-varying volatility model AdaVol, (b) obtain probabilistic forecasts of future returns, and (c) optimize the competition metrics using stochastic gradient-based algorithms. We contend that the simplicity of our approach contributes to its robustness and consistency. Our performance in the M6 competition resulted in an overall 7th place, with a 5th place in the forecasting task. Considering our approach’s perceived simplicity, this achievement underscores the efficacy of our adaptive volatility method in probabilistic forecasting.

Electricity price forecasting using quantile regression averaging with nonconvex regularization

AbstractElectricity price forecasting (EPF) is an emergent research domain that focuses on forecasting the future electricity market price both deterministically and probabilistically. EPF has attracted enormous interest from both practitioners and scholars since the deregulation of the power market and wide applications of renewable energy sources, such as wind and solar energy. However, forecasting the electricity price accurately and efficiently is an extremely challenging task because of its high volatility, randomness, and fluctuation. Although quantile regression averaging (QRA) has been demonstrated to be efficacious in probabilistic EPF since the global energy forecasting competition in 2014 (GEFCom2014), it is sensitive to nuisance variables especially when the number of variables is large. The forecasting accuracy will be negatively affected by these nuisance variables. To address these challenges, this study investigates a nonconvex regularized QRA in probabilistic forecasting. Two types of nonconvex regularized QRA select the important inputs obtained from point forecasting to obtain more accurate forecasting outcomes. To demonstrate the effectiveness of the proposed EPF model, two real datasets from the European power market are considered.

Tourism forecasts after COVID-19: Evidence of Portugal

Based on a comprehensive tourism forecasting competition using Portugal's regional data, we study the impact of COVID-19 on the ability of time series models to forecast tourism demand. We find that the stable seasonal patterns observed before the pandemic did not persist in 2020, but regions with higher weights of domestic tourism showed much lower tourism declines and seasonal shifts. Although this change was temporary, it caused significant forecast breakdowns in all methods. However, the intensity of the break differed across models leading to important changes in model rankings, especially in the most affected regions. We discuss the effectiveness and implications of applying straightforward data adjustments and how they can attenuate the pandemic impact on ex-post assessment of tourism forecasts.

Predicting the magnitude and timing of peak electricity demand: A competition case study

AbstractAs weather dependence of the electricity network grows, there is an increasing need to predict the time at which the network peak load will occur. Improving forecasts of peak hour can lead to more accurate scheduling of generation as well as the ability to use flexibility to improve system utilisation or defer capital investment. While there are extensive benchmark models for forecasting electricity demand, their efficacy at forecasting the time or shape of the peak remains to be seen. Global forecasting competitions provide a unique opportunity to compare multiple methodologies under common performance criteria and incentives. The methodology and results are detailed from the Big Data and Energy Analytics Laboratory Challenge 2022 used by the team ‘peaky‐finders’ and investigates the suitability of using hourly methods to forecast daily peak magnitude, time, and shape. The resulting approach provides a reproducible ensemble benchmark against which to evaluate more complex methods. Results indicate that simple regression techniques can perform well and outperform more complicated methods during seasons with low hourly variability, however ensemble methods show higher accuracy overall. The results also highlight the significant impact of extreme weather on forecast accuracy, demonstrating the importance of forecasting processes that are resilient to extreme weather.

Combining varying training data-based artificial intelligence models for energy price forecasting

ABSTRACT In this work, the price-forecasting accuracy of an ensemble of seasonal auto-regressive integrated moving average (SARIMA), multiple linear regression (MLR), feed-forward neural network (FFNN), and radial basis function (RBF) network models has been assessed in a single series modeling framework. The methodology involved training the individual models using past data windows of varying sizes and making the forecast for the next five days on a daily rolling basis. The hourly spot price data of the Iberian Electricity Market (MIBEL) is selected as the test case system. All the models have been tested on high and low-volatile data sets to assess their forecasting abilities. The forecast performance has also been assessed by participating in a real-time forecasting competition and comparing it with the earlier models proposed in the literature. The results show that the ensemble model is effective in producing better forecasts. But its forecast accuracy is greatly affected by the size of the training window and the combination of various models. The ensemble of the FFNN and RBF network models performs best when conditions are volatile. Moreover, during volatile periods it is better to use a small window size for training as compared to a large window size.

Trends in Undergraduate WxChallenge Performance

Abstract The WxChallenge is a national forecasting competition in which participants at colleges and universities across North America attempt to accurately predict daily high and low temperatures, maximum wind speed, and precipitation accumulation at a variety of locations each year. Undergraduate students make up the majority of participants. In this study, we observed trends from 11 seasons of WxChallenge data and related them to existing literature on local forecasting contests. Normalized scores were calculated each day for any participant that submitted a forecast. On average, undergraduate scores improved with continued participation in the contest. Significant gains are made during their first year by forecasters who participated for multiple years. Duration of participation in the contest plays a role, but significant improvements in performance were also noted with higher academic standing, potentially as a result of forecasting experience gained through other curricular or extracurricular activities. Significance Statement One avenue that aspiring meteorologists have outside the classroom to hone their forecasting skills is through the WxChallenge, a national collegiate forecasting contest. We wanted to see whether, on average, students who participated longer in the contest outperformed those who did not forecast for as long. We found this to be true. Mean scores by students who participated for more than one year improved significantly by the end of the first year, and they outperformed those without prior experience in the WxChallenge in following years. These findings are important because they can be used as evidence to motivate students to join a forecasting contest early in their academic careers, allowing them to gain skill before pursuing a career in forecasting.

How many fundamentals should we include in the behavioral equilibrium exchange rate model?

This paper evaluates the reliability of alternative equilibrium exchange rate assessments. The literature has predominantly relied on the behavioral equilibrium exchange rate (BEER) model, which attempts to establish a long-term link between the real exchange rate and economic fundamentals. We challenge the recent tendency among BEER modelers to increase the number of regressors. The wisdom of this choice is evaluated with the criterion that equilibrium exchange rates should help predict the future trajectory of real exchange rates. Using panel data for 30 countries during the period 1991–2018, we show that it is hard to outperform both a parsimonious BEER model (with only three fundamentals) and the relative PPP model (which excludes a role for fundamentals in the long run). This finding calls for caution in the ability of economists to closely map the evolution of equilibrium exchange rates and economic fundamentals. • Behavioral equilibrium exchange rate (EER) models are widely used by policy makers. • Large BEER models are commonly believed to be more reliable than simple ones. • We assess BEER model reliability within a forecasting competition framework. • We find that the increased model complexity is counterproductive out-of-sample. • The claim that we understand long-term drivers of real exchange rates beyond PPP is doubtful.

Nonparametric Probabilistic Forecasting for Wind Power Generation Using Quadratic Spline Quantile Function and Autoregressive Recurrent Neural Network

Compared with commonly-used point forecasting, probabilistic forecasting provides quantitative information on the uncertainty associated with wind power output. However, most studies focus on modelling only a fixed set of quantiles or intervals, which cannot provide the entire information of wind power distribution. In this paper, we propose a non-parametric and flexible method for probabilistic wind power forecasting. First, the distribution of wind power output is specified by spline quantile function, which avoids assuming a parametric form and also provides flexible shape for wind power density. Then, autoregressive recurrent neural network is used to build the non-linear mapping from input features to the parameters of quadratic spline quantile function. A novel loss function based on continuous ranked probability score (CRPS) is designed to train the forecasting model. We also derive the closed-form solution of the integral required in computing the CRPS-based loss function in order to improve the computational efficiency in training. The effectiveness of our proposed method has been validated with public real-world data from Global Energy Forecasting Competition 2014. Compared with some advanced benchmarks, our proposed approach outperforms them at least six percent in terms of CRPS, indicating that it can provide probabilistic wind power forecasting with high quality.

Forecasting office rents with ensemble models – the case for European real estate markets

PurposeCommercial real estate and office rental values, in particular, have long been the focus of research. Several forecasting frameworks for office rental values in multivariate and univariate fashions have been proposed. Recent developments in time series forecasting using machine learning and deep learning methods offer an opportunity to update traditional univariate forecasting frameworks.Design/methodology/approachWith the aim to extend research on univariate rent forecasting a hybrid methodology combining both ARIMA and a neural network model is proposed to exploit the unique strengths of both methods in linear and nonlinear modelling. N-BEATS, a deep learning algorithm that has demonstrated state-of-the-art forecasting performance in major forecasting competitions, are explained. With the ARIMA model, it is jointly applied to the office rental dataset to produce forecasts for four-quarters ahead.FindingsWhen the approach is applied to a dataset of 21 major European office cities, the results show that the ensemble model can be an effective approach to improve the prediction accuracy achieved by each of the models used separately.Practical implicationsReal estate forecasting is essential for assessing the value of managing portfolios and for evaluating investment strategies. The approach applied in this paper confirms the heterogeneity of real estate markets. The application of mixed modelling via linear and nonlinear methods decreases the uncertainty of abrupt changes in rents.Originality/valueTo the best of the authors' knowledge, no such application of a hybrid model updating classical statistical forecasting with a deep learning neural network approach in the field of commercial real estate rent forecasting has been undertaken.

Complex exponential smoothing

AbstractExponential smoothing has been one of the most popular forecasting methods used to support various decisions in organizations, in activities such as inventory management, scheduling, revenue management, and other areas. Although its relative simplicity and transparency have made it very attractive for research and practice, identifying the underlying trend remains challenging with significant impact on the resulting accuracy. This has resulted in the development of various modifications of trend models, introducing a model selection problem. With the aim of addressing this problem, we propose the complex exponential smoothing (CES), based on the theory of functions of complex variables. The basic CES approach involves only two parameters and does not require a model selection procedure. Despite these simplifications, CES proves to be competitive with, or even superior to existing methods. We show that CES has several advantages over conventional exponential smoothing models: it can model and forecast both stationary and non‐stationary processes, and CES can capture both level and trend cases, as defined in the conventional exponential smoothing classification. CES is evaluated on several forecasting competition datasets, demonstrating better performance than established benchmarks. We conclude that CES has desirable features for time series modeling and opens new promising avenues for research.

An advanced deep neuroevolution model for probabilistic load forecasting

Probabilistic load forecasting (PLF) is necessary for power system operations and control as it assists in proper scheduling and dispatch. Moreover, PLF adequately captures the uncertainty whether that uncertainty is related to load data or the forecasting model. And there are not many PLF models, and those which exist are very complex or difficult to interpret. This paper proposes a novel neuroevolution algorithm for handling the uncertainty associated with load forecasting. In this paper, a new modified evolutionary algorithm is proposed which is used to find the optimal hyperparameters of 1D-Convolutional neural network (CNN). The probabilistic forecasts are produced by minimizing the mean scaled interval score loss function at 50%, 90% and 95% prediction intervals. The proposed neuroevolution algorithm is tested on a global energy forecasting competition (GEFCom-2014) load dataset, and two different experiments are conducted considering load only and one with load and temperature. Strong conclusions are drawn from these experiments. Also, the proposed model is compared with other benchmark models, and it has been shown that it outperforms the other models.

Tourism forecasting competition in the time of COVID-19: An assessment of ex ante forecasts

• This commentary summarises Stage 2 forecasting results of the organised competition. • The Asia and Pacific team won both stages and the overall competition. • Future research should fully account for uncertainties associated with a crisis.

A robust support vector regression model for electric load forecasting

Electric load forecasting is a crucial part of business operations in the energy industry. Various load forecasting methods and techniques have been proposed and tested. With growing concerns about cybersecurity and malicious data manipulations, an emerging topic is to develop robust load forecasting models. In this paper, we propose a robust support vector regression (SVR) model to forecast the electricity demand under data integrity attacks. We first introduce a weight function to calculate the relative importance of each observation in the load history. We then construct a weighted quadratic surface SVR model. Some theoretical properties of the proposed model are derived. Extensive computational experiments are based on the publicly available data from Global Energy Forecasting Competition 2012 and ISO New England. To imitate data integrity attacks, we have deliberately increased or decreased the historical load data. Finally, the computational results demonstrate better accuracy of the proposed robust model over other recently proposed robust models in the load forecasting literature.

Fathoming empirical forecasting competitions’ winners

The M5 forecasting competition has provided strong empirical evidence that machine learning methods can outperform statistical methods: in essence, complex methods can be more accurate than simple ones. Regardless, this result challenges the flagship empirical result that led the forecasting discipline for the last four decades: keep methods sophisticatedly simple. Nevertheless, this was a first, and we can argue that this will not happen again. There has been a different winner in each forecasting competition. This inevitably raises the question: can a method win more than once (and should it be expected to)? Furthermore, we argue for the need to elaborate on the perks of competing methods, and what makes them winners?

Forecast Competition in Energy Imbalance Market

Uncertainties in renewable generation make accurate load forecast essential for reliable power system operation. This paper considers energy imbalance market (EIM), where market players are allowed to procure energy ahead of time and trade the mismatch due to forecast error and strategic behaviors. The ISO sets the trading prices according to the market conditions, and pursues various system-level objectives. We first identify the power-law relationship between data volume and forecast accuracy, which enables the formulation of forecast cost model. Then, we cast the interactions in the EIM in the Stackelberg game framework with the ISO acting as the leader. We offer explicit subgame perfect equilibrium among the players in EIM, and derive the sufficient condition for the existence of unique equilibrium. Then, we show that this equilibrium, if exists, supports the maximal social welfare and under certain conditions, minimizes the total mismatch. We further examine the local and global impacts of the forecast errors under mild conditions, together with robustness analysis. Such analysis provides mechanism design guidelines for the ISO to enable the data sharing and forecast method sharing among market players in the EIM. Numerical studies further examine the effectiveness, robustness and sensitivity of the subgame.

A simple model for mixing intuition and analysis

Firefighters, emergency paramedics, and airplane pilots are able to make correct judgments and choices in challenging situations of scarce information and time pressure. Experts often attribute such successes to intuition and report that they avoid analysis. Similarly, laypeople can effortlessly perform tasks that confuse machine algorithms. OR should ideally respect human intuition while supporting and improving it with analytical modelling. We utilize research on intuitive decision making from psychology to build a model of mixing intuition and analysis over a set of interrelated tasks, where the choice of intuition or analysis in one task affects the choice in other tasks. In this model, people may use any analytical method, such as multi-attribute utility, or a single-cue heuristic, such as availability or recognition. The article makes two contributions. First, we study the model and derive a necessary and sufficient condition for the optimality of using a positive proportion of intuition (i.e., for some tasks): Intuition is more frequently accurate than analysis to a larger extent than analysis is more frequently accurate than guessing. Second, we apply the model to synthetic data and also natural data from a forecasting competition for a Wimbledon tennis tournament and a King’s Fund study on how patients choose a London hospital: The optimal proportion of intuition is estimated to range from 25% to 53%. The accuracy benefit of using the optimal mix over analysis alone is estimated between 3% and 27%. Such improvements would be impactful over large numbers of choices as in public health.

Temporal inception convolutional network based on multi‐head attention for ultra‐short‐term load forecasting

Accurate load forecasting is essential for ensuring safe, stable, and economical operation of energy internet. Temporal convolutional networks (TCNs) have demonstrated superior performance, when compared to recurrent neural network models, since their introduction in electrical load forecasting. However, the current TCN-based models are unable to obtain a large receptive field and strong long-time feature extraction capability owing to the specific kernel size of the 1D convolution structure. This paper proposes a temporal inception convolutional network based on multi-head attention (TICN-Att) for ultra-short-term load prediction. By introducing an inception structure into the TCN, the proposed model can extract multi-dimensional information from the input features, through the multiple hidden convolutional kernels of different scales, without stacking layers depth-wise. Simultaneously, by introducing a multi-head attention mechanism, the TICN-Att model has a long time-dependent extraction capability, similar to that of the long short-term memory network models. The generalization and validity of the model are tested using the global energy forecasting competition (GEFCOM2014) dataset, electrical load data of a city in Jiangsu (China), and PJM power system dataset. The experimental results demonstrated that the proposed model has the best prediction effect, compared to the other state-of-the-art models.

Common factors and the dynamics of cereal prices. A forecasting perspective

This article investigates what determines the price dynamics of the main cereals: barley, maize, rice and wheat. Using an extensive dataset of monthly time series covering the years 1980–2019, we extract four different common factors explaining the dynamics of commodity prices, exchange rates, financial and macroeconomic indicators. Next, we examine whether these factors are useful in explaining the movements of cereal prices. We show that models incorporating all four factors outperform significantly the naive random walk model in out-of-sample forecasting competition, especially for longer horizons. However, they have only marginally better performance than a simpler model based on the commodity factor alone.

The Future of Forecasting Competitions: Design Attributes and Principles

Forecasting competitions are the equivalent of laboratory experimentation widely used in physical and life sciences. They provide useful, objective information to improve the theory and practice of forecasting, advancing the field, expanding its usage, and enhancing its value to decision and policymakers. We describe 10 design attributes to be considered when organizing forecasting competitions, taking into account trade-offs between optimal choices and practical concerns, such as costs, as well as the time and effort required to participate in them. Consequently, we map all major past competitions in respect to their design attributes, identifying similarities and differences between them, as well as design gaps, and making suggestions about the principles to be included in future competitions, putting a particular emphasis on learning as much as possible from their implementation in order to help improve forecasting accuracy and uncertainty. We discuss that the task of forecasting often presents a multitude of challenges that can be difficult to capture in a single forecasting contest. To assess the caliber of a forecaster, we, therefore, propose that organizers of future competitions consider a multicontest approach. We suggest the idea of a forecasting-“athlon” in which different challenges of varying characteristics take place.

Forecast with forecasts: Diversity matters

Forecast combinations have been widely applied in the last few decades to improve forecasting. Estimating optimal weights that can outperform simple averages is not always an easy task. In recent years, the idea of using time series features for forecast combinations has flourished. Although this idea has been proved to be beneficial in several forecasting competitions, it may not be practical in many situations. For example, the task of selecting appropriate features to build forecasting models is often challenging. Even if there was an acceptable way to define the features, existing features are estimated based on the historical patterns, which are likely to change in the future. Other times, the estimation of the features is infeasible due to limited historical data. In this work, we suggest a change of focus from the historical data to the produced forecasts to extract features. We use out-of-sample forecasts to obtain weights for forecast combinations by amplifying the diversity of the pool of methods being combined. A rich set of time series is used to evaluate the performance of the proposed method. Experimental results show that our diversity-based forecast combination framework not only simplifies the modeling process but also achieves superior forecasting performance in terms of both point forecasts and prediction intervals. The value of our proposition lies on its simplicity, transparency, and computational efficiency, elements that are important from both an optimization and a decision analysis perspective.