Past Forecast Errors Research Articles

Leveraging Deterministic Weather Forecasts for In Situ Probabilistic Temperature Predictions via Deep Learning

Abstract We propose a neural network approach to produce probabilistic weather forecasts from a deterministic numerical weather prediction. Our approach is applied to operational surface temperature outputs from the Global Deterministic Prediction System up to 10-day lead times, targeting METAR observations in Canada and the United States. We show how postprocessing performance is improved by training a single model for multiple lead times. Multiple strategies to condition the network for the lead time are studied, including a supplementary predictor and an embedding. The proposed model is evaluated for accuracy, spread, distribution calibration, and its behavior under extremes. The neural network approach decreases the continuous ranked probability score (CRPS) by 15% and has improved distribution calibration compared to a naive probabilistic model based on past forecast errors. Our approach increases the value of a deterministic forecast by adding information about the uncertainty, without incurring the cost of simulating multiple trajectories. It applies to any gridded forecast including the recent machine learning–based weather prediction models. It requires no information regarding forecast spread and can be trained to generate probabilistic predictions from any deterministic forecast. Significance Statement Weather is difficult to predict a long time in advance because we cannot measure the state of the atmosphere precisely enough. Consequently, it is common practice to run forecasts several times and look at the differences to evaluate how uncertain the prediction is. This process of running ensemble forecasts is expensive and consequently not always feasible. We propose a middle ground where we add uncertainty information to forecasts that were run only once, using artificial intelligence. Our method increases the value of these forecasts by adding information about the uncertainty without incurring the cost of multiple full simulations.

Forecasting Tourism Demand for Medical Services

ABSTRACT: Medical tourism is considered nowadays as a multi-billion industry which can promote a country's economic growth. Therefore, forecasting the scheduled tourism demand for medical services is of great importance for policy makers. Doing so, however, is not an easy task due to the following reasons: Data on medical tourism are (i) not easily accessible; (ii) not typically distinguished from tourists' non-scheduled (unintentional) use of a country's medical services; and (iii) usually not publicly available for long time periods. In this paper, we present a novel way to forecast tourism demand (intentional and unintentional) foro medical services —a rough but informative proxy of medical tourism— using limited data. To perform the analysis, we use data on the percentage of hospital discharges of non-residents for 17 European countries over the period 2008-2019 retrieved from Eurostat. Our methodological approach is based on a forecasting technique recently developed by Kyriazi, Thomakos and Guerard ; the adaptive learning forecasting. According to this method, MSE (Mean Squared Error)-performance enhancements can be achieved using any forecast as input —as long as that input is not a 'perfect' forecast— by learning from past forecast errors. Within this context, even the most basic forecast, the no-change or naïve forecast, can be used as input to the adaptive learning procedure. Kyriazi, Thomakos and Guerard approach is very well suited to our research question because (i) the no-change forecast is a natural candidate in a short time series where models cannot be estimated with sufficient accuracy, (ii) the no-change forecast is obviously far from being the 'perfect' forecast, and (iii) the adaptive learning process can be initialized by the no-change forecast and then learn by its own past forecast errors. Our results show that adaptive learning forecasting leads to performance enhancements that range from the 5% to more than 20% relative to the no-change benchmark. This finding indicates the efficiency of the adaptive learning method in forecasting medical tourism demand; an important subcategory of tourism demand for which data are not easily accessible and freely available historical data are existing for short time periods.

Empirical probabilistic forecasting: An approach solely based on deterministic explanatory variables for the selection of past forecast errors

Empirical probabilistic forecasts based on out-of-sample forecast errors have the advantage of incorporating all sources of forecast uncertainty but the drawback of being compute-intensive. Hence, selecting the past timestamps for which errors are generated may be crucial in “big data” settings. The existing error-based empirical methods either select errors based on their corresponding point forecasts—not addressing the scalability issue—or do not consider information regarding the target timestamps. We propose an approach solely based on deterministic explanatory variables for selecting past errors, thus exploiting information on the target timestamps without generating any forecasts beforehand. The proposed method was evaluated on the M5 competition’s dataset, compared to the competition’s top 50 submissions and several benchmarks. The results indicate that—given an efficient strategy for selecting past errors—empirical methods can offer a scalable alternative with a performance comparable to the state-of-the-art’s.

Evaluation of an inflow forecast correction method based on Multi-Scenarios division

Accurate inflow forecasting is very important for the operation of reservoirs. However, in the actual inflow forecasting, the differences between various parameters and external conditions will affect the forecast results of the model, so it is necessary to correct the forecast results. Most of the existing researches on the correction of forecast results focus on considering the model uncertainty as a whole, and there are few studies on the influence of different external factors on the forecast results. In view of this, the differences of external environment when the forecast occurs and the differences of forecast errors under different forecast scenarios are both considered in this paper. According to the differences in key influencing factors, such as rainfall conditions and foresight periods, the forecast scenarios are divided, and the forecast error distribution law under different scenarios is deduced based on the past forecast error data. On this basis, using Variational Mode Decomposition (VMD) and Long Short-Term Memory (LSTM) Neural Network Model, a new multi-dimensional and multi-attribute inflow forecast correction method considering forecast errors and forecast scenarios is established. This method breaks through the single-dimensional, single-attribute time series limitations of the traditional inflow forecast correction methods. Through the case study of Three Gorges Reservoir (TGR), it was found that compared with the actual situation, the average relative error of the inflow forecast was reduced from the actual 8.32% to 6.36%, a decrease of 1.96%, and the reduction rate reached 23.6%. In addition, other indexes such as mean absolute error, root mean square error and Nash-Sutcliffe efficiency coefficient have all been improved to varying degrees. It showed that the proposed method in this paper can simultaneously consider both the forecast error information of the previous periods and its corresponding forecast scenario information, and increase the effective information input of the correction model to improve the accuracy of the inflow forecast model.

How can independent fiscal institutions make the most of assessing past economic forecasts?

Forecasts of economic variables like GDP, inflation and unemployment are the starting points for budget planning. Most OECD countries have designated an independent fiscal institutions to scrutinise or endorse the economic forecasts prepared by finance ministries. Although external assessments of past forecast errors can help uncover systematic bias or underperformance in official forecasts, there are several conceptual and practical problems with doing so that limit their usefulness. This paper describes the limitations of ex post forecast assessments and offers some suggestions on how independent fiscal institutions can add the most value in providing them.

Analysis of USDA Livestock Price Forecasting Accuracy for Cattle, Hogs, and Broilers

Agricultural markets, compared to other sectors, are typically characterized by uncertainty and high price fluctuations. High price volatility in livestock markets leads to inefficient resource allocation and production planning. Expert price forecasts are not always affordable for all market players, so readily available public forecasts have risen in popularity. This study uses accuracy-based testing methods to evaluate the accuracy of the United States Department of Agriculture (USDA) livestock price forecasting by utilizing the World Agricultural Supply and Demand Estimates (WASDE) quarterly data for slaughter cattle, hogs, and broilers. The study also employs a vector error correction (VEC) model to compare USDA price forecasts. Results suggest that the USDA forecast was more accurate than the competing VEC model across three sectors, suggested by low RMSE and MAE. The beta efficiency test results showed that USDA price forecasts were efficient for all three price series, whereas VEC forecasts were biased for hogs and broiler prices. The findings of the study also confirm that USDA price forecasts are biased for cattle prices with a tendency to repeat past forecast error in all three markets. Results from the forecast encompassing tests showed that USDA cattle and broiler forecasts captured the information contained in VEC forecasts. However, because the hog prices did not show any improvement over time, there is room for improvement of the USDA price forecasts. Overall, results suggest that USDA price forecasts for slaughter cattle, hogs and broilers provide useful information to the market. However, the results also indicate that USDA price forecasts reduce forecast error by economically significant levels.

Lest We Forget: Learn from Out-of-Sample Forecast Errors When Optimizing Portfolios

Abstract Portfolio optimization often struggles in realistic out-of-sample contexts. We deconstruct this stylized fact by comparing historical forecasts of portfolio optimization inputs with subsequent out-of-sample values. We confirm that historical forecasts are imprecise guides of subsequent values, but we discover the resultant forecast errors are not entirely random. They have predictable patterns and can be partially reduced using their own history. Learning from past forecast errors to calibrate inputs (akin to empirical Bayesian learning) generates portfolio performance that reinforces the case for optimization. Furthermore, the portfolios achieve performance that meets expectations, a desirable yet elusive feature of optimization methods. Authors have furnished an Internet Appendix, which is available on the Oxford University Press Web site next to the link to the final published paper online.

Fiscal stabilisation in real time: An exercise in risk management

Fiscal policy in the EU has been largely pro-cyclical. We propose an approach that supports more informed decision making aimed at stabilising output. Rather than relying on notoriously uncertain point estimates of the cycle, our approach is built around the management of risks: (i) launching discretionary measures to support or dampen aggregate demand when, in hindsight, no measures would have been required, versus (ii) remaining inactive when, in hindsight, a stabilisation measure would have been warranted. A rational policymaker can manage these risks by using information on past forecast errors and take stabilisation measures only when real-time estimates of the cycle exceed an optimal threshold. We show that the observed tendency to run pro-cyclical fiscal policies can reflect two complementary factors: a preference for activism combined with the desire to avert downside risks to growth, while putting a blind eye on upside risks.

Improving Wind Power Forecasts: Combination through Multivariate Dimension Reduction Techniques

Wind energy and wind power forecast errors have a direct impact on operational decision problems involved in the integration of this form of energy into the electricity system. As the relationship between wind and the generated power is highly nonlinear and time-varying, and given the increasing number of available forecasting techniques, it is possible to use alternative models to obtain more than one prediction for the same hour and forecast horizon. To increase forecast accuracy, it is possible to combine the different predictions to obtain a better one or to dynamically select the best one in each time period. Hybrid alternatives based on combining a few selected forecasts can be considered when the number of models is large. One of the most popular ways to combine forecasts is to estimate the coefficients of each prediction model based on its past forecast errors. As an alternative, we propose using multivariate reduction techniques and Markov chain models to combine forecasts. The combination is thus not directly based on the forecast errors. We show that the proposed combination strategies based on dimension reduction techniques provide competitive forecasting results in terms of the Mean Square Error.

Adaptive learning forecasting, with applications in forecasting agricultural prices

We introduce a new forecasting methodology, referred to as adaptive learning forecasting, that allows for both forecast averaging and forecast error learning. We analyze its theoretical properties and demonstrate that it provides a priori MSE improvements under certain conditions. The learning rate based on past forecast errors is shown to be non-linear. This methodology is of wide applicability and can provide MSE improvements even for the simplest benchmark models. We illustrate the method’s application using data on agricultural prices for several agricultural products, as well as on real GDP growth for several of the corresponding countries. The time series of agricultural prices are short and show an irregular cyclicality that can be linked to economic performance and productivity, and we consider a variety of forecasting models, both univariate and bivariate, that are linked to output and productivity. Our results support both the efficacy of the new method and the forecastability of agricultural prices.

Forecast to grow: Aviation demand forecasting in an era of demand uncertainty and optimism bias

Errors in forecasting airport passenger demand arise from uncertain economic climates and planners’ optimism, leading airport planners to make misinformed infrastructure investments. We use publicly available data to develop and test methodologies that enable airport planners to (1) predict the probability of a severe contraction in passenger volumes and (2) improve forecast accuracy by systematically incorporating past forecast errors of airport peers thus “grounding” optimistic forecasts. By incorporating past forecast errors from like airports into airport forecasting models, we build a methodology that is grounded in established demand forecasting practices and is able to significantly improve the accuracy of aviation demand forecasting models.

Forecasting the duration of short‐term deflation episodes

AbstractThe paper proposes a simulation‐based approach to multistep probabilistic forecasting, applied for predicting the probability and duration of negative inflation. The essence of this approach is in counting runs simulated from a multivariate distribution representing the probabilistic forecasts, which enters the negative inflation regime. The marginal distributions of forecasts are estimated using the series of past forecast errors, and the joint distribution is obtained by a multivariate copula approach. This technique is applied for estimating the probability of negative inflation in China and its expected duration, with the marginal distributions computed by fitting weighted skew‐normal and two‐piece normal distributions to autoregressive moving average ex post forecast errors and using the multivariate Student t copula.

Insights from the Evaluation of Past Local Area Population Forecasts

Local area population forecasts have a wide variety of uses in the public and private sectors. But not enough is known about the errors of such forecasts, particularly over the longer term (20 years or more). Understanding past errors is valuable for both forecast producers and users. This paper (i) evaluates the forecast accuracy of past local area population forecasts published by Australian State and Territory Governments over the last 30 years and (ii) illustrates the ways in which past error distributions can be employed to quantify the uncertainty of current forecasts. Population forecasts from the past 30 years were sourced from State and Territory Governments. Estimated resident populations to which the projections were compared were created for the geographical regions of the past projections. The key features of past forecast error patterns are described. Forecast errors mostly confirm earlier findings with regard to the relationship between error and length of projection horizon and population size. The paper then introduces the concept of a forecast ‘shelf life’, which indicates how far into the future a forecast is likely to remain reliable. It also illustrates how past error distributions can be used to create empirical prediction intervals for current forecasts. These two complementary measures provide a simple way of communicating the likely magnitude of error that can be expected with current local area population forecasts.

Recent monetary policy effects on Japanese macroeconomy

This paper empirically analyzes the effects of recent monetary policy based on expected and real-time data for Japan. Also, expected data between different time periods would have impacts on the economy and are taken into account. Since the 2000s, Japan has experienced a serious recession and low or zero interest rates policy has been conducted, so such a special situation might have influence on the economy. The empirical results show that past forecast error data of GDP plays a significant role on macro economy in Japan, however, impacts on the economy are not found during a longer period, over a quarter. Furthermore, forecast data is not crucial to the economy.

Forecast-error-based estimation of forecast uncertainty when the horizon is increased

Past forecast errors are employed frequently in the estimation of the unconditional forecast uncertainty, and several institutions have increased their forecast horizons in recent times. This work addresses the question of how forecast-error-based estimation can be performed if there are very few errors available for the new forecast horizons. It extends the results of Knüppel (2014) in order to relax the condition on the data structure that is required for the SUR estimator to be independent of unknown quantities. It turns out that the SUR estimator of the forecast uncertainty, which estimates the forecast uncertainty for all horizons jointly, tends to deliver large efficiency gains relative to the OLS estimator (i.e., the sample mean of the squared forecast errors for each individual horizon) in the case of increased forecast horizons. The SUR estimator is applied to the forecast errors of the Bank of England, the US Survey of Professional Forecasters, and the FOMC.

Learning About Banks' Net Worth and the Slow Recovery after the Financial Crisis

In this paper, we examine the influence of information rigidities concerning the net worth of banks on the real economy over time. In a first part, we show empirically that expectations about the net earnings of banks (as growth of net worth) are truly biased, particularly during the financial crisis. The forecast error of professional investors cannot be attributed to sticky information but rather to noisy information. Investors display a learning behavior with regard to past forecast errors in forming their expectations about future earnings during the crisis. In a second part, by drawing on a New Keynesian general equilibrium model with a banking sector, we demonstrate that, by quantitatively incorporating this type of information updating and expectations formation about the net worth of banks, noisy information can produce a slow recovery compared to a full information rational expectation case.

Performance and robustness of probabilistic river forecasts computed with quantile regression based on multiple independent variables

Abstract. This study applies quantile regression (QR) to predict exceedance probabilities of various water levels, including flood stages, with combinations of deterministic forecasts, past forecast errors and rates of water level rise as independent variables. A computationally cheap technique to estimate forecast uncertainty is valuable, because many national flood forecasting services, such as the National Weather Service (NWS), only publish deterministic single-valued forecasts. The study uses data from the 82 river gauges, for which the NWS' North Central River Forecast Center issues forecasts daily. Archived forecasts for lead times of up to 6 days from 2001 to 2013 were analyzed. Besides the forecast itself, this study uses the rate of rise of the river stage in the last 24 and 48 h and the forecast error 24 and 48 h ago as predictors in QR configurations. When compared to just using the forecast as an independent variable, adding the latter four predictors significantly improved the forecasts, as measured by the Brier skill score and the continuous ranked probability score. Mainly, the resolution increases, as the forecast-only QR configuration already delivered high reliability. Combining the forecast with the other four predictors results in a much less favorable performance. Lastly, the forecast performance does not strongly depend on the size of the training data set but on the year, the river gauge, lead time and event threshold that are being forecast. We find that each event threshold requires a separate configuration or at least calibration.

Estimates of uncertainty around Australian budget forecasts

In this article, past forecast errors are used to construct confidence intervals around Australian Government Budget forecasts of key economic and fiscal variables. These confidence intervals provide an indication of the extent of uncertainty around the point estimate forecasts presented in the Budget. JEL classification numbers: E17, H68. Keywords: Confidence intervals, forecast errors, government budget, nominal GDP, real GDP, treasury, uncertainty.

Inefficiency in Survey Exchange Rates Forecasts

We use a novel database of a panel of quarterly survey of exchange rates forecasts available on the Bloomberg platform, for the following five bilateral exchange rates: EUR/GBP, EUR/JPY, EUR/USD, GBP/USD and USD/JPY, for the timespan ranging from the third quarter 2006 up to the fourth quarter of 2011. We find that forecasters are on average irrational, failing to identify the true data generating process of bilateral exchange rates and generally overreacting to past observed information. Moreover, exploring individual performance, we can state that financial analysts irrationally do not look at their past forecast errors to improve the quality of their later forecasts.

Efficient estimation of forecast uncertainty based on recent forecast errors

Multi-step-ahead forecasts of the forecast uncertainty of an individual forecaster are often based on the horizon-specific sample means of his recent squared forecast errors, where the number of past forecast errors available decreases one-to-one with the forecast horizon. In this paper, the efficiency gains from the joint estimation of forecast uncertainty for all horizons in such samples are investigated. If the forecast uncertainty is estimated by seemingly unrelated regressions, it turns out that the covariance matrix of the squared forecast errors does not have to be estimated, but simply needs to have a certain structure, which is a very useful property in small samples. Considering optimal and non-optimal forecasts, it is found that the efficiency gains can be substantial for longer horizons in small samples. The superior performance of the seemingly-unrelated-regressions approach is confirmed in several empirical applications.