Ranking Of Forecasts Research Articles

Volatility forecasts, proxies and loss functions

Abstract The main problem in volatility forecasting is that the variable of interest is unobservable, which complicates not only the construction of forecasts but also their comparison. This article challenges the common practice of using only proxy-robust loss functions, which have the nice property that they lead to the same ranking of forecasts regardless whether the unobservable true volatility is used or some unbiased proxy. It is shown that two proxy-robust loss functions need not necessarily produce similar rankings but may even produce completely contradictory rankings. Two likelihood-based loss functions are proposed instead, which are not exactly proxy-robust but are still robust in the classical sense. The first is based on a t-distribution and is meant for daily data. The second is based on an F-distribution and is meant for high-frequency data. In the latter case, the squared error loss function may also be used when a logarithmic transformation is applied to the realized variances in order to achieve approximate normality. An alternative transformation is proposed which allows the adaptation to the degree of non-normality. The forecasting procedures that are compared by the different loss functions include GARCH, HAR, HARQ, and MIDAS models as well as nonparametric techniques. Finally, the economic relevance of choosing the right forecast is illustrated with the problem of establishing the intertemporal risk–return tradeoff. All theoretical arguments are backed up with empirical evidence obtained from daily data as well as from high-frequency data.

Additive Scoring Rules for Discrete Sample Spaces

In this paper, we develop strictly proper scoring rules that may be used to evaluate the accuracy of a sequence of probabilistic forecasts. In practice, when forecasts are submitted for multiple uncertainties, competing forecasts are ranked by their cumulative or average score. Alternatively, one could score the implied joint distributions. We demonstrate that these measures of forecast accuracy disagree under some commonly used rules. Furthermore, and most importantly, we show that forecast rankings can depend on the selected scoring procedure. In other words, under some scoring rules, the relative ranking of probabilistic forecasts does not depend solely on the information content of those forecasts and the observed outcome. Instead, the relative ranking of forecasts is a function of the process by which those forecasts are evaluated. As an alternative, we describe additive and strongly additive strictly proper scoring rules, which have the property that the score for the joint distribution is equal to a sum of scores for the associated marginal and conditional distributions. We give methods for constructing additive rules and demonstrate that the logarithmic score is the only strongly additive rule. Finally, we connect the additive properties of scoring rules with analogous properties for a general class of entropy measures.

Evaluating multi-step system forecasts with relatively few forecast-error observations

This paper develops a new approach for evaluating multi-step system forecasts with relatively few forecast-error observations. It extends the work of Clements and Hendry (1993) by using that of Abadir et al. (2014) to generate “design-free” estimates of the general matrix of the forecast-error second-moment when there are relatively few forecast-error observations. Simulations show that the usefulness of alternative methods deteriorates when their assumptions are violated. The new approach compares well with these methods and provides correct forecast rankings.

ROBUST FORECAST COMPARISON

Forecast accuracy is typically measured in terms of a given loss function. However, as a consequence of the use of misspecified models in multiple model comparisons, relative forecast rankings are loss function dependent. In order to address this issue, a novel criterion for forecast evaluation that utilizes the entire distribution of forecast errors is introduced. In particular, we introduce the concepts of general-loss (GL) forecast superiority and convex-loss (CL) forecast superiority; and we develop tests for GL (CL) superiority that are based on an out-of-sample generalization of the tests introduced by Linton, Maasoumi, and Whang (2005, Review of Economic Studies 72, 735–765). Our test statistics are characterized by nonstandard limiting distributions, under the null, necessitating the use of resampling procedures to obtain critical values. Additionally, the tests are consistent and have nontrivial local power, under a sequence of local alternatives. The above theory is developed for the stationary case, as well as for the case of heterogeneity that is induced by distributional change over time. Monte Carlo simulations suggest that the tests perform reasonably well in finite samples, and an application in which we examine exchange rate data indicates that our tests can help identify superior forecasting models, regardless of loss function.

Of Quantiles and Expectiles: Consistent Scoring Functions, Choquet Representations and Forecast Rankings

SummaryIn the practice of point prediction, it is desirable that forecasters receive a directive in the form of a statistical functional. For example, forecasters might be asked to report the mean or a quantile of their predictive distributions. When evaluating and comparing competing forecasts, it is then critical that the scoring function used for these purposes be consistent for the functional at hand, in the sense that the expected score is minimized when following the directive. We show that any scoring function that is consistent for a quantile or an expectile functional can be represented as a mixture of elementary or extremal scoring functions that form a linearly parameterized family. Scoring functions for the mean value and probability forecasts of binary events constitute important examples. The extremal scoring functions admit appealing economic interpretations of quantiles and expectiles in the context of betting and investment problems. The Choquet-type mixture representations give rise to simple checks of whether a forecast dominates another in the sense that it is preferable under any consistent scoring function. In empirical settings it suffices to compare the average scores for only a finite number of extremal elements. Plots of the average scores with respect to the extremal scoring functions, which we call Murphy diagrams, permit detailed comparisons of the relative merits of competing forecasts.

Robust Forecast Comparison

Forecast accuracy is typically measured in terms of a given loss function. However, as a consequence of the use of misspecified models in multiple model comparisons, relative forecast rankings are loss function dependent. This paper addresses this issue by using a novel criterion for forecast evaluation which is based on the entire distribution of forecast errors. We introduce the concepts of general-loss (GL) forecast superiority and convex-loss (CL) forecast superiority; and develop tests for GL (CL) superiority that are based on an out-of-sample generalization of the tests introduced by Linton, Maasoumi and Whang (2005). The asymptotic null distributions of our test statistics are nonstandard, and resampling procedures are used to obtain critical values. Additionally, the tests are consistent and have nontrivial local power under a sequence of local alternatives. In addition to the stationary case, we outline theory extending our tests to the case of heterogeneity induced by distributional change over time. Monte Carlo simulations suggest that the tests perform reasonably well in finite samples; and an application to exchange rate data indicates that our tests can help identify superior forecasting models, regardless of loss function.

Measuring and adjusting for overconfidence

To evaluate density forecasts, the applied scoring rule is often arbitrarily chosen. The selection of the scoring rule strongly influences the ranking of forecasts. This paper identifies overconfidence as the main driver for scoring differences. A novel approach to measure overconfidence is proposed. Based on a non-proper scoring rule, the forecasts can be individually adjusted toward a calibrated forecast. Applying the adjustment procedure to the survey of professional forecasters, it can be shown that out-of-sample forecasts can be significantly improved. Also the ranking of the adjusted forecasts becomes less sensitive to the selection of scoring rules.

Measuring and Adjusting for Overconfidence

To evaluate density forecasts the applied scoring rule is often arbitrarily chosen. The selection of the scoring rule strongly influences the ranking of forecasts. This paper identifies overconfidence as the main driver for scoring differences. A novel approach to measure overconfidence is proposed. Based on a non-proper scoring rule the forecasts can be individually adjusted towards a calibrated forecast. Applying the adjustment procedure to the Survey of Professional Forecasters it can be shown that out-of-sample forecasts can be significantly improved. Also the ranking of the adjusted forecasts becomes less sensitive to the selection of scoring rules.

A High-Low Model of Daily Stock Price Ranges

We observe that daily highs and lows of stock prices do not diverge over time and, hence, adopt the cointegration concept and the related vector error correction model (VECM) to model the daily high, the daily low, and the associated daily range data. The in-sample results attest the importance of incorporating high-low interactions in modeling the range variable. In evaluating the out-of-sample forecast performance using both mean-squared forecast error and direction of change criteria, it is found that the VECM-based low and high forecasts offer some advantages over some alternative forecasts. The VECM-based range forecasts, on the other hand, do not always dominate - the forecast rankings depend on the choice of evaluation criterion and the variables being forecasted.

A high–low model of daily stock price ranges

AbstractWe observe that daily highs and lows of stock prices do not diverge over time and, hence, adopt the cointegration concept and the related vector error correction model (VECM) to model the daily high, the daily low, and the associated daily range data. The in‐sample results attest to the importance of incorporating high–low interactions in modeling the range variable. In evaluating the out‐of‐sample forecast performance using both mean‐squared forecast error and direction of change criteria, it is found that the VECM‐based low and high forecasts offer some advantages over alternative forecasts. The VECM‐based range forecasts, on the other hand, do not always dominate—the forecast rankings depend on the choice of evaluation criterion and the variables being forecast. Copyright © 2008 John Wiley & Sons, Ltd.

A High-Low Model of Daily Stock Price Ranges

We observe that daily highs and lows of stock prices do not diverge over time and, hence, adopt the cointegration concept and the related vector error correction model (VECM) to model the daily high, the daily low, and the associated daily range data. The in-sample results attest the importance of incorporating high-low interactions in modeling the range variable. In evaluating the out-of-sample forecast performance using both mean-squared forecast error and direction of change criteria, it is found that the VECM-based low and high forecasts offer some advantages over some alternative forecasts. The VECM-based range forecasts, on the other hand, do not always dominate - the forecast rankings depend on the choice of evaluation criterion and the variables being forecasted.