Assessing the Value of Probability Forecasts

Abstract

The purpose of this paper is to explore the effects of sampling variability—in estimating forecast probabilities—on optimal decision making and the economic value of probability forecasts in the context of the cost–loss decision model. One effect is that the strategy, “protect if and only if the forecast probability is greater than C/L,” is not necessarily optimal. A second is that failure to adjust for sampling variability leads to biased estimates of the value of probability forecasts. The bias, which is small when probability forecasts are based on large training samples, can be substantial when data are scarce—as may be the case in the context of long-range forecasting. The problems associated with assessing the value of (objectively estimated) probability forecasts are discussed and a method of estimation is presented. As is illustrated in an example, this methodology can be used to compare different probability forecasting procedures and, in particular, to determine which, if any, of a number of potential predictors can be expected to enhance the value of the forecasts.