On the Relationship between the Quality and Value of Forecasts in the Generalized Cost-Loss Ratio Situation

Abstract

This paper investigates the relationship between the quality and value of forecast in the context of a generalized N-action, N-event model of the cost-loss ratio situation. The forecasts of interest are imperfect categorical forecasts, calibrated according to past performance and represented by multidimensional sets of conditional and predictive probabilities. Forecasts quality is measured by the ranked probability score (RPS), a natural measure of the accuracy of forecasts in the context of this model. The measure of value is the difference between the expected expense associated with climatological information and the expected expense associated with imperfect forecasts. Thus, climatological and perfect information define lower and upper bounds, respectively, on the quality and value of the imperfect forecasts. Quality-value relationships are explored in the three-action, three-event situation, using brute form and mathematical programming methods. Numerical results are presented for several specific cases. In all cases, the relationships are described by envelopes of values rather than by single-valued functions, indicating that a range of forecast value is generally associated with a given level of forecast quality (and vice versa). The existence of these envelopes reveals two important deficiencies in scalar (i.e., one-dimensional) measures of forecast quality, such as the RPS, when they an used as surrogates for measures of value: 1) these quality measures generally provide only imprecise estimates of forecast value and 2) increases in forecast quality, as reflected by such measures may actually be associated with decreases in forecast value.