Reply on RC2

Abstract

Forecasts have the potential to improve decision-making but have not been widely evaluated because current forecast value methods have critical limitations. The ubiquitous Relative Economic Value (REV) metric is limited to binary decisions, cost-loss economic model, and risk neutral decision-makers. Expected Utility Theory can flexibly model more real-world decisions, but its application in forecasting has been limited and the findings are difficult to compare with those from REV. A new metric, Relative Utility Value (RUV), is developed using Expected Utility Theory. RUV has the same interpretation as REV which enables a systematic comparison of results, but RUV is more flexible and able to handle a wider range of real-world decisions because all aspects of the decision-context are user-defined. In addition, when specific assumptions are imposed it is shown that REV and RUV are equivalent. We demonstrate the key differences and similarities between the methods with a case study using probabilistic subseasonal streamflow forecasts in a catchment in the southern Murray-Darling Basin of Australia. The ensemble forecasts were more valuable than a reference climatology for all lead-times (max 30 days), decision types (binary, multi-categorical, and continuous-flow), and levels of risk aversion for most decision-makers. Beyond the second week however, decision-makers who were highly exposed to damages should use the reference climatology for the binary decision, and forecasts for the multi-categorical and continuous-flow decision. Risk aversion impact was governed by the relationship between the decision thresholds and the damage function, leading to a mixed impact across the different decision-types. The generality of RUV makes it applicable to any domain where forecast information is used for making decisions, and the flexibility enables forecast assessment tailored to specific decisions and decision-makers. It complements forecast verification and enables assessment of forecast systems through the lens of customer impact.