Forecast accuracy measures for exception reporting using receiver operating characteristic curves

Abstract

The exception principle of management reporting suggests that, under ordinary conditions, operational staff persons make decisions, but that the same staff refer decisions to upper-level managers under exceptional conditions. Forecasts of large changes or extreme values in product or service demand are potential triggers for such reporting. Seasonality estimates in univariate forecast models and leading independent variables in multivariate forecast models are among the approaches to forecasting exceptional demand, a forecast activity that this paper identifies as requiring new accuracy measures based on the tails of sampled forecast error distributions, rather than conventional measures which use the central tendency. For this purpose, the paper introduces the application of the receiver operating characteristic (ROC) framework, which has been used for the assessment of exceptional behavior in many fields. In a case study on serious violent crime in Pittsburgh, Pennsylvania, the simplest, non-naïve univariate forecast method is best for forecasting ordinary conditions using conventional forecast accuracy measures, but the most complex multivariate model is best for forecasting exceptional conditions using ROC forecast accuracy measures.