Renewal‐theoretic considerations in evaluation of forecast monitoring schemes

Abstract

AbstractTo evaluate the performance of a scheme for monitoring forecasts that are generated by exponential smoothing, forecasters have traditionally used a simulation‐based estimator of some characteristic of the associated run‐length distribution. The most frequently cited performance measures are the average run length and the probability that the run length does not exceed a user‐specified cut‐off point. However, there is disagreement about the definition of run length that is appropriate in the context of forecasting. In this paper we use fundamental results from renewal theory to establish the precise relationships between conflicting formulations both of the average run length and of the probability density function for the run length. More generally we derive the asymptotic mean and the asymptotic distribution function of the forward recurrence time for both ordinary and delayed renewal processes whose interoccurrence distributions are arithmetic with a given span. We discuss the practical significance of these results in the context of forecasting. Our findings bear directly on the way in which simulation experiments should be designed and executed to compare alternative forecast monitoring schemes.