Sampling distributions of the Brier score and Brier skill score under serial dependence

Abstract

AbstractRelatively little attention has been given to the effects of serial correlation of forecasts and observations on the sampling properties of forecast verification statistics. An assumption of serial independence for low‐quality forecasts may be reasonable. However, forecasts of sufficient quality for autocorrelated events must themselves be autocorrelated: as quality approaches the limit of perfect forecasts, the forecasts become increasingly similar to the corresponding observations.The effects of forecast serial correlation on the sampling properties of the Brier Score (BS) and Brier Skill Score (BSS), for probability forecasts of dichotomous events, are examined here. As in other settings, the effect of serial correlation is to inflate the variances of the sampling distributions of the two statistics, so that uncorrected confidence intervals are too narrow, and uncorrected hypothesis tests yield p‐values that are too small. Expressions are given for ‘effective sample size’ corrections for the sampling variances of both BS and BSS, in which it can be seen that the effects of serial correlation on the sampling variances increase with increasing forecast accuracy, and with decreasing climatological event probability. The sampling variance of BSS is more robust to serial correlation than that of BS. Hypothesis tests based on BSS are seen to be more powerful (i.e. more sensitive) than those based on BS, and substantially so for lower‐accuracy forecasts of lower‐probability events, for both serially correlated and temporally independent forecasts. Copyright © 2010 Royal Meteorological Society