The fractions skill score for ensemble forecast verification

Abstract

AbstractThe fractions skill score (FSS) is a neighbourhood verification method originally designed to verify deterministic forecasts of binary events. Previous studies employed different approaches for computing an ensemble‐based FSS for probabilistic forecast verification. We show that the formulation of an ensemble‐based FSS substantially affects verification results. Comparing four possible approaches, we determine how different ensemble‐based FSS variants depend on ensemble size, neighbourhood size, and forecast event frequency of occurrence. We demonstrate that only one ensemble‐based FSS, which we call the probabilistic FSS (pFSS), is well behaved and reasonably dependent on ensemble size. Furthermore, we derive a relationship to describe how the pFSS behaves with ensemble size. The proposed relationship is similar to a known result for the Brier skill score. Our study uses high‐resolution 1000‐member ensemble precipitation forecasts from a high‐impact weather period. The large ensemble enables us to study the influence of ensemble and neighbourhood size on forecast skill by deriving probabilistic skilful spatial scales.