Analysis of Fractions Skill Score properties for random precipitation fields and ECMWF forecasts

Abstract

The Fractions Skill Score (FSS) is a spatial verification measure that is used for assessing the performance of precipitation forecasts from numerical weather prediction models. Previous studies have shown that the FSS is able to give a direct measure of the error in the placement of the rain. This article takes the approach further and derives analytical expressions and uses Monte‐Carlo simulations for randomly positioned observed and forecast rainfall to reveal further characteristics of the FSS in both infinite and bounded domains. It reveals that the definition of an FSS value that determines the minimum scale at which a forecast should be deemed ‘useful’ (useful forecast criteria) is a meaningful concept and shows how this value increases with increasing fractional rainfall coverage. A study of real forecast data is also presented using 8 years of European Centre for Medium‐Range Weather Forecasts (ECMWF) model forecasts, out to a lead time of 9 days, over domains of differing sizes covering parts of Europe and North Africa. The FSS is examined using different strategies for dealing with the domain boundary and is compared with the analytical study. The findings give practical guidance on how to use the FSS. For most situations a FSS value of >0.5 serves as a good indicator of a useful forecast. The choice of domain size for rainfall forecast verification should consider the typical spatial errors of the forecast. For a domain that is large compared to the typical spatial error, the boundaries have little adverse affect, but this is not the case if the spatial errors start to become comparable to the size of the domain. The evaluation of ECMWF forecasts reveals the extent of the spatial errors that emerge for medium‐range forecasts and show the value of verifying those forecasts using the FSS over an appropriately sized region.