Understanding changes of the continuous ranked probability score using a homogeneous Gaussian approximation

Abstract

AbstractImproving ensemble forecasts is a complex process which involves proper scores such as the continuous ranked probability score (CRPS). A homogeneous Gaussian (hoG) model is introduced in order to better understand the characteristics of the CRPS. An analytical formula is derived for the expected CRPS of an ensemble in the hoG model. The score is a function of the variance of the error of the ensemble mean, the mean error of the ensemble mean and the ensemble variance. The hoG model also provides a score decomposition into reliability and resolution components. We examine whether the hoG model provides a useful approximation of the CRPS when applied to operational ECMWF medium‐range ensemble forecasts. The hoG approximation describes the spatial variations of the CRPS well while moderately overestimating the mean score. Seasonal averages over large domains are within 10% of the actual CRPS. Furthermore, the ability to approximate score changes is evaluated by (a) comparing raw ensemble forecasts with postprocessed ensemble forecasts, and (b) by examining score changes associated with a recent upgrade of the IFS. Overall, the hoG approximation predicts the actual CRPS changes well. One of the main anticipated applications of the hoG approximation are new diagnostics in verification software used by NWP developers routinely. The purpose of the diagnostics is to help developers explain impacts of forecast system changes on the CRPS in terms of the changes in mean error, changes in error variance and changes in ensemble variance. The diagnostics require little additional computational resources compared to the alternative of verifying postprocessed versions of the ensemble forecasts. Therefore, it will be feasible to apply the diagnostics easily to all variables that are examined as part of the model development process.