Filter Likelihood as an Observation-Based Verification Metric in Ensemble Forecasting

Abstract

In numerical weather prediction (NWP), ensemble forecasting aims to quantify the flow-dependent forecast uncertainty. The focus here is on observation-based verification of the reliability of ensemble forecasting systems. In particular, at short forecast lead times, forecast errors tend to be relatively small compared to observation errors and hence it is very important that the verification metric also accounts for observational uncertainties. This paper studies the so-called <em>filter likelihood score</em> which is deep-rooted in Bayesian estimation theory and fits naturally to the filtering setup of NWP. The filter likelihood score considers observation errors, ensemble mean skill, and ensemble spread in one metric. Importantly, it can be made multivariate and effortlessly expanded to simultaneous verification against all observation types through the observation operators contained in the parental data assimilation scheme. Here observations from the global radiosonde network and satellites (AMSU-A channel 5) are included in the verification of OpenIFS-based ensemble forecasts using different types of initial state perturbations. Our results show that the filter likelihood score is sensitive to the ensemble prediction system quality and compares consistently with other verification metrics such as the relationships between ensemble spread and ensemble mean forecast error, and Dawid-Sebastiani score. Our conclusion is that the filter likelihood score provides a very well-behaving verification metric, that can be made truly multivariate by including covariances, for ensemble prediction systems with a strong foundation in estimation theory.