Observation‐based evaluation of ensemble reliability

Abstract

In order to obtain new insight into the reliability of the ensemble prediction system of the European Centre for Medium‐range Weather Forecasts (ECMWF), we compare the ensemble spread‐error relationship obtained from an observation‐based verification to the one obtained from an analysis‐based verification. Observations used in this study are mainly radiosonde temperatures and radiance measurements from the AMSU‐A channel 5 microwave temperature sounder. The observation operators from the 4D‐Var data assimilation scheme are used to map the forecasts into observation space. In ‘observation‐space’, observed radiances are compared with forecast radiances, derived from the ensemble's atmospheric profiles of temperature, gas concentrations, cloud, and surface properties using the ‘RTTOV’ radiative transfer code.The observation‐space assessment yields different results than the analysis‐based assessment in the extratropics for short‐range forecasts (1‐day), and in the Tropics in general. In the extratropics, for 5‐day forecasts the discrepancy between the analysis‐based and observation‐based verification is small and the ensemble variances are quite reliable.The observation‐based diagnostics indicate that the stochastic model error schemes contribute to the well‐tuned ensemble spread in the extratropics, but can degrade the reliability of the ensemble in the Tropics. It is suggested that observation‐based diagnostics should be used more routinely to diagnose the ensemble performance, and help diagnosing the effectiveness of model error schemes and estimating the amplitude of the initial perturbations.