Why does EnKF suffer from analysis overconfidence? An insight into exploiting the ever‐increasing volume of observations

Abstract

AbstractEnsemble Kalman filters (EnKF) are empirically known to suffer from insufficient posterior spread and this issue is aggravated when assimilating a large volume of observations. This problem, commonly called analysis underdispersion or analysis overconfidence, has been well recognized, but why it occurs seems to be rather poorly understood. Inspired by the theory of the degrees of freedom for signal, this article investigates this problem by analyzing the trace of the matrix HK, where H and K represent, respectively, the observation operator and the gain matrix. A simple mathematical argument shows that tr HK for EnKF is bounded from above by the ensemble size, which entails that assimilating many more observations than the ensemble size leads automatically to tr HK underestimation, as long as the observations are of accuracy comparable to the background. Since tr HK can be interpreted as the squared spread of the posterior ensemble measured in the normalized observation space, underestimated tr HK implies overconfidence in the analysis spread, which, in a cycled context, requires covariance inflation to be applied. The theory is then extended to cases where covariance localization schemes (either B‐localization or R‐localization) are applied to show how they alleviate the analysis underdispersion. These findings from the mathematical argument are demonstrated with a simple one‐dimensional covariance model. Finally, the findings described above are used to form speculative arguments about how to interpret several puzzling features of the local ensemble transform Kalman filter (LETKF) previously reported in the literature, such as why using fewer observations can lead to better performance, when optimal localization scales tend to occur, and why covariance inflation methods based on the relaxation to prior information approach are particularly successful when observations are distributed inhomogeneously.