A Statistical Investigation of the Sensitivity of Ensemble-Based Kalman Filters to Covariance Filtering

Abstract

This paper investigates the effects of spatial filtering on the ensemble-based estimate of the background error covariance matrix in an ensemble-based Kalman filter (EnKF). In particular, a novel kernel smoothing method with variable bandwidth is introduced and its performance is compared to that of the widely used Gaspari–Cohn filter, which uses a fifth-order kernel function with a fixed localization length. Numerical experiments are carried out with the 40-variable Lorenz-96 model. The results of the experiments show that the nonparametric approach provides a more accurate estimate of the background error covariance matrix than the Gaspari–Cohn filter with any localization length. It is also shown that the Gaspari–Cohn filter tends to provide more accurate estimates of the covariance with shorter localization lengths. However, the analyses obtained by using longer localization lengths tend to be more accurate than those produced by using short localization lengths or the nonparametric approach. This seemingly paradoxical result is explained by showing that localization with longer localization lengths produces filtered estimates whose time mean is the most similar to the time mean of both the unfiltered estimate and the true covariance. This result suggests that a better metric of covariance filtering skill would be one that combined a measure of closeness to the sample covariance matrix for a very large ensemble with a measure of similarity between the climatological averages of the filtered and sample covariance.