Randomized Subensembles: An Approach to Reduce the Risk of Divergence in an Ensemble Kalman Filter Using Cross Validation

Abstract

Abstract In an ensemble Kalman filter, when the analysis update of an ensemble member is computed using error statistics estimated from an ensemble that includes the background of the member being updated, the spread of the resulting ensemble systematically underestimates the uncertainty of the ensemble mean analysis. This problem can largely be avoided by applying cross validation: using an independent subset of ensemble members for updating each member. However, in some circumstances cross validation can lead to the divergence of one or more ensemble members from observations. This can culminate in catastrophic filter divergence in which the analyzed or forecast states become unrealistic in the diverging members. So far, such instabilities have been reported only in the context of highly nonlinear low-dimensional models. The first known manifestation of catastrophic filter divergence caused by the use of cross validation in an NWP context is reported here. To reduce the risk of such filter divergence, a modification to the traditional cross-validation approach is proposed. Instead of always assigning the ensemble members to the same subensembles, the members forming each subensemble are randomly chosen at every analysis step. It is shown that this new approach can prevent filter divergence and also brings a cycling ensemble data assimilation system containing divergent members back to a state consistent with Gaussianity. The randomized subensemble approach was implemented in the operational global ensemble prediction system at Environment and Climate Change Canada on 1 December 2021.