AbstractEnsemble-based filtering or data assimilation methods have proved to be indispensable tools in atmosphere and ocean science as they allow computationally cheap, low-dimensional ensemble state approximation for extremely high-dimensional turbulent dynamical systems. For sparse, accurate, and infrequent observations, which are typical in data assimilation of geophysical systems, ensemble filtering methods can suffer from catastrophic filter divergence, which frequently drives the filter predictions to machine infinity. A two-layer quasigeostrophic equation, which is a classical idealized model for geophysical turbulence, is used to demonstrate catastrophic filter divergence. The mathematical theory of adaptive covariance inflation by Tong et al. and covariance localization are investigated to stabilize the ensemble methods and prevent catastrophic filter divergence. Two forecast models—a coarse-grained ocean code, which ignores the small-scale parameterization, and stochastic superparameterization (SP), which is a seamless multiscale method developed for large-scale models without scale gap between the resolved and unresolved scales—are applied to generate large-scale forecasts with a coarse spatial resolution compared to the full resolution . The methods are tested in various dynamical regimes in ocean with jets and vorticities, and catastrophic filter divergence is documented for the standard filter without inflation. Using the two forecast models, various kinds of covariance inflation with or without localization are compared. It shows that proper adaptive additive inflation can effectively stabilize the ensemble methods without catastrophic filter divergence in all regimes. Furthermore, stochastic SP achieves accurate filtering skill with localization while the ocean code performs poorly even with localization.