Stochastic Ensemble Kalman Filter Research Articles

Comparison of uncertainty quantification methods for cloud simulation

AbstractQuantification of evolving uncertainties is required for both probabilistic forecasting and data assimilation in weather prediction. In current practice, the ensemble of model simulations is often used as a primary tool to describe the required uncertainties. In this work, we explore an alternative approach, the so‐called stochastic Galerkin (SG) method, which integrates uncertainties forward in time using a spectral approximation in stochastic space. In an idealized two‐dimensional model that couples nonhydrostatic weakly compressible Navier–Stokes equations to cloud variables, we first investigate the propagation of initial uncertainty. Propagation of initial perturbations is followed through time for all model variables during two types of forecast: the ensemble forecast and the SG forecast. Series of experiments indicate that differences in performance of the two methods depend on the system state and truncations used. For example, in more stable conditions, the SG method outperforms the ensemble of simulations for every truncation and every variable. However, in unstable conditions, the ensemble of simulations would need more than 100 members (depending on the model variable) and the SG method more than a truncation at five to produce comparable but not identical results. As estimates of the uncertainty are crucial for data assimilation, secondly we instigate the use of these two methods with the stochastic ensemble Kalman filter. The use of the SG method avoids evolution of a large ensemble, which is usually the most expensive component of the data assimilation system, and provides results comparable with the ensemble Kalman filter in the cases investigated.

Efficient derivative-free Bayesian inference for large-scale inverse problems

We consider Bayesian inference for large-scale inverse problems, where computational challenges arise from the need for repeated evaluations of an expensive forward model. This renders most Markov chain Monte Carlo approaches infeasible, since they typically require model runs, or more. Moreover, the forward model is often given as a black box or is impractical to differentiate. Therefore derivative-free algorithms are highly desirable. We propose a framework, which is built on Kalman methodology, to efficiently perform Bayesian inference in such inverse problems. The basic method is based on an approximation of the filtering distribution of a novel mean-field dynamical system, into which the inverse problem is embedded as an observation operator. Theoretical properties are established for linear inverse problems, demonstrating that the desired Bayesian posterior is given by the steady state of the law of the filtering distribution of the mean-field dynamical system, and proving exponential convergence to it. This suggests that, for nonlinear problems which are close to Gaussian, sequentially computing this law provides the basis for efficient iterative methods to approximate the Bayesian posterior. Ensemble methods are applied to obtain interacting particle system approximations of the filtering distribution of the mean-field model; and practical strategies to further reduce the computational and memory cost of the methodology are presented, including low-rank approximation and a bi-fidelity approach. The effectiveness of the framework is demonstrated in several numerical experiments, including proof-of-concept linear/nonlinear examples and two large-scale applications: learning of permeability parameters in subsurface flow; and learning subgrid-scale parameters in a global climate model. Moreover, the stochastic ensemble Kalman filter and various ensemble square-root Kalman filters are all employed and are compared numerically. The results demonstrate that the proposed method, based on exponential convergence to the filtering distribution of a mean-field dynamical system, is competitive with pre-existing Kalman-based methods for inverse problems.

Ensemble Riemannian data assimilation: towards large-scale dynamical systems

Abstract. This paper presents the results of the ensemble Riemannian data assimilation for relatively high-dimensional nonlinear dynamical systems, focusing on the chaotic Lorenz-96 model and a two-layer quasi-geostrophic (QG) model of atmospheric circulation. The analysis state in this approach is inferred from a joint distribution that optimally couples the background probability distribution and the likelihood function, enabling formal treatment of systematic biases without any Gaussian assumptions. Despite the risk of the curse of dimensionality in the computation of the coupling distribution, comparisons with the classic implementation of the particle filter and the stochastic ensemble Kalman filter demonstrate that, with the same ensemble size, the presented methodology could improve the predictability of dynamical systems. In particular, under systematic errors, the root mean squared error of the analysis state can be reduced by 20 % (30 %) in the Lorenz-96 (QG) model.

Ensemble Kalman filter based data assimilation for tropical waves in the MJO skeleton model

AbstractThe Madden–Julian oscillation (MJO) is the dominant component of tropical intraseasonal variability, with wide‐reaching impacts even on extratropical weather and climate patterns. However, predicting the MJO is challenging. One reason is the suboptimal state estimates obtained with standard data assimilation (DA) approaches. These are typically based on filtering methods with Gaussian approximations and do not take into account physical properties that are important specifically for the MJO. In this article, a constrained ensemble DA method is applied to study the impact of different physical constraints on the state estimation and prediction of the MJO. The quadratic programming ensemble (QPEns) algorithm utilized extends the standard stochastic ensemble Kalman filter (EnKF) with specifiable constraints on the updates of all ensemble members. This allows us to recover physically more consistent states and to respect possible associated non‐Gaussian statistics. The study is based on identical twin experiments with an adopted nonlinear model for tropical intraseasonal variability. This so‐called skeleton model succeeds in reproducing the main large‐scale features of the MJO and closely related tropical waves, while keeping adequate simplicity for fast experiments on intraseasonal time‐scales. Conservation laws and other crucial physical properties from the model are examined as constraints in the QPEns. Our results demonstrate an overall improvement in the filtering and forecast skill when the model's total energy is conserved in the initial conditions. The degree of benefit is found to be dependent on the observational setup and the strength of the model's nonlinear dynamics. It is also shown that, even in cases where the statistical error in some waves remains comparable with the stochastic EnKF during the DA stage, their prediction is improved remarkably when using the initial state resulting from the QPEns.

Data Assimilation by Stochastic Ensemble Kalman Filtering to Enhance Turbulent Cardiovascular Flow Data From Under-Resolved Observations.

We propose a data assimilation methodology that can be used to enhance the spatial and temporal resolution of voxel-based data as it may be obtained from biomedical imaging modalities. It can be used to improve the assessment of turbulent blood flow in large vessels by combining observed data with a computational fluid dynamics solver. The methodology is based on a Stochastic Ensemble Kalman Filter (SEnKF) approach and geared toward pulsatile and turbulent flow configurations. We describe the observed flow fields by a mean value and its covariance. These flow fields are combined with forecasts obtained from a direct numerical simulation of the flow field. The method is validated against canonical pulsatile and turbulent flows. Finally, it is applied to a clinically relevant configuration, namely the flow downstream of a bioprosthetic valve in an aorta phantom. It is demonstrated how the 4D flow field obtained from experimental observations can be enhanced by the data assimilation algorithm. Results show that the presented method is promising for future use with in vivo data from 4D Flow Magnetic Resonance Imaging (4D Flow MRI). 4D Flow MRI returns spatially and temporally averaged flow fields that are limited by the spatial and the temporal resolution of the tool. These averaged flow fields and the associated uncertainty might be used as observation data in the context of the proposed methodology.

Ensemble Kalman filter for vortex models of disturbed aerodynamic flows

The task of dynamic flow estimation is to construct an approximation of an evolving flow---and particularly, its response to disturbances---using measurements from available sensors. Building from previous work by Darakananda et al.~(Phys Rev Fluids 2018), we further develop an ensemble Kalman filter (EnKF) framework for aerodynamic flows based on an ensemble of randomly-perturbed inviscid vortex models of flow about an infinitely-thin plate. In the forecast step, vortex elements in each ensemble member are advected by the flow and new elements are released from each edge of the plate; the elements are aggregated to maintain an efficient representation. The vortex elements and leading edge constraint are corrected in the analysis step by assimilating the surface pressure differences across the plate measured from the truth system. We show that the overall framework can be physically interpreted as a series of adjustments to the position and shape of an elliptical region of uncertainty associated with each vortex element. In this work, we compare the previously-used stochastic EnKF with the ensemble transform Kalman filter (ETKF), which uses a deterministic analysis step. We examine the response of the flat plate at $20^\circ$ in two perturbed flows, with truth data obtained from high-fidelity simulation at Reynolds number 500. In the first case, we apply a sequence of large-amplitude pulses near the leading edge of the plate to mimic flow actuation. In the second, we place the plate in a vortex street wake behind a cylinder. In both cases, we show that the vortex-based framework accurately estimates the pressure distribution and normal force, with no {\em a priori} knowledge of the perturbations. We show that the ETKF is consistently more robust than the stochastic EnKF. Finally, we examine the mapping from measurements to state update in the analysis step through SVD of the Kalman gain.

An Efficient Algorithm for Stochastic Ensemble Smoothing

The state of the environment is assessed with a mathematical model and observational data by using a data assimilation procedure. At present the ensemble Kalman filter is one of the most popular data assimilation algorithms. An important component of the data assimilation procedure is assessment not only of the predicted values, but also of parameters that are not described by the model. A single correction procedure from observational data in the ensemble Kalman filter may not provide the required accuracy. In this regard, an ensemble smoothing algorithm in which data from a certain time interval are used to estimate values at a given time is becoming increasingly popular. This paper considers a generalization of a previously proposed algorithm which is a version of the stochastic ensemble Kalman filter. The generalized algorithm is an ensemble smoothing algorithm in which smoothing is performed for the sample mean, and then the ensemble of perturbations is transformed. The transformation matrix proposed in this paper is used to estimate both the predicted value and the parameter. An important advantage of the algorithm is its locality, which makes it possible to estimate the parameter in a given domain. The paper provides a justification of the applicability of this algorithm to ensemble smoothing. Test calculations are performed with a one-dimensional model of transport and diffusion of a passive pollutant. The algorithm is efficient and can be used to assess the state of the environment.

Ensemble Transform Algorithms for Nonlinear Smoothing Problems

Several numerical tools designed to overcome the challenges of smoothing in a nonlinear and non-Gaussian setting are investigated for a class of particle smoothers. The considered family of smoothers is induced by the class of linear ensemble transform filters which contains classical filters such as the stochastic ensemble Kalman filter, the ensemble square root filter, and the recently introduced nonlinear ensemble transform filter. Further the ensemble transform particle smoother is introduced and particularly highlighted as it is consistent in the particle limit and does not require assumptions with respect to the family of the posterior distribution. The linear update pattern of the considered class of linear ensemble transform smoothers allows one to implement important supplementary techniques such as adaptive spread corrections, hybrid formulations, and localization in order to facilitate their application to complex estimation problems. These additional features are derived and numerically investigated for a sequence of increasingly challenging test problems.

A Stochastic Ensemble Kalman Filter with Perturbation Ensemble Transformation

The Kalman filter is currently one of the most popular approaches to solving the data assimilation problem. A major line of the application of the Kalman filter to data assimilation is the ensemble approach. In this paper, a version of the stochastic ensemble Kalman filter is considered. In this algorithm, an ensemble of analysis errors is obtained by transforming an ensemble of forecast errors. The analysis step is made only for a mean value. Thus, the ensemble π-algorithm combines the advantages of stochastic filters and the efficiency and locality of square root filters. A numerical method of implementing the ensemble π-algorithm is proposed, and the validity of this method is proved. This algorithm is used for a test problem in a three-dimensional domain. The results of numerical experiments with model data for estimating the efficiency of the algorithm are presented. A comparative analysis of the behavior of the root-mean-square errors of the ensemble π-algorithm and the Kalman ensemble filter by means of numerical experiments with a one-dimensional Lorentz model is performed.

Application of the ensemble Kalman filter to environmental data assimilation

Assessment of the state of the environment with observational data is one of the most urgent modern issues. Such an assessment can be made using forecast models based on data assimilation systems. One of the most popular algorithms for data assimilation is the ensemble Kalman filter, in which the forecast error covariance is estimated using an ensemble of forecasts for perturbed initial fields. Parameter estimation is an important part of atmospheric chemistry modelling. In particular, pollutant emission may be a parameter to be estimated. A single-time estimation based on observations may not give the required accuracy. In this context, the method of ensemble smoothing (EnKS), which uses data from the entire time interval to estimate the parameter at a given time, is becoming increasingly popular. In this paper, we consider a generalization of a previously proposed method called the ensemble π-algorithm, which is a variant of stochastic ensemble Kalman filter. The generalized algorithm is an ensemble smoothing algorithm in which ensemble smoothing is performed for the sample average value and then the ensemble of perturbations is transformed. The proposed algorithm is stochastic. Numerical experiments with a 1-dimensional advection-diffusion model are carried out with the smoothing algorithm proposed in the article.

Assimilation of semi‐qualitative observations with a stochastic ensemble Kalman filter

The ensemble Kalman filter assumes observations to be Gaussian random variables with a pre‐specified mean and variance. In practice, observations may also have detection limits, for instance when a gauge has a minimum or maximum value. In such cases, most data assimilation schemes discard out‐of‐range values, treating them as “not a number,” with the loss of possibly useful qualitative information.The current work focuses on the development of a data assimilation scheme that tackles observations with a detection limit. We present the Ensemble Kalman Filter Semi‐Qualitative (EnKF‐SQ) and test its performance against the Partial Deterministic Ensemble Kalman Filter (PDEnKF) of Borup et al. Both are designed to assimilate out‐of‐range observations explicitly: the out‐of‐range values are qualitative by nature (inequalities), but one can postulate a probability distribution for them and then update the ensemble members accordingly. The EnKF‐SQ is tested within the framework of twin experiments, using both linear and nonlinear toy models. Different sensitivity experiments are conducted to assess the influence of the ensemble size, observation detection limit and number of observations on the performance of the filter. Our numerical results show that assimilating qualitative observations using the proposed scheme improves the overall forecast mean, making it viable for testing on more realistic applications such as sea‐ice models.

Ensemble Kalman Filtering with One-Step-Ahead Smoothing

The ensemble Kalman filter (EnKF) is widely used for sequential data assimilation. It operates as a succession of forecast and analysis steps. In realistic large-scale applications, EnKFs are implemented with small ensembles and poorly known model error statistics. This limits their representativeness of the background error covariances and, thus, their performance. This work explores the efficiency of the one-step-ahead (OSA) smoothing formulation of the Bayesian filtering problem to enhance the data assimilation performance of EnKFs. Filtering with OSA smoothing introduces an updated step with future observations, conditioning the ensemble sampling with more information. This should provide an improved background ensemble in the analysis step, which may help to mitigate the suboptimal character of EnKF-based methods. Here, the authors demonstrate the efficiency of a stochastic EnKF with OSA smoothing for state estimation. They then introduce a deterministic-like EnKF-OSA based on the singular evolutive interpolated ensemble Kalman (SEIK) filter. The authors show that the proposed SEIK-OSA outperforms both SEIK, as it efficiently exploits the data twice, and the stochastic EnKF-OSA, as it avoids observational error undersampling. They present extensive assimilation results from numerical experiments conducted with the Lorenz-96 model to demonstrate SEIK-OSA’s capabilities.

A local ensemble transform Kalman particle filter for convective‐scale data assimilation

Ensemble data assimilation methods such as the ensemble Kalman filter (EnKF) are a key component of probabilistic weather forecasting. They represent the uncertainty in the initial conditions by an ensemble that incorporates information coming from the physical model with the latest observations. High‐resolution numerical weather prediction models run at operational centres are able to resolve nonlinear and non‐Gaussian physical phenomena such as convection. There is therefore a growing need to develop ensemble assimilation algorithms able to deal with non‐Gaussianity while staying computationally feasible. In the present article, we address some of these needs by proposing a new hybrid algorithm based on the ensemble Kalman particle filter. It is fully formulated in ensemble space and uses a deterministic scheme such that it has the ensemble transform Kalman filter (ETKF) instead of the stochastic EnKF as a limiting case. A new criterion for choosing the proportion of particle filter and ETKF updates is also proposed. The new algorithm is implemented in the Consortium for Small‐scale Modeling (COSMO) framework and numerical experiments in a quasi‐operational convective‐scale set‐up are conducted. The results show the feasibility of the new algorithm in practice and indicate the strong potential of such local hybrid methods, in particular for forecasting non‐Gaussian variables such as wind and hourly precipitation.

Comparison and combination of EAKF and SIR-PF in the Bayesian filter framework

Bayesian estimation theory provides a general approach for the state estimate of linear or nonlinear and Gaussian or non-Gaussian systems. In this study, we first explore two Bayesian-based methods: ensemble adjustment Kalman filter (EAKF) and sequential importance resampling particle filter (SIR-PF), using a well-known nonlinear and non-Gaussian model (Lorenz '63 model). The EAKF, which is a deterministic scheme of the ensemble Kalman filter (EnKF), performs better than the classical (stochastic) EnKF in a general framework. Comparison between the SIR-PF and the EAKF reveals that the former outperforms the latter if ensemble size is so large that can avoid the filter degeneracy, and vice versa. The impact of the probability density functions and effective ensemble sizes on assimilation performances are also explored. On the basis of comparisons between the SIR-PF and the EAKF, a mixture filter, called ensemble adjustment Kalman particle filter (EAKPF), is proposed to combine their both merits. Similar to the ensemble Kalman particle filter, which combines the stochastic EnKF and SIR-PF analysis schemes with a tuning parameter, the new mixture filter essentially provides a continuous interpolation between the EAKF and SIR-PF. The same Lorenz '63 model is used as a testbed, showing that the EAKPF is able to overcome filter degeneracy while maintaining the non-Gaussian nature, and performs better than the EAKF given limited ensemble size.

Mitigating Observation Perturbation Sampling Errors in the Stochastic EnKF

Abstract The stochastic ensemble Kalman filter (EnKF) updates its ensemble members with observations perturbed with noise sampled from the distribution of the observational errors. This was shown to introduce noise into the system and may become pronounced when the ensemble size is smaller than the rank of the observational error covariance, which is often the case in real oceanic and atmospheric data assimilation applications. This work introduces an efficient serial scheme to mitigate the impact of observations’ perturbations sampling in the analysis step of the EnKF, which should provide more accurate ensemble estimates of the analysis error covariance matrices. The new scheme is simple to implement within the serial EnKF algorithm, requiring only the approximation of the EnKF sample forecast error covariance matrix by a matrix with one rank less. The new EnKF scheme is implemented and tested with the Lorenz-96 model. Results from numerical experiments are conducted to compare its performance with the EnKF and two standard deterministic EnKFs. This study shows that the new scheme enhances the behavior of the EnKF and may lead to better performance than the deterministic EnKFs even when implemented with relatively small ensembles.

A Comparison of Ensemble Kalman Filters for Storm Surge Assimilation

Abstract This study evaluates and compares the performances of several variants of the popular ensemble Kalman filter for the assimilation of storm surge data with the advanced circulation (ADCIRC) model. Using meteorological data from Hurricane Ike to force the ADCIRC model on a domain including the Gulf of Mexico coastline, the authors implement and compare the standard stochastic ensemble Kalman filter (EnKF) and three deterministic square root EnKFs: the singular evolutive interpolated Kalman (SEIK) filter, the ensemble transform Kalman filter (ETKF), and the ensemble adjustment Kalman filter (EAKF). Covariance inflation and localization are implemented in all of these filters. The results from twin experiments suggest that the square root ensemble filters could lead to very comparable performances with appropriate tuning of inflation and localization, suggesting that practical implementation details are at least as important as the choice of the square root ensemble filter itself. These filters also perform reasonably well with a relatively small ensemble size, whereas the stochastic EnKF requires larger ensemble sizes to provide similar accuracy for forecasts of storm surge.

Derivative‐free optimization for large‐scale nonlinear data assimilation problems

AbstractThe computation of derivatives and the development of tangent and adjoint codes represent a challenging issue and a major human time‐consuming task when solving operational data assimilation problems. The ensemble Kalman filter provides a suitable derivative‐free adaptation for the sequential approach by using an ensemble‐based implementation of the Kalman filter equations. This article proposes a derivative‐free variant for the variational approach, based on an iterative subspace minimization (ISM) technique. At each iteration, a subspace of low dimension is built from the relevant information contained in the empirical orthogonal functions (EOFs), allowing us to define a reduced 4D‐Var subproblem which is then solved using a derivative‐free optimization (DFO) algorithm. Strategies to improve the quality of the selected subspaces are presented, together with two numerical illustrations. The ISM technique is first compared with a basic stochastic ensemble Kalman filter on an academic shallow‐water problem. The DFO algorithm embedded in the ISM technique is then validated in the NEMO framework, using its GYRE configuration.

Ensemble clustering in deterministic ensemble Kalman filters

ABSTRACTEnsemble clustering (EC) can arise in data assimilation with ensemble square root filters (EnSRFs) using non-linear models: an M-member ensemble splits into a single outlier and a cluster of M–1 members. The stochastic Ensemble Kalman Filter does not present this problem. Modifications to the EnSRFs by a periodic resampling of the ensemble through random rotations have been proposed to address it. We introduce a metric to quantify the presence of EC and present evidence to dispel the notion that EC leads to filter failure. Starting from a univariate model, we show that EC is not a permanent but transient phenomenon; it occurs intermittently in non-linear models. We perform a series of data assimilation experiments using a standard EnSRF and a modified EnSRF by a resampling though random rotations. The modified EnSRF thus alleviates issues associated with EC at the cost of traceability of individual ensemble trajectories and cannot use some of algorithms that enhance performance of standard EnSRF. In the non-linear regimes of low-dimensional models, the analysis root mean square error of the standard EnSRF slowly grows with ensemble size if the size is larger than the dimension of the model state. However, we do not observe this problem in a more complex model that uses an ensemble size much smaller than the dimension of the model state, along with inflation and localisation. Overall, we find that transient EC does not handicap the performance of the standard EnSRF.