Iterative Ensemble Kalman Smoother Research Articles

Using the (Iterative) Ensemble Kalman Smoother to Estimate the Time Correlation in Model Error

Numerical weather prediction systems contain model errors related to missing and simplified physical processes, and limited model resolution. While it has been widely recognized that these model errors need to be included in the data assimilation formulation, providing prior estimates of their spatio-temporal characteristics is a hard problem. We follow a systematic path to estimate parameters in the model error formulation, specifically related to time-correlated model errors. This problem is more difficult than the standard parameter estimation problem because the model error parameters are only visible through the random model error realisations. By concentrating on linear and nonlinear low-dimensional systems, we are able to highlight the many aspects of this problem, using state augmentation in an ensemble Kalman smoother (EnKS) and its iterative variant (IEnKS). It is not possible to estimate the model error parameters in one assimilation window because enough information has to be gathered to see the parameters through the random errors, even when every time step is observed. If only one parameter is estimated in a linear one-dimensional system the EnKS works well, but when we try to estimate two parameters the method fails. An IEnKS is able to find the correct parameter values for the linear system. For the highly nonlinear logistic map the IEnKS can get stuck in local minima, but with careful tuning of the step length in the iterations and careful transformation of the solution space the correct parameter values can be found. The main conclusion is that estimating model error parameters –even in low-dimensional systems– is a difficult problem, but via careful reformulation of the problem practical solutions can be found.

Iterative Ensemble Kalman Smoother Applied to History-Matching Coalbed Methane Wells

This article, written by JPT Technology Editor Chris Carpenter, contains highlights of paper URTEC 208291, “Introducing Iterative Ensemble Kalman Smoother to History-Matching Horizontal Coalbed Methane Wells,” by Rubakumar Sankararaj, SPE, and Sylvain Ducroux, SPE, Resoptima, and Dan Kuznetsov, SPE, Arrow Energy, et al. The paper has not been peer reviewed. This study presents the iterative ensemble Kalman smoother applied to a low-permeability coalbed methane (CBM) field in Australia. All wells in the studied reservoir are completed with an artificial lift system and permanent downhole gauges. A forecast study was conducted to validate the history-matched ensemble. The results showed a good match of 12 months of the new production data not used in history matching, which highlights the robust prediction capabilities of the approach. Geology of the Study Area Because of its strategic geologic and geographic situation in Queensland State, the Bowen-Surat area constitutes one of the more economically significant sedimentary provinces of the Australian mainland and one of the world’s major coal and coal-seam-gas-producing regions. This study considers the Q seam from the Late Permian. Although its thickness varies in the basin because of merging and splitting, it is continuous locally, with an approximate thickness of 5–6 m.

Technology Focus: History Matching and Forecasting (April 2022)

This year’s history matching and forecasting selections, made by reviewer Gopi Nalla of DeGolyer and MacNaughton, reflect the importance of accurate and innovative methodology in the approach toward development of unconventional or challenging plays, from tight oil to highly heterogeneous gas fields to coalbed methane. The authors of paper URTEC 208352 evaluate and compare the performance of rate-normalization and pressure-deconvolution techniques for both synthetic and tight-oil examples. While the synopsis is devoted mostly to the authors’ work in applying these techniques to synthetic cases, much of the complete paper is devoted to tight-oil examples. Ultimately, the authors recommend the pressure-deconvolution approach generally. In paper SPE 207933, the authors apply an integrated approach of using reservoir pressure/gas compressibility (P/Z) calculations to obtain a field gas initially in place (FGIIP) estimation that is then incorporated into an integrated asset model. The technique is applied to a giant onshore gas field. The authors conclude that the new FGIIP estimation can be applied as a reference to re-review the static modeling legacy and to narrow static modeling uncertainties, leading to reliable forecasting and more-efficient field development. An application of the iterative ensemble Kalman smoother to a scenario involving horizontal coalbed-methane wells for a low-permeability field in Australia is the subject of paper URTEC 208291. A forecast study was conducted to validate the history-matched ensemble, with the results showing a good match of 12 months of the new production data not used in history matching, highlighting the robust prediction capabilities of the presented approach. SPE papers continue to a be a vital resource for industry professionals; arguably, such work is more important than ever as the industry and the world adjusts to new modes of collaboration and realities in both the office and the field. I invite you to read the full text of these papers on OnePetro and to find further recent works that advance the literature of this specialized but critical Tech Focus topic. Recommended additional reading at OnePetro: www.onepetro.org. URTEC 208361 - Effect of Relative Permeability on Modeling of Shale Oil and Gas Production by Hamid Behmanesh, University of Calgary, et al. SPE 204835 - Successful Case Study of Machine-Learning Application To Streamline and Improve History-Matching Process for Complex Gas-Condensate Reservoirs in Hai Thach Field, Offshore Vietnam by Son Hoang, Bien Dong Petroleum Operating Company, et al. SPE 207855 - Unleashing the Potential of Relative Permeability Using Artificial Intelligence by Abdur Rahman Shah, Schlumberger, et al.

Characterizing viscoelastic materials via ensemble-based data assimilation of bubble collapse observations

Viscoelastic material properties at high strain rates are needed to model many biological and medical systems. Bubble cavitation can induce such strain rates, and the resulting bubble dynamics are sensitive to the material properties. Thus, in principle, these properties can be inferred via measurements of the bubble dynamics. Estrada et al. (2018) demonstrated such bubble-dynamic high-strain-rate rheometry by using least-squares shooting to minimize the difference between simulated and experimental bubble radius histories. We generalize their technique to account for additional uncertainties in the model, initial conditions, and material properties needed to uniquely simulate the bubble dynamics. Ensemble-based data assimilation minimizes the computational expense associated with the bubble cavitation model , providing a more efficient and scalable numerical framework for bubble-collapse rheometry. We test an ensemble Kalman filter (EnKF), an iterative ensemble Kalman smoother (IEnKS), and a hybrid ensemble-based 4D-Var method (En4D-Var) on synthetic data, assessing their estimations of the viscosity and shear modulus of a Kelvin–Voigt material. Results show that En4D-Var and IEnKS provide better moduli estimates than EnKF. Applying these methods to the experimental data of Estrada et al. (2018) yields similar material property estimates to those they obtained, but provides additional information about uncertainties. In particular, the En4D-Var yields lower viscosity estimates for some experiments, and the dynamic estimators reveal a potential mechanism that is unaccounted for in the model, whereby the apparent viscosity is reduced in some cases due to inelastic behavior, possibly in the form of material damage occurring at bubble collapse.

Batch seismic inversion using the iterative ensemble Kalman smoother

An ensemble-based method for seismic inversion to estimate elastic attributes is considered, namely the iterative ensemble Kalman smoother. The main focus of this work is the challenge associated with ensemble-based inversion of seismic waveform data. The amount of seismic data is large and, depending on ensemble size, it cannot be processed in a single batch. Instead a solution strategy of partitioning the data recordings in time windows and processing these sequentially is suggested. This work demonstrates how this partitioning can be done adaptively, with a focus on reliable and efficient estimation. The adaptivity relies on an analysis of the update direction used in the iterative procedure, and an interpretation of contributions from prior and likelihood to this update. The idea is that these must balance; if the prior dominates, the estimation process is inefficient while the estimation is likely to overfit and diverge if data dominates. Two approaches to meet this balance are formulated and evaluated. One is based on an interpretation of eigenvalue distributions and how this enters and affects weighting of prior and likelihood contributions. The other is based on balancing the norm magnitude of prior and likelihood vector components in the update. Only the latter is found to sufficiently regularize the data window. Although no guarantees for avoiding ensemble divergence are provided in the paper, the results of the adaptive procedure indicate that robust estimation performance can be achieved for ensemble-based inversion of seismic waveform data.

Improving Numerical Dispersion Modelling in Built Environments with Data Assimilation Using the Iterative Ensemble Kalman Smoother

Air-pollution modelling at the local scale requires accurate meteorological inputs such as from the velocity field. These meteorological fields are generally simulated with microscale models (here Code_Saturne), which are forced with boundary conditions provided by larger scale models or observations. Local atmospheric simulations are very sensitive to the boundary conditions, whose accurate estimation is difficult but crucial. When observations of the wind speed and turbulence or pollutant concentration are available inside the domain, they provide supplementary information via data assimilation, to enhance the simulation accuracy by modifying the boundary conditions. Among the existing data assimilation methods, the iterative ensemble Kalman smoother (IEnKS) is adapted to urban-scale simulations. This method has already been found to increase the accuracy of wind-resource assessment. Here we assess the ability of the IEnKS method to improve scalar-dispersion modelling—an important component of air-quality modelling—by assimilating perturbed measurements inside the urban canopy. To test the data assimilation method in urban conditions, we use the observations provided by the Mock Urban Setting Test field campaign and consider cases with neutral and stable conditions, and the boundary conditions consisting of the horizontal velocity components and turbulence. We prove the capacity of the IEnKS method to assimilate observations of velocity as well as pollutant concentration. In both cases, the accuracy of pollutant concentration estimates is enhanced by 40–60%. We also show that assimilating both types of observations allows further improvements of turbulence predictions by the model.

An Iterative Ensemble Kalman Smoother in Presence of Additive Model Error

Ensemble variational methods are being increasingly used in the field of geophysical data assimilation. Their efficiency comes from the combined use of ensembles, which provide statistics estimates, and a variational analysis, which handles nonlinear operators through iterative optimization techniques. Taking model error into account in four-dimensional ensemble variational algorithms is challenging because the state trajectory over the data assimilation window (DAW) is no longer determined by its sole initial condition. In particular, the control variable dimension scales with the DAW length, which yields a high numerical complexity. This is unfortunate since accuracy improvement is expected with longer DAWs. Building upon the work of [P. Sakov and M. Bocquet, Tellus A, 70 (2018), 1414545], this paper discusses how to algorithmically construct and numerically test an iterative ensemble Kalman smoother with additive model error (IEnKS-Q) which is thought to be the natural weak constraint generalization of the IEnKS [M. Bocquet and P. Sakov, Quart. J. Roy. Meteorol. Soc., 140 (2014), pp. 1521--1535], as well as the generalization of IEnKF-Q [P. Sakov, J. Haussaire, and M. Bocquet, Quart. J. Roy. Meteorol. Soc., 144 (2018), pp. 1297--1309] to general DAWs. The number of model evaluations per cycle of the IEnKS-Q is also examined. Solutions based on perturbation decomposition are proposed to dissociate those numerically costly evaluations from the control variable dimension.

Revising the stochastic iterative ensemble smoother

Abstract. Ensemble randomized maximum likelihood (EnRML) is an iterative (stochastic) ensemble smoother, used for large and nonlinear inverse problems, such as history matching and data assimilation. Its current formulation is overly complicated and has issues with computational costs, noise, and covariance localization, even causing some practitioners to omit crucial prior information. This paper resolves these difficulties and streamlines the algorithm without changing its output. These simplifications are achieved through the careful treatment of the linearizations and subspaces. For example, it is shown (a) how ensemble linearizations relate to average sensitivity and (b) that the ensemble does not lose rank during updates. The paper also draws significantly on the theory of the (deterministic) iterative ensemble Kalman smoother (IEnKS). Comparative benchmarks are obtained with the Lorenz 96 model with these two smoothers and the ensemble smoother using multiple data assimilation (ES-MDA).

Improving CFD atmospheric simulations at local scale for wind resource assessment using the iterative ensemble Kalman smoother

Accurate wind fields simulated by CFD models are necessary for many environmental and safety micro-meteorological applications, such as wind resource assessment. Atmospheric simulations at local scale are largely determined by boundary conditions (BCs), which are generally provided by mesoscale models (e.g., WRF). In order to improve the accuracy of the BCs, especially in the lowest levels, data assimilation methods might be used to take available observations into account. Among the existing data assimilation methods, the iterative ensemble Kalman smoother (IEnKS) has been chosen and adapted to micro-meteorology by taking BCs into account. In the present study, we assess the ability of the IEnKS to improve wind simulations over a very complex topography, by assimilating a few in situ observations. The IEnKS is tested with the CFD model Code_Saturne in 2D and 3D using both twin experiments and field observations. We propose a method to determine the first estimate of the BCs and to construct the associated background error covariance matrix, from the statistical analysis of three years of WRF simulations. The IEnKS is proved to greatly reduce the error and the uncertainty of the BCs and thus of the simulated wind field. Consequently, the wind potential is more accurately estimated.

Quasi-static ensemble variational data assimilation: a theoretical and numerical study with the iterative ensemble Kalman smoother

Abstract. The analysis in nonlinear variational data assimilation is the solution of a non-quadratic minimization. Thus, the analysis efficiency relies on its ability to locate a global minimum of the cost function. If this minimization uses a Gauss–Newton (GN) method, it is critical for the starting point to be in the attraction basin of a global minimum. Otherwise the method may converge to a local extremum, which degrades the analysis. With chaotic models, the number of local extrema often increases with the temporal extent of the data assimilation window, making the former condition harder to satisfy. This is unfortunate because the assimilation performance also increases with this temporal extent. However, a quasi-static (QS) minimization may overcome these local extrema. It accomplishes this by gradually injecting the observations in the cost function. This method was introduced by Pires et al. (1996) in a 4D-Var context. We generalize this approach to four-dimensional strong-constraint nonlinear ensemble variational (EnVar) methods, which are based on both a nonlinear variational analysis and the propagation of dynamical error statistics via an ensemble. This forces one to consider the cost function minimizations in the broader context of cycled data assimilation algorithms. We adapt this QS approach to the iterative ensemble Kalman smoother (IEnKS), an exemplar of nonlinear deterministic four-dimensional EnVar methods. Using low-order models, we quantify the positive impact of the QS approach on the IEnKS, especially for long data assimilation windows. We also examine the computational cost of QS implementations and suggest cheaper algorithms.

Data assimilation at local scale to improve CFD simulations of atmospheric dispersion: application to 1D shallow-water equations and method comparisons

Atmospheric dispersion modelling requires meteorological inputs over local domains with possibly complex topographies. These local wind fields may be difficult to simulate with CFD models, in particular because of their sensitivity to geometrical features and to model inputs, especially the boundary conditions which are generally provided by larger-scale models or measurements. Using data assimilation, a few measurements inside the domain could add information to the imprecise boundary conditions and thus greatly enhance the precision of the dispersion simulations. Three data assimilation techniques (3DVar, the back and forth nudging algorithm, and the iterative ensemble Kalman smoother) have been adapted to local scale simulations by taking boundary conditions into account instead of initial conditions for which they are usually applied. Their performances have been evaluated at small scales, with a simple representation of the atmosphere into two layers, using 1D solution of the shallow-water equations.

Four-dimensional ensemble variational data assimilation and the unstable subspace

The performance of (ensemble) Kalman filters used for data assimilation in the geosciences critically depends on the dynamical properties of the evolution model. A key aspect is that the error covariance matrix is asymptotically supported by the unstable–neutral subspace only, i.e. it is spanned by the backward Lyapunov vectors with non-negative exponents. The analytic proof of such a property for the Kalman filter error covariance has been recently given, and in particular that of its confinement to the unstable–neutral subspace. In this paper, we first generalize those results to the case of the Kalman smoother in a linear, Gaussian and perfect model scenario. We also provide square-root formulae for the filter and smoother that make the connection with ensemble formulations of the Kalman filter and smoother, where the span of the error covariance is described in terms of the ensemble deviations from the mean. We then discuss how this neat picture is modified when the dynamics are nonlinear and chaotic, and for which analytic results are precluded or difficult to obtain. A numerical investigation is carried out to study the approximate confinement of the anomalies for both a deterministic ensemble Kalman filter (EnKF) and a four-dimensional ensemble variational method, the iterative ensemble Kalman smoother (IEnKS), in a perfect model scenario. The confinement is characterized using geometrical angles that determine the relative position of the anomalies with respect to the unstable–neutral subspace. The alignment of the anomalies and of the unstable–neutral subspace is more pronounced when observation precision or frequency, as well as the data assimilation window length for the IEnKS, are increased. These results also suggest that the IEnKS and the deterministic EnKF realize in practice (albeit implicitly) the paradigm behind the approach of Anna Trevisan and co-authors known as the assimilation in the unstable subspace.

Estimating model evidence using data assimilation

We review the field of data assimilation (DA) from a Bayesian perspective and show that, in addition to its by now common application to state estimation, DA may be used for model selection. An important special case of the latter is the discrimination between a factual model–which corresponds, to the best of the modeller's knowledge, to the situation in the actual world in which a sequence of events has occurred–and a counterfactual model, in which a particular forcing or process might be absent or just quantitatively different from the actual world. Three different ensemble‐DA methods are reviewed for this purpose: the ensemble Kalman filter (EnKF), the ensemble four‐dimensional variational smoother (En‐4D‐Var), and the iterative ensemble Kalman smoother (IEnKS). An original contextual formulation of model evidence (CME) is introduced. It is shown how to apply these three methods to compute CME, using the approximated time‐dependent probability distribution functions (pdfs) each of them provide in the process of state estimation. The theoretical formulae so derived are applied to two simplified nonlinear and chaotic models: (i) the Lorenz three‐variable convection model (L63), and (ii) the Lorenz 40‐variable midlatitude atmospheric dynamics model (L95). The numerical results of these three DA‐based methods and those of an integration based on importance sampling are compared. It is found that better CME estimates are obtained by using DA, and the IEnKS method appears to be best among the DA methods. Differences among the performance of the three DA‐based methods are discussed as a function of model properties. Finally, the methodology is implemented for parameter estimation and for event attribution.

Stochastic history matching to time-lapse seismic of a CO2-EOR project sector model

A stochastic seismic history matching workflow has been applied to a single well pattern sector from the Phase 1A development area of the Weyburn CO2-EOR project. Several experiments were conducted aimed at updating permeability and porosity at all grid cells in the model such that mismatches with time-lapse seismic data interpretations of CO2 spreading over the period 1999–2005 are minimized. The applied history matching methodology consists of the combination of an iterative Ensemble Kalman Smoother and a novel seismic data parameterization based on front positions. Synthetic data experiments were first conducted to verify the suitability of the methodology. These synthetic experiments showed that the method is able to produce multiple models that simulate CO2 front positions that are consistent with those determined from time-lapse seismic data. The algorithm was subsequently applied to inversion of the real data. A number of indicators such as low signal to noise ratio, non-monotonic CO2 spreading, differences in published interpretations, and inconsistency with simulations, suggest that the quality of the real seismic data is rather low. As a result, use of the real data set produced results of a more diverse nature. It was found that the 2002 time-lapse seismic map could be reproduced more accurately after history matching than before history matching and more accurately than the available reference model. The 2001 and 2004 maps could not be reproduced with the same accuracy. A combined match to the 2002 seismic and production data showed a 78% reduction of the total data mismatch. These results could be achieved at relatively low computational cost and thus demonstrate the potential for use of assisted history matching workflows in the management of CO2-EOR projects. The results also point towards limitations in both the seismic data and model quality for this particular case. The validity of some of the assumptions used in constructing the sector model was investigated in more detail. It was concluded that they cannot fully explain the remaining mismatch. Low grid resolution and possibly the absence of a fracture network in the model are identified as possible candidates for further investigation.

A low-order coupled chemistry meteorology model for testing online and offline data assimilation schemes: L95-GRS (v1.0)

Abstract. Bocquet and Sakov (2013) introduced a low-order model based on the coupling of the chaotic Lorenz-95 (L95) model, which simulates winds along a mid-latitude circle, with the transport of a tracer species advected by this zonal wind field. This model, named L95-T, can serve as a playground for testing data assimilation schemes with an online model. Here, the tracer part of the model is extended to a reduced photochemistry module. This coupled chemistry meteorology model (CCMM), the L95-GRS (generic reaction set) model, mimics continental and transcontinental transport and the photochemistry of ozone, volatile organic compounds and nitrogen oxides. Its numerical implementation is described. The model is shown to reproduce the major physical and chemical processes being considered. L95-T and L95-GRS are specifically designed and useful for testing advanced data assimilation schemes, such as the iterative ensemble Kalman smoother (IEnKS), which combines the best of ensemble and variational methods. These models provide useful insights prior to the implementation of data assimilation methods into larger models. We illustrate their use with data assimilation schemes on preliminary yet instructive numerical experiments. In particular, online and offline data assimilation strategies can be conveniently tested and discussed with this low-order CCMM. The impact of observed chemical species concentrations on the wind field estimate can be quantitatively assessed. The impacts of the wind chaotic dynamics and of the chemical species non-chaotic but highly nonlinear dynamics on the data assimilation strategies are illustrated.

Localization and the iterative ensemble Kalman smoother

The iterative ensemble Kalman smoother (IEnKS) is a data assimilation method meant for tracking the state of nonlinear geophysical models efficiently. It combines an ensemble of model states to estimate the errors similarly to the ensemble square‐root Kalman filter, with a four‐dimensional variational analysis performed within the ensemble space. As such, it belongs to the class of four‐dimensional ensemble variational methods. It could require the use of localization of the analysis when the state‐space dimension is high. However, its localization needs to be defined across time and to be as consistent as possible with the dynamical flow within the data assimilation window where the four‐dimensional variational analysis is performed. We show that a Liouville equation governs the time evolution of the localization operator, which is linked to the evolution of the error correlations. It is argued that the time evolution of the localization operator depends strongly on the forecast dynamics. Using either covariance localization or domain localization, we propose and test several localization strategies meant to address the issue: (i) a static and uniform localization, (ii) propagation through the window of a restricted set of dominant modes of the error covariance matrix and (iii) the approximate propagation of the localization operator using covariant local domains that are moved in accordance with the dynamical flow. These schemes are illustrated with the one‐dimensional Lorenz 40 variable model and with a two‐dimensional barotropic vorticity model. In both cases, local analysis based on the covariant local domains leads to a systematic improvement of the data assimilation performance.

Iterative regularization for ensemble data assimilation in reservoir models

We propose the application of iterative regularization for the development of ensemble methods for solving Bayesian inverse problems. In concrete, we construct (i) a variational iterative regularizing ensemble Levenberg-Marquardt method (IR-enLM) and (ii) a derivative-free iterative ensemble Kalman smoother (IR-ES). The aim of these methods is to provide a robust ensemble approximation of the Bayesian posterior. The proposed methods are based on fundamental ideas from iterative regularization methods that have been widely used for the solution of deterministic inverse problems (Katltenbacher et al. de Gruyter, Berlin 2008). In this work, we are interested in the application of the proposed ensemble methods for the solution of Bayesian inverse problems that arise in reservoir modeling applications. The proposed ensemble methods use key aspects of the regularizing Levenberg-Marquardt scheme developed by Hanke (Inverse Problems 13,79–95 1997) and that we recently applied for history matching in Iglesias (Comput. Geosci. 1–21 2013). Unlike most existing methods where the stopping criteria and regularization parameters are typically selected heuristically, in the proposed ensemble methods, the discrepancy principle is applied for (i) the selection of the regularization parameters and (ii) the early termination of the scheme. The discrepancy principle is key for the theory of iterative regularization, and the purpose of the present work is to apply this principle for the development of ensemble methods defined as iterative updates of solutions to linear ill-posed inverse problems. The regularizing and convergence properties of iterative regularization methods for deterministic inverse problems have long been established. However, the approximation properties of the proposed ensemble methods in the context of Bayesian inverse problems is an open problem. In the case where the forward operator is linear and the prior is Gaussian, we show that the tunable parameters of the proposed IR-enLM and IR-ES can be chosen so that the resulting schemes coincide with the standard randomized maximum likelihood (RML) and the ensemble smoother (ES), respectively. Therefore, the proposed methods sample from the posterior in the linear-Gaussian case. Similar to RML and ES methods, in the nonlinear case, one may not conclude that the proposed methods produce samples from the posterior. The present work provides a numerical investigation of the performance of the proposed ensemble methods at capturing the posterior. In particular, we aim at understanding the role of the tunable parameters that arise from the application of iterative regularization techniques. The numerical framework for our investigations consists of using a state-of-the art Markov chain Monte Carlo (MCMC) method for resolving the Bayesian posterior from synthetic experiments. The resolved posterior via MCMC then provides a gold standard against to which compare the proposed IR-enLM and IR-ES. Our numerical experiments show clear indication that the regularizing properties of the regularization methods applied for the computation of each ensemble have significant impact of the approximation properties of the proposed ensemble methods at capturing the Bayesian posterior. Furthermore, we provide a comparison of the proposed regularizing methods with respect to some unregularized methods that have been typically used in the literature. Our numerical experiments showcase the advantage of using iterative regularization for obtaining more robust and stable approximation of the posterior than unregularized methods.

An iterative ensemble Kalman smoother

The iterative ensemble Kalman filter (IEnKF) was recently proposed in order to improve the performance of ensemble Kalman filtering with strongly nonlinear geophysical models. The IEnKF can be used as a lag‐one smoother and extended to a fixed‐lag smoother: the iterative ensemble Kalman smoother (IEnKS). The IEnKS is an ensemble variational method. It does not require the use of the tangent linear of the evolution and observation models, nor the adjoint of these models: the required sensitivities (gradient and Hessian) are obtained from the ensemble. Looking for optimal performance, out of the many possible extensions we consider a quasi‐static algorithm. The IEnKS is explored for the Lorenz '95 model and for a two‐dimensional turbulence model. As the logical extension of the IEnKF, the IEnKS significantly outperforms standard Kalman filters and smoothers in strongly nonlinear regimes. In mildly nonlinear regimes (typically synoptic‐scale meteorology), its filtering performance is marginally but clearly better than the standard ensemble Kalman filter and it keeps improving as the length of the temporal data assimilation window is increased. For long windows, its smoothing performance outranks the standard smoothers very significantly, a result that is believed to stem from the variational but flow‐dependent nature of the algorithm. For very long windows, the use of a multiple data assimilation variant of the scheme, where observations are assimilated several times, is advocated. This paves the way for finer reanalysis, freed from the static prior assumption of 4D‐Var but also partially freed from the Gaussian assumptions that usually impede standard ensemble Kalman filtering and smoothing.

Joint state and parameter estimation with an iterative ensemble Kalman smoother

Abstract. Both ensemble filtering and variational data assimilation methods have proven useful in the joint estimation of state variables and parameters of geophysical models. Yet, their respective benefits and drawbacks in this task are distinct. An ensemble variational method, known as the iterative ensemble Kalman smoother (IEnKS) has recently been introduced. It is based on an adjoint model-free variational, but flow-dependent, scheme. As such, the IEnKS is a candidate tool for joint state and parameter estimation that may inherit the benefits from both the ensemble filtering and variational approaches. In this study, an augmented state IEnKS is tested on its estimation of the forcing parameter of the Lorenz-95 model. Since joint state and parameter estimation is especially useful in applications where the forcings are uncertain but nevertheless determining, typically in atmospheric chemistry, the augmented state IEnKS is tested on a new low-order model that takes its meteorological part from the Lorenz-95 model, and its chemical part from the advection diffusion of a tracer. In these experiments, the IEnKS is compared to the ensemble Kalman filter, the ensemble Kalman smoother, and a 4D-Var, which are considered the methods of choice to solve these joint estimation problems. In this low-order model context, the IEnKS is shown to significantly outperform the other methods regardless of the length of the data assimilation window, and for present time analysis as well as retrospective analysis. Besides which, the performance of the IEnKS is even more striking on parameter estimation; getting close to the same performance with 4D-Var is likely to require both a long data assimilation window and a complex modeling of the background statistics.