Implementation Of Ensemble Kalman Filter Research Articles

Impact of assimilating dam outflow measurements to update distributed hydrological model states: Localization for improving ensemble Kalman filter performance

This paper presents an investigation on the effect of dam operation on the ensemble Kalman filter (EnKF) performance in a distributed hydrological model based on kinematic wave theory. For this purpose, two flood events in a mountainous catchment in Japan are selected, and the model error parameters for the EnKF implementation are first determined through a series of experiments by varying the input error magnitude, state variable perturbation, and its correlation length. Then, through the assimilation of dam outflow observations, it is shown that special treatment of such measurements is critical in ensuring the success of the filter when flood control operation significantly changes the flow hydrograph and when the covariances between distant grids are high. Localization methods based on Euclidean distance, flow accumulation values, and river distance are all able to limit the deleterious updates to model states at distant grids and improve filter efficacy. However, no single localization approach is found to be consistently optimal for varying noise configurations, number (and type) of assimilated stations, or flood events. Separate localization of dam outflows from streamflow measurements by assuming the dam grids to be only related to their downstream grids shows potential in improving prediction at locations near the dams themselves while minimizing disruptions at long distances.

Estimation of Evapotranspiration Rates and Root Water Uptake Profiles From Soil Moisture Sensor Array Data

AbstractEvapotranspiration is arguably the least quantified component of the hydrologic cycle. We propose two complementary strategies for estimation of evapotranspiration rates and root water uptake profiles from soil‐moisture sensor‐array data. One is our implementation of ensemble Kalman filter (EnKF); it treats the evapotranspiration sink term in the Richards equation, rather than soil moisture, as the observable to update. The other is a maximum likelihood estimator (MLE) applied to the same observable; it is supplemented with the Fisher information matrix to quantify uncertainty in its predictions. We use numerical experiments to demonstrate the accuracy and computational efficiency of these techniques. We found our EnKF implementation to be two orders of magnitude faster than either the standard EnKF or MLE, and our MLE procedure to require an order of magnitude fewer iterations to converge than its counterpart applied to soil moisture. These findings render our methodologies a viable and practical tool for estimation of the root water uptake profiles and evaporation rates, with the MLE technique to be used when the prior knowledge about evapotranspiration at the site is elusive.

An ensemble Kalman filter implementation based on the Ledoit and Wolf covariance matrix estimator

In this paper, we propose an efficient and practical implementation of the ensemble Kalman filter (EnKF) via the distribution-free Ledoit and Wolf (LW) covariance matrix estimator. Initially, we develop a tractable implementation of the LW estimator in high-dimensional probability spaces such as those found in the context of operational data assimilation. We employ this well-conditioned, full-rank covariance matrix estimator to approximate background error covariances and to mitigate the impact of spurious correlations during assimilation steps. The proposed formulation can be coupled within the EnKF framework to derive a matrix-free implementation via an iterative Woodbury formula. Experimental tests are performed by using an Atmospheric General Circulation Model. The numerical results are compared with those of the EnKF based on the Rao–Blackwell Ledoit, and Wolf covariance matrix estimator (EnKF-RBLW) wherein Gaussian assumptions are a must on prior members. The outcomes reveal that the use of the proposed filter can mitigate the impact of spurious correlations during assimilation stages, and even more, the proposed method can improve on the results of the EnKF-RBLW as a consequence of the Gaussian relaxation on prior errors.

Impact of Bias-Correction Methods on Effectiveness of Assimilating SMAP Soil Moisture Data into NCEP Global Forecast System Using the Ensemble Kalman Filter

Improving numerical weather prediction was one of the main justifications for National Aeronautics and Space Administration's Soil Moisture Active/Passive (SMAP) Mission. The ensemble Kalman filter (EnKF) has been extensively applied to assimilate the SM observations into numerical weather predication models. Implementation of EnKF requires the observations and model simulations to be Gaussian distributed and not biased from each other. In this letter, we tested the impacts of three bias-correction methods on effectiveness of assimilating SMAP retrievals into the National Oceanic and Atmospheric Administration-National Centers for Environmental Prediction Global Forecast System (GFS). They are: 1) global cumulative distribution function (CDF) matching with only one CDF for all grids and time series; 2) monthly CDF matching with one CDF for each grid; 3) the linear transformation technique that matches monthly mean and standard deviation of the SMAP retrievals and model simulations for each grid; and 4) assimilating SMAP SM data into GFS without any bias-correction procedure. With respect to the global land data assimilation (DA) system precipitation product, the results demonstrate that the effectiveness of assimilating SMAP retrievals into GFS is significantly impacted by the bias-correction methods. Relative to other DA cases, the monthly CDF matching produces the best precipitation forecast performance. Improvements of the three-hourly GFS precipitation prediction with SMAP assimilation using the monthly CDF matching can reach to 8% and 10% in sparsely and densely vegetated areas, respectively, and marginally positive in medium vegetation areas. Based on these results, assimilating SMAP retrievals into the GFS with the EnKF algorithm using the monthly CDF matching method is suggested for enhancing accuracy of the precipitation forecasts.

Performance Analysis of Local Ensemble Kalman Filter

Ensemble Kalman filter (EnKF) is an important data assimilation method for high dimensional geophysical systems. Efficient implementation of EnKF in practice often involves the localization technique, which updates each component using only information within a local radius. This paper rigorously analyzes the local EnKF (LEnKF) for linear systems, and shows that the filter error can be dominated by the ensemble covariance, as long as 1) the sample size exceeds the logarithmic of state dimension and a constant that depends only on the local radius; 2) the forecast covariance matrix admits a stable localized structure. In particular, this indicates that with small system and observation noises, the filter error will be accurate in long time even if the initialization is not. The analysis also reveals an intrinsic inconsistency caused by the localization technique, and a stable localized structure is necessary to control this inconsistency. While this structure is usually taken for granted for the operation of LEnKF, it can also be rigorously proved for linear systems with sparse local observations and weak local interactions. These theoretical results are also validated by numerical implementation of LEnKF on a simple stochastic turbulence in two dynamical regimes.

Constraining the ensemble Kalman filter for improved streamflow forecasting

Data assimilation techniques such as the Ensemble Kalman Filter (EnKF) are often applied to hydrological models with minimal state volume/capacity constraints enforced during ensemble generation. Flux constraints are rarely, if ever, applied. Consequently, model states can be adjusted beyond physically reasonable limits, compromising the integrity of model output. In this paper, we investigate the effect of constraining the EnKF on forecast performance. A “free run” in which no assimilation is applied is compared to a completely unconstrained EnKF implementation, a ‘typical’ hydrological implementation (in which mass constraints are enforced to ensure non-negativity and capacity thresholds of model states are not exceeded), and then to a more tightly constrained implementation where flux as well as mass constraints are imposed to force the rate of water movement to/from ensemble states to be within physically consistent boundaries. A three year period (2008–2010) was selected from the available data record (1976–2010). This was specifically chosen as it had no significant data gaps and represented well the range of flows observed in the longer dataset. Over this period, the standard implementation of the EnKF (no constraints) contained eight hydrological events where (multiple) physically inconsistent state adjustments were made. All were selected for analysis. Mass constraints alone did little to improve forecast performance; in fact, several were significantly degraded compared to the free run. In contrast, the combined use of mass and flux constraints significantly improved forecast performance in six events relative to all other implementations, while the remaining two events showed no significant difference in performance. Placing flux as well as mass constraints on the data assimilation framework encourages physically consistent state estimation and results in more accurate and reliable forward predictions of streamflow for robust decision-making. We also experiment with the observation error, which has a profound effect on filter performance. We note an interesting tension exists between specifying an error which reflects known uncertainties and errors in the measurement versus an error that allows “optimal” filter updating.

An Ensemble Kalman Filter for the Thermosphere‐Ionosphere

AbstractA principal limitation of physics‐based modeling in the Thermosphere‐Ionosphere system is the large uncertainty associated with the implementation of the external forcing of the system, including high‐latitude convection and particle precipitation, solar UV/EUV fluxes, and waves propagating from below. As measuring these quantities with sufficient spatial and temporal resolution is prohibitively costly, a more practical approach to improve model results is to assimilate more readily available measurements of the system. We discuss considerations in implementing an Ensemble Kalman Filter (EnKF) for a strongly forced system versus a chaotic system, overview an EnKF implementation for the strongly forced Coupled Thermosphere, Ionosphere, Plasmasphere, and Electrodynamics model, and present encouraging improvements to neutral density specification obtained by assimilating Challenging Minisatellite Payload (CHAMP) neutral density measurements. The model results show improvement in comparisons with both CHAMP and Gravity Recovery and Climate Experiment (GRACE) measurements during a geomagnetically quiet period at solar minimum.

The Influence of Assimilated Upstream, Preconvective Dropsonde Observations on Ensemble Forecasts of Convection Initiation during the Mesoscale Predictability Experiment

This study tests the hypothesis that assimilating mid- to upper-tropospheric, meso- α- to synoptic-scale observations collected in upstream, preconvective environments is insufficient to improve short-range ensemble convection initiation (CI) forecast skill over the set of cases considered by the 2013 Mesoscale Predictability Experiment (MPEX) because of a limited influence upon the lower-tropospheric phenomena that modulate CI occurrence, timing, and location. The ensemble Kalman filter implementation within the Data Assimilation Research Testbed as coupled to the Advanced Research Weather Research and Forecasting (WRF) Model is used to initialize two nearly identical 30-member ensembles of short-range forecasts for each case: one initial condition set that incorporates MPEX dropsonde observations and one that excludes these observations. All forecasts for a given mission begin at 1500 UTC and are integrated for 15 h on a convection-permitting grid encompassing much of the conterminous United States. Forecast verification is conducted probabilistically using fractions skill score and deterministically using a 2 × 2 contingency table approach at multiple neighborhood sizes and spatiotemporal event-matching thresholds to assess forecast skill and support hypothesis testing. The probabilistic verification represents the first of its kind for numerical CI forecasts. Forecasts without MPEX observations have high fractions skill score and probabilities of detection on the meso- α scale but exhibit a considerable high bias for forecast CI event count. Assimilating MPEX observations has a negligible impact upon forecast skill for the cases considered, independent of verification metric, as the MPEX observations result in only subtle differences primarily manifest in the position and intensity of atmospheric features responsible for focusing and/or triggering deep, moist convection.

A Matrix-Free Posterior Ensemble Kalman Filter Implementation Based on a Modified Cholesky Decomposition

In this paper, a matrix-free posterior ensemble Kalman filter implementation based on a modified Cholesky decomposition is proposed. The method works as follows: the precision matrix of the background error distribution is estimated based on a modified Cholesky decomposition. The resulting estimator can be expressed in terms of Cholesky factors which can be updated based on a series of rank-one matrices in order to approximate the precision matrix of the analysis distribution. By using this matrix, the posterior ensemble can be built by either sampling from the posterior distribution or using synthetic observations. Furthermore, the computational effort of the proposed method is linear with regard to the model dimension and the number of observed components from the model domain. Experimental tests are performed making use of the Lorenz-96 model. The results reveal that, the accuracy of the proposed implementation in terms of root-mean-square-error is similar, and in some cases better, to that of a well-known ensemble Kalman filter (EnKF) implementation: the local ensemble transform Kalman filter. In addition, the results are comparable to those obtained by the EnKF with large ensemble sizes.

Evaluation of snow data assimilation using the ensemble Kalman filter for seasonal streamflow prediction in the western United States

Abstract. In this study, we examine the potential of snow water equivalent data assimilation (DA) using the ensemble Kalman filter (EnKF) to improve seasonal streamflow predictions. There are several goals of this study. First, we aim to examine some empirical aspects of the EnKF, namely the observational uncertainty estimates and the observation transformation operator. Second, we use a newly created ensemble forcing dataset to develop ensemble model states that provide an estimate of model state uncertainty. Third, we examine the impact of varying the observation and model state uncertainty on forecast skill. We use basins from the Pacific Northwest, Rocky Mountains, and California in the western United States with the coupled Snow-17 and Sacramento Soil Moisture Accounting (SAC-SMA) models. We find that most EnKF implementation variations result in improved streamflow prediction, but the methodological choices in the examined components impact predictive performance in a non-uniform way across the basins. Finally, basins with relatively higher calibrated model performance (> 0.80 NSE) without DA generally have lesser improvement with DA, while basins with poorer historical model performance show greater improvements.

Fast linearized forecasts for subsurface flow data assimilation with ensemble Kalman filter

Ensemble methods present a practical framework for parameter estimation, performance prediction, and uncertainty quantification in subsurface flow and transport modeling. In particular, the ensemble Kalman filter (EnKF) has received significant attention for its promising performance in calibrating heterogeneous subsurface flow models. Since an ensemble of model realizations is used to compute the statistical moments needed to perform the EnKF updates, large ensemble sizes are needed to provide accurate updates and uncertainty assessment. However, for realistic problems that involve large-scale models with computationally demanding flow simulation runs, the EnKF implementation is limited to small-sized ensembles. As a result, spurious numerical correlations can develop and lead to inaccurate EnKF updates, which tend to underestimate or even eliminate the ensemble spread. Ad hoc practical remedies, such as localization, local analysis, and covariance inflation schemes, have been developed and applied to reduce the effect of sampling errors due to small ensemble sizes. In this paper, a fast linear approximate forecast method is proposed as an alternative approach to enable the use of large ensemble sizes in operational settings to obtain more improved sample statistics and EnKF updates. The proposed method first clusters a large number of initial geologic model realizations into a small number of groups. A representative member from each group is used to run a full forward flow simulation. The flow predictions for the remaining realizations in each group are approximated by a linearization around the full simulation results of the representative model (centroid) of the respective cluster. The linearization can be performed using either adjoint-based or ensemble-based gradients. Results from several numerical experiments with two-phase and three-phase flow systems in this paper suggest that the proposed method can be applied to improve the EnKF performance in large-scale problems where the number of full simulation is constrained.

Ensemble Kalman filter implementations based on shrinkage covariance matrix estimation

This paper develops efficient ensemble Kalman filter (EnKF) implementations based on shrinkage covariance estimation. The forecast ensemble members at each step are used to estimate the background error covariance matrix via the Rao-Blackwell Ledoit and Wolf estimator, which has been specifically developed to approximate high-dimensional covariance matrices using a small number of samples. Two implementations are considered: in the EnKF full-space (EnKF-FS) approach, the assimilation process is performed in the model space, while the EnKF reduce-space (EnKF-RS) formulation performs the analysis in the subspace spanned by the ensemble members. In the context of EnKF-RS, additional samples are taken from the normal distribution described by the background ensemble mean and the estimated background covariance matrix, in order to increase the size of the ensemble and reduce the sampling error of the filter. This increase in the size of the ensemble is obtained without running the forward model. After the assimilation step, the additional samples are discarded and only the model-based ensemble members are propagated further. Methodologies to reduce the impact of spurious correlations and under-estimation of sample variances in the context of the EnKF-FS and EnKF-RS implementations are discussed. An adjoint-free four-dimensional extension of EnKF-RS is also discussed. Numerical experiments carried out with the Lorenz-96 model and a quasi-geostrophic model show that the use of shrinkage covariance matrix estimation can mitigate the impact of spurious correlations during the assimilation process.

Optimal ensemble size of ensemble Kalman filter in sequential soil moisture data assimilation

AbstractThe ensemble Kalman filter (EnKF) has been extensively applied in sequential soil moisture data assimilation to improve the land surface model performance and in turn weather forecast capability. Usually, the ensemble size of EnKF is determined with limited sensitivity experiments. Thus, the optimal ensemble size may have never been reached. In this work, based on a series of mathematical derivations, we demonstrate that the maximum efficiency of the EnKF for assimilating observations into the models could be reached when the ensemble size is set to 12. Simulation experiments are designed in this study under ensemble size cases 2, 5, 12, 30, 50, 100, and 300 to support the mathematical derivations. All the simulations are conducted from 1 June to 30 September 2012 over southeast USA (from −90°W, 30°N to −80°W, 40°N) at 25 km resolution. We found that the simulations are perfectly consistent with the mathematical derivation. This optical ensemble size may have theoretical implications on the implementation of EnKF in other sequential data assimilation problems.

Facies estimation through data assimilation and structure parameterization

Data assimilation in a reservoir with facies description using the ensemble Kalman filter (EnKF) is challenging. An important reason is that probability density functions for pixel-based representations of facies fields seldom follow the unimodal Gaussian assumption underlying traditional EnKF implementations. Different approaches for identification of facies fields, aiming to overcome this challenge, have been proposed within the EnKF framework. Level set (LS) representations of the facies field have been reported to alleviate the problems of multimodality. Several authors have, however, pointed out that the most commonly applied LS representation suffers from topological constraints that can create difficulties in an estimation setting. An alternative LS representation, that overcomes these topological constraints, leads to instabilities in the assimilated ensemble members. To overcome topological constraints, the recently proposed hierarchical LS representation is applied in an estimation setting for the first time in this paper. To improve stability and to alleviate challenges associated with model nonlinearities, we apply regularization by reduced representation of the LS functions and adjustable smoothing of the LS representation. The resolution of the reduced LS representation is selected based on the variability of the initial ensemble, aiming at preserving enough flexibility to disclose unexpected features. 2D and 3D estimation results demonstrate that the hierarchical LS representation does avoid topological constraints and that instabilities are avoided. The results suggest that the method is capable of handling estimation of facies fields while preserving geological plausibility.

Estimation of Facies Boundaries Using Categorical Indicators with P-Field Simulation and Ensemble Kalman Filter (EnKF)

The Ensemble Kalman Filter (EnKF) often fails to reproduce realistic facies distribution and proportion corresponding to the prior spatial distribution. In this paper, a new step is proposed for inclusion in the history matching of multiple facies reservoir models using EnKF. The new step consists of constructing a facies probability map and application of probability field (P-Field) simulation to re-sample a new ensemble. After certain number of assimilation steps in EnKF, the updated ensemble members are used to propose a probability map for facies distribution. P-Field simulation is performed subsequently using the facies probability map to generate a new ensemble, which honors the static geologic data and is more consistent with the early production data. After the re-sampling step, the forecast model is applied to the new ensemble from the beginning. EnKF is again applied on the ensemble members to assimilate the remaining production history. Two case studies with different facies distribution and well configurations were conducted. Initial ensemble was created using known facies classification at the well locations and populating binary facies data throughout the reservoir using prior information of spatial facies distribution and facies proportions. The qualities of the history-matched models were assessed by comparing the spatial facies distribution and proportions of the updated ensemble, in addition to the root mean square error of the predicted data mismatch. Implementation of EnKF together with re-sampling of the new realizations using probability maps demonstrates reasonable improvement in the history matching and uncertainty assessment of multiple facies reservoir models.

An efficient implementation of the ensemble Kalman filter based on an iterative Sherman–Morrison formula

We present a practical implementation of the ensemble Kalman filter (EnKF) based on an iterative Sherman–Morrison formula. The new direct method exploits the special structure of the ensemble-estimated error covariance matrices in order to efficiently solve the linear systems involved in the analysis step of the EnKF. The computational complexity of the proposed implementation is equivalent to that of the best EnKF implementations available in the literature when the number of observations is much larger than the number of ensemble members, as typically is case in practice. Moreover, the proposed method provides the best theoretical complexity when it is compared to generic formulations of matrix inversion based on the Sherman–Morrison formula. The stability analysis of the proposed method is carried out and a pivoting strategy is discussed in order to reduce the accumulation of round-off errors without increasing the computational effort. A parallel implementation is discussed as well. Computational experiments carried out using an oceanic quasi-geostrophic model reveal that the proposed algorithm yields the same accuracy as other EnKF implementations, but scales better with regard to the number of observations.

AN ENSEMBLE KALMAN FILTER USING THE CONJUGATE GRADIENT SAMPLER

The ensemble Kalman filter (EnKF) is a technique for dynamic state estimation. EnKF approximates the standard extended Kalman filter (EKF) by creating an ensemble of model states whose mean and empirical covariance are then used within the EKF formulas. The technique has a number of advantages for large-scale, nonlinear problems. First, large-scale covariance matrices required within EKF are replaced by low-rank and low-storage approximations, making implementation of EnKF more efficient. Moreover, for a nonlinear state space model, implementation of EKF requires the associated tangent linear and adjoint codes, while implementation of EnKF does not. However, for EnKF to be effective, the choice of the ensemble members is extremely important. In this paper, we show how to use the conjugate gradient (CG) method, and the recently introduced CG sampler, to create the ensemble members at each filtering step. This requires the use of a variational formulation of EKF. The effectiveness of the method is demonstrated on both a large-scale linear, and a small-scale, nonlinear, chaotic problem. In our examples, the CG-EnKF performs better than the standard EnKF, especially when the ensemble size is small.

Combining sensitivities and prior information for covariance localization in the ensemble Kalman filter for petroleum reservoir applications

Sampling errors can severely degrade the reliability of estimates of conditional means and uncertainty quantification obtained by the application of the ensemble Kalman filter (EnKF) for data assimilation. A standard recommendation for reducing the spurious correlations and loss of variance due to sampling errors is to use covariance localization. In distance-based localization, the prior (forecast) covariance matrix at each data assimilation step is replaced with the Schur product of a correlation matrix with compact support and the forecast covariance matrix. The most important decision to be made in this localization procedure is the choice of the critical length(s) used to generate this correlation matrix. Here, we give a simple argument that the appropriate choice of critical length(s) should be based both on the underlying principal correlation length(s) of the geological model and the range of the sensitivity matrices. Based on this result, we implement a procedure for covariance localization and demonstrate with a set of distinctive reservoir history-matching examples that this procedure yields improved results over the standard EnKF implementation and over covariance localization with other choices of critical length.

Integration of Well-Test Pressure Data Into Heterogeneous Geological Reservoir Models

Summary This paper presents the application of the ensemble Kalman filter (EnKF) method to the integration of well-test data into heterogeneous reservoir models generated from geological and geophysical data. EnKF does not require computing the gradient of an objective function and, hence, can be applied easily with any reservoir simulator, and more importantly, it is far more efficient than a gradient-based history-matching procedure when the forward model is represented by a reservoir simulator. In the procedure for integrating pressure-transient data considered here, the static geological/geophysical data are assumed to be encapsulated in a multivariate probability-density function (pdf) characterized by a prior mean and covariance for the joint distribution of the porosity and permeability fields. Because the prior mean of the property fields obtained from the core and log data can sometimes be erroneous, a partially doubly stochastic model is applied to account for the uncertainty of the prior mean. In the doubly stochastic model, a correction to the prior mean is adjusted together with the heterogeneous field during history matching. The method is tested with synthetic heterogeneous single-layer and two-layer reservoirs. Excellent data matches are obtained with EnKF in a small fraction of the time that would be required for a gradient-based history-matching process, and the observed data fall within the uncertainty bounds of the ensemble data predictions. In the layered-reservoir case, the uncertainty in rock-property fields is reduced further by integration of production-log data (layer flow rates). We show that problems of statistical inconsistency and poor data matches that are sometimes encountered when matching production data with EnKF do not occur when matching pressure data with the EnKF implementation used here. This undoubtedly occurs because the dynamical system (reservoir-simulator equations) is much more nearly linear for the single-phase-flow problems considered here than for multiphase-flow cases.

Ensemble data assimilation with the CNMCA regional forecasting system

AbstractThe Ensemble Kalman Filter (EnKF) is likely to become a viable alternative to variational methods for the next generation of meteorological and oceanographic data assimilation systems. In this work we present results from real‐data assimilation experiments using the CNMCA regional numerical weather prediction (NWP) forecasting system and compare them to the currently operational variational‐based analysis. The set of observations used is the same as the one ingested in the operational data stream, with the exception of satellite radiances and scatterometer winds. Results show that the EnKF‐based assimilation cycle is capable of producing analyses and forecasts of consistently superior skill in the root mean square error metric than CNMCA operational 3D‐Var.One of the most important issues in EnKF implementations lies in the filter tendency to become underdispersive for practical ensemble sizes. To combat this problem a number of different parametrizations of the model error unaccounted for in the assimilation cycle have been proposed. In the CNMCA system a combination of adaptive multiplicative and additive background covariance inflations has been found to give adequate results and to be capable of avoiding filter divergence in extended assimilation trials. The additive component of the covariance inflation has been implemented through the use of scaled forecast differences.Following suggestions that ensemble square‐root filters can violate the gaussianity assumption when used with nonlinear prognostic models, the statistical distribution of the forecast and analysis ensembles has been studied. No sign of the ensemble collapsing onto one or a few model states has been found, and the forecast and analysis ensembles appear to stay remarkably close to the assumed probability distribution functions. Copyright © 2010 Royal Meteorological Society