Conventional Ensemble Kalman Filter Research Articles

Particle network EnKF for large-scale data assimilation

The Ensemble Kalman filter (EnKF) is a classic method of data assimilation. For distributed sampling, the conventional EnKF usually requires a centralized server to integrate the predictions of all particles or a fully-connected communication network, causing traffic jams and low bandwidth utilization in high-performance computing. In this paper, we propose a novel distributed scheme of EnKF based on network setting of sampling, called Particle Network EnKF. Without a central server, every sampling particle communicates with its neighbors over a sparsely connected network. Unlike the existing work, this method focuses on the distribution of sampling particles instead of sensors and has been proved effective and robust on numerous tasks. The numerical experiments on the Lorenz-63 and Lorenz-96 systems indicate that, with proper communication rounds, even on a sparse particle network, this method achieves a comparable performance to the standard EnKF. A detailed analysis of effects of the network topology and communication rounds is performed. Another experiment demonstrating a trade-off between the particle homogeneity and performance is also provided. The experiments on the whole-brain neuronal network model show promises for applications in large-scale assimilation problems.

A meteorologically adjusted ensemble Kalman filter approach for inversing daily emissions: A case study in the Pearl River Delta, China

The conventional Ensemble Kalman filter (EnKF), which is now widely used to calibrate emission inventories and to improve air quality simulations, is susceptible to simulation errors of meteorological inputs, making accurate updates of high temporal-resolution emission inventories challenging. In this study, we developed a novel meteorologically adjusted inversion method (MAEInv) based on the EnKF to improve daily emission estimations. The new method combines sensitivity analysis and bias correction to alleviate the inversion biases caused by errors of meteorological inputs. For demonstration, we used the MAEInv to inverse daily carbon monoxide (CO) emissions in the Pearl River Delta (PRD) region, China. In the case study, 60% of the total CO simulation biases were associated with sensitive meteorological inputs, which would lead to the overestimation of daily variations of posterior emissions. Using the new inversion method, daily variations of emissions shrank dramatically, with the percentage change decreased by 30%. Also, the total amount of posterior CO emissions estimated by the MAEInv decreased by 14%, indicating that posterior CO emissions might be overestimated using the conventional EnKF. Model evaluations using independent observations revealed that daily CO emissions estimated by MAEInv better reproduce the magnitude and temporal patterns of ambient CO concentration, with a higher correlation coefficient (R, +37.0%) and lower normalized mean bias (NMB, -17.9%). Since errors of meteorological inputs are major sources of simulation biases for both low-reactive and reactive pollutants, the MAEInv is also applicable to improve the daily emission inversions of reactive pollutants.

An effective method for point pollution source identification in rivers with performance-improved ensemble Kalman filter

To manage a river sudden pollution incident, a fundamental task is to quickly identify when, where, and how much pollutant was released. The source identification of this kind of pollution incident due to instantaneously spilled pollutant from point source remains a challenging problem because of the limited observations and the ill-posedness. In this paper, the ensemble Kalman filter (EnKF) is coupled with the backward location probability (BLP) to improve its performance in identifying the river pollution source. When applied in practice, the proposed BLP-EnKF method has two important advantages. First, it supports on-line identification and can mitigate the limitation from shortage of observations in the sudden river pollution case. Second, BLP-EnKF performs much better than conventional EnKF in terms of computation time, because it can identify the pollution source without restart forecast process. The effectiveness and efficiency of BLP-EnKF is tested and demonstrated by a synthetic case study. The case results show that the BLP-EnKF can identify all the source parameters with a relative error approximately 1.00% or smaller. Its performance can be further improved with more accurate estimation of observational error or dispersion coefficient. A real-world case in Ganjiang River further demonstrates the applicability of BLP-EnKF in practice. The pollution source is identified successfully with relative errors smaller than 3.0% for all the source parameters, while the computation time is sharply shorted from 8.0 h of conventional EnKF to 22.0 s of BLP-EnKF.

Integrated Data Assimilation and Distance-Based Model Selection with Ensemble Kalman Filter for Characterization of Uncertain Geological Scenarios

This paper develops a reliable workflow for multi-objective history matching coupled with a distance-based model selection and ensemble-based data assimilation at a clastic channel reservoir with uncertain geological scenarios. The distance map allocated to each producer determines the probabilities of training images which assess the uncertainty of geological scenarios. K-medoids clustering selects the reservoir models within the ensemble set applied with some training images with less error. These geo-models play as initial ensembles suitable to explain the geological scenarios and ensemble Kalman filter recursively assimilates the oil rates of each producer. The developed workflow, updating reliable reservoir models suitable for well-performance-based history matching, more accurately forecasts water breakthrough and improves the predictability of unknown oil rates with a lower error than those of the conventional ensemble Kalman filter. This framework is able to preserve the spatial characteristics of facies models and reservoir properties without interpreting one fixed scenario. The proposed method can contribute to a reasonable design for data analytics with uncertain geological scenarios and for matching different-scaled well production histories.

Bearings-Only Tracking Using Augmented Ensemble Kalman Filter

Tracking targets with bearings-only measurement is a great challenge caused by poor observability and highly nonlinear estimation. In this brief, a novel augmented ensemble Kalman filter (AEnKF) is presented to address this bearings-only tracking problem. Different from the conventional ensemble Kalman filter (EnKF), the AEnKF overcomes the limitation of the linear measurement update rule in the linear minimum mean-square error (LMMSE) framework. The AEnKF utilizes a nonlinear transform of the measurement, called uncorrelated conversion (UC), to augment the measurement space. This conversion serves as a pseudomeasurement and is uncorrelated with the original measurement statistically. Unlike other UC filters based on the Gaussian assumption in the existing literature, the AEnKF does not impose any assumption on the probability density of the measurement by using generalized orthogonal polynomials to construct the UCs in a systematic way. The simulation results show that the AEnKF outperforms the conventional EnKF and other UC filters in the bearings-only tracking problem.

Recursive update of channel information for reliable history matching of channel reservoirs using EnKF with DCT

Ensemble Kalman filter (EnKF) is one of the promising reservoir characterization schemes for history matching. It has been widely researched due to its excellence in recursive data processing and dependable uncertainty quantification. However, EnKF has critical limitations for applying to non-Gaussian distribution fields such as channel reservoirs.In this study, we propose a novel scheme to characterize channel reservoirs reliably. The scheme, recursive update of channel information (RUCI), updates key channel information such as facies ratio and permeabilities of each rock facies as components of the state vector. Since permeabilities assimilated by EnKF follow a Gaussian distribution, EnKF itself is not adequate to preserve channel structure. To overcome this limitation, we use cumulative distribution function mapping to maintain its bimodal distribution. In addition to RUCI, discrete cosine transform (DCT) is utilized to preserve clear channel continuities by figuring out essential channel trend.The proposed method, RUCI in EnKF with DCT, is applied to three different channel reservoir cases and is compared with typical three methods: EnKF, EnKF with DCT, and EnKF with normal score transform (NST). Both of conventional EnKF and EnKF with DCT show overshooting and fail to present channel details. Furthermore, their permeability distributions have a tendency to follow a Gaussian, not a typical bimodal distribution of sand and shale. Even if EnKF with NST maintains a bimodal distribution after updates, it cannot figure out overall channel trend. The proposed method not only prevents overshooting but also preserves a bimodal distribution of permeability. It can clearly characterize true channel trend even though initial reservoir models are poorly made. In addition, it gives better estimation of future oil and water productions than the other 3 methods by reducing uncertainties and matching true performances.

Orthogonal Matching Pursuit for Enhanced Recovery of Sparse Geological Structures With the Ensemble Kalman Filter

Estimating the locations and the structures of subsurface channels holds significant importance for forecasting the subsurface flow and reservoir productivity. These channels exhibit high permeability and are easily contrasted from the low-permeability rock formations in their surroundings. This enables formulating the flow channels estimation problem as a sparse field recovery problem. The ensemble Kalman filter (EnKF) is a widely used technique for the estimation and calibration of subsurface reservoir model parameters, such as permeability. However, the conventional EnKF framework does not provide an efficient mechanism to incorporate prior information on the wide varieties of subsurface geological structures, and often fails to recover and preserve flow channel structures. Recent works in the area of compressed sensing (CS) have shown that estimating in a sparse domain, using algorithms such as the orthogonal matching pursuit (OMP), may significantly improve the estimation quality when dealing with such problems. We propose two new, and computationally efficient, algorithms combining OMP with the EnKF to improve the estimation and recovery of the subsurface geological channels. Numerical experiments suggest that the proposed algorithms provide efficient mechanisms to incorporate and preserve structural information in the EnKF and result in significant improvements in recovering flow channel structures.

Groundwater Contaminant Transport Modeling Using Multiple Adaptive Data Assimilation Techniques

In modeling the behavior of contaminants in a subsurface environment using data assimilation schemes, accurate assignment of model and observation errors are significant for the successful application of the techniques. In this study, a three‐dimensional transport model was used to simulate the advection and dispersiontransport of contaminant in the subsurface. Stochastic data assimilation schemes were coupled with the subsurface contaminant transport model to predict the state of the contaminant. Three data assimilation techniques namely the conventional Ensemble KalmanFilter, the Adaptive Ensemble Kalman Filter and the Hybrid Adaptive Ensemble Kalman Filter were adopted to improve the prediction of the contaminant fate and transport in the groundwater. The Ensemble Kalman Filter applies a Monte Carlo approach to the filtering problem. The adaptive filtering technique employs the diagnostic approach to fine tune the model and observation covariance matrix. The hybrid technique uses combination of the forecast covariance matrix and the invariant background covariance matrix to explore the Ensemble Kalman filter. The impact of the filters on the numerical model is examined by using the Normalized Root Mean Square Error (NRMSE), Average Absolute Bias (AAB) metric, and Maximum Absolute Deviation (MAD) techniques. The AAB evaluation of Adaptive Ensemble Kalman Filter and Hybrid Adaptive Ensemble Kalman Filter shows error reduction of 93% and 90%, respectively, while the MAD assessment recorded 94% and 89% improvement respectively, in the numerical model. The results of simulations show that the prediction accuracy of the filters is better than numerical model. The proposed Adaptive Ensemble Kalman Filter and Hybrid Adaptive Ensemble Kalman Filter, takes advantage of the adaptive factor and the weighting factor, respectively to improve the prediction efficiency of the Ensemble Kalman filter. Sensitivity analysis performed on Adaptive Ensemble Kalman Filter and Hybrid Adaptive Ensemble Kalman Filter using NRMSE indicates that the adaptive factor does not affect the prediction accuracy of the former whereas the weighting factor has influence on the later.

A comparison of the performance of the 3-D super-ensemble and an ensemble Kalman filter for short-range regional ocean prediction

This study compares the ability of two approaches integrating models and data to forecast the Ligurian Sea regional oceanographic conditions in the short-term range (0–72 hours) when constrained by a common observation dataset. The post-processing 3-D super-ensemble (3DSE) algorithm, which uses observations to optimally combine multi-model forecasts into a single prediction of the oceanic variable, is first considered. The 3DSE predictive skills are compared to those of the Regional Ocean Modeling System model in which observations are assimilated through a more conventional ensemble Kalman filter (EnKF) approach. Assimilated measurements include sea surface temperature maps, and temperature and salinity subsurface observations from a fleet of five underwater gliders. Retrospective analyses are carried out to produce daily predictions during the 11-d period of the REP10 sea trial experiment. The forecast skill evaluation based on a distributed multi-sensor validation dataset indicates an overall superior performance of the EnKF, both at the surface and at depth. While the 3DSE and EnKF perform comparably well in the area spanned by the incorporated measurements, the 3DSE accuracy is found to rapidly decrease outside this area. In particular, the univariate formulation of the method combined with the absence of regular surface salinity measurements produces large errors in the 3DSE salinity forecast. On the contrary, the EnKF leads to more homogeneous forecast errors over the modelling domain for both temperature and salinity. The EnKF is found to consistently improve the predictions with respect to the control solution without assimilation and to be positively skilled when compared to the climatological estimate. For typical regional oceanographic applications with scarce subsurface observations, the lack of physical spatial and multivariate error covariances applicable to the individual model weights in the 3DSE formulation constitutes a major limitation for the performance of this multi-model-data fusion approach compared to conventional advanced data assimilation strategies.

Ensemble Kalman Filtering with a Divided State-Space Strategy for Coupled Data Assimilation Problems*

Abstract This study considers the data assimilation problem in coupled systems, which consists of two components (subsystems) interacting with each other through certain coupling terms. A straightforward way to tackle the assimilation problem in such systems is to concatenate the states of the subsystems into one augmented state vector, so that a standard ensemble Kalman filter (EnKF) can be directly applied. This work presents a divided state-space estimation strategy, in which data assimilation is carried out with respect to each individual subsystem, involving quantities from the subsystem itself and correlated quantities from other coupled subsystems. On top of the divided state-space estimation strategy, the authors also consider the possibility of running the subsystems separately. Combining these two ideas, a few variants of the EnKF are derived. The introduction of these variants is mainly inspired by the current status and challenges in coupled data assimilation problems and thus might be of interest from a practical point of view. Numerical experiments with a multiscale Lorenz 96 model are conducted to evaluate the performance of these variants against that of the conventional EnKF. In addition, specific for coupled data assimilation problems, two prototypes of extensions of the presented methods are also developed in order to achieve a trade-off between efficiency and accuracy.

Improving the ensemble transform Kalman filter using a second-order Taylor approximation of the nonlinear observation operator

Abstract. The ensemble transform Kalman filter (ETKF) assimilation scheme has recently seen rapid development and wide application. As a specific implementation of the ensemble Kalman filter (EnKF), the ETKF is computationally more efficient than the conventional EnKF. However, the current implementation of the ETKF still has some limitations when the observation operator is strongly nonlinear. One problem in the minimization of a nonlinear objective function similar to 4D-Var is that the nonlinear operator and its tangent-linear operator have to be calculated iteratively if the Hessian is not preconditioned or if the Hessian has to be calculated several times. This may be computationally expensive. Another problem is that it uses the tangent-linear approximation of the observation operator to estimate the multiplicative inflation factor of the forecast errors, which may not be sufficiently accurate. This study attempts to solve these problems. First, we apply the second-order Taylor approximation to the nonlinear observation operator in which the operator, its tangent-linear operator and Hessian are calculated only once. The related computational cost is also discussed. Second, we propose a scheme to estimate the inflation factor when the observation operator is strongly nonlinear. Experimentation with the Lorenz 96 model shows that using the second-order Taylor approximation of the nonlinear observation operator leads to a reduction in the analysis error compared with the traditional linear approximation method. Furthermore, the proposed inflation scheme leads to a reduction in the analysis error compared with the procedure using the traditional inflation scheme.

Characterization of Non-Gaussian Geologic Facies Distribution Using Ensemble Kalman Filter with Probability Weighted Re-Sampling

The Ensemble Kalman Filter (EnKF) is a Monte Carlo-based technique for assisted history matching and real-time updating of reservoir models. However, it often fails to detect precise locations of distinct facies boundaries and their proportions, as the facies distributions are non-Gaussian, while geologic data for reservoir modeling is usually insufficient. In this paper, a new re-sampling step is introduced to the conventional EnKF formulation; after certain number of assimilation steps, the updated ensemble is used to generate a new ensemble with a novel probability weighted re-sampling scheme. The new ensemble samples from a probability density function that is conditional to both the geological information and the early production data. After the re-sampling step, the forecast model is applied to the new ensemble from the beginning up to the last update step (without any intermediate Kalman updates). Full EnKF is again applied on the ensemble members to assimilate the remaining production history. Combination of EnKF and regenerating new members using the re-sampling method demonstrates reasonable improvement and reduction of uncertainty in history matching of reservoir models with multiple facies. The histogram and the experimental variogram of the updated ensemble members are more consistent with the static geologic information. Moreover, the technique helps maintaining ensemble variance which is essential for uncertainty estimation in the posterior probability distribution of facies proportions.

Storm-Centered Ensemble Data Assimilation for Tropical Cyclones

Abstract A significant challenge for tropical cyclone ensemble data assimilation is that storm-scale observations tend to make analyses that are more asymmetric than the prior forecasts. Compromised structure and intensity, such as an increase of amplitude across the azimuthal Fourier spectrum, are a routine property of ensemble-based analyses, even with accurate position observations and frequent assimilation. Storm dynamics in subsequent forecasts evolve these states toward axisymmetry, creating difficulty in distinguishing between model-induced and actual storm asymmetries for predictability studies and forecasting. To address this issue, a novel algorithm using a storm-centered approach is proposed. The method is designed for use with existing ensemble filters with little or no modification, facilitating its adoption and maintenance. The algorithm consists of 1) an analysis of the environment using conventional coordinates, 2) a storm-centered analysis using storm-relative coordinates, and 3) a merged analysis that combines the large-scale and storm-scale fields together at an updated storm location. This algorithm is evaluated in two sets of observing system simulation experiments (OSSEs): first, no-cycling tests of the update step for idealized three-dimensional storms in radiative–convective equilibrium; second, full cycling tests of data assimilation applied to a shallow-water model for a field of interacting vortices. Results are compared against a control experiment based on a conventional ensemble Kalman filter (EnKF) scheme as well as an alternative EnKF scheme proposed by Lawson and Hansen. The storm-relative method yields vortices that are more symmetric and exhibit finer inner-core structure than either approach, with errors reduced by an order of magnitude over a control case with prior spread consistent with the National Hurricane Center (NHC)’s mean 5-yr forecast track error at 12 h. Azimuthal Fourier error spectra exhibit much-reduced noise associated with data assimilation as compared to both the control and the Lawson and Hansen approach. An assessment of free-surface height tendency of model forecasts after the merge step reveals a balanced trend between the storm-centered and conventional approaches, with storm-centered values more closely resembling the reference state.

Growing‐error correction of ensemble Kalman filter using empirical singular vectors

AbstractIn this study, a new Ensemble Kalman Filter (EnKF) algorithm called EnKF with growing‐error correction (EnKF‐GEC) is developed for minimizing the growing component of the forecast error; for this purpose, prospective observations are assimilated using empirical singular vectors (ESVs). Unlike the Ensemble Kalman Smoother (EnKS) or four‐dimensional EnKF (4DEnKF), the EnKF‐GEC is designed to reduce the analysis error at the last analysis time (errors of initial condition for prediction). By performing assimilation experiments using the CZ‐SPEEDY coupled model within a perfect model framework, it is shown that the analysis errors obtained using the EnKF‐GEC are significantly reduced as compared to those obtained using the conventional EnKF until the last analysis time as well as during the middle of analysis time. This indicates that the new algorithm is beneficial for prediction. Seasonal prediction results show that the prediction skill when initial conditions are generated by the EnKF‐GEC is superior to when initial conditions are generated by the conventional EnKF or EnKS, particularly during the early forecast lead month. For example, correlation skill improvement with 16 ensemble members is about 0.1 for a 3‐month lead forecast. In addition, it is shown that the new EnKF algorithm is more effective for unpredictable regions, where the value of the unstable singular vector is robust. Copyright © 2010 Royal Meteorological Society

An Automatic History Matching Module with Distributed and Parallel Computing

The data assimilation process of adjusting variables in a reservoir simulation model to honor observations of field data is known as history matching and has been extensively studied for a few decades. However, limited success has been achieved due to the high complexity of the problem and the large computational effort required in the real fields. Successful applications of the ensemble Kalman filter (EnKF) to reservoir history matching have been reported in various publications. The EnKF is a sequential method: once new data are available, only these data are used to update all the unknown reservoir properties while previous geological information is unused directly. In this method, multiple reservoir models rather than one single model are implemented, and each model is called a member. Conventionally, the impact of each member on the updating is equally treated. Another approach is the weighted EnKF. During the updating, the method weighs the contribution of each member through the comparison between the simulation response and the measurements. Better matching performance has been found in the weighted EnKF than in the conventional EnKF. To improve computational efficiency, two-level high-performance computing for reservoir history matching process is implemented in this research, distributing ensemble members simultaneously while simulating each member in a parallel style. An automatic history-matching module based on the weighted EnKF and high-performance computing is developed and validated through a synthetic case operating from primary, waterflooding to flooding of water alternating with gas. The study shows that the weighted EnKF improves the matching results, and the high-performance computing process significantly reduces the history matching execution time.

Correcting for forecast bias in soil moisture assimilation with the ensemble Kalman filter

Land surface models are usually biased in at least a subset of the simulated variables even after calibration. Bias estimation may therefore be needed for data assimilation. Here, in situ soil moisture profile observations in a small agricultural field were merged with Community Land Model (CLM2.0) simulations using different algorithms for state and forecast bias estimation with and without bias correction feedback. Simple state updating with the conventional ensemble Kalman filter (EnKF) allows for some implicit forecast bias correction. It is possible to estimate the soil moisture bias explicitly and derive superior soil moisture estimates with a generalized EnKF that uses a simple persistence model for the bias and assumes that the a priori bias error covariance is proportional to the a priori state error covariance. For the case of bi‐weekly assimilation of the entire profile of soil moisture observations, bias estimation and correction typically reduces the RMSE in soil moisture (over the standard EnKF without bias correction) by around 60 percent. However, under the above assumptions, significant improvements are limited to state variables for which observations are available. Therefore, it is crucial to measure the state variables of interest. The best variant for state and bias estimation depends on the nature of the model bias and the output of interest to the user. In a model that is only biased for soil moisture, large and frequent increments for soil moisture updating may be required, which in turn may negatively impact the water balance and output fluxes. It is then better to post‐process the soil moisture with the bias analysis without updating the model state.