Ensemble Data Assimilation Methods Research Articles

A Quantile-Conserving Ensemble Filter Framework. Part III: Data Assimilation for Mixed Distributions with Application to a Low-Order Tracer Advection Model

Abstract The uncertainty associated with many observed and modeled quantities of interest in Earth system prediction can be represented by mixed probability distributions that are neither discrete nor continuous. For instance, a forecast probability of precipitation can have a finite probability of zero precipitation, consistent with a discrete distribution. However, nonzero values are not discrete and are represented by a continuous distribution; the same is true for rainfall rate. Other examples include snow depth, sea ice concentration, amount of a tracer or the source rate of a tracer. Some Earth system model parameters may also have discrete or mixed distributions. Most ensemble data assimilation methods do not explicitly consider the possibility of mixed distributions. The Quantile Conserving Ensemble Filtering Framework (Anderson 2022, 2023) is extended to explicitly deal with discrete or mixed distributions. An example is given using bounded normal rank histogram probability distributions applied to observing system simulation experiments in a low-order tracer advection model. Analyses of tracer concentration and tracer source are shown to be improved when using the extended methods. A key feature of the resulting ensembles is that there can be ensemble members with duplicate values. An extension of the rank histogram diagnostic method to deal with potential duplicates shows that the ensemble distributions from the extended assimilation methods are more consistent with the truth.

Dynamical Gaussian, lognormal, and reverse lognormal maximum‐likelihood ensemble filter

AbstractWe present a non‐Gaussian ensemble data assimilation method based on the maximum‐likelihood ensemble filter, which allows for any combination of Gaussian, lognormal, and reverse lognormal errors in both the background and the observations. The technique is fully nonlinear, does not require a tangent linear model, and uses a Hessian preconditioner to minimise the cost function efficiently in ensemble space. When the Gaussian assumption is relaxed, the results show significant improvements in the analysis skill within two atmospheric toy models, and the performance of data assimilation systems for (semi)bounded variables is expected to improve.

Ensemble forecasts in reproducing kernel Hilbert space family

A methodological framework for ensemble-based estimation and simulation of high dimensional dynamical systems such as the oceanic or atmospheric flows is proposed. To that end, the dynamical system is embedded in a family of reproducing kernel Hilbert spaces (RKHS) with kernel functions driven by the dynamics. In the RKHS family, the Koopman and Perron–Frobenius operators are unitary and uniformly continuous. This property warrants they can be expressed in exponential series of diagonalizable bounded evolution operators defined from their infinitesimal generators. Access to Lyapunov exponents and to exact ensemble based expressions of the tangent linear dynamics are directly available as well. The RKHS family enables us the devise of strikingly simple ensemble data assimilation methods for trajectory reconstructions in terms of constant-in-time linear combinations of trajectory samples. Such an embarrassingly simple strategy is made possible through a fully justified superposition principle ensuing from several fundamental theorems.

Estimating the occurrence of slow slip events and earthquakes with an ensemble Kalman filter

SUMMARY Our ability to forecast earthquakes and slow slip events is hampered by limited information on the current state of stress on faults. Ensemble data assimilation methods permit estimating the state by combining physics-based models and observations, while considering their uncertainties. We use an ensemble Kalman filter (EnKF) to estimate shear stresses, slip rates and the state θ acting on a fault point governed by rate-and-state friction embedded in a 1-D elastic medium. We test the effectiveness of data assimilation by conducting perfect model experiments. We assimilate noised shear-stress and velocity synthetic values acquired at a small distance to the fault. The assimilation of uncertain shear stress observations improves in particular the estimates of shear stress on fault segments hosting slow slip events, while assimilating observations of velocity improves their slip-rate estimation. Both types of observations help equally well to better estimate the state θ. For earthquakes, the shear stress observations improve the estimation of shear stress, slip rates and the state θ, whereas the velocity observations improve in particular the slip-rate estimation. Data assimilation significantly improves the estimates of the temporal occurrence of slow slip events and to a large extent also of earthquakes. Rapid and abrupt changes in velocity and shear stress during earthquakes lead to non-Gaussian priors for subsequent assimilation steps, which breaks the assumption of Gaussian priors of the EnKF. In spite of this, the EnKF still provides estimates that are unexpectedly close to the true evolution. In fact, the forecastability for earthquakes for the same alarm duration is very similar to slow slip events, having a very low miss rate with an alarm duration of just 10 per cent of the recurrence interval of the events. These results confirm that data assimilation is a promising approach for the combination of uncertain physics and indirect, noisy observations for the forecasting of both slow slip events and earthquakes.

Ensemble-based data assimilation for predictable zones and application for non-linear deep-water waves

The ensemble-based variational method is easier to implement and parallelize than the adjoint method. For circumstances in which observed data are too limited and sparse for oceanographic data assimilation, the surface wave reconstruction by ensemble adjoint-free data assimilation (SWEAD) method was developed in a previous study. SWEAD generates ensembles of search directions from Fourier modes to numerically differentiate the squared error between observed data and a physical model. However, Fourier modes are global bases and could be redundant for a narrow predictable zone confined by a dispersion relationship. To concentrate ensembles on the predictable zone, we propose using singular value decomposition (SVD) of the approximated Jacobian of the squared error. Here, the Jacobian was first approximated by the linear dispersion relationship and successively updated to consider the non-linearity of the physical system. A new criterion for reusing the ensemble was also devised for this new method, increasing the dimension of search directions. A twin experiment was conducted for non-linear deep-water waves, and the optimization efficiency of the new method—SWEAD using SVD (SWEAD-S)—was significantly greater than that of SWEAD. Expansion of the predictable zone caused by the effect of non-linearity on the wave group velocity is thought to be essential for this improvement.

Analog ensemble data assimilation in a quasigeostrophic coupled model

AbstractThe ensemble forecast dominates the computational cost of many data assimilation methods, especially for high‐resolution and coupled models. In situations where the cost is prohibitive, one can either use a lower‐cost model or a lower‐cost data assimilation method, or both. Ensemble optimal interpolation (EnOI) is a classical example of a lower‐cost ensemble data assimilation method that replaces the ensemble forecast with a single forecast and then constructs an ensemble about this single forecast by adding perturbations drawn from climatology. This research develops lower‐cost ensemble data assimilation methods that add perturbations to a single forecast, where the perturbations are obtained from analogs of the single model forecast. These analogs can either be found from a catalog of model states, constructed using linear combinations of model states from a catalog, or constructed using generative machine‐learning methods. Four analog ensemble data assimilation methods, including two new ones, are compared with EnOI in the context of a coupled model of intermediate complexity: Q‐GCM. Depending on the method and on the physical variable, analog methods can be up to 40% more accurate than EnOI.

A local particle filter and its Gaussian mixture extension implemented with minor modifications to the LETKF

Abstract. A particle filter (PF) is an ensemble data assimilation method that does not assume Gaussian error distributions. Recent studies proposed local PFs (LPFs), which use localization, as in the ensemble Kalman filter, to apply the PF efficiently for high-dimensional dynamics. Among others, Penny and Miyoshi (2016) developed an LPF in the form of the ensemble transform matrix of the local ensemble transform Kalman filter (LETKF). The LETKF has been widely accepted for various geophysical systems, including numerical weather prediction (NWP) models. Therefore, implementing the LPF consistently with an existing LETKF code is useful. This study develops a software platform for the LPF and its Gaussian mixture extension (LPFGM) by making slight modifications to the LETKF code with a simplified global climate model known as Simplified Parameterizations, Primitive Equation Dynamics (SPEEDY). A series of idealized twin experiments were accomplished under the ideal-model assumption. With large inflation by the relaxation to prior spread, the LPF showed stable filter performance with dense observations but became unstable with sparse observations. The LPFGM showed a more accurate and stable performance than the LPF with both dense and sparse observations. In addition to the relaxation parameter, regulating the resampling frequency and the amplitude of Gaussian kernels was important for the LPFGM. With a spatially inhomogeneous observing network, the LPFGM was superior to the LETKF in sparsely observed regions, where the background ensemble spread and non-Gaussianity were larger. The SPEEDY-based LETKF, LPF, and LPFGM systems are available as open-source software on GitHub (https://github.com/skotsuki/speedy-lpf, last access: 16 November 2022) and can be adapted to various models relatively easily, as in the case of the LETKF.

An ensemble data assimilation modeling system for operational outdoor microalgae growth forecasting.

Microalgae have received increasing attention as a potential feedstock for biofuel or biobased products. Forecasting the microalgae growth is beneficial for managers in planning pond operations and harvesting decisions. This study proposed a biomass forecasting system comprised of the Huesemann Algae Biomass Growth Model (BGM), the Modular Aquatic Simulation System in Two Dimensions (MASS2), ensemble data assimilation (DA), and numerical weather prediction Global Ensemble Forecast System (GEFS) ensemble meteorological forecasts. The novelty of this study is to seek the use of ensemble DA to improve both BGM and MASS2 model initial conditions with the assimilation of biomass and water temperature measurements and consequently improve short-term biomass forecasting skills. This study introduces the theory behind the proposed integrated biomass forecasting system, with an application undertaken in pseudo-real-time in three outdoor ponds cultured with Chlorella sorokiniana in Delhi, California, United States. Results from all three case studies demonstrate that the biomass forecasting system improved the short-term (i.e., 7-day) biomass forecasting skills by about 60% on average, comparing to forecasts without using the ensemble DA method. Given the satisfactory performances achieved in this study, it is probable that the integrated BGM-MASS2-DA forecasting system can be used operationally to inform managers in making pond operation and harvesting planning decisions.

Ensemble data assimilation using optimal control in the Wasserstein metric

An ensemble data assimilation method is proposed that is based on optimal control minimizing the cost of mismatch in the Wasserstein metric on the observation space. The new method achieved the optimal state without calculating the posterior distribution of the particle state and the particle states are evolved deterministically, which is easy to be implemented. The method is appropriate for systems in which multiple, noisy, partial observations are available (e.g. citizen weather stations or smart phones). The method is demonstrated for: (i) deterministic dynamics with uncertain initial conditions, (ii) multiple noisy observations of a randomly forced ordinary differential equation (ODE), (iii) observations from multiple sample paths from a stochastic differential equation (SDE). A bi-modal measure and a measure supported on a strange attractor are tested. The numerical results show that our method performs a relatively small Wasserstein distance which measures the approximation performance. But numerical implementation is a bit expensive due to the complexity of Wasserstein distance computation, especially with large set of particles.

Real-time thermoacoustic data assimilation

Low-order thermoacoustic models are qualitatively correct, but typically, they are quantitatively inaccurate. We propose a time-domain bias-aware method to make qualitatively low-order models quantitatively (more) accurate. First, we develop a Bayesian ensemble data assimilation method for a low-order model to self-adapt and self-correct any time that reference data become available. Second, we apply the methodology to infer the thermoacoustic states and heat-release parameters on the fly without storing data (real time). We perform twin experiments using synthetic acoustic pressure measurements to analyse the performance of data assimilation in all nonlinear thermoacoustic regimes, from limit cycles to chaos, and interpret the results physically. Third, we propose practical rules for thermoacoustic data assimilation. An increase, reject, inflate strategy is proposed to deal with the rich nonlinear behaviour; and physical time scales for assimilation are proposed in non-chaotic regimes (with the Nyquist–Shannon criterion) and in chaotic regimes (with the Lyapunov time). Fourth, we perform data assimilation using data from a higher-fidelity model. We introduce an echo state network to estimate in real time the forecast bias, which is the model error of the low-fidelity model. We show that: (i) the correct acoustic pressure, parameters, and model bias can be inferred accurately; (ii) the learning is robust as it can tackle large uncertainties in the observations (up to 50 % of the mean values); (iii) the uncertainty of the prediction and parameters is naturally part of the output; and (iv) both the time-accurate solution and statistics can be inferred successfully. Data assimilation opens up new possibility for real-time prediction of thermoacoustics by combining physical knowledge and experimental data synergistically.

Calibration of RANS model constant based on data assimilation and accurate simulation of separated flow

To improve the prediction accuracy of separated flow based on the Reynolds Averaged Navier–Stokes model, the model constants of the baseline Reynolds stress model are calibrated by the ensemble Kalman filter data assimilation method. The separated flow in a diffuser is taken as the object, and the wall pressure coefficients of the diffuser are used as the driving data. The results show that the method that recalibrates the model constants based on data assimilation is easy to implement and is an effective method. The wall pressure coefficients and the separation regions of the diffuser predicted by the baseline Reynolds stress model with the default model constants deviate greatly from the experimental observations. By recalibrating the model constants, the prediction accuracy of separated flow based on the baseline Reynolds stress model is improved. This provides an idea for the accurate simulation of separated flow based on the Reynolds Averaged Navier–Stokes model in engineering applications.

A One-Step Blended Soundproof-Compressible Model with Balanced Data Assimilation: Theory and Idealized Tests

Abstract A challenge arising from the local Bayesian assimilation of data in an atmospheric flow simulation is the imbalances it may introduce. Acoustic fast-mode imbalances of the order of the slower dynamics can be negated by employing a blended numerical model with seamless access to the compressible and the soundproof pseudo-incompressible dynamics. Here, the blended modeling strategy by Benacchio et al. is upgraded in an advanced numerical framework and extended with a Bayesian local ensemble data assimilation method. Upon assimilation of data, the model configuration is switched to the pseudo-incompressible regime for one time step. After that, the model configuration is switched back to the compressible model for the duration of the assimilation window. The switching between model regimes is repeated for each subsequent assimilation window. An improved blending strategy for the numerical model ensures that a single time step in the pseudo-incompressible regime is sufficient to suppress imbalances coming from the initialization and data assimilation. This improvement is based on three innovations: (i) the association of pressure fields computed at different stages of the numerical integration with actual time levels, (ii) a conversion of pressure-related variables between the model regimes derived from low Mach number asymptotics, and (iii) a judicious selection of the pressure variables used in converting numerical model states when a switch of models occurs. Idealized two-dimensional traveling vortex and buoyancy-driven bubble convection experiments show that acoustic imbalances arising from data assimilation can be eliminated by using this blended model, thereby achieving balanced analysis fields. Significance Statement Weather forecasting models use a combination of physics-based algorithms and meteorological measurements. A problem with combining outputs from the model with measurements of the atmosphere is that insignificant signals may generate noise and compromise the physical soundness of weather-relevant processes. By selecting atmospheric processes through the toggling of parameters in a mixed model, we propose to suppress the undesirable signals in an efficient way and retain the physical features of solutions produced by the model. The approach is validated here for acoustic imbalances using a compressible/pseudo-incompressible model pair. This development has the potential to improve the techniques used to bring observations into models and with them the quality of atmospheric model output.

Super-resolution data assimilation

Increasing model resolution can improve the performance of a data assimilation system because it reduces model error, the system can more optimally use high-resolution observations, and with an ensemble data assimilation method the forecast error covariances are improved. However, increasing the resolution scales with a cubical increase of the computational costs. A method that can more effectively improve performance is introduced here. The novel approach called “Super-resolution data assimilation” (SRDA) is inspired from super-resolution image processing techniques and brought to the data assimilation context. Starting from a low-resolution forecast, a neural network (NN) emulates the fields to high-resolution, assimilates high-resolution observations, and scales it back up to the original resolution for running the next model step. The SRDA is tested with a quasi-geostrophic model in an idealized twin experiment for configurations where the model resolution is twice and four times lower than the reference solution from which pseudo-observations are extracted. The assimilation is performed with an Ensemble Kalman Filter. We show that SRDA outperforms both the low-resolution data assimilation approach and a version of SRDA with cubic spline interpolation instead of NN. The NN’s ability to anticipate the systematic differences between low- and high-resolution model dynamics explains the enhanced performance, in particular by correcting the difference of propagation speed of eddies. With a 25-member ensemble at low resolution, the SRDA computational overhead is 55% and the errors reduce by 40%, making the performance very close to that of the high-resolution system (52% of error reduction) that increases the cost by 800%. The reliability of the ensemble system is not degraded by SRDA.

Estimation of Ocean Biogeochemical Parameters in an Earth System Model Using the Dual One Step Ahead Smoother: A Twin Experiment

Ocean biogeochemical (BGC) models utilise a large number of poorly-constrained global parameters to mimic unresolved processes and reproduce the observed complex spatio-temporal patterns. Large model errors stem primarily from inaccuracies in these parameters whose optimal values can vary both in space and time. This study aims to demonstrate the ability of ensemble data assimilation (DA) methods to provide high-quality and improved BGC parameters within an Earth system model in an idealized perfect twin experiment framework. We use the Norwegian Climate Prediction Model (NorCPM), which combines the Norwegian Earth System Model with the Dual-One-Step ahead smoothing-based Ensemble Kalman Filter (DOSA-EnKF). We aim to estimate five spatially varying BGC parameters by assimilating salinity and temperature profiles and surface BGC (Phytoplankton, Nitrate, Phosphate, Silicate, and Oxygen) observations in a strongly coupled DA framework—i.e., jointly updating ocean and BGC state-parameters during the assimilation. We show how BGC observations can effectively constrain error in the ocean physics and vice versa. The method converges quickly (less than a year) and largely reduces the errors in the BGC parameters. Some parameter error remains, but the resulting state variable error using the estimated parameters for a free ensemble run and for a reanalysis performs nearly as well as with true parameter values. Optimal parameter values can also be recovered by assimilating climatological BGC observations or sparse observational networks. The findings of this study demonstrate the applicability of the DA approach for tuning the system in a real framework.

Ensemble Kalman filter based data assimilation for tropical waves in the MJO skeleton model

AbstractThe Madden–Julian oscillation (MJO) is the dominant component of tropical intraseasonal variability, with wide‐reaching impacts even on extratropical weather and climate patterns. However, predicting the MJO is challenging. One reason is the suboptimal state estimates obtained with standard data assimilation (DA) approaches. These are typically based on filtering methods with Gaussian approximations and do not take into account physical properties that are important specifically for the MJO. In this article, a constrained ensemble DA method is applied to study the impact of different physical constraints on the state estimation and prediction of the MJO. The quadratic programming ensemble (QPEns) algorithm utilized extends the standard stochastic ensemble Kalman filter (EnKF) with specifiable constraints on the updates of all ensemble members. This allows us to recover physically more consistent states and to respect possible associated non‐Gaussian statistics. The study is based on identical twin experiments with an adopted nonlinear model for tropical intraseasonal variability. This so‐called skeleton model succeeds in reproducing the main large‐scale features of the MJO and closely related tropical waves, while keeping adequate simplicity for fast experiments on intraseasonal time‐scales. Conservation laws and other crucial physical properties from the model are examined as constraints in the QPEns. Our results demonstrate an overall improvement in the filtering and forecast skill when the model's total energy is conserved in the initial conditions. The degree of benefit is found to be dependent on the observational setup and the strength of the model's nonlinear dynamics. It is also shown that, even in cases where the statistical error in some waves remains comparable with the stochastic EnKF during the DA stage, their prediction is improved remarkably when using the initial state resulting from the QPEns.

A scalable model-independent iterative data assimilation tool for sequential and batch estimation of high dimensional model parameters and states

Ensemble-based data assimilation (DA) methods have displayed strong potential to improve model state and parameter estimation across several disciplines due to their computational efficiency, scalability, and ability to estimate uncertainty in the dynamic states and the parameters. However, a barrier to adoption of ensemble DA methods remains. Namely, there is currently a lack of available tools that enable efficient and scalable DA in a non-intrusive fashion and that support implementation flexibility. This paper presents an open-source software tool (PESTPP-DA) that implements a range of data assimilation methods—Ensemble Kalman filter, Ensemble Kalman Smoother and Ensemble Smoother—using the widely known PEST model-interface protocols, to interact with any model. Two iterative solutions can be used for nonlinear and/or non-Gaussian assimilation problems. To demonstrate the broad range of PESTPP-DA applications, two synthetic case studies are presented: (1) the Lorenz model and (2) a groundwater pumping test in the presence of a non-Gaussian hydraulic conductivity field.

Technical Note: Sequential ensemble data assimilation in convergent and divergent systems

Abstract. Data assimilation methods are used throughout the geosciences to combine information from uncertain models and uncertain measurement data. However, the characteristics of geophysical systems differ and may be distinguished between divergent and convergent systems. In divergent systems initially nearby states will drift apart, while they will coalesce in convergent systems. This difference has implications for the application of sequential ensemble data assimilation methods. This study explores these implications on two exemplary systems, i.e., the divergent Lorenz 96 model and the convergent description of soil water movement by the Richards equation. The results show that sequential ensemble data assimilation methods require a sufficient divergent component. This makes the transfer of the methods from divergent to convergent systems challenging. We demonstrate, through a set of case studies, that it is imperative to represent model errors adequately and incorporate parameter uncertainties in ensemble data assimilation in convergent systems.

Ensemble Kalman filter for nonconservative moving mesh solvers with a joint physics and mesh location update

AbstractNumerical solvers using adaptive meshes can focus computational power on important regions of a model domain capturing important or unresolved physics. The adaptation can be informed by the model state or external information, or made to depend on the model physics. In this latter case, one can think of the mesh configuration as part of the model state. If observational data are to be assimilated into the model, the question of updating the mesh configuration with the physical values arises. Adaptive meshes present significant challenges when using popular ensemble data assimilation (DA) methods. We develop a novel strategy for ensemble‐based DA, for which the adaptive mesh is updated along with the physical values. This involves including the node locations as a part of the model state itself, allowing them to be updated automatically at the analysis step. This poses a number of challenges, which we resolve to produce an effective approach that promises to apply with some generality. We evaluate our strategy with two testbed models in one dimension (1‐d), comparing them with a strategy we previously developed that does not update the mesh configuration. We find that updating the mesh improves the fidelity and convergence of the filter. An extensive analysis on the performance of our scheme beyond just the root‐mean‐squared error (RMSE) is also presented.

Using a machine learning proxy for localization in ensemble data assimilation

Ensemble data assimilation methods, particularly iterative forms of ensemble smoother, are very useful assisted history matching techniques. One of the main challenges in the application of these methods is the excessive variance loss due to sampling errors occasioned by the limited size ensembles used in practical cases. The standard procedure to mitigate this problem is called distance-based localization. However, in this case, the data assimilation performance becomes highly dependent on the choice of the localization region. Moreover, there are several relevant examples of non-local model parameters and data types, in which cases, distance-based localization does not apply. This paper proposes a distance-free localization procedure that combines a least-squares support vector (LS-SVR) proxy with a non-isotropic tapering function proposed in the literature. We tested the proposed method in two versions of the PUNQ-S3 problem. In the first version, we tested the method to localize scalar (non-local) parameters. The results show relevant improvements, especially in terms of preserving the posterior variance of the ensemble compared to methods investigated in Lacerda et al. (J. Pet. Sci. Eng. 172,690–706, 2019). The second version considered the problem of localization of grid parameters. In this case, the proposed method outperformed a distance-based localization approach proposed in the literature.

Comparison of ensemble data assimilation methods for the estimation of time-varying soil hydraulic parameters

The hydraulic properties of the soil top layer may change during the growth period due to various factors such as wetting and drying cycles, tillage practices, and crop root growth. In this study, the potential of the assimilation method to estimate time-varying soil hydraulic parameters is explored. Four assimilation schemes, including the simultaneous update state augmentation method, the partitioned update state augmentation method, the simultaneous update parameter correction method, and the partitioned update parameter correction method, are compared. The performance of four assimilation schemes on parameter estimations and soil moisture simulations is tested first using the synthetic case. The influence of initial parameter values and a parameter update order on assimilation is also analyzed. Finally, the partitioned update parameter correction method is applied to a real case involving a field drip irrigation experiment. The results show that when the analyzed parameter has either a periodical or a linear variation, there is a time lag between the assimilation value and the true value. The assimilation method can respond immediately to an abrupt change of the parameter value. Using the simultaneous update method leads to an obvious parameter correlation problem. In contrast, the partitioned update method can relieve the parameter correlation problems, thereby improving the accuracy of parameter estimations and pressure head simulations. However, when the initial values of the parameters deviate from their true values to a certain extent, the partitioned update method cannot obtain accurate parameter estimations. Compared with the partitioned update state augmentation method, the partitioned update parameter correction method is not sensitive to the parameter update order. The partitioned update parameter correction method has higher computational efficiency and assimilation stability, and it can obtain more accurate parameter estimations and soil moisture predictions in comparison with the traditional state augmentation method. The partitioned update parameter correction method provides an assimilation tool for improving the predictions of soil moisture by considering the time-varying parameters.