Ensemble Data Assimilation Methods Research Articles

Sequential model identification with reversible jump ensemble data assimilation method

In data assimilation (DA) schemes, the form representing the processes in the evolution models are pre-determined except some parameters to be estimated. In some applications, such as the contaminant solute transport model and the gas reservoir model, the modes in the equations within the evolution model cannot be predetermined from the outset and may change with the time. We propose a framework of sequential DA method named Reversible Jump Ensemble Filter (RJEnF) to identify the governing modes of the evolution model over time. The main idea is to introduce the Reversible Jump Markov Chain Monte Carlo (RJMCMC) method to the DA schemes to fit the situation where the modes of the evolution model are unknown and the dimension of the parameters is changing. Our framework allows us to identify the modes in the evolution model and their changes, as well as estimate the parameters and states of the dynamic system. Numerical experiments are conducted and the results show that our framework can effectively identify the underlying evolution models and increase the predictive accuracy of DA methods.

Using Satellite Data Assimilation Techniques to Combine Infrasound Observations and a Full Ray-Tracing Model to Constrain Stratospheric Variables

Abstract Infrasound waves generated at Earth’s surface can reach high altitudes before returning to the surface to be recorded by microbarometer array stations. These waves carry information about the propagation medium, in particular temperature and winds in the atmosphere. It is only recently that studies on the assimilation of such data into atmospheric models have been published. Intending to advance this line of research, we here use the modulated ensemble transform Kalman filter (METKF)—commonly used in satellite data assimilation—to assimilate infrasound-related observations in order to update a column of three vertically varying variables: temperature and horizontal wind speeds. This includes stratospheric and mesospheric heights, which are otherwise poorly observed. The numerical experiments on synthetic data but with realistic reanalysis product atmospheric specifications (following the observing system simulation experiment paradigm) reveal that a large ensemble is capable of reducing errors, especially for wind speeds in stratospheric heights close to 30–60 km. While using a small ensemble leads to incorrect analysis increments and large estimation errors, the METKF ameliorates this problem and even achieves error reduction from the prior to the posterior mean estimator. Significance Statement The stratosphere and mesosphere have significantly less observational coverage compared to the troposphere, especially for the winds. This lack of information can reduce the accuracy of medium-range weather forecasts. By mimicking a realistic setup, this study paves the way for including novel observations in the estimation of the atmospheric state in these heights using an ensemble data assimilation method. These observations come from a dataset of opportunity containing infrasound-related measurements that are routinely carried out at several stations around the world.

A Quantile-Conserving Ensemble Filter Framework. Part III: Data Assimilation for Mixed Distributions with Application to a Low-Order Tracer Advection Model

Abstract The uncertainty associated with many observed and modeled quantities of interest in Earth system prediction can be represented by mixed probability distributions that are neither discrete nor continuous. For instance, a forecast probability of precipitation can have a finite probability of zero precipitation, consistent with a discrete distribution. However, nonzero values are not discrete and are represented by a continuous distribution; the same is true for rainfall rate. Other examples include snow depth, sea ice concentration, amount of a tracer or the source rate of a tracer. Some Earth system model parameters may also have discrete or mixed distributions. Most ensemble data assimilation methods do not explicitly consider the possibility of mixed distributions. The Quantile Conserving Ensemble Filtering Framework (Anderson 2022, 2023) is extended to explicitly deal with discrete or mixed distributions. An example is given using bounded normal rank histogram probability distributions applied to observing system simulation experiments in a low-order tracer advection model. Analyses of tracer concentration and tracer source are shown to be improved when using the extended methods. A key feature of the resulting ensembles is that there can be ensemble members with duplicate values. An extension of the rank histogram diagnostic method to deal with potential duplicates shows that the ensemble distributions from the extended assimilation methods are more consistent with the truth.

Dynamical Gaussian, lognormal, and reverse lognormal maximum‐likelihood ensemble filter

AbstractWe present a non‐Gaussian ensemble data assimilation method based on the maximum‐likelihood ensemble filter, which allows for any combination of Gaussian, lognormal, and reverse lognormal errors in both the background and the observations. The technique is fully nonlinear, does not require a tangent linear model, and uses a Hessian preconditioner to minimise the cost function efficiently in ensemble space. When the Gaussian assumption is relaxed, the results show significant improvements in the analysis skill within two atmospheric toy models, and the performance of data assimilation systems for (semi)bounded variables is expected to improve.

GPU-enabled ensemble data assimilation for mesh-refined lattice Boltzmann method

We implemented the ensemble data assimilation (DA) method, the local ensemble transform Kalman filter (LETKF), into the mesh-refined lattice Boltzmann method (LBM) for turbulent flows. Both the LETKF and the mesh-refined LBM were fully implemented on GPUs, so that they are efficiently computed on modern GPU-based supercomputers. We examined the DA accuracy against the flow around a cylinder. The result showed that our method enabled accurate DA with spatially- and temporarily-sparse observation data; the error of the assimilated velocity field with the observation interval of τK/2 and the observation resolution D/16 (1.56% of the total computational grids) was smaller than the amplitude of the observation noise, where τK is the period of the Kármán vortex and D is diameter of the square cylinder.

Ensemble forecasts in reproducing kernel Hilbert space family

A methodological framework for ensemble-based estimation and simulation of high dimensional dynamical systems such as the oceanic or atmospheric flows is proposed. To that end, the dynamical system is embedded in a family of reproducing kernel Hilbert spaces (RKHS) with kernel functions driven by the dynamics. In the RKHS family, the Koopman and Perron–Frobenius operators are unitary and uniformly continuous. This property warrants they can be expressed in exponential series of diagonalizable bounded evolution operators defined from their infinitesimal generators. Access to Lyapunov exponents and to exact ensemble based expressions of the tangent linear dynamics are directly available as well. The RKHS family enables us the devise of strikingly simple ensemble data assimilation methods for trajectory reconstructions in terms of constant-in-time linear combinations of trajectory samples. Such an embarrassingly simple strategy is made possible through a fully justified superposition principle ensuing from several fundamental theorems.

Estimating the occurrence of slow slip events and earthquakes with an ensemble Kalman filter

SUMMARY Our ability to forecast earthquakes and slow slip events is hampered by limited information on the current state of stress on faults. Ensemble data assimilation methods permit estimating the state by combining physics-based models and observations, while considering their uncertainties. We use an ensemble Kalman filter (EnKF) to estimate shear stresses, slip rates and the state θ acting on a fault point governed by rate-and-state friction embedded in a 1-D elastic medium. We test the effectiveness of data assimilation by conducting perfect model experiments. We assimilate noised shear-stress and velocity synthetic values acquired at a small distance to the fault. The assimilation of uncertain shear stress observations improves in particular the estimates of shear stress on fault segments hosting slow slip events, while assimilating observations of velocity improves their slip-rate estimation. Both types of observations help equally well to better estimate the state θ. For earthquakes, the shear stress observations improve the estimation of shear stress, slip rates and the state θ, whereas the velocity observations improve in particular the slip-rate estimation. Data assimilation significantly improves the estimates of the temporal occurrence of slow slip events and to a large extent also of earthquakes. Rapid and abrupt changes in velocity and shear stress during earthquakes lead to non-Gaussian priors for subsequent assimilation steps, which breaks the assumption of Gaussian priors of the EnKF. In spite of this, the EnKF still provides estimates that are unexpectedly close to the true evolution. In fact, the forecastability for earthquakes for the same alarm duration is very similar to slow slip events, having a very low miss rate with an alarm duration of just 10 per cent of the recurrence interval of the events. These results confirm that data assimilation is a promising approach for the combination of uncertain physics and indirect, noisy observations for the forecasting of both slow slip events and earthquakes.

Ensemble-based data assimilation for predictable zones and application for non-linear deep-water waves

The ensemble-based variational method is easier to implement and parallelize than the adjoint method. For circumstances in which observed data are too limited and sparse for oceanographic data assimilation, the surface wave reconstruction by ensemble adjoint-free data assimilation (SWEAD) method was developed in a previous study. SWEAD generates ensembles of search directions from Fourier modes to numerically differentiate the squared error between observed data and a physical model. However, Fourier modes are global bases and could be redundant for a narrow predictable zone confined by a dispersion relationship. To concentrate ensembles on the predictable zone, we propose using singular value decomposition (SVD) of the approximated Jacobian of the squared error. Here, the Jacobian was first approximated by the linear dispersion relationship and successively updated to consider the non-linearity of the physical system. A new criterion for reusing the ensemble was also devised for this new method, increasing the dimension of search directions. A twin experiment was conducted for non-linear deep-water waves, and the optimization efficiency of the new method—SWEAD using SVD (SWEAD-S)—was significantly greater than that of SWEAD. Expansion of the predictable zone caused by the effect of non-linearity on the wave group velocity is thought to be essential for this improvement.

Analog ensemble data assimilation in a quasigeostrophic coupled model

AbstractThe ensemble forecast dominates the computational cost of many data assimilation methods, especially for high‐resolution and coupled models. In situations where the cost is prohibitive, one can either use a lower‐cost model or a lower‐cost data assimilation method, or both. Ensemble optimal interpolation (EnOI) is a classical example of a lower‐cost ensemble data assimilation method that replaces the ensemble forecast with a single forecast and then constructs an ensemble about this single forecast by adding perturbations drawn from climatology. This research develops lower‐cost ensemble data assimilation methods that add perturbations to a single forecast, where the perturbations are obtained from analogs of the single model forecast. These analogs can either be found from a catalog of model states, constructed using linear combinations of model states from a catalog, or constructed using generative machine‐learning methods. Four analog ensemble data assimilation methods, including two new ones, are compared with EnOI in the context of a coupled model of intermediate complexity: Q‐GCM. Depending on the method and on the physical variable, analog methods can be up to 40% more accurate than EnOI.

A local particle filter and its Gaussian mixture extension implemented with minor modifications to the LETKF

Abstract. A particle filter (PF) is an ensemble data assimilation method that does not assume Gaussian error distributions. Recent studies proposed local PFs (LPFs), which use localization, as in the ensemble Kalman filter, to apply the PF efficiently for high-dimensional dynamics. Among others, Penny and Miyoshi (2016) developed an LPF in the form of the ensemble transform matrix of the local ensemble transform Kalman filter (LETKF). The LETKF has been widely accepted for various geophysical systems, including numerical weather prediction (NWP) models. Therefore, implementing the LPF consistently with an existing LETKF code is useful. This study develops a software platform for the LPF and its Gaussian mixture extension (LPFGM) by making slight modifications to the LETKF code with a simplified global climate model known as Simplified Parameterizations, Primitive Equation Dynamics (SPEEDY). A series of idealized twin experiments were accomplished under the ideal-model assumption. With large inflation by the relaxation to prior spread, the LPF showed stable filter performance with dense observations but became unstable with sparse observations. The LPFGM showed a more accurate and stable performance than the LPF with both dense and sparse observations. In addition to the relaxation parameter, regulating the resampling frequency and the amplitude of Gaussian kernels was important for the LPFGM. With a spatially inhomogeneous observing network, the LPFGM was superior to the LETKF in sparsely observed regions, where the background ensemble spread and non-Gaussianity were larger. The SPEEDY-based LETKF, LPF, and LPFGM systems are available as open-source software on GitHub (https://github.com/skotsuki/speedy-lpf, last access: 16 November 2022) and can be adapted to various models relatively easily, as in the case of the LETKF.

An ensemble data assimilation modeling system for operational outdoor microalgae growth forecasting.

Microalgae have received increasing attention as a potential feedstock for biofuel or biobased products. Forecasting the microalgae growth is beneficial for managers in planning pond operations and harvesting decisions. This study proposed a biomass forecasting system comprised of the Huesemann Algae Biomass Growth Model (BGM), the Modular Aquatic Simulation System in Two Dimensions (MASS2), ensemble data assimilation (DA), and numerical weather prediction Global Ensemble Forecast System (GEFS) ensemble meteorological forecasts. The novelty of this study is to seek the use of ensemble DA to improve both BGM and MASS2 model initial conditions with the assimilation of biomass and water temperature measurements and consequently improve short-term biomass forecasting skills. This study introduces the theory behind the proposed integrated biomass forecasting system, with an application undertaken in pseudo-real-time in three outdoor ponds cultured with Chlorella sorokiniana in Delhi, California, United States. Results from all three case studies demonstrate that the biomass forecasting system improved the short-term (i.e., 7-day) biomass forecasting skills by about 60% on average, comparing to forecasts without using the ensemble DA method. Given the satisfactory performances achieved in this study, it is probable that the integrated BGM-MASS2-DA forecasting system can be used operationally to inform managers in making pond operation and harvesting planning decisions.

Ensemble data assimilation using optimal control in the Wasserstein metric

An ensemble data assimilation method is proposed that is based on optimal control minimizing the cost of mismatch in the Wasserstein metric on the observation space. The new method achieved the optimal state without calculating the posterior distribution of the particle state and the particle states are evolved deterministically, which is easy to be implemented. The method is appropriate for systems in which multiple, noisy, partial observations are available (e.g. citizen weather stations or smart phones). The method is demonstrated for: (i) deterministic dynamics with uncertain initial conditions, (ii) multiple noisy observations of a randomly forced ordinary differential equation (ODE), (iii) observations from multiple sample paths from a stochastic differential equation (SDE). A bi-modal measure and a measure supported on a strange attractor are tested. The numerical results show that our method performs a relatively small Wasserstein distance which measures the approximation performance. But numerical implementation is a bit expensive due to the complexity of Wasserstein distance computation, especially with large set of particles.

Real-time thermoacoustic data assimilation

Low-order thermoacoustic models are qualitatively correct, but typically, they are quantitatively inaccurate. We propose a time-domain bias-aware method to make qualitatively low-order models quantitatively (more) accurate. First, we develop a Bayesian ensemble data assimilation method for a low-order model to self-adapt and self-correct any time that reference data become available. Second, we apply the methodology to infer the thermoacoustic states and heat-release parameters on the fly without storing data (real time). We perform twin experiments using synthetic acoustic pressure measurements to analyse the performance of data assimilation in all nonlinear thermoacoustic regimes, from limit cycles to chaos, and interpret the results physically. Third, we propose practical rules for thermoacoustic data assimilation. An increase, reject, inflate strategy is proposed to deal with the rich nonlinear behaviour; and physical time scales for assimilation are proposed in non-chaotic regimes (with the Nyquist–Shannon criterion) and in chaotic regimes (with the Lyapunov time). Fourth, we perform data assimilation using data from a higher-fidelity model. We introduce an echo state network to estimate in real time the forecast bias, which is the model error of the low-fidelity model. We show that: (i) the correct acoustic pressure, parameters, and model bias can be inferred accurately; (ii) the learning is robust as it can tackle large uncertainties in the observations (up to 50 % of the mean values); (iii) the uncertainty of the prediction and parameters is naturally part of the output; and (iv) both the time-accurate solution and statistics can be inferred successfully. Data assimilation opens up new possibility for real-time prediction of thermoacoustics by combining physical knowledge and experimental data synergistically.

Calibration of RANS model constant based on data assimilation and accurate simulation of separated flow

To improve the prediction accuracy of separated flow based on the Reynolds Averaged Navier–Stokes model, the model constants of the baseline Reynolds stress model are calibrated by the ensemble Kalman filter data assimilation method. The separated flow in a diffuser is taken as the object, and the wall pressure coefficients of the diffuser are used as the driving data. The results show that the method that recalibrates the model constants based on data assimilation is easy to implement and is an effective method. The wall pressure coefficients and the separation regions of the diffuser predicted by the baseline Reynolds stress model with the default model constants deviate greatly from the experimental observations. By recalibrating the model constants, the prediction accuracy of separated flow based on the baseline Reynolds stress model is improved. This provides an idea for the accurate simulation of separated flow based on the Reynolds Averaged Navier–Stokes model in engineering applications.

A One-Step Blended Soundproof-Compressible Model with Balanced Data Assimilation: Theory and Idealized Tests

Abstract A challenge arising from the local Bayesian assimilation of data in an atmospheric flow simulation is the imbalances it may introduce. Acoustic fast-mode imbalances of the order of the slower dynamics can be negated by employing a blended numerical model with seamless access to the compressible and the soundproof pseudo-incompressible dynamics. Here, the blended modeling strategy by Benacchio et al. is upgraded in an advanced numerical framework and extended with a Bayesian local ensemble data assimilation method. Upon assimilation of data, the model configuration is switched to the pseudo-incompressible regime for one time step. After that, the model configuration is switched back to the compressible model for the duration of the assimilation window. The switching between model regimes is repeated for each subsequent assimilation window. An improved blending strategy for the numerical model ensures that a single time step in the pseudo-incompressible regime is sufficient to suppress imbalances coming from the initialization and data assimilation. This improvement is based on three innovations: (i) the association of pressure fields computed at different stages of the numerical integration with actual time levels, (ii) a conversion of pressure-related variables between the model regimes derived from low Mach number asymptotics, and (iii) a judicious selection of the pressure variables used in converting numerical model states when a switch of models occurs. Idealized two-dimensional traveling vortex and buoyancy-driven bubble convection experiments show that acoustic imbalances arising from data assimilation can be eliminated by using this blended model, thereby achieving balanced analysis fields. Significance Statement Weather forecasting models use a combination of physics-based algorithms and meteorological measurements. A problem with combining outputs from the model with measurements of the atmosphere is that insignificant signals may generate noise and compromise the physical soundness of weather-relevant processes. By selecting atmospheric processes through the toggling of parameters in a mixed model, we propose to suppress the undesirable signals in an efficient way and retain the physical features of solutions produced by the model. The approach is validated here for acoustic imbalances using a compressible/pseudo-incompressible model pair. This development has the potential to improve the techniques used to bring observations into models and with them the quality of atmospheric model output.

Super-resolution data assimilation

Increasing model resolution can improve the performance of a data assimilation system because it reduces model error, the system can more optimally use high-resolution observations, and with an ensemble data assimilation method the forecast error covariances are improved. However, increasing the resolution scales with a cubical increase of the computational costs. A method that can more effectively improve performance is introduced here. The novel approach called “Super-resolution data assimilation” (SRDA) is inspired from super-resolution image processing techniques and brought to the data assimilation context. Starting from a low-resolution forecast, a neural network (NN) emulates the fields to high-resolution, assimilates high-resolution observations, and scales it back up to the original resolution for running the next model step. The SRDA is tested with a quasi-geostrophic model in an idealized twin experiment for configurations where the model resolution is twice and four times lower than the reference solution from which pseudo-observations are extracted. The assimilation is performed with an Ensemble Kalman Filter. We show that SRDA outperforms both the low-resolution data assimilation approach and a version of SRDA with cubic spline interpolation instead of NN. The NN’s ability to anticipate the systematic differences between low- and high-resolution model dynamics explains the enhanced performance, in particular by correcting the difference of propagation speed of eddies. With a 25-member ensemble at low resolution, the SRDA computational overhead is 55% and the errors reduce by 40%, making the performance very close to that of the high-resolution system (52% of error reduction) that increases the cost by 800%. The reliability of the ensemble system is not degraded by SRDA.

Estimation of Ocean Biogeochemical Parameters in an Earth System Model Using the Dual One Step Ahead Smoother: A Twin Experiment

Ocean biogeochemical (BGC) models utilise a large number of poorly-constrained global parameters to mimic unresolved processes and reproduce the observed complex spatio-temporal patterns. Large model errors stem primarily from inaccuracies in these parameters whose optimal values can vary both in space and time. This study aims to demonstrate the ability of ensemble data assimilation (DA) methods to provide high-quality and improved BGC parameters within an Earth system model in an idealized perfect twin experiment framework. We use the Norwegian Climate Prediction Model (NorCPM), which combines the Norwegian Earth System Model with the Dual-One-Step ahead smoothing-based Ensemble Kalman Filter (DOSA-EnKF). We aim to estimate five spatially varying BGC parameters by assimilating salinity and temperature profiles and surface BGC (Phytoplankton, Nitrate, Phosphate, Silicate, and Oxygen) observations in a strongly coupled DA framework—i.e., jointly updating ocean and BGC state-parameters during the assimilation. We show how BGC observations can effectively constrain error in the ocean physics and vice versa. The method converges quickly (less than a year) and largely reduces the errors in the BGC parameters. Some parameter error remains, but the resulting state variable error using the estimated parameters for a free ensemble run and for a reanalysis performs nearly as well as with true parameter values. Optimal parameter values can also be recovered by assimilating climatological BGC observations or sparse observational networks. The findings of this study demonstrate the applicability of the DA approach for tuning the system in a real framework.

Ensemble Kalman filter based data assimilation for tropical waves in the MJO skeleton model

AbstractThe Madden–Julian oscillation (MJO) is the dominant component of tropical intraseasonal variability, with wide‐reaching impacts even on extratropical weather and climate patterns. However, predicting the MJO is challenging. One reason is the suboptimal state estimates obtained with standard data assimilation (DA) approaches. These are typically based on filtering methods with Gaussian approximations and do not take into account physical properties that are important specifically for the MJO. In this article, a constrained ensemble DA method is applied to study the impact of different physical constraints on the state estimation and prediction of the MJO. The quadratic programming ensemble (QPEns) algorithm utilized extends the standard stochastic ensemble Kalman filter (EnKF) with specifiable constraints on the updates of all ensemble members. This allows us to recover physically more consistent states and to respect possible associated non‐Gaussian statistics. The study is based on identical twin experiments with an adopted nonlinear model for tropical intraseasonal variability. This so‐called skeleton model succeeds in reproducing the main large‐scale features of the MJO and closely related tropical waves, while keeping adequate simplicity for fast experiments on intraseasonal time‐scales. Conservation laws and other crucial physical properties from the model are examined as constraints in the QPEns. Our results demonstrate an overall improvement in the filtering and forecast skill when the model's total energy is conserved in the initial conditions. The degree of benefit is found to be dependent on the observational setup and the strength of the model's nonlinear dynamics. It is also shown that, even in cases where the statistical error in some waves remains comparable with the stochastic EnKF during the DA stage, their prediction is improved remarkably when using the initial state resulting from the QPEns.

A scalable model-independent iterative data assimilation tool for sequential and batch estimation of high dimensional model parameters and states

Ensemble-based data assimilation (DA) methods have displayed strong potential to improve model state and parameter estimation across several disciplines due to their computational efficiency, scalability, and ability to estimate uncertainty in the dynamic states and the parameters. However, a barrier to adoption of ensemble DA methods remains. Namely, there is currently a lack of available tools that enable efficient and scalable DA in a non-intrusive fashion and that support implementation flexibility. This paper presents an open-source software tool (PESTPP-DA) that implements a range of data assimilation methods—Ensemble Kalman filter, Ensemble Kalman Smoother and Ensemble Smoother—using the widely known PEST model-interface protocols, to interact with any model. Two iterative solutions can be used for nonlinear and/or non-Gaussian assimilation problems. To demonstrate the broad range of PESTPP-DA applications, two synthetic case studies are presented: (1) the Lorenz model and (2) a groundwater pumping test in the presence of a non-Gaussian hydraulic conductivity field.

Technical Note: Sequential ensemble data assimilation in convergent and divergent systems

Abstract. Data assimilation methods are used throughout the geosciences to combine information from uncertain models and uncertain measurement data. However, the characteristics of geophysical systems differ and may be distinguished between divergent and convergent systems. In divergent systems initially nearby states will drift apart, while they will coalesce in convergent systems. This difference has implications for the application of sequential ensemble data assimilation methods. This study explores these implications on two exemplary systems, i.e., the divergent Lorenz 96 model and the convergent description of soil water movement by the Richards equation. The results show that sequential ensemble data assimilation methods require a sufficient divergent component. This makes the transfer of the methods from divergent to convergent systems challenging. We demonstrate, through a set of case studies, that it is imperative to represent model errors adequately and incorporate parameter uncertainties in ensemble data assimilation in convergent systems.