Sequential Data Assimilation Methods Research Articles

Sequential model identification with reversible jump ensemble data assimilation method

In data assimilation (DA) schemes, the form representing the processes in the evolution models are pre-determined except some parameters to be estimated. In some applications, such as the contaminant solute transport model and the gas reservoir model, the modes in the equations within the evolution model cannot be predetermined from the outset and may change with the time. We propose a framework of sequential DA method named Reversible Jump Ensemble Filter (RJEnF) to identify the governing modes of the evolution model over time. The main idea is to introduce the Reversible Jump Markov Chain Monte Carlo (RJMCMC) method to the DA schemes to fit the situation where the modes of the evolution model are unknown and the dimension of the parameters is changing. Our framework allows us to identify the modes in the evolution model and their changes, as well as estimate the parameters and states of the dynamic system. Numerical experiments are conducted and the results show that our framework can effectively identify the underlying evolution models and increase the predictive accuracy of DA methods.

A novel estimation method for microstructural evolution based on data assimilation and phase field crystal model

In the simulation of microstructural evolutions, detailed priori knowledge for the model parameters and initial states is difficult to be observed by experiments. For an improved simulation, we present a data assimilation framework for the phase field crystal model with the effect of stochastic noise. A sequential data assimilation method based on the ensemble Kalman filter is applied to integrate phase field simulation and observational data from experiments. Furthermore, we couple the present framework with a second-order accurate unconditional-energy stable scheme, and thus the issue of stability restriction can be avoided. A series of twin experiments are performed to demonstrate the performance of our present framework. The results reveal that our framework can successfully improve the simulation accuracy, even when some parameters are uncertain and the initial value significantly deviates from the observed situation.

Streamflow simulation in poorly gauged basins with regionalised assimilation using Kalman filter

The streamflow estimation in ungauged or poorly gauged basins is a fundamental and challenging problem in hydrology, which has often been solved by transferring hydrological information from gauged basins (i.e. by regionalisation). Most studies on streamflow regionalisation focus on identifying the best methods to transfer the hydrologic model parameters and uses primarily physiographic attributes or climate information for these purposes. In the present study, the sequential data assimilation method – the Kalman filter – has been used to determine streamflow in a poorly gauged (unknown) basin to combine, in an optimal way, observations of neighbouring basins and model simulation of a given basin. The methodology is based on the concept of concatenated upstream catchments, where the aggregation of unobserved states can be estimated. The streamflow estimate is further divided between the unknown sub-catchments using linear regression on the catchments’ hydrological characteristics, which are subsequently used to approximate error statistics and operators in the Kalman filter application. The results were evaluated on 165 catchments in the Czech Republic using RMSE, MASE and MAE criteria and indicate that in 87.3% of the cases, the proposed methodology improved the accuracy of streamflow estimations by an average of 40% (in combined evaluated measurements) compared to the original simulations within the system for drought monitoring and forecasting in the Czech Republic ‘HAMR’.

Regulating the time of the crop model clock: A data assimilation framework for regions with high phenological heterogeneity

in crop growth data assimilation systems, the mismatch between simulated and observed phenology significantly deteriorates the performance of crop growth modeling. This situation may be more severe for smallholder farmers-managed fields, where the phenological heterogeneity was high even when climate condition was relatively uniform. Previous studies investigated the non-sequential methods to retrospectively assimilate historical phenology observations. However, approaches to dynamically assimilating phenological measurements through sequential data assimilation methods remain unexplored one of the most intractable challenges of dynamic phenology assimilation is that a considerable proportion of model parameters and variables are entangled with phenology, therefore simply assimilating phenological measurements could disturb the model clock. This study aims to establish a robust crop data assimilation framework capable of assimilating phenological measurements in real time without disturbing the model clock the framework used an open-source version of the AquaCrop model to simulate crop growth and used the ensemble Kalman filter (EnKF) to assimilate observations sequentially. A parameter refresh method was proposed to restore the phenological consistency of model parameters after updating the phenology state. Assimilation strategies with different observation types and compositions of state vectors were designed after a global sensitivity analysis of model parameters. These strategies were evaluated through the Observing System Simulation Experiments (OSSE), and the selected strategies were tested in a real-world case. the results of the OSS Experiments show that the phenological mismatch problem greatly affects crop growth simulation, and this mismatch could not be narrowed effectively by assimilating non-phenological observations. Assimilating phenological measurements with the proposed parameters refresh method and assimilation strategies closed this mismatch and produced better performance compared to the Restart-EnKF method. In the real-world paddy rice case, assimilating phenology with the proposed strategies significantly improved yield estimation in low-yield plots (less than 4 ton/ha) compared to assimilating canopy cover (CC) alone, with an R2 increase from 0.07 to 0.48. Assimilating CC, biomass and phenology simultaneously produced the best yield estimation for all plots, with R2 = 0.57 and RMSE = 1.00 ton/ha. assimilating phenology under a consistent model clock significantly improved yield estimation when the phenological heterogeneity of plots was high. the results highlight the effectiveness and robustness of the established data assimilation framework for dynamic crop growth simulation, indicating the potentials of the proposed data assimilation framework for regional in-season crop modeling and yield forecasting.

Ensemble based methods for leapfrog integration in the simplified parameterizations, primitive‐equation dynamics model

AbstractThis paper presents efficient and practical implementations of sequential data assimilation methods for the Simplified Parameterizations, primitive‐Equation DYnamics (SPEEDY) Model, a well‐known numerical model, into the data assimilation community for climate prediction. In the SPEEDY model, the time evolution of dynamics is performed via the second‐order Leapfrog integration scheme; this time integrator relies on two steps: the position and the velocity. The computational implementation of SPEEDY blends the time integrator and the spatial discretization of dynamics to accelerate algebraic computations. Thus, there is no access to the right‐hand side function of the ordinary differential equations governing the time evolution of model dynamics. Consequently, the SPEEDY model is often treated as a black box wherein positions and velocities work as inputs and outputs. Since observations in operational data assimilation only match position states, we can exploit augmented vector states to propagate analysis innovations from positions to velocities. For this purpose, we formulate three variants of ensemble‐based filters and perform numerical experiments to assess their accuracies. We consider two scenarios for the experiments: an ideal case wherein positions and velocities can be observed and a more realistic one wherein measurements are only accessible for position states. Besides, we discuss the effects of the ensemble size on the accuracies of our formulations and, even more, the typical case in which velocities are not updated across assimilation steps. The results reveal that all filter formulations' accuracies remain the same in terms of Root‐Mean‐Square‐Error by neglecting observations from velocities (a realistic scenario) even for cases wherein the number of measurements decreases to 6% of model components. Furthermore, for all discussed filter implementations, the propagation of analysis increments from position to velocities improves up to 100% the performance of filter implementations wherein velocities are not updated, a typical operational scenario.

Comparison of sequential and variational assimilation methods to improve hydrological predictions in snow dominated mountainous catchments

• Assimilation of streamflow and snow data using variational and sequential techniques in a hydrological model. • Satellite snow coverage products are assimilated together with streamflow data. • Variational techniques outperform sequential techniques in two mountainous basins. • Average relative gains in variational assimilation are 49% for streamflow and 28% for snow data. Snowmelt modeling is vital issue for proper operational flow forecasting systems but still challenging in upstream mountainous basins due to limited observations and associated uncertainties. Data assimilation is a valuable tool to improve model state estimates to improve hydrological forecast. While assimilation of snow has been increasingly implemented in research, particularly using sequential data assimilation methods, less attention has been dedicated to compare different types of assimilation techniques. This study aims at comparing the well-known Ensemble Kalman Filter (EnKF) sequential data assimilation and the less known Moving Horizon Estimation (MHE) variational assimilation. Initial conditions of the hydrological model states are improved by assimilating in-situ streamflow and remotely sensed snow covered area data into the Hydrologiska Byråns Vattenbalansavdelning (HBV) conceptual rainfall-runoff model. The study is applied to improve runoff predictions over two mountainous basins: one in the uppermost catchment of the Karasu river, in Eastern Turkey, and the second in an alpine basin located in the headwaters of the Genil River in the northern flank of the Sierra Nevada Mountain in Southern Spain. The forecast models are tested in a hindcasting condition, i.e. closed loop environment, which simulates real-time forecasting systems. The results are analyzed using forecast verification metrics. Results showed that streamflow and snow state predictions using MHE outperform the commonly used EnKF based on the continuous ranked probability score (CRPS).

Integrating visual and acoustic observations to build an “intelligent” killer whale movement forecast system

Shipping traffic continues to grow in the Salish Sea with considerable marine industrial developments planned for the region. With this comes an increase in the risk of cetaceans being disturbed, harassed, and potentially colliding with commercially operating vessels. One key conservation concern is these waters coincide with designated Critical Habitat for the endangered population of Southern Resident killer whales (SRKW). Monitoring technologies such as hydrophones provide the promise of real-time animal detection and localisation. To use these continuous streams of real-time whale location data, we are developing algorithms that automate acoustic detection and classification of SRKW calls using deep learning models trained on extensive new and updated annotated datasets from the region. We are developing open-source models for pod-level classification, while also differentiating SRKW from other ecotypes and species. We are developing a forecasting system to merge these whale detections with opportunistic visual observations. This is based on sequential data assimilation methods using advanced animal movement models to estimate SRKW pod locations with probabilistic predictions of whale directional movement. These methods will be transformative for shipping by providing forecasting with a lead time of a few hours allowing vessels to adjust their speed or pathway to minimize whale-vessel interactions.

Pressure reconstruction of a planar turbulent flow field within a multiply-connected domain with arbitrary boundary shapes

Over the past two decades, it has been demonstrated that the instantaneous spatial pressure distribution in a turbulent flow field can be reconstructed from the pressure gradient field non-intrusively measured by Particle Image Velocimetry (PIV). Representative pressure reconstruction methods include the omnidirectional integration (Liu and Katz, 2006; Liu et al., 2016; Liu and Moreto, 2020), the Poisson equation approach (Violato et al., 2011; De Kat and Van Oudheusden, 2012), the least-square method (Jeon et al., 2015), and most recently, the adjoint-based sequential data assimilation method, which also essentially utilizes the Poisson equation to reconstruct the pressure(He et al., 2020). Most of these previous pressure reconstruction examples, however, were applied to simply-connected domains (Gluzman et al., 2017) only. None of these previous studies have discussed how to apply the pressure reconstruction procedures to a multiply-connected domain (Gluzman et al., 2017). To fill in this gap, this paper presents a detailed report for the first time documenting the implementation procedures and validation results for pressure reconstruction of a planar turbulent flow field within a multiply-connected domain that has arbitrary inner and outer boundary shapes. The pressure reconstruction algorithm used in the current study is the rotating parallelray omni-directional integration algorithm, which, as demonstrated in reference (Liu and Moreto, 2020) based on simply-connected flow domains, offers high-level of accuracy in the reconstructed pressure. While preserving the nature and advantage of the parallel ray omni-directional pressure reconstruction at places with flow data, the new implementation of the algorithm is capable of processing an arbitrary number of inner void areas with arbitrary boundary shapes. Validation of the multiply-connected domain pressure reconstruction code is conducted using the DNS (Direct Numerical Simulation) isotropic turbulence field available at the Johns Hopkins Turbulence Databases, with 1000 statistically independent pressure gradient field realizations embedded with random noise used to gauge the code performance. For further validation, the code is also applied for pressure reconstruction from the DNS pressure gradient in the ambient flow field of a shock-induced non-spherical bubble collapse in water (Johnsen and Colonius, 2009). The successful implementation of the parallel ray pressure reconstruction method to multiply-connected domains paves the way for a variety of important applications including, for example, experimental characterization of pressure field changes during the process of cavitation bubble inception, growth and collapse, non-intrusive unsteady aerodynamic force assessment for an arbitrary body shape immersed in flows, and multi-phase flow investigations, etc. In particular, as an immediate follow-up effort, the parallel ray pressure code will be used for the instantaneous pressure distribution reconstruction of the turbulent flow surrounding cavitation inception bubbles occurring on top of a cavity trailing corner based on high-speed PIV measurements.

Analyzing the effects of observation function selection in ensemble Kalman filtering for epidemic models

The Ensemble Kalman Filter (EnKF) is a popular sequential data assimilation method that has been increasingly used for parameter estimation and forecast prediction in epidemiological studies. The observation function plays a critical role in the EnKF framework, connecting the unknown system variables with the observed data. Key differences in observed data and modeling assumptions have led to the use of different observation functions in the epidemic modeling literature. In this work, we present a novel computational analysis demonstrating the effects of observation function selection when using the EnKF for state and parameter estimation in this setting. In examining the use of four epidemiologically-inspired observation functions of different forms in connection with the classic Susceptible–Infectious–Recovered (SIR) model, we show how incorrect observation modeling assumptions (i.e., fitting incidence data with a prevalence model, or neglecting under-reporting) can lead to inaccurate filtering estimates and forecast predictions. Results demonstrate the importance of choosing an observation function that well interprets the available data on the corresponding EnKF estimates in several filtering scenarios, including state estimation with known parameters, and combined state and parameter estimation with both constant and time-varying parameters. Numerical experiments further illustrate how modifying the observation noise covariance matrix in the filter can help to account for uncertainty in the observation function in certain cases.

Technical Note: Sequential ensemble data assimilation in convergent and divergent systems

Abstract. Data assimilation methods are used throughout the geosciences to combine information from uncertain models and uncertain measurement data. However, the characteristics of geophysical systems differ and may be distinguished between divergent and convergent systems. In divergent systems initially nearby states will drift apart, while they will coalesce in convergent systems. This difference has implications for the application of sequential ensemble data assimilation methods. This study explores these implications on two exemplary systems, i.e., the divergent Lorenz 96 model and the convergent description of soil water movement by the Richards equation. The results show that sequential ensemble data assimilation methods require a sufficient divergent component. This makes the transfer of the methods from divergent to convergent systems challenging. We demonstrate, through a set of case studies, that it is imperative to represent model errors adequately and incorporate parameter uncertainties in ensemble data assimilation in convergent systems.

Data assimilation empowered neural network parametrizations for subgrid processes in geophysical flows

Modeling the effect of subgrid-scale processes is one of the main obstacles in the accurate prediction of multiscale systems. An investigation considers how machine learning methods can be applied to model subgrid-scale processes and integrated within sequential data assimilation methods. It is found that the use of machine-learning-based closure modeling in conjunction with data assimilation improves the prediction of multiscale systems and can be considered a promising approach to numerical weather prediction tasks in the age of data.

Sequential Data Assimilation of the Stochastic SEIR Epidemic Model for Regional COVID-19 Dynamics

Newly emerging pandemics like COVID-19 call for predictive models to implement precisely tuned responses to limit their deep impact on society. Standard epidemic models provide a theoretically well-founded dynamical description of disease incidence. For COVID-19 with infectiousness peaking before and at symptom onset, the SEIR model explains the hidden build-up of exposed individuals which creates challenges for containment strategies. However, spatial heterogeneity raises questions about the adequacy of modeling epidemic outbreaks on the level of a whole country. Here, we show that by applying sequential data assimilation to the stochastic SEIR epidemic model, we can capture the dynamic behavior of outbreaks on a regional level. Regional modeling, with relatively low numbers of infected and demographic noise, accounts for both spatial heterogeneity and stochasticity. Based on adapted models, short-term predictions can be achieved. Thus, with the help of these sequential data assimilation methods, more realistic epidemic models are within reach.

Thermophysical modelling and parameter estimation of small Solar system bodies via data assimilation

ABSTRACT Deriving thermophysical properties such as thermal inertia from thermal infrared observations provides useful insights into the structure of the surface material on planetary bodies. The estimation of these properties is usually done by fitting temperature variations calculated by thermophysical models to infrared observations. For multiple free model parameters, traditional methods such as least-squares fitting or Markov chain Monte Carlo methods become computationally too expensive. Consequently, the simultaneous estimation of several thermophysical parameters, together with their corresponding uncertainties and correlations, is often not computationally feasible and the analysis is usually reduced to fitting one or two parameters. Data assimilation (DA) methods have been shown to be robust while sufficiently accurate and computationally affordable even for a large number of parameters. This paper will introduce a standard sequential DA method, the ensemble square root filter, for thermophysical modelling of asteroid surfaces. This method is used to re-analyse infrared observations of the MARA instrument, which measured the diurnal temperature variation of a single boulder on the surface of near-Earth asteroid (162173) Ryugu. The thermal inertia is estimated to be 295 ± 18 $\mathrm{J\, m^{-2}\, K^{-1}\, s^{-1/2}}$, while all five free parameters of the initial analysis are varied and estimated simultaneously. Based on this thermal inertia estimate the thermal conductivity of the boulder is estimated to be between 0.07 and 0.12,$\mathrm{W\, m^{-1}\, K^{-1}}$ and the porosity to be between 0.30 and 0.52. For the first time in thermophysical parameter derivation, correlations and uncertainties of all free model parameters are incorporated in the estimation procedure that is more than 5000 times more efficient than a comparable parameter sweep.

Instantaneous pressure determination from unsteady velocity fields using adjoint-based sequential data assimilation

A sequential data assimilation (DA) method is developed for pressure determination of turbulent velocity fields measured by particle image velocimetry (PIV), based on the unsteady adjoint formulation. A forcing term F, which is optimized using the adjoint system, is added to the primary Navier–Stokes (N–S) equations to drive the assimilated flow toward the observations at each time step. Compared with the conventional unsteady adjoint method, which requires the forward integration of the primary system and the backward integration of the adjoint system, the present approach integrates the primary-adjoint system all the way forward, discarding the requirement of data storage at every time step, being less computationally resource-consuming, and saving space. The pressure determination method of integration from eight paths [J. O. Dabiri et al., “An algorithm to estimate unsteady and quasi-steady pressure fields from velocity field measurements,” J. Exp. Biol. 217, 331 (2014)] is also evaluated for comparison. Using synthetic PIV data of a turbulent jet as the observational data, the present DA method is able to determine the instantaneous pressure field precisely using the three-dimensional velocity fields, regardless of the observational noise. For the two-dimensional three-component (3C) or two-component (2C) velocity fields, which are not sufficient for pressure determination by the integration method due to the lack of off-plane derivatives, the present DA method is able to reproduce pressure fields whose statistics agree reasonably well with those of the referential results. The 3C and 2C velocity fields yield quite similar results, indicating the possibility of pressure determination from only planar-PIV measurements in turbulent flows. The tomography PIV measurements are also used as observational data, and a clear pressure pattern is obtained with the present DA method.

An efficient automatic history matching method through the probabilistic collocation based particle filter for shale gas reservoir

Shale gas can be economically developed with the advancement of horizontal well drilling and hydraulic fracturing. The complex multiscale flow mechanisms associated with the shale gas production simulation model can lead to severe model non-linearity, and the model parameters may not have the Gaussian distribution. Particle filter is a sequential data assimilation method derived from Bayesian theorem and can be an effective automatic history matching method for shale gas reservoir without the limitation on the model linearity and Gaussian assumption. However, particle filter may require a large amount of model evaluations to obtain the stable and accurate results. To improve the computational efficiency of the particle filter, we propose the probabilistic collocation based particle filter method (PCPF). An illustrative case study on shale gas production is conducted to validate the efficiency and accuracy of the proposed PCPF method. The results show that the computational efficiency can be greatly improved and it makes the full particle filter analysis computationally affordable. With more measurement data available, the parameter estimations and the corresponding shale gas production predictions can be more accurate and the uncertainties associated with the parameter estimations and shale gas production predictions are significantly reduced during the history matching process.

Approximate filtering of conditional intensity process for Poisson count data: Application to urban crime

The primary focus is a sequential data assimilation method for count data modelled by an inhomogeneous Poisson process. In particular, a quadratic approximation technique similar to the extended Kalman filter is applied to develop a sub-optimal, discrete-time, filtering algorithm, called the extended Poisson–Kalman filter (ExPKF), where only the mean and covariance are sequentially updated using count data via the Poisson likelihood function. The performance of ExPKF is investigated in several synthetic experiments where the true solution is known. In numerical examples, ExPKF provides a good estimate of the “true” posterior mean, which can be well-approximated by the particle filter (PF) algorithm in the very large sample size limit. In addition, the experiments demonstrate that the ExPKF algorithm can be conveniently used to track parameter changes; on the other hand, a non-filtering framework such as a maximum likelihood estimation (MLE) would require a statistical test for change points or implement time-varying parameters. Finally, to demonstrate the model on real-world data, the ExPKF is used to approximate the uncertainty of urban crime intensity and parameters for self-exciting crime models. The Chicago Police Department’s CLEAR (Citizen Law Enforcement Analysis and Reporting) system data is used as a case study for both univariate and multivariate Hawkes models. An improved goodness of fit measured by the Kolmogorov–Smirnov (KS) statistics is achieved by the filtered intensity. The potential of using filtered intensity to improve police patrolling prioritisation is also tested. By comparing with the prioritisation based on MLE-derived intensity and historical frequency, the result suggests an insignificant difference between them. While the filter is developed and tested in the context of urban crime, it has the potential to make a contribution to data assimilation in other application areas.

Estimation of frictional properties and slip evolution on a long-term slow slip event fault with the ensemble Kalman filter: numerical experiments

SummaryA variety of slow slip events at subduction zones have been observed. They can be stress meters for monitoring the stress state of megathrust faults during their earthquake cycles. In this study, we focus on long-term slow slip events (LSSEs) recurring at downdip portions of megathrust faults among such slow earthquakes. Data analyses and simulation studies of LSSEs have so far been executed independently. In atmosphere and ocean sciences, data assimilations that optimally combine data analyses and simulation studies have been developed. We develop a method for estimating frictional properties and monitoring slip evolution on an LSSE fault, with a sequential data assimilation method, the ensemble Kalman filter (EnKF). We executed numerical twin experiments for the Bungo Channel LSSE fault in southwest Japan to validate the method. First, based on a rate- and state-dependent friction law, we set a rate-weakening circular LSSE patch on the rate-strengthening flat plate interface, whose critical nucleation size is larger than that of the patch, and reproduced the observed Bungo Channel LSSEs with recurrence times of approximately 7 yr and slip durations of 1 yr. Then, we synthesized the observed data of surface displacement rates at uniformly distributed stations with noises from the simulated slip model. Using our EnKF method, we successfully estimated the frictional parameters and the slip rate evolution after a few cycles. Secondly, we considered the effect of the megathrust fault existing in the updip portion of the LSSE fault, as revealed by kinematic inversion studies of Global Navigation Satellite System (GNSS) data and added this locked region with a slip deficit rate in the model. We estimated the slip rate on the locked region only kinematically, but the quasi-dynamic equation of motion in each LSSE fault cell includes the stress term arising from the locked region. Based on this model, we synthesized the observed surface displacement rate data for the actual distribution of GNSS stations and executed EnKF estimations including the slip rate on the locked region. The slip rate on the locked region could be quickly retrieved. Even for the actual distribution of GNSS stations, we could successfully estimate frictional parameters and slip evolution on the LSSE fault. Thus, our twin numerical experiments showed the validity of our EnKF method, although we need further studies for actual GNSS data analyses.

Evaluation of Estimation Accuracy of Sequential Data Assimilation Methods for Phase-field Models

Evaluation of Estimation Accuracy of Sequential Data Assimilation Methods for Phase-field Models

A detectability criterion and data assimilation for nonlinear differential equations**Submitted to the editors.

In this paper we propose a new sequential data assimilation method for nonlinear ordinary differential equations with compact state space. The method is designed so that the Lyapunov exponents of the corresponding estimation error dynamics are negative, i.e. the estimation error decays exponentially fast. The latter is shown to be the case for generic regular flow maps if and only if the observation matrix H satisfies detectability conditions. In particular this implies that the rank of H must be at least as great as the number of nonnegative Lyapunov exponents of the underlying attractor. Numerical experiments illustrate the exponential convergence of the method and the sharpness of the theory for the case of Lorenz ’96 and Burgers equations with incomplete and noisy observations.

Efficient Formulation and Implementation of Data Assimilation Methods

This Special Issue presents efficient formulations and implementations of sequential and variational data assimilation methods. The methods address three important issues in the context of operational data assimilation: efficient implementation of localization methods, sampling methods for approaching posterior ensembles under non-linear model errors, and adjoint-free formulations of four dimensional variational methods.