Online Data Assimilation Research Articles

Model predictive control of stormwater basins coupled with real-time data assimilation enhances flood and pollution control under uncertainty

Smart stormwater systems equipped with real-time controls are transforming urban drainage management by enhancing the flood control and water treatment potential of previously static infrastructure. Real-time control of detention basins, for instance, has been shown to improve contaminant removal by increasing hydraulic retention times while also reducing downstream flood risk. However, to date, few studies have explored optimal real-time control strategies for achieving both water quality and flood control targets. This study advances a new model predictive control (MPC) algorithm for stormwater detention ponds that determines the outlet valve control schedule needed to maximize pollutant removal and minimize flooding using forecasts of the incoming pollutograph and hydrograph. Comparing MPC against three rule-based control strategies, MPC is found to be more effective at balancing between multiple competing control objectives such as preventing overflows, reducing peak discharges, and improving water quality. Moreover, when paired with an online data assimilation scheme based on Extended Kalman Filtering (EKF), MPC is found to be robust to uncertainty in both pollutograph forecasts and water quality measurements. By providing an integrated control strategy that optimizes both water quality and quantity goals while remaining robust to uncertainty in hydrologic and pollutant dynamics, this study paves the way for real-world smart stormwater systems that will achieve improved flood and nonpoint source pollution management.

Simulation of COVID-19 Spread Scenarios in the Republic of Kazakhstan Based on Regularization of the Agent-Based Model

We propose an algorithm for modeling scenarios for newly diagnosed cases of COVID-19 in the Republic of Kazakhstan. The algorithm is based on treating incomplete epidemiological data and solving the inverse problem of reconstructing the parameters of the agent-based model (ABM) using the set of available epidemiological data. The main tool for constructing the ABM is the Covasim open library. In the event of a drastic change in the situation (appearance of a new strain, removal or introduction of restrictive measures, etc.), the model parameters are updated taking into account additional information for the previous month (online data assimilation). The inverse problem is solved by stochastic global optimization (of tree-structured Parzen estimators). As an example, we give two scenarios of COVID-19 propagation calculated on December 12, 2021 for the period up to January 20, 2022. The scenario that took into account the New Year holidays (published on December 12, 2021 on http://covid19-modeling.ru ) almost coincided with what happened in reality (the error was 0.2%).

An Efficient Estimation of Time‐Varying Parameters of Dynamic Models by Combining Offline Batch Optimization and Online Data Assimilation

AbstractIt is crucially important to estimate unknown parameters in process‐based models by integrating observation and numerical simulation. For many applications in earth system sciences, a parameter estimation method which allows parameters to temporally change is required. In the present paper, an efficient and practical method to estimate time‐varying parameters of relatively low dimensional models is presented. In the newly proposed method, called Hybrid Offline Online Parameter Estimation with Particle Filtering (HOOPE‐PF), an inflation method to maintain the spread of ensemble members in a Sampling‐Importance‐Resampling Particle Filter (SIRPF) is improved using a non‐parametric posterior probabilistic distribution of time‐invariant parameters obtained by comparing simulated and observed climatology. HOOPE‐PF outperforms the original SIRPF in synthetic experiments with toy models and a real‐data experiment with a conceptual hydrological model when an ensemble size is small. The advantage of HOOPE‐PF is that its performance is not greatly affected by the size of perturbation to be added to ensemble members to maintain their spread while it is important to get the optimal performance in the original particle filter. Since HOOPE‐PF is the extension of the existing particle filter which has been extensively applied to many models in earth system sciences such as land, ecosystem, hydrology, and paleoclimate reconstruction, HOOPE‐PF can be applied to improve the simulation of these process‐based models by considering time‐varying model parameters.

An aerosol vertical data assimilation system (NAQPMS-PDAF v1.0): development and application

Abstract. Aerosol vertical stratification is important for global climate and planetary boundary layer (PBL) stability, and no single method can obtain spatiotemporally continuous vertical profiles. This paper develops an online data assimilation (DA) framework for the Eulerian atmospheric chemistry-transport model (CTM) Nested Air Quality Prediction Model System (NAQPMS) with the Parallel Data Assimilation Framework (PDAF) as the NAQPMS-PDAF for the first time. Online coupling occurs based on a memory-based way with two-level parallelization, and the arrangement of state vectors during the filter is specifically designed. Scaling tests demonstrate that the NAQPMS-PDAF can make efficient use of parallel computational resources for up to 25 000 processors with a weak scaling efficiency of up to 0.7. The 1-month long aerosol extinction coefficient profiles measured by the ground-based lidar and the concurrent hourly surface PM2.5 are solely and simultaneously assimilated to investigate the performance and application of the DA system. The hourly analysis and subsequent 1 h simulation are validated through lidar and surface PM2.5 measurements assimilated and not assimilated. The results show that lidar DA can significantly improve the underestimation of aerosol loading, especially at a height of approximately 400 m in the free-running (FR) experiment, with the mean bias (BIAS) changing from −0.20 (−0.14) km−1 to −0.02 (−0.01) km−1 and correlation coefficients increasing from 0.33 (0.28) to 0.91 (0.53) averaged over sites with measurements assimilated (not assimilated). Compared with the FR experiment, simultaneously assimilating PM2.5 and lidar can have a more consistent pattern of aerosol vertical profiles with a combination of surface PM2.5 and lidar, independent extinction coefficients from the Cloud-Aerosol Lidar with Orthogonal Polarization (CALIOP), and aerosol optical depth (AOD) from the Aerosol Robotic Network (AERONET). Lidar DA has a larger temporal impact than that in PM2.5 DA but has deficiencies in subsequent quantification on the surface PM2.5. The proposed NAQPMS-PDAF has great potential for further research on the impact of aerosol vertical distribution.

Revisiting Online and Offline Data Assimilation Comparison for Paleoclimate Reconstruction: An Idealized OSSE Study

AbstractData assimilation (DA) has been applied to estimate the time‐mean state, such as annual mean surface temperature for paleoclimate reconstruction. There are two types of DA for this purpose: online‐DA and offline‐DA. The online‐DA estimates both time‐mean states (analyses) and initial conditions for subsequent DA cycles, while the offline‐DA only estimates the time‐mean analyses. If there is sufficiently long predictability in the system of interest compared to the temporal resolution of the observations, online‐DA is expected to outperform offline‐DA by utilizing information in the initial conditions. However, previous studies failed to show the superiority of online‐DA when time‐averaged observations are assimilated, and the reason has not been investigated thoroughly. This study compares online‐DA and offline‐DA and investigates the relation to the predictability using an intermediate complexity general circulation model with perfect‐model observing system simulation experiments. The result shows that the online‐DA outperforms offline‐DA when the length of predictability is longer than the averaging time of the observations. We also found that the longer the predictability, the more skillful the online‐DA. Here, the ocean plays a crucial role in extending predictability, which helps online‐DA to outperform offline‐DA. Interestingly, the observations of near‐surface air temperature over land are highly valuable to update the ocean variables in the analysis steps, suggesting the importance of using cross‐domain covariance information between the atmosphere and the ocean when online‐DA is applied to reconstruct paleoclimate.

Coupled Atmosphere–Ocean Reconstruction of the Last Millennium Using Online Data Assimilation

AbstractWe use online data assimilation to combine information from a linear inverse model of coupled atmosphere‐ocean dynamics with proxy records to create a new annual‐resolution reconstruction of atmosphere and ocean fields over the last millennium. Instrumental validation of reconstructed sea‐surface temperature and 0–700 m ocean heat content shows broad regions of positive spatial correlations, and high correlations (∼0.6–0.9) for global averages and indices of large‐scale modes of atmospheric variability. Compared to previous reconstructions, the online reconstructions show global and hemispheric averages with little‐to‐no millennial‐scale trend and global‐mean temperatures ∼0.25–0.5 K cooler during early periods (1000–1400 C.E.). The spatial anomaly differences of average temperature between an early (1000–1250 C.E.) and later (1400–1700 C.E.) period show warm anomalies over high‐latitude Europe and cool tropical conditions in partial agreement with previous assessments. The addition of online data assimilation, which provides dynamical memory to climate proxy information, is shown to be crucial for adequately characterizing decadal‐to‐centennial‐scale variability of 0–700 m ocean heat content. Furthermore, the climate forecasts provide model‐based physical constraints for atmosphere–ocean interaction, which become increasingly important during early periods when less proxy information is available for assimilation.

Online data assimilation of a hybrid flow stress model by particle filtering

Models for the evolution of hidden microstructural states are needed for fast prediction and closed-loop control of workpiece properties. Machine learning allows to obtain models by learning from experimental data, avoiding the limitations of explicitly defined physics-based models. However, the identification of the parameters of deep network structures, reliable extrapolation and fast online assimilation to new measurements are open problems. At the example of titanium forging, a new approach is investigated that combines a hybrid physics-informed microstructure and flow stress model that draws upon a long short-term memory network with a particle filter for online data assimilation to new measurements.

A Kalman Filter-Based Approach for Online Source-Term Estimation in Accidental Radioactive Dispersion Events

In the present work, a online data assimilation approach, based on the Kalman filter algorithm, is proposed for the source term reconstruction in accidental events with dispersion of radioactive agents in air. For this purpose a Gaussian plume model of dispersion in air is embedded in the Kalman filter algorithm to estimate unknown scenario parameters, such as the coordinates and the intensity of the source, on the basis of measurements collected by a mobile sensor. The approach was tested against pseudo-experimental data produced with both the Gaussian plume model and the Lagrangian puff model SCIPUFF. The results show the good capabilities of the proposed approach in retrieving the values of the unknown parameters when (i) one or more release parameters are poorly known and (ii) a sufficient number of experimental measurements describing the evolution of the dispersion process can be collected in a short time by means of mobile sensors. Thanks to its flexibility and computational efficiency, and due to the exploitation of the Kalman filter potentialities through the use of a simplified model of dispersion in air, the proposed approach can constitute a useful tool for the management of emergency scenarios.

Data Assimilation and Online Optimization With Performance Guarantees

This article considers a class of real-time stochastic optimization problems dependent on an unknown probability distribution. In the considered scenario, data are streaming frequently while trying to reach a decision. Thus, we aim to devise a procedure that incorporates samples (data) of the distribution sequentially and adjusts decisions accordingly. We approach this problem in a distributionally robust optimization framework and propose a novel Online Data Assimilation Algorithm (OnDA Algorithm) for this purpose. This algorithm guarantees out-of-sample performance of decisions with high probability, and gradually improves the quality of the decisions by incorporating the streaming data. We show that the OnDA Algorithm converges under a sufficiently slow data streaming rate, and provide a criteria for its termination after certain number of data have been collected. Simulations illustrate the results.

Particle filter-based data assimilation technique for the evaluation of transport of pollutants in small rivers

The discharge of pollutants into water resources has caused many undesirable effects on human health and the environment. In this context, meta-heuristic methods, such as the Simulated Annealing algorithm, allow for only offline estimation of the concentration and dispersion coefficient of pollutant. However, online data assimilation is a very important for decision support in real time. In this regard, the present paper focuses on the use of the so-called Particle Filter as an online observer. Our studies were performed through the algorithms Sampling Importance Resampling (SIR) and Liu and West (LW), using tracer experimental observations from a small river in Brazil. Such approach allowed for tracking satisfactorily the pollutant concentration, longitudinal dispersion coefficient, flow velocity, pollutant load released and distance from the pollution source. The results can be enhanced ever more through the LW filter, which reduces the impoverishment of the samples as well as the computational time due to the use of a smoothing kernel.

Hybrid Gain Data Assimilation Using Variational Corrections in the Subspace Orthogonal to the Ensemble

AbstractThe viability of a parameterless hybrid data assimilation algorithm is investigated. As an alternative to the traditional hybrid covariance scheme, hybrid gain data assimilation (HGDA) was proposed to blend the gain matrix derived from the variational method and the ensemble-based Kalman filter (EnKF). A previously proposed HGDA algorithm uses a two-step process applying the EnKF with a variational update. The algorithm is modified here to limit the variational correction to the subspace orthogonal to the ensemble perturbation subspace without the use of a hybrid weighting parameter, as the optimization of such a parameter is nontrivial. The modified HGDA algorithm is investigated with a quasigeostrophic (QG) model. Results indicate that when the climatological background error covariance matrix B and the observation error covariance R are well estimated, state estimates from the parameterless HGDA are more accurate than the parameter-dependent HGDA. The parameterless HGDA not only has potential advantages over the standard HGDA as an online data assimilation algorithm but can also serve as a valuable diagnostic tool for tuning the B and R matrices. It is also found that in this QG model, the empirically best static B matrix for the stand-alone 3DVAR has high variance at larger spatial scales, which degrades the accuracy of the HGDA systems and may not be the best choice for hybrid methods in general. A comparison of defining the orthogonal subspace globally or locally demonstrates that global orthogonality is more advantageous for stabilizing the hybrid system and maintains large-scale balances.

Spatial-temporal variations in surface ozone over Ushuaia and the Antarctic region: observations from in situ measurements, satellite data, and global models.

The Antarctic continent is known to be an unpopulated region due to its extreme weather and climate conditions. However, the air quality over this continent can be affected by long-lived anthropogenic pollutants from the mainland. The Argentinian region of Ushuaia is often the main source area of accumulated hazardous gases over the Antarctic Peninsula. The main objective of this study is to report the first in situ observations yet known of surface ozone (O3) over Ushuaia, the Drake Passage, and Coastal Antarctic Peninsula (CAP) on board the RV Australis during the Malaysian Antarctic Scientific Expedition Cruise 2016 (MASEC'16). Hourly O3 data was measured continuously for 23days using an EcoTech O3 analyzer. To understand more about the distribution of surface O3 over the Antarctic, we present the spatial and temporal of surface O3 of long-term data (2009-2015) obtained online from the World Meteorology Organization of World Data Centre for greenhouse gases (WMO WDCGG). Furthermore, surface O3 satellite data from the free online NOAA-Atmospheric Infrared Sounder (AIRS) database and online data assimilation from the European Centre for Medium-Range Weather Forecasts (ECMWF)-Monitoring Atmospheric Composition and Climate (MACC) were used. The data from both online products are compared to document the data sets and to give an indication of its quality towards in situ data. Finally, we used past carbon monoxide (CO) data as a proxy of surface O3 formation over Ushuaia and the Antarctic region. Our key findings were that the surface O3 mixing ratio during MASEC'16 increased from a minimum of 5ppb to ~10-13ppb approaching the Drake Passage and the Coastal Antarctic Peninsula (CAP) region. The anthropogenic and biogenic O3 precursors from Ushuaia and the marine region influenced the mixing ratio of surface O3 over the Drake Passage and CAP region. The past data from WDCGG showed that the annual O3 cycle has a maximum during the winter of 30 to 35ppb between June and August and a minimum during the summer (January to February) of 10 to 20ppb. The surface O3 mixing ratio during the summer was controlled by photochemical processes in the presence of sunlight, leading to the depletion process. During the winter, the photochemical production of surface O3 was more dominant. The NOAA-AIRS and ECMWF-MACC analysis agreed well with the MASEC'16 data but twice were higher during the expedition period. Finally, the CO past data showed the surface O3 mixing ratio was influenced by the CO mixing ratio over both the Ushuaia and Antarctic regions. Peak surface O3 and CO hourly mixing ratios reached up to ~38ppb (O3) and ~500ppb (CO) over Ushuaia. High CO over Ushuaia led to the depletion process of surface O3 over the region. Monthly CO mixing ratio over Antarctic (South Pole) were low, leading to the production of surface O3 over the Antarctic region.

Assimilation of pseudo-tree-ring-width observations into an atmospheric general circulation model

Abstract. Paleoclimate data assimilation (DA) is a promising technique to systematically combine the information from climate model simulations and proxy records. Here, we investigate the assimilation of tree-ring-width (TRW) chronologies into an atmospheric global climate model using ensemble Kalman filter (EnKF) techniques and a process-based tree-growth forward model as an observation operator. Our results, within a perfect-model experiment setting, indicate that the "online DA" approach did not outperform the "off-line" one, despite its considerable additional implementation complexity. On the other hand, it was observed that the nonlinear response of tree growth to surface temperature and soil moisture does deteriorate the operation of the time-averaged EnKF methodology. Moreover, for the first time we show that this skill loss appears significantly sensitive to the structure of the growth rate function, used to represent the principle of limiting factors (PLF) within the forward model. In general, our experiments showed that the error reduction achieved by assimilating pseudo-TRW chronologies is modulated by the magnitude of the yearly internal variability in the model. This result might help the dendrochronology community to optimize their sampling efforts.

Sparse-grid, reduced-basis Bayesian inversion: Nonaffine-parametric nonlinear equations

We extend the reduced basis (RB) accelerated Bayesian inversion methods for affine-parametric, linear operator equations which are considered in 16,17 to non-affine, nonlinear parametric operator equations. We generalize the analysis of sparsity of parametric forward solution maps in 20 and of Bayesian inversion in 48,49 to the fully discrete setting, including Petrov-Galerkin high-fidelity (HiFi) discretization of the forward maps. We develop adaptive, stochastic collocation based reduction methods for the efficient computation of reduced bases on the parametric solution manifold. The nonaffinity and nonlinearity with respect to (w.r.t.) the distributed, uncertain parameters and the unknown solution is collocated; specifically, by the so-called Empirical Interpolation Method (EIM). For the corresponding Bayesian inversion problems, computational efficiency is enhanced in two ways: first, expectations w.r.t. the posterior are computed by adaptive quadratures with dimension-independent convergence rates proposed in 49; the present work generalizes 49 to account for the impact of the PG discretization in the forward maps on the convergence rates of the Quantities of Interest (QoI for short). Second, we propose to perform the Bayesian estimation only w.r.t. a parsimonious, RB approximation of the posterior density. Based on the approximation results in 49, the infinite-dimensional parametric, deterministic forward map and operator admit N-term RB and EIM approximations which converge at rates which depend only on the sparsity of the parametric forward map. In several numerical experiments, the proposed algorithms exhibit dimension-independent convergence rates which equal, at least, the currently known rate estimates for N-term approximation. We propose to accelerate Bayesian estimation by first offline construction of reduced basis surrogates of the Bayesian posterior density. The parsimonious surrogates can then be employed for online data assimilation and for Bayesian estimation. They also open a perspective for optimal experimental design.

Predicting the Cloud Patterns for the Boreal Summer Intraseasonal Oscillation Through a Low-Order Stochastic Model

AbstractWe assess the predictability limits of the large-scale cloud patterns in the boreal summer intraseasonal variability (BSISO), which are measured by the infrared brightness temperature, a proxy for convective activity. A recent developed nonlinear data analysis technique, nonlinear Laplacian spectrum analysis (NLSA), is applied to the brightness temperature data, defining two spatial modes with high intermittency associated with the BSISO time series. Then a recent developed data-driven physics-constrained low-ordermodeling strategy is applied to these time series. The result is a four dimensional system with two observed BSISO variables and two hidden variables involving correlated multiplicative noise through the nonlinear energyconserving interaction. With the optimal parameters calibrated by information theory, the non-Gaussian fat tailed probability distribution functions (PDFs), the autocorrelations and the power spectrum of the model signals almost perfectly match those of the observed data. An ensemble prediction scheme incorporating an effective on-line data assimilation algorithm for determining the initial ensemble of the hidden variables shows the useful prediction skill in the non-El Niño years is at least 30 days and even reaches 55 days in those years with regular oscillations and the skillful prediction lasts for 18 days in the strong El Niño year (year 1998). Furthermore, the ensemble spread succeeds in indicating the forecast uncertainty. Although the reduced linear model with time-periodic stable-unstable damping is able to capture the non-Gaussian fat tailed PDFs, it is less skillful in forecasting the BSISO in the years with irregular oscillations. The failure of the ensemble spread to include the truth also indicates failure in quantification of the uncertainty. In addition, without the energy-conserving nonlinear interactions, the linear model is sensitive with parameter variations. mcwfnally, the twin experiment with nonlinear stochastic model has comparable skill as the observed data, suggesting the nonlinear stochastic model has significant skill for determining the predictability limits of the large-scale cloud patterns of the BSISO.

Application of dynamic data driven application system in environmental science

The paradigm of dynamic data driven application system (DDDAS) has been proposed as a framework to analyze and predict the character and behavior of complex systems that influence computational models significantly. Its accuracy and efficiency lies in its ability to integrate observations on different temporal and spatial scales from real-time sensors, and in its measurement steering and controlling capabilities. Many problems in environmental sciences are nonlinear and complex, impossible to solve by using input/output sequence flows without feedback control. Nonlinear system efficiency depends on measurement control and steering, on-line data assimilation, and model selection with dynamic optimization. Compared with traditional methods, DDDAS possesses the capacity to overcome these limitations. This paper discusses DDDAS and classifies typical cases of its application in environmental sciences into three levels of paradigm. Short reviews of multi-model simulation and data assimilation are provided for practical use. Recent developments and future perspectives are reviewed. Future work may address determining automatically where, when, and how to acquire real-time data, and its integration with GIS, to improve efficiency and accuracy. User-generated content will find wide application in the future. Considering the differences between DDDAS and other data-driven methods in solving the same nonlinear complex system problems, a combination of nonlinear science and chaos theory is advocated.

A Simple Technique to Infer the Missing Gravity Wave Drag in the Middle Atmosphere Using a General Circulation Model: Potential Vorticity Budget

AbstractA simple technique to infer the missing momentum forcing in a general circulation model is developed and evaluated. The response of the large-scale dynamic equations to an external momentum forcing presents a nonlocal response in the zonal and meridional wind. On the other hand, the response to the external momentum forcing in the potential vorticity (PV) is a local growing geostrophic mode, so that there is a direct relationship between the external momentum forcing and the response in PV. In this work, this fact is exploited to diagnose the missing momentum forcing in the extratropics using a general circulation model. The capability of the simple technique to estimate a concentrated gravity wave forcing is evaluated. A dynamical model is evolved with prescribed sources and sinks of PV and then the technique is used to estimate these known momentum sources and sinks. PV is found to give a much better diagnostic of gravity wave drag compared to the more traditional zonal wind differences. The technique is also used in a realistic environment, in which the sources and sinks of PV in Met Office analyses are determined. The estimation of this missing forcing with this simple technique is compared with the estimation given by a more complex data assimilation technique developed by Pulido and Thuburn and, in general, a good agreement is found. The simple gravity wave drag estimation technique can be used in an online data assimilation cycle, using the increments of the analysis, and also offline, using a general circulation model and observations.

An unconventional adaptation of a classical Gaussian plume dispersion scheme for the fast assessment of external irradiation from a radioactive cloud

This article focuses on derivation of an effective algorithm for the fast estimation of cloudshine doses/dose rates induced by a large mixture of radionuclides discharged into the atmosphere. A certain special modification of the classical Gaussian plume approach is proposed for approximation of the near-field dispersion problem. Specifically, the accidental radioactivity release is subdivided into consecutive one-hour Gaussian segments, each driven by a short-term meteorological forecast for the respective hours. Determination of the physical quantity of photon fluence rate from an ambient cloud irradiation is coupled to a special decomposition of the Gaussian plume shape into the equivalent virtual elliptic disks. It facilitates solution of the formerly used time-consuming 3-D integration and provides advantages with regard to acceleration of the computational process on a local scale. An optimal choice of integration limit is adopted on the basis of the mean free path of γ-photons in the air. An efficient approach is introduced for treatment of a wide range of energetic spectrum of the emitted photons when the usual multi-nuclide approach is replaced by a new multi-group scheme. The algorithm is capable of generating the radiological responses in a large net of spatial nodes. It predetermines the proposed procedure such as a proper tool for online data assimilation analysis in the near-field areas. A specific technique for numerical integration is verified on the basis of comparison with a partial analytical solution. Convergence of the finite cloud approximation to the tabulated semi-infinite cloud values for dose conversion factors was validated.

Data Assimilation Schemes For Non-linear Shallow Water Flow Models

In theory K aim an filters can be used to solve many on-line data assimilation problems. However, for models resulting from the discretization of partial differential equations the number of state variables is usually very large, leading to a huge computational burden. Therefore approximation of the Kalman filter equations is in general necessary. In this paper two new algorithms are proposed, that extend the idea of the Reduced Rank Square Root filter [15] for use with non-linear models. The algorithms are based on a low rank approximation of the error covariance matrix and use a square root representation of the error covariance. For both algorithms the tangent linear model is not needed. The first algorithm proposed is accurate up to first order terms, which is comparable to the extended Kalman filter. The second, at the cost of twice the number of computations, is second order accurate, which may be important for strongly nonlinear models. Several experiments were performed on a model of the southern part of the North Sea to measure the performance of both algorithms. Both algorithms perform well when the the number of modes, i.e. the rank of the approximation, is set to 30. This corresponds to a computation time of approximately 30 model runs for the first order algorithm and 60 for the second order algorithm.