Linear Observation Operator Research Articles

Online state and time‐varying parameter estimation using the implicit equal‐weights particle filter

AbstractA method is proposed for resilient and efficient estimation of the states and time‐varying parameters in nonlinear high‐dimensional systems through a sequential data assimilation process. The importance of estimating time‐varying parameters lies not only in improving prediction accuracy but also in determining when model characteristics change. We propose a particle‐filter‐based method that incorporates nudging techniques inspired by optimization algorithms in machine learning by taking advantage of the flexibility of the proposal density in particle filtering. However, as the model resolution and number of observations increase, filter degeneracy tends to be the obstacle to implementing the particle filter. Therefore, this proposed method is combined with the implicit equal‐weights particle filter (IEWPF), in which all particle weights are equal. The method is validated using the 1000‐dimensional linear model with an additive parameter and the 1000‐dimensional Lorenz‐96 model, where the forcing term is parameterized. The method is shown to be capable of resilient and efficient parameter estimation for parameter changes over time in our application with a linear observation operator. This leads to the conjecture that it applies to realistic geophysical, climate, and other problems.

Development of New Observation Operators for Assimilating GOES-R Geostationary Lightning Mapper Flash Extent Density Data Using GSI EnKF: Tests with Two Convective Events over the United States

Abstract In a prior study, GOES-R Geostationary Lightning Mapper (GLM) flash extent density (FED) data were assimilated using ensemble Kalman filter into a convection-allowing model for a mesoscale convective system (MCS) and a supercell storm. The FED observation operator based on a linear relation with column graupel mass was tuned by multiplying a factor to avoid large FED forecast bias. In this study, new observation operators are developed by fitting a third-order polynomial to GLM FED observations and the corresponding FED forecasts of graupel mass of the MCS and/or supercell cases. The new operators are used to assimilate the FED data for both cases, in three sets of experiments called MCSFit, SupercellFit, and CombinedFit, and their performances are compared with the prior results using the linear operator and with a reference simulation assimilating no FED data. The new nonlinear operators reduce the frequency biases (root-mean-square innovations) in the 0–4-h forecasts of the FED (radar reflectivity) relative to the results using the linear operator for both storm cases. The operator obtained by fitting data from the same case performs slightly better than fitting to data from the other case, while the operator obtained by fitting forecasts of both cases produce intermediate but still very similar results, and the latter is considered more general. In practice, a more general operator can be developed by fitting data from more cases. Significance Statement Prior studies found that assimilation of satellite lightning observation can benefit storm forecasts for up to 4 h. A linear lightning observation operator originally developed for assimilating pseudo-satellite lightning observations was tuned earlier through sensitivity experiments when assimilating real lightning data. However, the linear relation does not fit the model and observational data well and significant bias can exist. This study develops new lightning observation operators by fitting a high-order polynomial to satellite lightning observations and model-predicted quantities that directly relate to lightning. The new operator was found to reduce the frequency biases and root-mean-square innovations for lightning and radar reflectivity forecasts, respectively, up to several hours relative to the linear operator. The methodology can be applied to larger data samples to obtain a more general operator for use in operational data assimilation systems.

Manifold Markov Chain Monte Carlo Methods for Bayesian Inference in Diffusion Models

AbstractBayesian inference for nonlinear diffusions, observed at discrete times, is a challenging task that has prompted the development of a number of algorithms, mainly within the computational statistics community. We propose a new direction, and accompanying methodology—borrowing ideas from statistical physics and computational chemistry—for inferring the posterior distribution of latent diffusion paths and model parameters, given observations of the process. Joint configurations of the underlying process noise and of parameters, mapping onto diffusion paths consistent with observations, form an implicitly defined manifold. Then, by making use of a constrained Hamiltonian Monte Carlo algorithm on the embedded manifold, we are able to perform computationally efficient inference for a class of discretely observed diffusion models. Critically, in contrast with other approaches proposed in the literature, our methodology is highly automated, requiring minimal user intervention and applying alike in a range of settings, including: elliptic or hypo-elliptic systems; observations with or without noise; linear or non-linear observation operators. Exploiting Markovianity, we propose a variant of the method with complexity that scales linearly in the resolution of path discretisation and the number of observation times. Python code reproducing the results is available at http://doi.org/10.5281/zenodo.5796148.

The Maximum Likelihood Ensemble Filter with State Space Localization

AbstractA new method for ensemble data assimilation that incorporates state space covariance localization, global numerical optimization, and implied Bayesian inference is presented. The method is referred to as the MLEF with state space localization (MLEF-SSL) due to its similarity with the maximum likelihood ensemble filter (MLEF). One of the novelties introduced in MLEF-SSL is the calculation of a reduced-rank localized forecast error covariance using random projection. The Hessian preconditioning is accomplished via Cholesky decomposition of the Hessian matrix, accompanied with solving triangular system of equations instead of directly inverting matrices. For ensemble update, the MLEF-SSL system employs resampling of posterior perturbations. The MLEF-SSL was applied to Lorenz model II and compared to ensemble Kalman filter with state space localization and to MLEF with observation space localization. The observations include linear and nonlinear observation operators, each applied to integrated and point observations. Results indicate improved performance of MLEF-SSL, particularly in assimilation of integrated nonlinear observations. Resampling of posterior perturbations for an ensemble update also indicates a satisfactory performance. Additional experiments were conducted to examine the sensitivity of the method to the rank of random matrix and to compare it to truncated eigenvectors of the localization matrix. The two methods are comparable in application to low-dimensional Lorenz model, except that the new method outperforms the truncated eigenvector method in case of severe rank reduction. The random basis method is simple to implement and may be more promising for realistic high-dimensional applications.

Background error covariance iterative updating with invariant observation measures for data assimilation

In order to leverage the information embedded in the background state and observations, covariance matrices modelling is a pivotal point in data assimilation algorithms. These matrices are often estimated from an ensemble of observations or forecast differences. Nevertheless, for many industrial applications the modelling still remains empirical based on some form of expertise and physical constraints enforcement in the absence of historical observations or predictions. We have developed two novel robust adaptive assimilation methods named CUTE (Covariance Updating iTerativE) and PUB (Partially Updating BLUE). These two non-parametric methods are based on different optimization objectives, both capable of sequentially adapting background error covariance matrices in order to improve assimilation results under the assumption of a good knowledge of the observation error covariances. We have compared these two methods with the standard approach using a misspecified background matrix in a shallow water twin experiments framework with a linear observation operator. Numerical experiments have shown that the proposed methods bear a real advantage both in terms of posterior error correlation identification and assimilation accuracy.

Non-linear data assimilation via trust region optimization

In this paper, we propose an efficient and practical implementation of the ensemble Kalman filter based on trust region method for non-Gaussian data assimilation. The proposed method works as follows: the empirical moments of an ensemble of model realizations are employed to estimate the parameters of the background error distribution, then an iterative method is proposed to, among iterations, build quadratic surrogate models of the three-dimensional variational (3D-Var) cost function, these models are utilized to approximate solutions of the 3D-Var optimization problem, and the quality of solutions are assessed using the trust region framework. Furthermore, the global convergence of the proposed trust region implementation is theoretically proven. Experimental tests are carried out making use the Lorenz 96 model. Even more, two different observation operators are considered during the experiments: a linear observation operator, and a non-convex and discontinuous one. Different ensemble sizes as well as number of observations (and their time frequencies) are employed to enrich the numerical results. The Maximum Likelihood Ensemble Filter (MLEF) is utilized as a reference filter to estimate the performance of our proposed filter implementation. Ten experiments (repetitions) are performed for each configuration of parameters. The results reveal that, in average, the proposed filter outperforms the MLEF in terms of root mean square errors and even more, it converges to posterior modes of error distributions in cases wherein filter divergence can be possible in the MLEF context.

Ultra Rapid Data Assimilation Based on Ensemble Filters

The goal of this work is to analyse and study an ultra-rapid data assimilation (URDA) method for adapting a given ensemble forecast for some particular variable of a dynamical system to given observation data which become available after the standard data assimilation and forecasting steps. Initial ideas have been suggested and tested by Etherthon 2006 and Madaus and Hakim 2015 in the framework of numerical weather prediction. The methods are, however, much more universally applicable to general non-linear dynamical systems as they arise in neuroscience, biology and medicine as well as numerical weather prediction. Here we provide a full analysis in the linear case, we formulate and analyse an ultra-rapid ensemble smoother and test the ideas on the Lorentz 63 dynamical system. In particular, we study the assimilation and preemptive forecasting step of an ultra-rapid data assimilation in comparison to a full ensemble data assimilation step as calculated by an ensemble Kalman square root filter. We show that for linear systems and observation operators, the ultra-rapid assimilation and forecasting is equivalent to a full ensemble Kalman filter step. For nonlinear systems this is no longer the case. However, we show that we obtain good results even when rather strong nonlinearities are part of the time interval $[t_0, t_n]$ under consideration. Then, an ultra-rapid ensemble Kalman smoother is formulated and numerically tested. We show that when the numerical model under consideration is different from the true model, used to generate the nature run and observations, errors in the correlations will also lead to errors in the smoother analysis. The numerical study is based on the popular Lorenz 1963 model system used in geophysics and life sciences. We investigate both the situation where the full system forecast is calculated and the situation important to practical applications where we study reduced data, when only one or two variables are known to the URDA scheme.

An ensemble Kalman filter approach based on operator splitting for solving nonlinear Hammerstein type ill-posed operator equations

In this paper, we consider the application of the ensemble Kalman filter for some nonlinear operator equations. According to the structural feature of the nonlinear operator, we construct an iteration process and the corresponding linear observation operator. This construction puts our problem into the frame of 3DVar. Based on the linear observation operator, the ensemble Kalman filter approach is adopted to blend the data into the dynamics to obtain an approximation of the unknown parameter. The method is applied to an inverse potential problem and a nonlinear Fredholm integral equation. The numerical results are compared with Bayesian approach, classical regularization methods including Tikhonov regularization and Landweber iteration, which shows that the proposed algorithm is effective and competitive.

Analysis of influence of observation operator on sequential data assimilation through soil temperature simulation with common land model

An observation operator is a bridge linking the system state vector and observations in a data assimilation system. Despite its importance, the degree to which an observation operator influences the performance of data assimilation methods is still poorly understood. This study aimed to analyze the influences of linear and nonlinear observation operators on the sequential data assimilation through soil temperature simulation using the unscented particle filter (UPF) and the common land model. The linear observation operator between unprocessed simulations and observations was first established. To improve the correlation between simulations and observations, both were processed based on a series of equations. This processing essentially resulted in a nonlinear observation operator. The linear and nonlinear observation operators were then used along with the UPF in three assimilation experiments: an hourly in situ soil surface temperature assimilation, a daily in situ soil surface temperature assimilation, and a moderate resolution imaging spectroradiometer (MODIS) land surface temperature (LST) assimilation. The results show that the filter improved the soil temperature simulation significantly with the linear and nonlinear observation operators. The nonlinear observation operator improved the UPF's performance more significantly for the hourly and daily in situ observation assimilations than the linear observation operator did, while the situation was opposite for the MODIS LST assimilation. Because of the high assimilation frequency and data quality, the simulation accuracy was significantly improved in all soil layers for hourly in situ soil surface temperature assimilation, while the significant improvements of the simulation accuracy were limited to the lower soil layers for the assimilation experiments with low assimilation frequency or low data quality.

A modified randomized maximum likelihood for improved Bayesian history matching

Randomized maximum likelihood is known in the petroleum reservoir community as a Bayesian history matching technique by means of minimizing a stochastic quadratic objective function. The algorithm is well established and has shown promising results in several applications. For linear models with linear observation operator, the algorithm samples the posterior density accurately. To improve the sampling for nonlinear models, we introduce a generalized version in its simplest form by re-weighting the prior. The weight term is motivated by a sufficiency condition on the expected gradient of the objective function. Recently, an ensemble version of the algorithm was proposed which can be implemented with any simulator. Unfortunately, the method has some practical implementation issues due to computation of low rank pseudo inverse matrices and in practice only the data mismatch part of the objective function is maintained. Here, we take advantage of the fact that the measurement space is often much smaller than the parameter space and project the prior uncertainty from the parameter space to the measurement space to avoid over fitting of data. The proposed algorithms show good performance on synthetic test cases including a 2D reservoir model.

The smoother extension of the nonlinear ensemble transform filter

The recently-proposed nonlinear ensemble transform filter (NETF) is extended to a fixed-lag smoother. The NETF approximates Bayes’ theorem by applying a square root update. The smoother (NETS) is derived and formulated in a joint framework with the filter. The new smoother method is evaluated using the low-dimensional, highly nonlinear Lorenz-96 model and a square-box configuration of the NEMO ocean model, which is nonlinear and has a higher dimensionality. The new smoother is evaluated within the same assimilation framework against the local error subspace transform Kalman filter (LESTKF) and its smoother extension (LESTKS), which are state-of-the-art ensemble square-root Kalman techniques. In the case of the Lorenz-96 model, both the filter NETF and its smoother extension NETS provide lower errors than the LESTKF and LESTKS for sufficiently large ensembles. In addition, the NETS shows a distinct dependence on the smoother lag, which results in a stronger error increase beyond the optimal lag of minimum error. For the experiment using NEMO, the smoothing in the NETS effectively reduces the errors in the state estimates, compared to the filter. For different state variables very similar optimal smoothing lags are found, which allows for a simultaneous tuning of the lag. In comparison to the LESTKS, the smoothing with the NETS yields a smaller relative error reduction with respect to the filter result, and the optimal lag of the NETS is shorter in both experiments. This is explained by the distinct update mechanisms of both filters. The comparison of both experiments shows that the NETS can provide better state estimates with similar smoother lags if the model exhibits a sufficiently high degree of nonlinearity or if the observations are not restricted to be Gaussian with a linear observation operator.

A null-space-based weightedl1minimization approach to compressed sensing

It has become an established fact that the constrained $\ell_1$ minimization is capable of recovering the sparse solution from a small number of linear observations and the reweighted version can significantly improve its numerical performance. The recoverability is closely related to the Restricted Isometry Constant (RIC) {of order $s$ ($s$ is an integer), often denoted as $\delta_{s}$. A class of sufficient conditions for successful $k$-sparse signal recovery often take the form $\delta_{tk} 1$, such a bound is often called RIC bound of high order.} There exist a number of such bounds of RICs, high order or not. For example, a high order bound {is recently given by Cai and Zhang \cite{CZ14}:} $\delta_{tk} < \sqrt{(t-1)/t}$, and this bound is known sharp for $t \ge 4/3$. In this paper, we propose a new weighted $\ell_1$ minimization which only requires the following RIC bound that is more relaxed (i.e., bigger) than the above mentioned bound: \[\delta_{tk} 1$ and $0< \omega \le 1$ is determined {by two optimizations of a similar type}over the null space of the linear observation operator. {In tackling the combinatorial nature of the two optimization problems,we develop a reweighted $\ell_1$ minimization that yields a sequence of approximate solutions,which enjoy strong convergence properties. Moreover, the numerical performance of the proposed method}is very satisfactory when compared to some of the state of-the-art methods incompressed sensing.

Towards the assimilation of tree-ring-width records using ensemble Kalman filtering techniques

This paper investigates the applicability of the Vaganov–Shashkin–Lite (VSL) forward model for tree-ring-width chronologies as observation operator within a proxy data assimilation (DA) setting. Based on the principle of limiting factors, VSL combines temperature and moisture time series in a nonlinear fashion to obtain simulated TRW chronologies. When used as observation operator, this modelling approach implies three compounding, challenging features: (1) time averaging, (2) “switching recording” of 2 variables and (3) bounded response windows leading to “thresholded response”. We generate pseudo-TRW observations from a chaotic 2-scale dynamical system, used as a cartoon of the atmosphere-land system, and attempt to assimilate them via ensemble Kalman filtering techniques. Results within our simplified setting reveal that VSL’s nonlinearities may lead to considerable loss of assimilation skill, as compared to the utilization of a time-averaged (TA) linear observation operator. In order to understand this undesired effect, we embed VSL’s formulation into the framework of fuzzy logic (FL) theory, which thereby exposes multiple representations of the principle of limiting factors. DA experiments employing three alternative growth rate functions disclose a strong link between the lack of smoothness of the growth rate function and the loss of optimality in the estimate of the TA state. Accordingly, VSL’s performance as observation operator can be enhanced by resorting to smoother FL representations of the principle of limiting factors. This finding fosters new interpretations of tree-ring-growth limitation processes.

A validation of the multivariate and minimum residual method for cloud retrieval using radiance from multiple satellites

The Multivariate and Minimum Residual (MMR) cloud detection and retrieval algorithm, previously developed and tested on simulated observations and Advanced Infrared Sounder radiance, was explored and validated using various radiances from multiple sensors. For validation, the cloud retrievals were compared to independent cloud products from CloudSat, MODIS (Moderate Resolution Imaging Spectroradiometer), and GOES (Geostationary Operational Environmental Satellites). We found good spatial agreement within a single instrument, although the cloud fraction on each pixel was estimated independently. The retrieved cloud properties showed good agreement using radiances from multiple satellites, especially for the vertically integrated cloud mask. The accuracy of the MMR scheme in detecting mid-level clouds was found to be higher than for higher and lower clouds. The accuracy in retrieving cloud top pressures and cloud profiles increased with more channels from observations. For observations with fewer channels, the MMR solution was an “overly smoothed” estimation of the true vertical profile, starting from a uniform clear guess. Additionally, the retrieval algorithm showed some meaningful skill in simulating the cloudy radiance as a linear observation operator, discriminating between numerical weather prediction (NWP) error and cloud effects. The retrieval scheme was also found to be robust when different radiative transfer models were used. The potential application of the MMR algorithm in NWP with multiple radiances is also discussed.

A Hybrid Monte Carlo Sampling Filter for Non-Gaussian Data Assimilation

Data assimilation combines information from models, measurements, and priors to obtain improved estimates of the state of a dynamical system such as the atmosphere. Ensemble-based data assimilation approaches such as the Ensemble Kalman filter (EnKF) have gained wide popularity due to their simple formulation, ease of implementation, and good practical results. Many of these methods are derived under the assumption that the underlying probability distributions are Gaussian. It is well accepted, however, that the Gaussianity assumption is too restrictive when applied to large nonlinear models, nonlinear observation operators, and large levels of uncertainty. When the Gaussianity assumptions are severely violated, the performance of EnKF variations degrades. This paper proposes a new ensemble-based data assimilation method, named the <em>sampling filter</em>, which obtains the analysis by sampling directly from the posterior distribution. The sampling strategy is based on a Hybrid Monte Carlo (HMC) approach that can handle non-Gaussian probability distributions. Numerical experiments are carried out using the Lorenz-96 model and observation operators with different levels of non-linearity and differentiability. The proposed filter is also tested with shallow water model on a sphere with linear observation operator. Numerical results show that the sampling filter performs well even in highly nonlinear situations where the traditional filters diverge.

Ensemble Kalman Filtering with Residual Nudging: An Extension to State Estimation Problems with Nonlinear Observation Operators

Abstract The ensemble Kalman filter (EnKF) is an efficient algorithm for many data assimilation problems. In certain circumstances, however, divergence of the EnKF might be spotted. In previous studies, the authors proposed an observation-space-based strategy, called residual nudging, to improve the stability of the EnKF when dealing with linear observation operators. The main idea behind residual nudging is to monitor and, if necessary, adjust the distances (misfits) between the real observations and the simulated ones of the state estimates, in the hope that by doing so one may be able to obtain better estimation accuracy. In the present study, residual nudging is extended and modified in order to handle nonlinear observation operators. Such extension and modification result in an iterative filtering framework that, under suitable conditions, is able to achieve the objective of residual nudging for data assimilation problems with nonlinear observation operators. The 40-dimensional Lorenz-96 model is used to illustrate the performance of the iterative filter. Numerical results show that, while a normal EnKF may diverge with nonlinear observation operators, the proposed iterative filter remains stable and leads to reasonable estimation accuracy under various experimental settings.

Estimating soil moisture in gullies from adjacent upland measurements through different observation operators

Summary Soil moisture datasets in large gullies are rare due to the difficulty of direct sampling in such landform. This study attempted to estimate spatial soil moisture averages in gullies from measurements of adjacent uplands by using observation operators, based on three-year soil moisture datasets in a gully catchment of the Loess Plateau. Soil moisture datasets in 2010 and 2011 were used for developing observation operators and those in 2012 were used for validation. Several nonlinear and linear methods including cumulative distribution function (CDF) matching method, linear regression (LRG) method, mean relative difference (MRD) method and linear rescaling (LRS) method were used to define observation operators. The results showed observation operators significantly improved the predictions compared to when using spatial averages of uplands as the direct surrogates for gullies. Among different methods, the CDF matching method performed best in estimating soil moisture in gullies followed by the LRG, LRS and MRD methods. Validation analysis showed that the linear observation operators such as LRS, MRD and LRG had better temporal transferability than the nonlinear operators. The MRD observation operators for various layers could successfully transfer in time whereas temporal transferability only succeeds to a limited extent for other observation operators. Furthermore, the MRD, LRG and LRS methods exhibited better vertical transferability than the CDF matching method. However, the transferability of observation operators across the whole root zone layers was not successful.

A Traffic Model for Velocity Data Assimilation

This article is motivated by the practical problem of highway traffic estimation using velocity measurements from GPS enabled mobile devices such as cell phones. In order to simplify the estimation procedure, a velocity model for highway traffic is constructed, which results in a dynamical system in which the observation operator is linear. This article presents a new scalar hyperbolic partial differential equation (PDE) model for traffic velocity evolution on highways, based on the seminal Lighthill-Whitham-Richards (LWR) PDE for density. Equivalence of the solution of the new velocity PDE and the solution of the LWR PDE is shown for quadratic flux functions. Because this equivalence does not hold for general flux functions, a discretized model of velocity evolution based on the Godunov scheme applied to the LWR PDE is proposed. Using an explicit instantiation of the weak boundary conditions of the PDE, the discrete velocity evolution model is generalized to a network, thus making the model applicable to arbitrary highway networks. The resulting velocity model is a nonlinear and nondifferentiable discrete time dynamical system with a linear observation operator, for which a Monte Carlo based ensemble Kalman filtering data assimilation algorithm is applied. The model and estimation technique is evaluated with experimental data obtained from a large-scale field experiment known as Mobile Century, which is available for download at http://traffic.berkeley.edu.

Comparison of sequential data assimilation methods for the Kuramoto–Sivashinsky equation

AbstractThe Kuramoto–Sivashinsky equation plays an important role as a low‐dimensional prototype for complicated fluid dynamics systems having been studied due to its chaotic pattern forming behavior. Up to now, efforts to carry out data assimilation with this 1‐D model were restricted to variational adjoint methods domain and only Chorin and Krause (Proc. Natl. Acad. Sci. 2004; 101(42):15013–15017) tested it using a sequential Bayesian filter approach. In this work we compare three sequential data assimilation methods namely the Kalman filter approach, the sequential Monte Carlo particle filter approach and the maximum likelihood ensemble filter methods. This comparison is to the best of our knowledge novel. We compare in detail their relative performance for both linear and nonlinear observation operators. The results of these sequential data assimilation tests are discussed and conclusions are drawn as to the suitability of these data assimilation methods in the presence of linear and nonlinear observation operators. Copyright © 2009 John Wiley &amp; Sons, Ltd.

A New TwIST: Two-Step Iterative Shrinkage/Thresholding Algorithms for Image Restoration

Iterative shrinkage/thresholding (IST) algorithms have been recently proposed to handle a class of convex unconstrained optimization problems arising in image restoration and other linear inverse problems. This class of problems results from combining a linear observation model with a nonquadratic regularizer (e.g., total variation or wavelet-based regularization). It happens that the convergence rate of these IST algorithms depends heavily on the linear observation operator, becoming very slow when this operator is ill-conditioned or ill-posed. In this paper, we introduce two-step IST (TwIST) algorithms, exhibiting much faster convergence rate than IST for ill-conditioned problems. For a vast class of nonquadratic convex regularizers (l(p) norms, some Besov norms, and total variation), we show that TwIST converges to a minimizer of the objective function, for a given range of values of its parameters. For noninvertible observation operators, we introduce a monotonic version of TwIST (MTwIST); although the convergence proof does not apply to this scenario, we give experimental evidence that MTwIST exhibits similar speed gains over IST. The effectiveness of the new methods are experimentally confirmed on problems of image deconvolution and of restoration with missing samples.