Ensemble Data Assimilation Methods Research Articles

Inflating transform matrices to mitigate assimilation errors with robust filtering based ensemble Kalman filters

AbstractAn error covariance matrix plays an important role in maintaining the statistical property of the ensemble in an ensemble Kalman filter method. However, data assimilation filter divergence may occur from an inaccurate estimate of the covariance matrix. In this study, based on an ensemble time‐local H‐infinity filter, which inflates the eigenvalues of the analysis error covariance matrix, a new robust ensemble data assimilation method is proposed, referred to as an inflation transform matrix eigenvalues. By design, new filters may be preferred over other traditional ensemble filters, when model performances are not well known, or change unpredictably. The primary aim is to improve the performance of the assimilation system in the framework of the ensemble filtering, according to the minimum/maximum rule of robust filtering. The proposed estimation method is tested using the well‐known Lorenz‐96 model, in order to investigate how the ensemble time‐local H‐infinity filter method of the inflation transform matrix impacts the robustness of the assimilation system under selected special conditions, such as the assimilation steps, force parameters, ensemble sizes, and observation information. The experiments show that the proposed inflation transform matrix method displays good robustness to the changes in the system's parameters. Also, when compared with the traditional filtering methods, this robust filtering method is found to improve the assimilation performance.

The TIGGE Project and Its Achievements

Abstract The International Grand Global Ensemble (TIGGE) was a major component of The Observing System Research and Predictability Experiment (THORPEX) research program, whose aim is to accelerate improvements in forecasting high-impact weather. By providing ensemble prediction data from leading operational forecast centers, TIGGE has enhanced collaboration between the research and operational meteorological communities and enabled research studies on a wide range of topics. The paper covers the objective evaluation of the TIGGE data. For a range of forecast parameters, it is shown to be beneficial to combine ensembles from several data providers in a multimodel grand ensemble. Alternative methods to correct systematic errors, including the use of reforecast data, are also discussed. TIGGE data have been used for a range of research studies on predictability and dynamical processes. Tropical cyclones are the most destructive weather systems in the world and are a focus of multimodel ensemble research. Their extratropical transition also has a major impact on the skill of midlatitude forecasts. We also review how TIGGE has added to our understanding of the dynamics of extratropical cyclones and storm tracks. Although TIGGE is a research project, it has proved invaluable for the development of products for future operational forecasting. Examples include the forecasting of tropical cyclone tracks, heavy rainfall, strong winds, and flood prediction through coupling hydrological models to ensembles. Finally, the paper considers the legacy of TIGGE. We discuss the priorities and key issues in predictability and ensemble forecasting, including the new opportunities of convective-scale ensembles, links with ensemble data assimilation methods, and extension of the range of useful forecast skill.

Data assimilation using a climatologically augmented local ensemble transform Kalman filter

Ensemble data assimilation methods are potentially attractive because they provide a computationally affordable (and computationally parallel) means of obtaining flow-dependent background-error statistics. However, a limitation of these methods is that the rank of their flow-dependent background-error covariance estimate, and hence the space of possible analysis increments, is limited by the number of forecast ensemble members. To overcome this deficiency ensemble methods typically use empirical localisation, which allows more degrees of freedom for the analysis increment by suppressing spatially distant background correlations. The method presented here improves the performance of an Ensemble Kalman filter by increasing the size of the ensemble at analysis time in order to boost the rank of its background-error covariance estimate. The additional ensemble members added to the forecast ensemble at analysis time are created by adding a collection of ‘climatological’ perturbations to the forecast ensemble mean. These perturbations are constant in time and provide state space directions, possibly missed by the dynamically forecasted background ensemble, in which the analysis increment can correct the forecast mean based on observations. As the climatological perturbations are calculated once, there is negligible computational cost in obtaining the additional ensemble members at each analysis cycle. Included here are a formulation of the method, results of numerical experiments conducted with a spatiotemporally chaotic model in one spatial dimension and discussion of possible future extensions and applications. The numerical tests indicate that the method presented here has significant potential for improving analyses and forecasts.

Assessment of an ensemble system that assimilates Jason-1/Envisat altimeter data in a probabilistic model of the North Atlantic ocean circulation

Abstract. A realistic circulation model of the North Atlantic ocean at 0.25° resolution (NATL025 NEMO configuration) has been adapted to explicitly simulate model uncertainties. This is achieved by introducing stochastic perturbations in the equation of state to represent the effect of unresolved scales on the model dynamics. The main motivation for this work is to develop ensemble data assimilation methods, assimilating altimetric data from past missions Jason-1 and Envisat. The assimilation experiment is designed to provide a description of the uncertainty associated with the Gulf Stream circulation for years 2005/2006, focusing on frontal regions which are predominantly affected by unresolved dynamical scales. An ensemble based on such stochastic perturbations is first produced and evaluated using along-track altimetry observations. Then each ensemble member is updated by a square root algorithm based on the SEEK (singular evolutive extended Kalman) filter (Brasseur and Verron, 2006). These three elements – stochastic parameterization, ensemble simulation and 4-D observation operator – are then used together to perform a 4-D analysis of along-track altimetry over 10-day windows. Finally, the results of this experiment are objectively evaluated using the standard probabilistic approach developed for meteorological applications (Toth et al., 2003; Candille et al., 2007). The results show that the free ensemble – before starting the assimilation process – correctly reproduces the statistical variability over the Gulf Stream area: the system is then pretty reliable but not informative (null probabilistic resolution). Updating the free ensemble with altimetric data leads to a better reliability with an information gain of around 30% (for 10-day forecasts of the SSH variable). Diagnoses on fully independent data (i.e. data that are not assimilated, like temperature and salinity profiles) provide more contrasted results when the free and updated ensembles are compared.

Linear Filtering of Sample Covariances for Ensemble-Based Data Assimilation. Part I: Optimality Criteria and Application to Variance Filtering and Covariance Localization

Abstract In data assimilation (DA) schemes for numerical weather prediction (NWP) systems, the estimation of forecast error covariances is a key point to get some flow dependency. As shown in previous studies, ensemble data assimilation methods are the most accurate for this task. However, their huge computational cost raises a strong limitation to the ensemble size. Consequently, covariances estimated with small ensembles are affected by random sampling errors. The aim of this study is to develop a theory of covariance filtering in order to remove most of the sampling noise while keeping the signal of interest and then to use it in the DA scheme of a real NWP system. This first part of a two-part study presents the theoretical aspects of such criteria for optimal filtering based on the merging of the theories of optimal linear filtering and of sample centered moments estimation. Its strength relies on the use of sample estimated quantities and filter output only. These criteria pave the way for new algorithms and interesting applications for NWP. Two of them are detailed here: spatial filtering of variances and covariance localization. Results obtained in an idealized 1D analytical framework are shown for illustration. Applications on real forecast error covariances deduced from ensembles at convective scale are discussed in a companion paper.

A joint data assimilation system (Tan-Tracker) to simultaneously estimate surface CO<sub>2</sub> fluxes and 3-D atmospheric CO<sub>2</sub> concentrations from observations

Abstract. We have developed a novel framework ("Tan-Tracker") for assimilating observations of atmospheric CO2 concentrations, based on the POD-based (proper orthogonal decomposition) ensemble four-dimensional variational data assimilation method (PODEn4DVar). The high flexibility and the high computational efficiency of the PODEn4DVar approach allow us to include both the atmospheric CO2 concentrations and the surface CO2 fluxes as part of the large state vector to be simultaneously estimated from assimilation of atmospheric CO2 observations. Compared to most modern top-down flux inversion approaches, where only surface fluxes are considered as control variables, one major advantage of our joint data assimilation system is that, in principle, no assumption on perfect transport models is needed. In addition, the possibility for Tan-Tracker to use a complete dynamic model to consistently describe the time evolution of CO2 surface fluxes (CFs) and the atmospheric CO2 concentrations represents a better use of observation information for recycling the analyses at each assimilation step in order to improve the forecasts for the following assimilations. An experimental Tan-Tracker system has been built based on a complete augmented dynamical model, where (1) the surface atmosphere CO2 exchanges are prescribed by using a persistent forecasting model for the scaling factors of the first-guess net CO2 surface fluxes and (2) the atmospheric CO2 transport is simulated by using the GEOS-Chem three-dimensional global chemistry transport model. Observing system simulation experiments (OSSEs) for assimilating synthetic in situ observations of surface CO2 concentrations are carefully designed to evaluate the effectiveness of the Tan-Tracker system. In particular, detailed comparisons are made with its simplified version (referred to as TT-S) with only CFs taken as the prognostic variables. It is found that our Tan-Tracker system is capable of outperforming TT-S with higher assimilation precision for both CO2 concentrations and CO2 fluxes, mainly due to the simultaneous estimation of CO2 concentrations and CFs in our Tan-Tracker data assimilation system. A experiment for assimilating the real dry-air column CO2 retrievals (XCO2) from the Japanese Greenhouse Gases Observation Satellite (GOSAT) further demonstrates its potential wide applications.

A comparison between the Local Ensemble Transform Kalman Filter and the Ensemble Square Root Filter for the assimilation of radar data in convective‐scale models

Two ensemble data assimilation methods are used to assimilate Doppler radar observations into a convection‐allowing model. The analyses and subsequent forecasts from the two systems are compared. The Local Ensemble Transform Kalman Filter (LETKF) simultaneously assimilates all observations that can impact the model state at a given location. It is compared to the Ensemble Square Root Filter (EnSRF), which assimilates observations sequentially and has commonly been used for convective‐scale Doppler radar data assimilation. While the filters should behave the same for ideal systems, a comparison between the serial and simultaneous filters has not previously been explored at the convective scale where significant nonlinear effects are present. Observing System Simulation Experiments (OSSEs) are first used to compare the assimilation systems for the analysis and forecast of a supercell thunderstorm. Both the EnSRF and LETKF produce reasonable analyses from the Doppler velocity and reflectivity observations of the true supercell. Small improvements in analysis errors and system noise from the LETKF simultaneous update do not significantly impact the subsequent forecasts. This result is consistent across a range of localization length‐scales and is independent of the manner in which localization is applied. Tests comparing the EnSRF and LETKF for a real‐data case also have small differences. The magnitudes of these differences are similar to those that arise from the sampling variability associated with a finite ensemble. Overall, the results suggest the EnSRF and LETKF approaches are equally capable methods for radar data assimilation at convective scales.

Intercomparison and Coupling of Ensemble and Four-Dimensional Variational Data Assimilation Methods for the Analysis and Forecasting of Hurricane Karl (2010)

AbstractThis study examines the performance of ensemble and variational data assimilation systems for the Weather Research and Forecasting (WRF) Model. These methods include an ensemble Kalman filter (EnKF), an incremental four-dimensional variational data assimilation (4DVar) system, and a hybrid system that uses a two-way coupling between the two approaches (E4DVar). The three methods are applied to assimilate routinely collected data and field observations over a 10-day period that spans the life cycle of Hurricane Karl (2010), including the pregenesis disturbance that preceded its development into a tropical cyclone. In general, forecasts from the E4DVar analyses are found to produce smaller 48–72-h forecast errors than the benchmark EnKF and 4DVar methods for all variables and verification methods tested in this study. The improved representation of low- and midlevel moisture and vorticity in the E4DVar analyses leads to more accurate track and intensity predictions by this system. In particular, E4DVar analyses provide persistently more skillful genesis and rapid intensification forecasts than the EnKF and 4DVar methods during cycling. The data assimilation experiments also expose additional benefits of the hybrid system in terms of physical balance, computational cost, and the treatment of asynoptic observations near the beginning of the assimilation window. These factors make it a practical data assimilation method for mesoscale analysis and forecasting, and for tropical cyclone prediction.

Ensemble variational data assimilation method based on regional successive analysis scheme

The ensemble variational data assimilation method may be subject to significant uncertainties due to the size of forecast ensemble. We found that this problem occurs because the analysis increment of this method is expressed as a linear combination of ensemble perturbation vectors or expansion of the orthogonal basis vectors. Though this method avoids introducing adjoint model while calculating the gradient of object function, the number of physical control variables is much larger than the sample size of forecast ensemble, which causes the assimilation results to be sensitive to the number of ensemble members. For this reason, the regional successive analysis scheme of ensemble variational method is proposed. By this scheme, the ratio between the number of physical control variables in analysis region and the sample size is decreased, so that it is expected that the problem can be solved. The results of numerical experiments using shallow water model show that the regional successive analysis scheme can give better assimilation results than traditional method, and the analysis precision is improved appreciably.

Joint state and parameter estimation with an iterative ensemble Kalman smoother

Abstract. Both ensemble filtering and variational data assimilation methods have proven useful in the joint estimation of state variables and parameters of geophysical models. Yet, their respective benefits and drawbacks in this task are distinct. An ensemble variational method, known as the iterative ensemble Kalman smoother (IEnKS) has recently been introduced. It is based on an adjoint model-free variational, but flow-dependent, scheme. As such, the IEnKS is a candidate tool for joint state and parameter estimation that may inherit the benefits from both the ensemble filtering and variational approaches. In this study, an augmented state IEnKS is tested on its estimation of the forcing parameter of the Lorenz-95 model. Since joint state and parameter estimation is especially useful in applications where the forcings are uncertain but nevertheless determining, typically in atmospheric chemistry, the augmented state IEnKS is tested on a new low-order model that takes its meteorological part from the Lorenz-95 model, and its chemical part from the advection diffusion of a tracer. In these experiments, the IEnKS is compared to the ensemble Kalman filter, the ensemble Kalman smoother, and a 4D-Var, which are considered the methods of choice to solve these joint estimation problems. In this low-order model context, the IEnKS is shown to significantly outperform the other methods regardless of the length of the data assimilation window, and for present time analysis as well as retrospective analysis. Besides which, the performance of the IEnKS is even more striking on parameter estimation; getting close to the same performance with 4D-Var is likely to require both a long data assimilation window and a complex modeling of the background statistics.

Reanalyzing temperature and salinity on decadal time scales using the ensemble optimal interpolation data assimilation method and a 3D ocean circulation model of the Baltic Sea

A 30-year (1970-1999) reanalysis of temperature and salinity is conducted by assimilating temperature and salinity profiles into an ocean model of the Baltic Sea with ensemble optimal interpolation ...

A comparison of 4DVar with ensemble data assimilation methods

AbstractThree data assimilation methods are compared for their ability to produce the best analysis: (i) 4DVar, four‐dimensional variational data assimilation using linear and adjoint models with either a (perfect) 3D climatological background‐error covariance or a 3D ensemble background‐error covariance; (ii) EDA, an ensemble of 4DEnVars, which is a variational method using a 4D ensemble covariance; and (iii) the deterministic ensemble Kalman filter (DEnKF, also using a 4D ensemble covariance).The accuracy of the deterministic analysis from each method was measured for both perfect and imperfect toy model experiments. With a perfect model, 4DVar with the climatological covariance is easily beaten by the ensemble methods, due to the importance of flow‐dependent background‐error covariances. When model error is present, 4DVar is more competitive and its relative performance is improved by increasing the observation density. This is related to the model error representation in the background‐error covariance.The dynamical time‐consistency of the 4D ensemble background‐error covariance is degraded by the localization, since the localization function and the nonlinear model do not commute. As a result, 4DVar with the ensemble covariance performs significantly better than the other ensemble methods when severe localization is required, i.e. for a small ensemble.

Assimilating multi-site measurements for semi-distributed hydrological model updating

Abstract Accurate estimates of the uncertainties associated with hydrological model are essential for better streamflow simulation. This paper explores the Ensemble Kalman Filter (EnKF), an ensemble data assimilation method, for semi-distributed hydrological model updating. The semi-distributed model is very practical and often used for moderate and large basin streamflow forecasting and water resources management. The studied area in this study is a large basin of Baohe, upper branch of Hanjiang River. The semi-distributed Xinanjiang model states are updated by assimilating several spatially distributed measurement points within the whole basin. The spatial pattern and ensemble of model states such as soil water content are derived. A lumped model updating case is taken for comparison. The results show that the semi-distributed model case does better in high flow simulation than the lumped case, with 16% and 25% improvement to the simulation performance at peak flow in two periods of heavy rain processes. The smaller streamflow uncertainty at main basin outlet is also found in the semi-distributed updating case.

Evaluating uncertainty estimates in hydrologic models: borrowing measures from the forecast verification community

Abstract. The hydrologic community is generally moving towards the use of probabilistic estimates of streamflow, primarily through the implementation of Ensemble Streamflow Prediction (ESP) systems, ensemble data assimilation methods, or multi-modeling platforms. However, evaluation of probabilistic outputs has not necessarily kept pace with ensemble generation. Much of the modeling community is still performing model evaluation using standard deterministic measures, such as error, correlation, or bias, typically applied to the ensemble mean or median. Probabilistic forecast verification methods have been well developed, particularly in the atmospheric sciences, yet few have been adopted for evaluating uncertainty estimates in hydrologic model simulations. In the current paper, we overview existing probabilistic forecast verification methods and apply the methods to evaluate and compare model ensembles produced from two different parameter uncertainty estimation methods: the Generalized Uncertainty Likelihood Estimator (GLUE), and the Shuffle Complex Evolution Metropolis (SCEM). Model ensembles are generated for the National Weather Service SACramento Soil Moisture Accounting (SAC-SMA) model for 12 forecast basins located in the Southeastern United States. We evaluate the model ensembles using relevant metrics in the following categories: distribution, correlation, accuracy, conditional statistics, and categorical statistics. We show that the presented probabilistic metrics are easily adapted to model simulation ensembles and provide a robust analysis of model performance associated with parameter uncertainty. Application of these methods requires no information in addition to what is already available as part of traditional model validation methodology and considers the entire ensemble or uncertainty range in the approach.

Maximum Likelihood Ensemble Filter: Theoretical Aspects

Abstract A new ensemble-based data assimilation method, named the maximum likelihood ensemble filter (MLEF), is presented. The analysis solution maximizes the likelihood of the posterior probability distribution, obtained by minimization of a cost function that depends on a general nonlinear observation operator. The MLEF belongs to the class of deterministic ensemble filters, since no perturbed observations are employed. As in variational and ensemble data assimilation methods, the cost function is derived using a Gaussian probability density function framework. Like other ensemble data assimilation algorithms, the MLEF produces an estimate of the analysis uncertainty (e.g., analysis error covariance). In addition to the common use of ensembles in calculation of the forecast error covariance, the ensembles in MLEF are exploited to efficiently calculate the Hessian preconditioning and the gradient of the cost function. A sufficient number of iterative minimization steps is 2–3, because of superior Hessian preconditioning. The MLEF method is well suited for use with highly nonlinear observation operators, for a small additional computational cost of minimization. The consistent treatment of nonlinear observation operators through optimization is an advantage of the MLEF over other ensemble data assimilation algorithms. The cost of MLEF is comparable to the cost of existing ensemble Kalman filter algorithms. The method is directly applicable to most complex forecast models and observation operators. In this paper, the MLEF method is applied to data assimilation with the one-dimensional Korteweg–de Vries–Burgers equation. The tested observation operator is quadratic, in order to make the assimilation problem more challenging. The results illustrate the stability of the MLEF performance, as well as the benefit of the cost function minimization. The improvement is noted in terms of the rms error, as well as the analysis error covariance. The statistics of innovation vectors (observation minus forecast) also indicate a stable performance of the MLEF algorithm. Additional experiments suggest the amplified benefit of targeted observations in ensemble data assimilation.

Ensemble Square Root Filters*

Ensemble data assimilation methods assimilate observations using state-space estimation methods and low-rank representations of forecast and analysis error covariances. A key element of such methods is the transformation of the forecast ensemble into an analysis ensemble with appropriate statistics. This transformation may be performed stochastically by treating observations as random variables, or deterministically by requiring that the updated analysis perturbations satisfy the Kalman filter analysis error covariance equation. Deterministic analysis ensemble updates are implementations of Kalman square root filters. The nonuniqueness of the deterministic transformation used in square root Kalman filters provides a framework to compare three recently proposed ensemble data assimilation methods.

Ensemble Data Assimilation without Perturbed Observations

The ensemble Kalman filter (EnKF) is a data assimilation scheme based on the traditional Kalman filter update equation. An ensemble of forecasts are used to estimate the background-error covariances needed to compute the Kalman gain. It is known that if the same observations and the same gain are used to update each member of the ensemble, the ensemble will systematically underestimate analysis-error covariances. This will cause a degradation of subsequent analyses and may lead to filter divergence. For large ensembles, it is known that this problem can be alleviated by treating the observations as random variables, adding random perturbations to them with the correct statistics. Two important consequences of sampling error in the estimate of analysis-error covariances in the EnKF are discussed here. The first results from the analysis-error covariance being a nonlinear function of the backgrounderror covariance in the Kalman filter. Due to this nonlinearity, analysis-error covariance estimates may be negatively biased, even if the ensemble background-error covariance estimates are unbiased. This problem must be dealt with in any Kalman filter‐based ensemble data assimilation scheme. A second consequence of sampling error is particular to schemes like the EnKF that use perturbed observations. While this procedure gives asymptotically correct analysis-error covariance estimates for large ensembles, the addition of perturbed observations adds an additional source of sampling error related to the estimation of the observation-error covariances. In addition to reducing the accuracy of the analysis-error covariance estimate, this extra source of sampling error increases the probability that the analysis-error covariance will be underestimated. Because of this, ensemble data assimilation methods that use perturbed observations are expected to be less accurate than those which do not. Several ensemble filter formulations have recently been proposed that do not require perturbed observations. This study examines a particularly simple implementation called the ensemble square root filter, or EnSRF. The EnSRF uses the traditional Kalman gain for updating the ensemble mean but uses a ‘‘reduced’’ Kalman gain to update deviations from the ensemble mean. There is no additional computational cost incurred by the EnSRF relative to the EnKF when the observations have independent errors and are processed one at a time. Using a hierarchy of perfect model assimilation experiments, it is demonstrated that the elimination of the sampling error associated with the perturbed observations makes the EnSRF more accurate than the EnKF for the same ensemble size.