Ensemble Variational Data Assimilation Research Articles

Application of 3-D ensemble variational data assimilation to a Baltic Sea reanalysis 1989–2013

A 3-D ensemble variational (3DEnVar) data assimilation method has been implemented and tested for oceanographic data assimilation of sea surface temperature (SST), sea surface salinity (SSS), sea ice concentration (SIC), and salinity and temperature profiles. To damp spurious long-range correlations in the ensemble statistics, horizontal and vertical localisation was implemented using empirical orthogonal functions. The results show that the 3DEnVar method is indeed possible to use in oceanographic data assimilation. So far, only a seasonally dependent ensemble has been used, based on historical model simulations. Near-surface experiments showed that the ensemble statistics gave inhomogeneous and anisotropic horizontal structure functions, and assimilation of real SST and SIC fields gave smooth, realistic increment fields. The implementation was multivariate, and results showed that the cross-correlations between variables work in an intuitive way, for example, decreasing SST where SIC was increased and vice versa. The profile data assimilation also gave good results. The results from a 25-year reanalysis showed that the vertical salinity and temperature structure were significantly improved, compared to both dependent and independent data.

A hybrid variational-ensemble data assimilation scheme with systematic error correction for limited-area ocean models

Abstract. A hybrid variational-ensemble data assimilation scheme to estimate the vertical and horizontal parts of the background error covariance matrix for an ocean variational data assimilation system is presented and tested in a limited-area ocean model implemented in the western Mediterranean Sea. An extensive data set collected during the Recognized Environmental Picture Experiments conducted in June 2014 by the Centre for Maritime Research and Experimentation has been used for assimilation and validation. The hybrid scheme is used to both correct the systematic error introduced in the system from the external forcing (initialisation, lateral and surface open boundary conditions) and model parameterisation, and improve the representation of small-scale errors in the background error covariance matrix. An ensemble system is run offline for further use in the hybrid scheme, generated through perturbation of assimilated observations. Results of four different experiments have been compared. The reference experiment uses the classical stationary formulation of the background error covariance matrix and has no systematic error correction. The other three experiments account for, or not, systematic error correction and hybrid background error covariance matrix combining the static and the ensemble-derived errors of the day. Results show that the hybrid scheme when used in conjunction with the systematic error correction reduces the mean absolute error of temperature and salinity misfit by 55 and 42 % respectively, versus statistics arising from standard climatological covariances without systematic error correction.

Ensemble–Variational Integrated Localized Data Assimilation

AbstractHybrid variational–ensemble data assimilation (hybrid DA) is widely used in research and operational systems, and it is considered the current state of the art for the initialization of numerical weather prediction models. However, hybrid DA requires a separate ensemble DA to estimate the uncertainty in the deterministic variational DA, which can be suboptimal both technically and scientifically. A new framework called the ensemble–variational integrated localized (EVIL) data assimilation addresses this inconvenience by updating the ensemble analyses using information from the variational deterministic system. The goal of EVIL is to encompass and generalize existing ensemble Kalman filter methods in a variational framework. Particular attention is devoted to the affordability and efficiency of the algorithm in preparation for operational applications.

雪氷圏の地域気候研究のための陸域大気結合データ同化システム(CLAVEDAS)開発

雪氷圏の地域気候研究のための陸域大気結合データ同化システム(CLAVEDAS)開発

Relationships among Four-Dimensional Hybrid Ensemble–Variational Data Assimilation Algorithms with Full and Approximate Ensemble Covariance Localization

Abstract Ensemble–variational data assimilation algorithms that can incorporate the time dimension (four-dimensional or 4D) and combine static and ensemble-derived background error covariances (hybrid) are formulated in general forms based on the extended control variable and the observation-space-perturbation approaches. The properties and relationships of these algorithms and their approximated formulations are discussed. The main algorithms discussed include the following: 1) the standard ensemble 4DVar (En4DVar) algorithm incorporating ensemble-derived background error covariance through the extended control variable approach, 2) the 4DEnVar neglecting the time propagation of the extended control variable (4DEnVar-NPC), 3) the 4D ensemble–variational algorithm based on observation space perturbation (4DEnVar), and 4) the 4DEnVar with no propagation of covariance localization (4DEnVar-NPL). Without the static background error covariance term, none of the algorithms requires the adjoint model except for En4DVar. Costly applications of the tangent linear model to localized ensemble perturbations can be avoided by making the NPC and NPL approximations. It is proven that En4DVar and 4DEnVar are mathematically equivalent, while 4DEnVar-NPC and 4DEnVar-NPL are mathematically equivalent. Such equivalences are also demonstrated by single-observation assimilation experiments with a 1D linear advection model. The effects of the non-flow-following or stationary localization approximations are also examined through the experiments. All of the above algorithms can include the static background error covariance term to establish a hybrid formulation. When the static term is included, all algorithms will require a tangent linear model and an adjoint model. The first guess at appropriate time (FGAT) approximation is proposed to avoid the tangent linear and adjoint models. Computational costs of the algorithms are also discussed.

Improving Large-Domain Convection-Allowing Forecasts with High-Resolution Analyses and Ensemble Data Assimilation

Analyses with 20-km horizontal grid spacing were produced from continuously cycling three-dimensional variational (3DVAR), ensemble square root Kalman filter (EnSRF), and “hybrid” variational–ensemble data assimilation (DA) systems over a domain spanning the conterminous United States. These analyses initialized 36-h Weather Research and Forecasting Model forecasts containing a large convection-allowing 4-km nested domain, where downscaled 20-km 3DVAR, EnSRF, and hybrid analyses initialized the 4-km forecasts. Overall, hybrid analyses initialized the best 4-km precipitation forecasts. Furthermore, whether 4-km precipitation forecasts could be improved by initializing them with true 4-km analyses was assessed. As it was computationally infeasible to produce 4-km continuously cycling ensembles over the large 4-km domain, several “dual-resolution” hybrid DA configurations were adopted where 4-km backgrounds were combined with 20-km ensembles to produce 4-km hybrid analyses. Additionally, 4-km 3DVAR analyses were produced. In both hybrid and 3DVAR frameworks, initializing 4-km forecasts with true 4-km analyses, rather than downscaled 20-km analyses, yielded superior precipitation forecasts over the first 12 h. Differences between forecasts initialized from 4-km and downscaled 20-km hybrid analyses were smaller for 18–36-h forecasts, but there were occasionally meaningful differences. Continuously cycling the 4-km backgrounds and using static background error covariances with larger horizontal length scales in the hybrid led to better forecasts. All hybrid-initialized forecasts, including those initialized from downscaled 20-km analyses, were more skillful than forecasts initialized from 4-km 3DVAR analyses, suggesting the analysis method was more important than analysis resolution.

Scale-dependent background-error covariance localisation

A new approach is presented and evaluated for efficiently applying scale-dependent spatial localisation to ensemble background-error covariances within an ensemble-variational data assimilation system. The approach is primarily motivated by the requirements of future data assimilation systems for global numerical weather prediction that will be capable of resolving the convective scale. Such systems must estimate the global and synoptic scales at least as well as current global systems while also effectively making use of information from frequent and spatially dense observation networks to constrain convective-scale features. Scale-dependent covariance localisation allows a wider range of scales to be efficiently estimated while simultaneously assimilating all available observations. In the context of an idealised numerical experiment, it is shown that using scale-dependent localisation produces an improved ensemble-based estimate of spatially varying covariances as compared with standard spatial localisation. When applied to an ensemble of Arctic sea-ice concentration, it is demonstrated that strong spatial gradients in the relative contribution of different spatial scales in the ensemble covariances result in strong spatial variations in the overall amount of spatial localisation. This feature is qualitatively similar to what might be expected when applying an adaptive localisation approach that estimates a spatially varying localisation function from the ensemble itself. When compared with standard spatial localisation, scale-dependent localisation also results in a lower analysis error for sea-ice concentration over all spatial scales.

Correlation operators based on an implicitly formulated diffusion equation solved with the Chebyshev iteration

Correlation operators are used in the formulation of background‐error covariance models in variational data assimilation (VDA) and for localizing low‐rank sample estimates of background‐error covariance matrices in ensemble VDA. This article describes new approaches for defining correlation operators based on diffusion operators. The starting point is a two‐dimensional (2D) implicitly formulated diffusion operator on the sphere, which has been shown in previous works to support symmetric and positive‐definite smoothing kernels that are closely related to those from the Matérn correlation family. Different methods are proposed for solving the 2D implicit diffusion problem and these are compared with respect to their efficiency, accuracy, memory cost, ease of implementation and parallelization properties on high‐performance computers. The methods described in this article are evaluated in a global ocean VDA system. An iterative algorithm based on the Chebyshev iteration, which uses a fixed number of iterations and pre‐computed eigenvalue bounds, is shown to be particularly promising. Techniques for improving the parallelization aspects of the algorithm further are discussed.

Implementation of Deterministic Weather Forecasting Systems Based on Ensemble–Variational Data Assimilation at Environment Canada. Part II: The Regional System

Abstract The modifications to the data assimilation component of the Regional Deterministic Prediction System (RDPS) implemented at Environment Canada operations during the fall of 2014 are described. The main change is the replacement of the limited-area four-dimensional variational data assimilation (4DVar) algorithm for the limited-area analysis and the associated three-dimensional variational data assimilation (3DVar) scheme for the synchronous global driver analysis by the four-dimensional ensemble–variational data assimilation (4DEnVar) scheme presented in the first part of this study. It is shown that a 4DEnVar scheme using global background-error covariances can provide RDPS forecasts that are slightly improved compared to the previous operational approach, particularly during the first 24 h of the forecasts and in the summertime convective regime. Further forecast improvements were also made possible by upgrades in the assimilated observational data and by introducing the improved global analysis presented in the first part of this study in the RDPS intermittent cycling strategy. The computational savings brought by the 4DEnVar approach are also discussed.

Implementation of Deterministic Weather Forecasting Systems Based on Ensemble–Variational Data Assimilation at Environment Canada. Part I: The Global System

Abstract A major set of changes was made to the Environment Canada global deterministic prediction system during the fall of 2014, including the replacement of four-dimensional variational data assimilation (4DVar) by four-dimensional ensemble–variational data assimilation (4DEnVar). The new system provides improved forecast accuracy relative to the previous system, based on results from two sets of two-month data assimilation and forecast experiments. The improvements are largest at shorter lead times, but significant improvements are maintained in the 120-h forecasts for most regions and vertical levels. The improvements result from the combined impact of numerous changes, in addition to the use of 4DEnVar. These include an improved treatment of radiosonde and aircraft observations, an improved radiance bias correction procedure, the assimilation of ground-based GPS data, a doubling of the number of assimilated channels from hyperspectral infrared sounders, and an improved approach for initializing model forecasts. Because of the replacement of 4DVar with 4DEnVar, the new system is also more computationally efficient and easier to parallelize, facilitating a doubling of the analysis increment horizontal resolution. Replacement of a full-field digital filter with the 4D incremental analysis update approach, and the recycling of several key variables that are not directly analyzed significantly reduced the model spinup during both the data assimilation cycle and in medium-range forecasts.

Enhanced ensemble-based 4DVar scheme for data assimilation

Ensemble based optimal control schemes combine the components of ensemble Kalman filters and variational data assimilation (4DVar). They are trendy because they are easier to implement than 4DVar. In this paper, we evaluate a modified version of an ensemble based optimal control strategy for image data assimilation. This modified method is assessed with a shallow water model combined with synthetic data and original incomplete experimental depth sensor observations. This paper shows that the modified ensemble technique is better in quality and can reduce the computational cost.

An OSSE-Based Evaluation of Hybrid Variational–Ensemble Data Assimilation for the NCEP GFS. Part I: System Description and 3D-Hybrid Results

Abstract An observing system simulation experiment (OSSE) has been carried out to evaluate the impact of a hybrid ensemble–variational data assimilation algorithm for use with the National Centers for Environmental Prediction (NCEP) global data assimilation system. An OSSE provides a controlled framework for evaluating analysis and forecast errors since a truth is known. In this case, the nature run was generated and provided by the European Centre for Medium-Range Weather Forecasts as part of the international Joint OSSE project. The assimilation and forecast impact studies are carried out using a model that is different than the nature run model, thereby accounting for model error and avoiding issues with the so-called identical-twin experiments. It is found that the quality of analysis is improved substantially when going from three-dimensional variational data assimilation (3DVar) to a hybrid 3D ensemble–variational (EnVar)-based algorithm. This is especially true in terms of the analysis error reduction for wind and moisture, most notably in the tropics. Forecast impact experiments show that the hybrid-initialized forecasts improve upon the 3DVar-based forecasts for most metrics, lead times, variables, and levels. An additional experiment that utilizes 3DEnVar (100% ensemble) demonstrates that the use of a 25% static error covariance contribution does not alter the quality of hybrid analysis when utilizing the tangent-linear normal mode constraint on the total hybrid increment.

An OSSE-Based Evaluation of Hybrid Variational–Ensemble Data Assimilation for the NCEP GFS. Part II: 4DEnVar and Hybrid Variants

Abstract This work describes the formulation of a hybrid four-dimensional ensemble--variational (4DEnVar) algorithm and initialization options utilized within the National Centers for Environmental Prediction global data assimilation system. Initialization schemes that are proposed for use are the tangent-linear normal mode constraint, weak constraint digital filter, and a combination thereof. An observing system simulation experiment is carried out to evaluate the impact of utilizing hybrid 4DEnVar with various initialization techniques. The experiments utilize a dual-resolution configuration, where the ensemble is run at roughly half the resolution of the deterministic component. It is found that by going from 3D to 4D, analysis error is reduced for most variables and levels. The inclusion of a time-invariant static covariance when used without a normal mode–based strong constraint is found to have a small, positive impact on the analysis. The experiments show that the weak constraint digital filter degrades the quality of analysis, due to the use of hourly states to prescribe high-frequency noise. It is found that going from 3D to 4D ensemble covariances has a relatively larger impact in the extratropics, whereas the original inclusion of ensemble-based covariances was found to have the largest impact in the tropics. The improvements found in going from 3D to 4D covariances in the hybrid EnVar formulation are not as large as was found in Part I from the original introduction of the hybrid algorithm. The analyses generated by the 4D hybrid scheme are found to yield slightly improved extratropical height and wind forecasts, with smaller impacts on other variables and in general in the tropics.

Comparison of Hybrid-4DEnVar and Hybrid-4DVar Data Assimilation Methods for Global NWP

Abstract The Met Office has developed an ensemble-variational data assimilation method (hybrid-4DEnVar) as a potential replacement for the hybrid four-dimensional variational data assimilation (hybrid-4DVar), which is the current operational method for global NWP. Both are four-dimensional variational methods, using a hybrid combination of a fixed climatological model of background error covariances with localized covariances from an ensemble of current forecasts designed to describe the structure of “errors of the day.” The fundamental difference between the methods is their modeling of the time evolution of errors within each data assimilation window: 4DVar uses a linear model and its adjoint and 4DEnVar uses a localized linear combination of nonlinear forecasts. Both hybrid-4DVar and hybrid-4DEnVar beat their three-dimensional versions, which are equivalent, in NWP trials. With settings based on the current operational system, hybrid-4DVar performs better than hybrid-4DEnVar. Idealized experiments designed to compare the time evolution of covariances in the methods are described: the basic 4DEnVar represents the evolution of ensemble errors as well as 4DVar. However, 4DVar also represents the evolution of errors from the climatological covariances, whereas 4DEnVar does not. This difference is the main cause of the superiority of hybrid-4DVar. Another difference is that the authors’ 4DVar explicitly penalizes rapid variations in the analysis increment trajectory, while the authors’ 4DEnVar contains no dynamical constaints on imbalance. The authors describe a four-dimensional incremental analysis update (4DIAU) method that filters out the high-frequency oscillations introduced by the poorly balanced 4DEnVar increments. Possible methods for improving hybrid-4DEnVar are discussed.

A joint data assimilation system (Tan-Tracker) to simultaneously estimate surface CO<sub>2</sub> fluxes and 3-D atmospheric CO<sub>2</sub> concentrations from observations

Abstract. We have developed a novel framework ("Tan-Tracker") for assimilating observations of atmospheric CO2 concentrations, based on the POD-based (proper orthogonal decomposition) ensemble four-dimensional variational data assimilation method (PODEn4DVar). The high flexibility and the high computational efficiency of the PODEn4DVar approach allow us to include both the atmospheric CO2 concentrations and the surface CO2 fluxes as part of the large state vector to be simultaneously estimated from assimilation of atmospheric CO2 observations. Compared to most modern top-down flux inversion approaches, where only surface fluxes are considered as control variables, one major advantage of our joint data assimilation system is that, in principle, no assumption on perfect transport models is needed. In addition, the possibility for Tan-Tracker to use a complete dynamic model to consistently describe the time evolution of CO2 surface fluxes (CFs) and the atmospheric CO2 concentrations represents a better use of observation information for recycling the analyses at each assimilation step in order to improve the forecasts for the following assimilations. An experimental Tan-Tracker system has been built based on a complete augmented dynamical model, where (1) the surface atmosphere CO2 exchanges are prescribed by using a persistent forecasting model for the scaling factors of the first-guess net CO2 surface fluxes and (2) the atmospheric CO2 transport is simulated by using the GEOS-Chem three-dimensional global chemistry transport model. Observing system simulation experiments (OSSEs) for assimilating synthetic in situ observations of surface CO2 concentrations are carefully designed to evaluate the effectiveness of the Tan-Tracker system. In particular, detailed comparisons are made with its simplified version (referred to as TT-S) with only CFs taken as the prognostic variables. It is found that our Tan-Tracker system is capable of outperforming TT-S with higher assimilation precision for both CO2 concentrations and CO2 fluxes, mainly due to the simultaneous estimation of CO2 concentrations and CFs in our Tan-Tracker data assimilation system. A experiment for assimilating the real dry-air column CO2 retrievals (XCO2) from the Japanese Greenhouse Gases Observation Satellite (GOSAT) further demonstrates its potential wide applications.

Development of a hybrid variational-ensemble data assimilation technique for observed lightning tested in a mesoscale model

Abstract. Lightning measurements from the Geostationary Lightning Mapper (GLM) that will be aboard the Geostationary Operational Environmental Satellite – R Series will bring new information that can have the potential for improving the initialization of numerical weather prediction models by assisting in the detection of clouds and convection through data assimilation. In this study we focus on investigating the utility of lightning observations in mesoscale and regional applications suitable for current operational environments, in which convection cannot be explicitly resolved. Therefore, we examine the impact of lightning observations on storm environment. Preliminary steps in developing a lightning data assimilation capability suitable for mesoscale modeling are presented in this paper. World Wide Lightning Location Network (WWLLN) data was utilized as a proxy for GLM measurements and was assimilated with the Maximum Likelihood Ensemble Filter, interfaced with the Nonhydrostatic Mesoscale Model core of the Weather Research and Forecasting system (WRF-NMM). In order to test this methodology, regional data assimilation experiments were conducted. Results indicate that lightning data assimilation had a positive impact on the following: information content, influencing several dynamical variables in the model (e.g., moisture, temperature, and winds), and improving initial conditions during several data assimilation cycles. However, the 6 h forecast after the assimilation did not show a clear improvement in terms of root mean square (RMS) errors.

GSI-Based Four-Dimensional Ensemble–Variational (4DEnsVar) Data Assimilation: Formulation and Single-Resolution Experiments with Real Data for NCEP Global Forecast System

AbstractA four-dimensional (4D) ensemble–variational data assimilation (DA) system (4DEnsVar) was developed, building upon the infrastructure of the gridpoint statistical interpolation (GSI)-based hybrid DA system. 4DEnsVar used ensemble perturbations valid at multiple time periods throughout the DA window to estimate 4D error covariances during the variational minimization, avoiding the tangent linear and adjoint of the forecast model. The formulation of its implementation in GSI was described. The performance of the system was investigated by evaluating the global forecasts and hurricane track forecasts produced by the NCEP Global Forecast System (GFS) during the 5-week summer period assimilating operational conventional and satellite data. The newly developed system was used to address a few questions regarding 4DEnsVar. 4DEnsVar in general improved upon its 3D counterpart, 3DEnsVar. At short lead times, the improvement over the Northern Hemisphere extratropics was similar to that over the Southern Hemisphere extratropics. At longer lead times, 4DEnsVar showed more improvement in the Southern Hemisphere than in the Northern Hemisphere. The 4DEnsVar showed less impact over the tropics. The track forecasts of 16 tropical cyclones initialized by 4DEnsVar were more accurate than 3DEnsVar after 1-day forecast lead times. The analysis generated by 4DEnsVar was more balanced than 3DEnsVar. Case studies showed that increments from 4DEnsVar using more frequent ensemble perturbations approximated the increments from direct, nonlinear model propagation better than using less frequent ensemble perturbations. Consistently, the performance of 4DEnsVar including both the forecast accuracy and the balances of analyses was in general degraded when less frequent ensemble perturbations were used. The tangent linear normal mode constraint had positive impact for global forecast but negative impact for TC track forecasts.

Intercomparison and Coupling of Ensemble and Four-Dimensional Variational Data Assimilation Methods for the Analysis and Forecasting of Hurricane Karl (2010)

AbstractThis study examines the performance of ensemble and variational data assimilation systems for the Weather Research and Forecasting (WRF) Model. These methods include an ensemble Kalman filter (EnKF), an incremental four-dimensional variational data assimilation (4DVar) system, and a hybrid system that uses a two-way coupling between the two approaches (E4DVar). The three methods are applied to assimilate routinely collected data and field observations over a 10-day period that spans the life cycle of Hurricane Karl (2010), including the pregenesis disturbance that preceded its development into a tropical cyclone. In general, forecasts from the E4DVar analyses are found to produce smaller 48–72-h forecast errors than the benchmark EnKF and 4DVar methods for all variables and verification methods tested in this study. The improved representation of low- and midlevel moisture and vorticity in the E4DVar analyses leads to more accurate track and intensity predictions by this system. In particular, E4DVar analyses provide persistently more skillful genesis and rapid intensification forecasts than the EnKF and 4DVar methods during cycling. The data assimilation experiments also expose additional benefits of the hybrid system in terms of physical balance, computational cost, and the treatment of asynoptic observations near the beginning of the assimilation window. These factors make it a practical data assimilation method for mesoscale analysis and forecasting, and for tropical cyclone prediction.

Some Observing System Simulation Experiments with a Hybrid 3DEnVAR System for Storm-Scale Radar Data Assimilation

AbstractA hybrid three-dimensional ensemble–variational data assimilation (3DEnVAR) algorithm is developed based on the 3D variational data assimilation (3DVAR) and ensemble Kalman filter (EnKF) programs with the Advanced Regional Prediction System (ARPS). The method uses the extended control variable approach to combine the static and ensemble-derived flow-dependent forecast error covariances. The method is applied to the assimilation of simulated data from two radars for a supercell storm. Some sensitivity experiments are performed to answer questions about how flow-dependent covariance estimated from the forecast ensemble can be best used in the hybrid 3DEnVAR scheme. When the ensemble size is relatively small (with 5 or 10 ensemble members), it is found that experiments with a weaker weighting value for the ensemble covariance leads to better analysis results. Even when severe sampling errors exist, introducing ensemble-estimated covariances into the variational method still benefits the analysis. For reasonably large ensemble sizes (50–100 members), a stronger relative weighting (>0.8) for the ensemble covariance leads to better analyses from the hybrid 3DEnVAR. In addition, the sensitivity experiments also indicate that the best results are obtained when the number of the augmented control variables is a function of three spatial dimensions and ensemble members, and is the same for all analysis variables.

Comparison of Sequential and Variational Streamflow Assimilation Techniques for Short-Term Hydrological Forecasting

This study compares sequential and variational streamflow assimilation techniques for short-term hydrological forecasting based on a lumped conceptual rainfall-runoff model and two dissimilar watersheds (Canada and Germany). The assessment targets the Ensemble Kalman filter (EnKF) and variational data assimilation (VDA). Deterministic streamflow forecasts are computed on a daily time step over a 10-day forecast horizon, using meteorological observations as inputs to the model. Results show that the EnKF leads to the highest performance for all forecast horizons while the optimal set-up for the VDA, which often competes with the EnKF, varies from one watershed to the other. EnKF surpasses forecasts without assimilation for all horizons and for both watersheds where the NSE varies between 0.88 and 0.79 on the au Saumon watershed in Canada and between 0.92 and 0.87 on the Schlehdorf watershed in Germany on a 10-day horizon, which is not always true for the VDA. The naïve output assimilation is also assessed and is only helpful for the first two days of the forecasts.