Analysis Error Covariance Matrix Research Articles

On the Foundation and Different Interpretations of Ensemble Sensitivity

Abstract In sensitivity analysis, ensemble sensitivity is defined as the regression coefficients resulting from a simple linear regression of changes of a response function on initial perturbations. One of the interpretations for ensemble sensitivity considers this a simplified version of regression-based adjoint sensitivity called univariate ensemble sensitivity whose derivation involves the so-called diagonal approximation. This approximation, which replaces the analysis error covariance matrix by a diagonal matrix with the same diagonal, helps to avoid inversion of the analysis error covariance, but, at the same time causes confusion in understanding and practical application of ensemble sensitivity. However, some authors have challenged such a controversial interpretation by showing that univariate ensemble sensitivity is multivariate in nature, which raises the necessity for the foundation of ensemble sensitivity. In this study, we have tried to resolve the confusion by establishing a robust foundation for ensemble sensitivity without relying on the controversial diagonality assumption. As employed in some studies, we adopt an impact-based definition for ensemble sensitivity by taking into account probability distributions of analysis perturbations. The mathematical results show that standardized ensemble sensitivity carries in itself three important quantities at the same time: 1) standardized changes of the forecast response with one standard deviation changes of individual state variables, 2) correlations between the forecast response and individual state variables, and 3) the most sensitive analysis perturbation. The theory guarantees validity of ensemble sensitivity, demonstrates its multivariate nature, and explains why ensemble sensitivity is effective in practice.

Inflating transform matrices to mitigate assimilation errors with robust filtering based ensemble Kalman filters

AbstractAn error covariance matrix plays an important role in maintaining the statistical property of the ensemble in an ensemble Kalman filter method. However, data assimilation filter divergence may occur from an inaccurate estimate of the covariance matrix. In this study, based on an ensemble time‐local H‐infinity filter, which inflates the eigenvalues of the analysis error covariance matrix, a new robust ensemble data assimilation method is proposed, referred to as an inflation transform matrix eigenvalues. By design, new filters may be preferred over other traditional ensemble filters, when model performances are not well known, or change unpredictably. The primary aim is to improve the performance of the assimilation system in the framework of the ensemble filtering, according to the minimum/maximum rule of robust filtering. The proposed estimation method is tested using the well‐known Lorenz‐96 model, in order to investigate how the ensemble time‐local H‐infinity filter method of the inflation transform matrix impacts the robustness of the assimilation system under selected special conditions, such as the assimilation steps, force parameters, ensemble sizes, and observation information. The experiments show that the proposed inflation transform matrix method displays good robustness to the changes in the system's parameters. Also, when compared with the traditional filtering methods, this robust filtering method is found to improve the assimilation performance.

Estimates of ocean forecast error covariance derived from Hessian Singular Vectors

Experience in numerical weather prediction suggests that singular value decomposition (SVD) of a forecast can yield useful a priori information about the growth of forecast errors. It has been shown formally that SVD using the inverse of the expected analysis error covariance matrix to define the norm at initial time yields the Empirical Orthogonal Functions (EOFs) of the forecast error covariance matrix at the final time. Because of their connection to the 2nd derivative of the cost function in 4-dimensional variational (4D-Var) data assimilation, the initial time singular vectors defined in this way are often referred to as the Hessian Singular Vectors (HSVs). In the present study, estimates of ocean forecast errors and forecast error covariance were computed using SVD applied to a baroclinically unstable temperature front in a re-entrant channel using the Regional Ocean Modeling System (ROMS). An identical twin approach was used in which a truth run of the model was sampled to generate synthetic hydrographic observations that were then assimilated into the same model started from an incorrect initial condition using 4D-Var. The 4D-Var system was run sequentially, and forecasts were initialized from each ocean analysis. SVD was performed on the resulting forecasts to compute the HSVs and corresponding EOFs of the expected forecast error covariance matrix. In this study, a reduced rank approximation of the inverse expected analysis error covariance matrix was used to compute the HSVs and EOFs based on the Lanczos vectors computed during the 4D-Var minimization of the cost function. This has the advantage that the entire spectrum of HSVs and EOFs in the reduced space can be computed. The associated singular value spectrum is found to yield consistent and reliable estimates of forecast error variance in the space spanned by the EOFs. In addition, at long forecast lead times the resulting HSVs and companion EOFs are able to capture many features of the actual realized forecast error at the largest scales. Forecast error growth via the HSVs was found to be significantly influenced by the non-normal character of the underlying forecast circulation, and is accompanied by a forward energy cascade, suggesting that forecast errors could be effectively controlled by reducing the error at the largest scales in the forecast initial conditions. A predictive relation for the amplitude of the basin integrated forecast error in terms of the mean aspect ratio of the forecast error hyperellipse (quantified in terms of the mean eccentricity) was also identified which could prove useful for predicting the level of forecast error a priori. All of these findings were found to be insensitive to the configuration of the 4D-Var data assimilation system and the resolution of the observing network.

On Domain Localization in Ensemble-Based Kalman Filter Algorithms

Abstract Ensemble Kalman filter methods are typically used in combination with one of two localization techniques. One technique is covariance localization, or direct forecast error localization, in which the ensemble-derived forecast error covariance matrix is Schur multiplied with a chosen correlation matrix. The second way of localization is by domain decomposition. Here, the assimilation is split into local domains in which the assimilation update is performed independently. Domain localization is frequently used in combination with filter algorithms that use the analysis error covariance matrix for the calculation of the gain like the ensemble transform Kalman filter (ETKF) and the singular evolutive interpolated Kalman filter (SEIK). However, since the local assimilations are performed independently, smoothness of the analysis fields across the subdomain boundaries becomes an issue of concern. To address the problem of smoothness, an algorithm is introduced that uses domain localization in combination with a Schur product localization of the forecast error covariance matrix for each local subdomain. On a simple example, using the Lorenz-40 system, it is demonstrated that this modification can produce results comparable to those obtained with direct forecast error localization. In addition, these results are compared to the method that uses domain localization in combination with weighting of observations. In the simple example, the method using weighting of observations is less accurate than the new method, particularly if the observation errors are small. Domain localization with weighting of observations is further examined in the case of assimilation of satellite data into the global finite-element ocean circulation model (FEOM) using the local SEIK filter. In this example, the use of observational weighting improves the accuracy of the analysis. In addition, depending on the correlation function used for weighting, the spectral properties of the solution can be improved.

The characteristics of Hessian singular vectors using an advanced data assimilation scheme

Initial condition uncertainty is a significant source of forecast error in numerical weather prediction. Singular vectors of the tangent linear propagator can identify directions in phase-space where initial errors are likely to make the largest contribution to forecast-error variance. The physical characteristics of these singular vectors depend on the choice of initial-time metric used to represent analysis-error covariances: the total-energy norm serves as a proxy to the analysis-error covariance matrix, whereas the Hessian of the cost function of a 4D-Var assimilation scheme represents a more sophisticated estimate of the analysis-error covariances, consistent with observation and background-error covariances used in the 4D-Var scheme. This study examines and compares the structure of singular vectors computed with the European Centre for Medium-Range Weather Forecasts (ECMWF) Integrated Forecasting System using these two types of initial metrics. Unlike earlier studies that use background errors derived from lagged forecast differences (the NMC method), the background-error covariance matrix in the Hessian metric is based on statistics from an ensemble of 4D-Vars using perturbed observations, which produces tighter correlations of background-error statistics than in previous formulations. In light of these new background-error statistics, this article re-examines the properties of Hessian singular vectors (and their relationship to total-energy singular vectors) using cases from different periods between 2003 and 2005. Energy profiles and wavenumber spectra reveal that the total-energy singular vectors are similar to Hessian singular vectors that use all observation types in the operational 4D-Var assimilation. This is in contrast to the structure of Hessian singular vectors without observations. Increasing the observation density tends to reduce the spatial scale of the Hessian singular vectors. Copyright © 2009 Royal Meteorological Society

Sampling Errors in Ensemble Kalman Filtering. Part I: Theory

Abstract This paper discusses the quality of the analysis given by the ensemble Kalman filter in a perfect model context when ensemble sizes are limited. The overall goal is to improve the theoretical understanding of the problem of systematic errors in the analysis variance due to the limited size of the ensemble, as well as the potential of the so-called double-ensemble Kalman filter, covariance inflation, and randomly perturbed analysis techniques to produce a stable analysis—that is to say, one not subject to filter divergence. This is achieved by expressing the error of the ensemble mean and the analysis error covariance matrix in terms of the sampling noise in the background error covariance matrix (owing to the finite ensemble estimation) and by comparing these errors for all methods. Theoretical predictions are confirmed with a simple scalar test case. In light of the analytical results obtained, the expression of the optimal covariance inflation factor is proposed in terms of the limited ensemble size and the Kalman gain.

A method for estimating the analysis error variance in a physical space data assimilation system

The error variance of a meteorological analysis is an important quantity, both as a measure of confidence in the analysed product, and as an initial condition for the forecast error variance in a cycling data assimilation system. Modem assimilation systems that rely on global solutions of the analysis equations typically do not include an estimate of this quantity, even though the equation for the full analysis error covariance matrix is well-known. The reason is that this equation is far too costly to implement directly. In this article we analyse the analysis error covariance equation of the Kalman filter, and we propose a computationally efficient method for estimating the analysis error variance, i.e. the diagonal elements of the covariance matrix. The estimate is shown to be conservative in the sense that the analysis error variance is overestimated everywhere. We illustrate the performance of the method in the context of simple two-dimensional analysis experiments as well as in a global, three-dimensional ozone data assimilation system. These experiments show that both quantitatively and qualitatively the method captures the main features of the variance calculated by the full covariance equation at a small fraction of the cost.

A Comparison of Breeding and Ensemble Transform Kalman Filter Ensemble Forecast Schemes

Abstract The ensemble transform Kalman filter (ETKF) ensemble forecast scheme is introduced and compared with both a simple and a masked breeding scheme. Instead of directly multiplying each forecast perturbation with a constant or regional rescaling factor as in the simple form of breeding and the masked breeding schemes, the ETKF transforms forecast perturbations into analysis perturbations by multiplying by a transformation matrix. This matrix is chosen to ensure that the ensemble-based analysis error covariance matrix would be equal to the true analysis error covariance if the covariance matrix of the raw forecast perturbations were equal to the true forecast error covariance matrix and the data assimilation scheme were optimal. For small ensembles (∼100), the computational expense of the ETKF ensemble generation is only slightly greater than that of the masked breeding scheme. Version 3 of the Community Climate Model (CCM3) developed at National Center for Atmospheric Research (NCAR) is used to test ...

A study on the optimization of the deployment of targeted observations using adjoint-based methods

A new adjoint-based method to find the optimal deployment of targeted observations, called Kalman Filter Sensitivity (KFS), is introduced. The major advantage of this adjoint-based method is that it allows direct computation of the reduction of the forecast-score error variance that would result from future deployment of targeted observations. This method is applied in a very simple one-dimensional context, and is then compared to other adjoint-based products, such as classical gradients and gradients with respect to observations. The major conclusion is that the deployment of targeted observation is strongly constrained by the aspect ratio between the length-scale of the sensitivity area and the length-scale of the analysis-error covariance matrix. This very simple example also clearly illustrates that the reduction of forecast-error variance is stronger for assimilation schemes which have a smaller characteristic length-scale. Finally, the KFS technique is applied in a diagnostic way (i.e. once the observations are done) to four FASTEX cases. For these cases, the reduction of the forecasterror variance is in agreement with the efficiency of targeted observations as previously studied. A preliminary step towards an operational use has been performed on FASTEX IOP18, and results seem to validate the KFS approach of targeting. Copyright © 2002 Royal Meteorological Society.

Low-dimensional representation of error covariance

Ensemble and reduced-rank approaches to prediction and assimilation rely on low-dimensionalapproximations of the estimation error covariances. Here stability properties of the forecast/analysis cycle for linear, time-independent systems are used to identify factors that cause thesteady-state analysis error covariance to admit a low-dimensional representation. A usefulmeasure of forecast/analysis cycle stability is the bound matrix, a function of the dynamics, observation operator and assimilation method. Upper and lower estimates for the steady-stateanalysis error covariance matrix eigenvalues are derived from the bound matrix. The estimatesgeneralize to time-dependent systems. If much of the steady-state analysis error variance is dueto a few dominant modes, the leading eigenvectors of the bound matrix approximate those ofthe steady-state analysis error covariance matrix. The analytical results are illustrated in twonumerical examples where the Kalman filter is carried to steady state. The first example usesthe dynamics of a generalized advection equation exhibiting non-modal transient growth.Failure to observe growing modes leads to increased steady-state analysis error variances.Leading eigenvectors of the steady-state analysis error covariance matrix are well approximatedby leading eigenvectors of the bound matrix. The second example uses the dynamics of adamped baroclinic wave model. The leading eigenvectors of a lowest-order approximation ofthe bound matrix are shown to approximate well the leading eigenvectors of the steady-stateanalysis error covariance matrix.

安定及び不安定な力学のための近似的なデータ同化スキーム

Two suboptimal data assimilation schemes for stable and unstable dynamics are introduced. The first scheme, the partial singular value decomposition filter, is based on the most dominant singular modes of the tangent linear propagator. The second scheme, the partial eigendecomposition filter, is based on the most dominant eigenmodes of the propagated analysis error covariance matrix. Both schemes rely on iterative procedures like the Lanczos algorithm to compute the relevant modes. The performance of these schemes is evaluated for a shallow-water model linearized about an unstable Bickley jet. The results are contrasted against those of a reduced resolution filter, in which the gains used to update the state vector are calculated from a lower-dimensional dynamics than the dynamics that evolve the state itself. The results are also contrasted against the exact results given by the Kalman filter. These schemes are validated for the case of stable dynamics as well. The two new approximate assimilation schemes are shown to perform well with relatively few modes computed. Adaptive tuning of a modeled trailing error covariance for all three of these low-rank approximate schemes enhances performance and compensates for the approximation employed.