Background Covariance Research Articles

Implications and mitigation of model mismatch and covariance contamination for hyperspectral chemical agent detection

Most chemical gas detection algorithms for long-wave infrared hyperspectral images assume a gas with a perfectly known spectral signature. In practice, the chemical signature is either imperfectly measured and/or exhibits spectral variability due to temperature variations and Beers law. The performance of these detection algorithms degrades further as a result of unavoidable contamination of the background covariance by the plume signal. The objective of this work is to explore robust matched filters that take the uncertainty and/or variability of the target signatures into account and mitigate performance loss resulting from different factors. We introduce various techniques that control the selectivity of the matched filter and we evaluate their performance in standoff LWIR hyperspectral chemical gas detection applications.

Unscented/ensemble transform-based variational filter

To deal with high dimensional nonlinear filtering problems, a hybrid scheme called Unscented/Ensemble transform Variational Filter (UEVF) is introduced. This is the combination of an Unscented Transform (UT), an Ensemble Transform (ET) and a rank-reduction method to compute the background covariance error matrices as well as a variational minimization to conduct the mean correction. The proposed UEVF is more efficient than the Unscented Kalman Filter (UKF) to estimate the ensemble mean and covariance by the blending of a variational optimization instead of a Kalman linear correction as well as the ET-like covariance estimation into the update step. Moreover, in order to tackle the high dimension dynamics, truncated singular value decomposition is applied to provide a size reduction of a sigma-points set with an adaptive fashion. For performance verifications, we present two numerical experiments with different dynamics. The first system is the chaotic and high dimensional Lorenz-95 model. We show the performance of different filters including the UEVF as the increasing of dimensionality or noise level. The second simulation is a model based on the 2D shallow water equation. The same tests are provided on this hydrodynamical system. All the numerical experiments confirm that the UEVF outperforms the widely applied Kalman-like filters explicitly.

Test models for filtering and prediction of moisture‐coupled tropical waves

AbstractThe filtering/data assimilation and prediction of moisture‐coupled tropical waves is a contemporary topic with significant implications for extended‐range forecasting. The development of efficient algorithms to capture such waves is limited by the unstable multiscale features of tropical convection which can organize large‐scale circulations and the sparse observations of the moisture‐coupled wave in both the horizontal and vertical. The approach proposed here is to address these difficult issues of data assimilation and prediction through a suite of analogue models which, despite their simplicity, capture key features of the observational record and physical processes in moisture‐coupled tropical waves. The analogue models emphasized here involve the multicloud convective parametrization based on three cloud types (congestus, deep, and stratiform) above the boundary layer. Two test examples involving an MJO‐like turbulent travelling wave and the initiation of a convectively coupled wave train are introduced to illustrate the approach. A suite of reduced filters with judicious model errors for data assimilation of sparse observations of tropical waves, based on linear stochastic models in a moisture‐coupled eigenmode basis is developed here and applied to the two test problems. Both the reduced filter and 3D‐Var with a full moist background covariance matrix can recover the unobserved troposphere humidity and precipitation rate; on the other hand, 3D‐Var with a dry background covariance matrix fails to recover these unobserved variables. The skill of the reduced filtering methods in recovering the unobserved precipitation, congestus, and stratiform heating rates as well as the front‐to‐rear tilt of the convectively coupled waves exhibits a subtle dependence on the sparse observation network and the observation time. Copyright © 2012 Royal Meteorological Society

Generation of analysis and consistent error fields using the Data Interpolating Variational Analysis (DIVA)

The Data Interpolating Variational Analysis (DIVA) is a method designed to interpolate irregularly-spaced, noisy data onto any desired location, in most cases on regular grids. It is the combination of a particular methodology, based on the minimisation of a cost function, and a numerically efficient method, based on a finite-element solver. The cost function penalises the misfit between the observations and the reconstructed field, as well as the regularity or smoothness of the field. The method bears similarities to the smoothing splines, where the second derivatives of the field are also penalised.The intrinsic advantages of the method are its natural way to take into account topographic and dynamic constraints (coasts, advection, etc.) and its capacity to handle large data sets, frequently encountered in oceanography. The method provides gridded fields in two dimensions, usually in horizontal layers. Three-dimension fields are obtained by stacking horizontal layers.In the present work, we summarize the background of the method and describe the possible methods to compute the error field associated to the analysis. In particular, we present new developments leading to a more consistent error estimation, by determining numerically the real covariance function in DIVA, which is never formulated explicitly, contrarily to Optimal Interpolation. The real covariance function is obtained by two concurrent executions of DIVA, the first providing the covariance for the second. With this improvement, the error field is now perfectly consistent with the inherent background covariance in all cases.A two-dimension application using salinity measurements in the Mediterranean Sea is presented. Applied on these measurements, Optimal Interpolation and DIVA provided very similar gridded fields (correlation: 98.6%, RMS of the difference: 0.02). The method using the real covariance produces an error field similar to the one of OI, except in the coastal areas.

Ensemble Methods for Dynamic Data Assimilation of Chemical Observations in Atmospheric Models

The task of providing an optimal analysis of the state of the atmosphere requires the development of dynamic data-driven systems (DDDAS) that efficiently integrate the observational data and the models. Data assimilation, the dynamic incorporation of additional data into an executing application, is an essential DDDAS concept with wide applicability. In this paper we discuss practical aspects of nonlinear ensemble Kalman data assimilation applied to atmospheric chemical transport models. We highlight the challenges encountered in this approach such as filter divergence and spurious corrections, and propose solutions to overcome them, such as background covariance inflation and filter localization. The predictability is further improved by including model parameters in the assimilation process. Results for a large scale simulation of air pollution in North-East United States illustrate the potential of nonlinear ensemble techniques to assimilate chemical observations.

Application of the WRF Hybrid ETKF–3DVAR Data Assimilation System for Hurricane Track Forecasts

Abstract A hybrid ensemble transform Kalman filter (ETKF)–three-dimensional variational data assimilation (3DVAR) system developed for the Weather Research and Forecasting Model (WRF) was studied for the forecasts of the tracks of two major hurricanes, Ike and Gustav, in 2008 over the Gulf of Mexico. The impacts of the flow-dependent ensemble covariance generated by the ETKF were revealed by comparing the forecasts, analyses, and analysis increments generated by the hybrid data assimilation method with those generated by the 3DVAR that used the static background covariance. The root-mean-square errors of the track forecasts by the hybrid data assimilation (DA) method were smaller than those by the 3DVAR for both Ike and Gustav. Experiments showed that such improvements were due to the use of the flow-dependent covariance provided by the ETKF ensemble in the hybrid DA system. Detailed diagnostics further revealed that the increments produced by the hybrid and the 3DVAR were different for both the analyses of the hurricane itself and its environment. In particular, it was found that the hybrid, using the flow-dependent covariance that gave the hurricane-specific error covariance estimates, was able to systematically adjust the position of the hurricane during the assimilation whereas the 3DVAR was not. The study served as a pilot study to explore and understand the potential of the hybrid method for hurricane data assimilation and forecasts. Caution needs to be taken to extrapolate the results to operational forecast settings.

Scaling between Landsat-7 and SAR images based on ensemble empirical mode decomposition

Scaling is one of the important methods that can be used to produce high-resolution multispectral images (HRMIs) from a synthetic aperture radar (SAR) image and low-resolution multispectral images (LRMIs). The purpose of this article is to present an alternative solution of scaling the LRMIs according to one SAR image. The proposed method uses à trous wavelet transform (AWT) to abstract the details from the SAR image, and ensemble empirical mode decomposition (EMD) to inject the spatial details of the approximation plane of the SAR image into the LRMIs. The intrinsic mode functions (IMFs) of the LRMIs are updated using the gain that is derived from the background and observation covariances estimated from the IMFs of the LRMIs and the approximation plane of the SAR image. Experimental results confirm that the proposed method can provide significant improvements over other tested approaches.

Time-expanded sampling approach for Ensemble Kalman Filter: Experiment assimilation of simulated soundings

In the Ensemble Kalman Filter (EnKF) data assimilation-prediction system, most of the computation time is spent on the prediction runs of ensemble members. A limited or small ensemble size does reduce the computational cost, but an excessively small ensemble size usually leads to filter divergence, especially when there are model errors. In order to improve the efficiency of the EnKF data assimilation-prediction system and prevent it against filter divergence, a time-expanded sampling approach for EnKF based on the WRF (Weather Research and Forecasting) model is used to assimilate simulated sounding data. The approach samples a series of perturbed state vectors from Nb member prediction runs not only at the analysis time (as the conventional approach does) but also at equally separated time levels (time interval is Δt) before and after the analysis time with M times. All the above sampled state vectors are used to construct the ensemble and compute the background covariance for the analysis, so the ensemble size is increased from Nb to Nb+2M×Nb=(1+2M)×Nb) without increasing the number of prediction runs (it is still Nb). This reduces the computational cost. A series of experiments are conducted to investigate the impact of Δt (the time interval of time-expanded sampling) and M (the maximum sampling times) on the analysis. The results show that if Δt and M are properly selected, the time-expanded sampling approach achieves the similar effect to that from the conventional approach with an ensemble size of (1+2M)×Nb, but the number of prediction runs is greatly reduced.

Coastal observing and forecasting system for the German Bight – estimates of hydrophysical states

Abstract. A coastal observing system for Northern and Arctic Seas (COSYNA) aims at construction of a long-term observatory for the German part of the North Sea, elements of which will be deployed as prototype modules in Arctic coastal waters. At present a coastal prediction system deployed in the area of the German Bight integrates near real-time measurements with numerical models in a pre-operational way and provides continuously state estimates and forecasts of coastal ocean state. The measurement suite contributing to the pre-operational set up includes in situ time series from stationary stations, a High-Frequency (HF) radar system measuring surface currents, a FerryBox system and remote sensing data from satellites. The forecasting suite includes nested 3-D hydrodynamic models running in a data-assimilation mode, which are forced with up-to-date meteorological forecast data. This paper reviews the present status of the system and its recent upgrades focusing on developments in the field of coastal data assimilation. Model supported data analysis and state estimates are illustrated using HF radar and FerryBox observations as examples. A new method combining radial surface current measurements from a single HF radar with a priori information from a hydrodynamic model is presented, which optimally relates tidal ellipses parameters of the 2-D current field and the M2 phase and magnitude of the radials. The method presents a robust and helpful first step towards the implementation of a more sophisticated assimilation system and demonstrates that even using only radials from one station can substantially benefit state estimates for surface currents. Assimilation of FerryBox data based on an optimal interpolation approach using a Kalman filter with a stationary background covariance matrix derived from a preliminary model run which was validated against remote sensing and in situ data demonstrated the capabilities of the pre-operational system. Data assimilation significantly improved the performance of the model with respect to both SST and SSS and demonstrated a good skill not only in the vicinity of the Ferry track, but also over larger model areas. The examples provided in this study are considered as initial steps in establishing new coastal ocean products enhanced by the integrated COSYNA-observations and numerical modelling.

Hyperspectral Anomaly Detection With Kurtosis-Driven Local Covariance Matrix Corruption Mitigation

Local background covariance matrix corruption due to outliers in the sample data may be one of the major causes that limit detection performance of those algorithms that detect local anomalies in hyperspectral images on the basis of the Mahalanobis distance. In this letter, an original detection scheme is presented that efficiently embeds covariance corruption mitigation. A kurtosis-based binary hypothesis test is first applied to each pixel to quickly determine the presence of outliers in the local neighborhood. Rejection of the null hypothesis triggers application of a robust-to-outlier covariance estimation technique. Results on real data exhibit good detection performance and robustness to outliers. Contrary to previous works, this is achieved without an unnecessary increase of the procedural complexity.

Global ocean re-analyses for climate applications

One of the main objectives of the global ocean modelling activities at Centro Euro-Mediterraneo per i Cambiamenti Climatici (CMCC) is the production of global ocean re-analyses over multidecadal periods to reconstruct the state of the ocean and the large scale circulation over the recent past. The re-analyses are used for climate applications and for the assessment of the benefits of assimilating ocean observations on seasonal and longer predictions. Here we present the main characteristics of an optimal interpolation based assimilation system used to produce a set of global ocean re-analyses validated against a set of high quality in situ observations and independent data. Differences among the experiments of the set are analyzed in terms of improvements in the method used to assimilate the data and the quality of observations themselves. For example, the integrated ocean heat content, which can be taken as an indicator of climate changes, is examined to detect possible sources of uncertainty of its long-term changes. Global and basin scale upper ocean heat content exhibits warming trends over the last few decades that still depend in a significant way on the assimilated observations and the formulation of the background covariances. However, all the re-analyses show a global warming trend of the oceanic uppermost 700 m over the last five decades that falls within the range of the most recent observation-based estimates. The largest discrepancies between our estimates and observational based ones are confined in the upwelling regions of the Pacific and Atlantic Oceans. Finally, the results show that the climatological heat and salt transports as a function of latitude also fall within the range of the estimates based on observations and atmospheric re-analyses.

Construction of non-diagonal background error covariance matrices for global chemical data assimilation

Abstract. Chemical data assimilation attempts to optimally use noisy observations along with imperfect model predictions to produce a better estimate of the chemical state of the atmosphere. It is widely accepted that a key ingredient for successful data assimilation is a realistic estimation of the background error distribution. Particularly important is the specification of the background error covariance matrix, which contains information about the magnitude of the background errors and about their correlations. As models evolve toward finer resolutions, the use of diagonal background covariance matrices is increasingly inaccurate, as they captures less of the spatial error correlations. This paper discusses an efficient computational procedure for constructing non-diagonal background error covariance matrices which account for the spatial correlations of errors. The correlation length scales are specified by the user; a correct choice of correlation lengths is important for a good performance of the data assimilation system. The benefits of using the non-diagonal covariance matrices for variational data assimilation with chemical transport models are illustrated.

The relevance of background-error covariance matrix localization: an application to the variational retrieval of vertical profiles from SEVIRI observations

A method to select local statistics for background-error covariance matrix (B) determination is presented and applied to the 1D-Var+nudging assimilation of SEVIRI radiances in the COSMO model. The system is designed as a post-processing algorithm of an ensemble forecast system based on multi-model initial and boundary conditions and perturbations of physical parametrization parameters. Ensemble spread maps are combined to identify regions (‘islands’) inside the model domain of uniform and large error. Thereafter B matrices are calculated using local statistics from the identified islands assuming the ensemble spread to be representative of the background-error dispersion. This calculation is repeated at the beginning of each assimilation cycle to ensure the time evolution of the selected regions and thus the flow-dependency of the background-error covariance matrix statistics. The benefit of calculating B using local statistics is then quantified by comparison to background covariance errors evaluated from global domain statistics. This is done using the same ensemble members over the whole domain, i.e. without applying the localization procedure. The standard NMC approach, which uses long time series of model departures calculated at different forecast times, is also compared for reference. Model departure statistics and comparison of the final analysis to independent observations from surface stations and satellite products highlight the relevance of the localization procedure even though the noisy structure of the variance profiles produces a substantial reduction in the number of valid retrieved profiles. The final findings show that the selective identification of areas of homogeneous spread/error is a suitable approach to characterize local error growth which can bring about sensible improvements in the forecast scores. Copyright © 2011 Royal Meteorological Society

Influence of resonance parameters' correlations on the resonance integral uncertainty; 55Mn case

For nuclides with a large number of resonances the covariance matrix of resonance parameters can become very large and expensive to process in terms of the computation time. By converting covariance matrix of resonance parameters into covariance matrices of background cross-section in a more or less coarse group structure a considerable amount of computer time and memory can be saved. The question is how important is the information that is discarded in the process. First, the uncertainty of the 55Mn resonance integral was estimated in narrow resonance approximation for different levels of self-shielding using Bondarenko method by random sampling of resonance parameters according to their covariance matrices from two different 55Mn evaluations: one from Nuclear Research and Consultancy Group NRG (with large uncertainties but no correlations between resonances), the other from Oak Ridge National Laboratory (with smaller uncertainties but full covariance matrix). We have found out that if all (or at least significant part of the) resonance parameters are correlated, the resonance integral uncertainty greatly depends on the level of self-shielding. Second, it was shown that the commonly used 640-group SAND-II representation cannot describe the increase of the resonance integral uncertainty. A much finer energy mesh for the background covariance matrix would have to be used to take the resonance structure into account explicitly, but then the objective of a more compact data representation is lost.

An Adaptive Approach to Mitigate Background Covariance Limitations in the Ensemble Kalman Filter

Abstract A new approach is proposed to address the background covariance limitations arising from undersampled ensembles and unaccounted model errors in the ensemble Kalman filter (EnKF). The method enhances the representativeness of the EnKF ensemble by augmenting it with new members chosen adaptively to add missing information that prevents the EnKF from fully fitting the data to the ensemble. The vectors to be added are obtained by back projecting the residuals of the observation misfits from the EnKF analysis step onto the state space. The back projection is done using an optimal interpolation (OI) scheme based on an estimated covariance of the subspace missing from the ensemble. In the experiments reported here, the OI uses a preselected stationary background covariance matrix, as in the hybrid EnKF–three-dimensional variational data assimilation (3DVAR) approach, but the resulting correction is included as a new ensemble member instead of being added to all existing ensemble members. The adaptive approach is tested with the Lorenz-96 model. The hybrid EnKF–3DVAR is used as a benchmark to evaluate the performance of the adaptive approach. Assimilation experiments suggest that the new adaptive scheme significantly improves the EnKF behavior when it suffers from small size ensembles and neglected model errors. It was further found to be competitive with the hybrid EnKF–3DVAR approach, depending on ensemble size and data coverage.

The Influence of Varying Background Regions on Statistical Matched Filter Detection Algorithms

Statistical matched filter detection algorithms can be applied to target detection in hyperspectral imagery.The calculation of the background covariance matrix is used in any statistical matched filters,including Adaptive Coherence Estimator(ACE) and Constrained Energy Minimization(CEM) presented in this paper.Experiments on the calculation of the background covariance matrix from different background regions and the utilization of ROC curve give a comparative analysis of the problem as to how different background regions affect the detection performance of ACE and CEM.

Improved estimation of local background covariance matrix for anomaly detection in hyperspectral images

Anomaly detection in hyperspectral images has proven valuable in many applications, such as hazardous material and mine detection. The benchmark anomaly detector is the Reed-Xiaoli (RX) detector, which is based on the local multivariate normality of background. The RX algorithm, along with its many modified versions, has been widely explored, and the main concerns identified are related to local background covariance matrix estimation. The small sample size, local background nonhomogeneity, and the presence of target pixels within the estimation window are factors that can deeply affect local background covariance matrix estimation. These critical aspects may occur together in the same operational scenario, and they may strongly impair the detection performance. However, due to their intrinsic difference, these aspects have been typically discussed within different frameworks, disregarding the possible existing connections while developing different approaches to solution. We investigate these critical aspects, along with their impact on the detection process, from an operational detection perspective. The approaches to solution are critically analyzed, discussing possible links and connections. Real hyperspectral data are employed for assessing if the algorithms, designed ad hoc to solve a specific problem, can either handle more complex situations, or bring about further complications.

A hybrid approach to estimating error covariances in variational data assimilation

Data assimilation (DA) involves the combination of observational data with the underlying dynamical principles governing the system under observation. In this work we combine the advantages of the two prominent DA systems: the 4D-Var and the ensemble methods. The hybrid method described in this paper consists of identifying the subspace spanned by the major 4D-Var error reduction directions. These directions are then removed from the background covariance through a Galerkin-type projection, and are replaced by estimates of the analysis error obtained through a low-rank Hessian inverse approximation. The updated error covariance in one window can be used as the background covariance for the next window thus better capturing the ‘error of the day’. The numerical results for a non-linear model demonstrate how the hybrid method leads to a good estimate of the true error covariance, and improves the 4D-Var analysis results.

A hybrid approach to estimating error covariances in variational data assimilation

Data assimilation (DA) involves the combination of observational data with the underlying dynamical principles governing the system under observation. In this work we combine the advantages of the two prominent DA systems: the 4D-Var and the ensemble methods. The hybrid method described in this paper consists of identifying the subspace spanned by the major 4D-Var error reduction directions. These directions are then removed from the background covariance through a Galerkin-type projection, and are replaced by estimates of the analysis error obtained through a low-rank Hessian inverse approximation. The updated error covariance in one window can be used as the background covariance for the next window thus better capturing the ‘error of the day’. The numerical results for a non-linear model demonstrate how the hybrid method leads to a good estimate of the true error covariance, and improves the 4D-Var analysis results.

Ensemble data assimilation with the CNMCA regional forecasting system

AbstractThe Ensemble Kalman Filter (EnKF) is likely to become a viable alternative to variational methods for the next generation of meteorological and oceanographic data assimilation systems. In this work we present results from real‐data assimilation experiments using the CNMCA regional numerical weather prediction (NWP) forecasting system and compare them to the currently operational variational‐based analysis. The set of observations used is the same as the one ingested in the operational data stream, with the exception of satellite radiances and scatterometer winds. Results show that the EnKF‐based assimilation cycle is capable of producing analyses and forecasts of consistently superior skill in the root mean square error metric than CNMCA operational 3D‐Var.One of the most important issues in EnKF implementations lies in the filter tendency to become underdispersive for practical ensemble sizes. To combat this problem a number of different parametrizations of the model error unaccounted for in the assimilation cycle have been proposed. In the CNMCA system a combination of adaptive multiplicative and additive background covariance inflations has been found to give adequate results and to be capable of avoiding filter divergence in extended assimilation trials. The additive component of the covariance inflation has been implemented through the use of scaled forecast differences.Following suggestions that ensemble square‐root filters can violate the gaussianity assumption when used with nonlinear prognostic models, the statistical distribution of the forecast and analysis ensembles has been studied. No sign of the ensemble collapsing onto one or a few model states has been found, and the forecast and analysis ensembles appear to stay remarkably close to the assumed probability distribution functions. Copyright © 2010 Royal Meteorological Society