Ensemble Filter Research Articles

Initial trials of convective‐scale data assimilation with a cheaply tunable ensemble filter

Ensemble methods are an attractive option for data assimilation (DA) in convection‐permitting models, providing flow‐dependent covariances which respect the complex balances that apply at these scales. The Met Office deterministic UK forecast currently uses variational assimilation based on predominantly climatological covariances, whilst the MOGREPS‐UK ensemble interpolates its initial perturbations from the corresponding global members. An experimental convective‐scale ensemble DA system has now been built based on a serial ensemble filter, where observations are assimilated sequentially. This has been tested in a series of one month trials assimilating sonde, surface and aircraft observations, exploring issues such as cycle length, filter type, localization and inflation. The tests use 44 perturbed ensemble members, plus an unperturbed control forecast that is updated using the same Kalman gain as the ensemble mean. The best configurations provide control forecasts which are competitive with interpolating global initial states like MOGREPS‐UK did at the time, even though the local assimilation has seen fewer observation types than the global DA. An hourly cycle performs much better than a 6‐hourly one, as suggested by theoretical arguments, even though both see the same boundary conditions and essentially the same total set of observations.The serial filter calculates the difference between each observation and its model equivalent as updated by all previous observations. We call this the Innovation After assimilating Prior Observations (IAPO). The mean square IAPO measures the error of the developing analysis using observations which are statistically independent of it, without the need to involve the forecast model. This diagnostic confirms the beneficial impact of assimilation, can be used to cheaply tune parameters such as localization radii, and contributes to evaluating the correctness of the ensemble spread.

Parallel assimilation of observed data in the hydrodynamic model of the ocean circulation

The parallel use of the Kalman ensemble filter technique for assimilating data from observations in theHYCOMmodel of theWorldOcean is described. Data from satellite observations of the sea’s surface temperature and the sea’s surface height are assimilated both separately and conjointly. Numerical experiments on correcting model calculations using data from observations are performed. The results from the corrections are compared to model calculations without assimilation. The effectiveness of the employed parallelization algorithm is confirmed.

Application of ensemble H-infinity filter in aquifer characterization and comparison to ensemble Kalman filter

Though the ensemble Kalman filter (EnKF) has been successfully applied in many areas, it requires explicit and accurate model and measurement error information, leading to difficulties in practice when only limited information on error mechanisms of observational instruments for subsurface systems is accessible. To handle the uncertain errors, we applied a robust data assimilation algorithm, the ensemble H-infinity filter (EnHF), to estimation of aquifer hydraulic heads and conductivities in a flow model with uncertain/correlated observational errors. The impacts of spatial and temporal correlations in measurements were analyzed, and the performance of EnHF was compared with that of the EnKF. The results show that both EnHF and EnKF are able to estimate hydraulic conductivities properly when observations are free of error; EnHF can provide robust estimates of hydraulic conductivities even when no observational error information is provided. In contrast, the estimates of EnKF seem noticeably undermined because of correlated errors and inaccurate error statistics, and filter divergence was observed. It is concluded that EnHF is an efficient assimilation algorithm when observational errors are unknown or error statistics are inaccurate.

Correction of inertial oscillations by assimilation of HF radar data in a model of the Ligurian Sea

This article aims at analyzing if high-frequency radar observations of surface currents allow to improve model forecasts in the Ligurian Sea, where inertial oscillations are a dominant feature. An ensemble of ROMS models covering the Ligurian Sea, and nested in the Mediterranean Forecasting System, is coupled with two WERA high-frequency radars. A sensitivity study allows to determine optimal parameters for the ensemble filter. By assimilating observations in a single point, the obtained correction shows that the forecast error covariance matrix represents the inertial oscillations, as well as large- and meso-scale processes. Furthermore, it is shown that the velocity observations can correct the phase and amplitude of the inertial oscillations. Observations are shown to have a strong effect during approximately half a day, which confirms the importance of using a high temporal observation frequency. In general, data assimilation of HF radar observations leads to a skill score of about 30% for the forecasts of surface velocity.

The Square Root Information Increment Ensemble Filter

Abstract A new type of ensemble filter is developed, one that stores and updates its state information in an efficient square root information filter form. It addresses two shortcomings of conventional ensemble Kalman filters: the coarse characterization of random forecast model error effects and the overly optimistic approximation of the estimation error statistics. The new filter uses an assumed a priori covariance approximation that is full rank but sparse, possibly with a dense low-rank increment. This matrix can be used to develop a nominal square root information equation for the system state and uncertainty. The measurements are used to develop an additional low-rank square root information equation. New algorithms provide forecasts and analyses of these increments at a computational cost comparable to that of existing ensemble Kalman filters. Model error effects are implicit in the a priori covariance time history, thereby obviating one of the reasons for including an inflation operation. The use of an a priori full-rank covariance allows the analysis operations to improve the state estimate without the need for a localization adjustment. This new filter exhibited worse performance than a typical covariance square root ensemble Kalman filter when operating on the Lorenz-96 problem in a chaotic regime. It excelled on a version of the Lorenz-96 problem where nonlinearities in the forecast model were weak, where the state vector uncertainty lay predominantly in a small subspace, and where the observations were spatially sparse. Such a problem might be representative of ionospheric space weather data assimilation where forcing variability can dominate the state uncertainty and where remote sensing data coverage can be sparse.

Quantum Noise Removal from Breast Mammograms Using Genetic Programming based Hybrid Ensemble Filter

Quantum noise are more likely to occur in mammographic images and it effects the accuracy of classification. In this paper, Genetic Programming (GP) based novel hybrid ensemble method designed for noise amputation from breast mammogram images has been proposed. Three different filters Frost filter, Weiner filter and Non Local Means has been fused by using Genetic Programming for noise removal. The fused GP based filter calculate the value by combining three filters and replacing them with the noisy pixels. Peak signal to noise ratio (PSNR) and Structure Similarity Index measure (SSIM) as a quantitative measures has been employed for evaluation of proposed method with different existing methods. Experimental results show that proposed method bring forward better results as compared to existing methods. Keywords: Frost, genetic programming, non local mean, quantum noise, weiner.

Ensemble Sensitivity Analysis for Mesoscale Forecasts of Dryline Convection Initiation

AbstractTwo cases of dryline convection initiation (CI) over north Texas have been simulated (3 April 2012 and 15 May 2013) from a 50-member WRF-DART ensemble adjustment Kalman filter (EAKF) ensemble. In this study, ensemble sensitivity analysis (ESA) is applied to a convective forecast metric, maximum composite reflectivity (referred to as the response function), as a simple proxy for CI to analyze dynamic mesoscale sensitivities at the surface and aloft. Analysis reveals positional and magnitude sensitivities related to the strength and placement of important dynamic features. Convection initiation is sensitive to the evolving temperature and dewpoint fields upstream of the forecast response region in the near-CI time frame (0–12 h), prior to initiation. The sensitivity to thermodynamics is also manifest in the magnitude of dewpoint gradients along the dryline that triggers the convection. ESA additionally highlights the importance of antecedent precipitation and cold pool generation that modifies the pre-CI environment. Aloft, sensitivity of CI to a weak short-wave trough and capping inversion-level temperature is coherent, consistent, and traceable through the entire forecast period. Notwithstanding the (often) non-Gaussian distribution of ensemble member forecasts of convection, which violate the underpinnings of ESA theory, ESA is demonstrated to sufficiently identify regions that influence dryline CI. These results indicate an application of ESA for severe storm forecasting at operational centers and forecast offices as well as other mesoscale forecasting applications.

MeLiF: Filter Ensemble Learning Algorithm for Gene Selection

MeLiF: Filter Ensemble Learning Algorithm for Gene Selection

Novel mislabeled training data detection algorithm

As a kind of noise, mislabeled training data exist in many applications. Because of their negative effects on learning, many filter techniques have been proposed to identify and eliminate them. Ensemble learning-based filter (EnFilter) is the most widely used filter which employs ensemble classifiers. In EnFilter, first the noisy training dataset is divided into several subsets. Each noisy subset is then checked by the multiple classifiers which are trained based on other noisy subsets. It is noted that since the training data used to train multiple classifiers are noisy, the quality of these classifiers cannot be guaranteed, which might generate poor noise identification result. This problem is more serious when the noise ratio in the training dataset is high. To solve this problem, a straightforward but effective approach is proposed in this work. Instead of using noisy data to train the classifiers, nearly noise-free (NNF) data are used since they are supposed to train more reliable classifiers. To this end, a novel NNF data extraction approach is also proposed. Experimental results on a set of benchmark datasets illustrate the utility of our proposed approach.

A Hierarchical Bayes Ensemble Kalman Filter

A new ensemble filter that allows for the uncertainty in the prior distribution is proposed and tested. The filter relies on the conditional Gaussian distribution of the state given the model-error and predictability-error covariance matrices. The latter are treated as random matrices and updated in a hierarchical Bayes scheme along with the state. The (hyper)prior distribution of the covariance matrices is assumed to be inverse Wishart. The new Hierarchical Bayes Ensemble Filter (HBEF) assimilates ensemble members as generalized observations and allows ordinary observations to influence the covariances. The actual probability distribution of the ensemble members is allowed to be different from the true one. An approximation that leads to a practicable analysis algorithm is proposed. The new filter is studied in numerical experiments with a doubly stochastic one-variable model of “truth”. The model permits the assessment of the variance of the truth and the true filtering error variance at each time instance. The HBEF is shown to outperform the EnKF and the HEnKF by Myrseth and Omre (2010) in a wide range of filtering regimes in terms of performance of its primary and secondary filters.

Inflating transform matrices to mitigate assimilation errors with robust filtering based ensemble Kalman filters

AbstractAn error covariance matrix plays an important role in maintaining the statistical property of the ensemble in an ensemble Kalman filter method. However, data assimilation filter divergence may occur from an inaccurate estimate of the covariance matrix. In this study, based on an ensemble time‐local H‐infinity filter, which inflates the eigenvalues of the analysis error covariance matrix, a new robust ensemble data assimilation method is proposed, referred to as an inflation transform matrix eigenvalues. By design, new filters may be preferred over other traditional ensemble filters, when model performances are not well known, or change unpredictably. The primary aim is to improve the performance of the assimilation system in the framework of the ensemble filtering, according to the minimum/maximum rule of robust filtering. The proposed estimation method is tested using the well‐known Lorenz‐96 model, in order to investigate how the ensemble time‐local H‐infinity filter method of the inflation transform matrix impacts the robustness of the assimilation system under selected special conditions, such as the assimilation steps, force parameters, ensemble sizes, and observation information. The experiments show that the proposed inflation transform matrix method displays good robustness to the changes in the system's parameters. Also, when compared with the traditional filtering methods, this robust filtering method is found to improve the assimilation performance.

Ensembles of label noise filters: a ranking approach

Label noise can be a major problem in classification tasks, since most machine learning algorithms rely on data labels in their inductive process. Thereupon, various techniques for label noise identification have been investigated in the literature. The bias of each technique defines how suitable it is for each dataset. Besides, while some techniques identify a large number of examples as noisy and have a high false positive rate, others are very restrictive and therefore not able to identify all noisy examples. This paper investigates how label noise detection can be improved by using an ensemble of noise filtering techniques. These filters, individual and ensembles, are experimentally compared. Another concern in this paper is the computational cost of ensembles, once, for a particular dataset, an individual technique can have the same predictive performance as an ensemble. In this case the individual technique should be preferred. To deal with this situation, this study also proposes the use of meta-learning to recommend, for a new dataset, the best filter. An extensive experimental evaluation of the use of individual filters, ensemble filters and meta-learning was performed using public datasets with imputed label noise. The results show that ensembles of noise filters can improve noise filtering performance and that a recommendation system based on meta-learning can successfully recommend the best filtering technique for new datasets. A case study using a real dataset from the ecological niche modeling domain is also presented and evaluated, with the results validated by an expert.

An Ensemble of Filters and Wrappers for Microarray Data Classification

The development of microarray technology has supplied a large volume of data to many fields. The gene microarray analysis and classification have demonstrated an effective way for the effective diagnosis of diseases and cancers. In as much as the data achieving from microarray technology is very noisy and also has thousands of features, feature selection plays an important role in removing irrelevant and redundant features and also reducing computational complexity. There are two important approaches for gene selection in microarray data analysis, the filters and the wrappers. To select a concise subset of informative genes, we introduce a hybrid feature selection which combines two approaches. The fact of the matter is that candidate’s features are first selected from the original set via several effective filters. The candidate feature set is further refined by more accurate wrappers. Thus, we can take advantage of both the filters and wrappers. Experimental results based on 11 microarray datasets show that our mechanism can be effected with a smaller feature set. Moreover, these feature subsets can be obtained in a reasonable time.

Localizing the impact of satellite radiance observations using a global group ensemble filter

AbstractAssimilation of satellite radiances has been proven to have positive impacts on the forecast skill, especially for regions with sparse conventional observations. Localization is an essential component to effectively assimilate satellite radiances in ensemble Kalman filters with affordable ensemble sizes. However, localizing the impact of radiance observations is not straightforward, since their location and separation from grid point model variables are not well defined. A global group filter (GGF) is applied here to provide a theoretical estimate of vertical localization functions for radiance observations being assimilated for global numerical weather prediction. As an extension of the hierarchical ensemble filter, the GGF uses groups of climatological ensembles to provide an estimated localization function that reduces the erroneous increments due to ensemble correlation sampling error. Results from an idealized simulation with known background error covariances show that the GGF localization function is superior to the optimal Gaspari and Cohn (GC) localization function. When the GGF is applied to the AMSU‐A radiances, it can provide different localization functions for different channels, which indicates the complexity and large computational cost of tuning the localization scales for radiance observations. The GC, GGF, and fitted GGF (FGGF) localization functions are compared using experiments with the NCEP GFS and the NOAA operational EnKF. Verifications relative to the conventional observations, AMSU‐A radiances, and the ECMWF analyses show that the GGF and FGGF have smaller errors than GC except in the tropics, and the advantages of the GGF and FGGF persist through 120 h forecast lead time.

The Role of Large-Scale Feedbacks in Cumulus Convection Parameter Estimation

AbstractUncertainty in cumulus convection parameterization is one of the most important causes of model climate drift through interactions between large-scale background and local convection that use empirically set parameters. Without addressing the large-scale feedback, the calibrated parameter values within a convection scheme are usually not optimal for a climate model. This study first designs a multiple-column atmospheric model that includes large-scale feedbacks for cumulus convection and then explores the role of large-scale feedbacks in cumulus convection parameter estimation using an ensemble filter. The performance of convection parameter estimation with or without the presence of large-scale feedback is examined. It is found that including large-scale feedbacks in cumulus convection parameter estimation can significantly improve the estimation quality. This is because large-scale feedbacks help transform local convection uncertainties into global climate sensitivities, and including these feedbacks enhances the statistical representation of the relationship between parameters and state variables. The results of this study provide insights for further understanding of climate drift induced from imperfect cumulus convection parameterization, which may help improve climate modeling.

Multiscale Estimation of Leaf Area Index from Satellite Observations Based on an Ensemble Multiscale Filter

Currently, multiple leaf area index (LAI) products retrieved from remote sensing data are widely used in crop growth monitoring, land-surface process simulation and studies of climate change. However, most LAI products are only retrieved from individual satellite observations, which may result in spatial-temporal discontinuities and low accuracy in these products. In this paper, a new method was developed to simultaneously retrieve multiscale LAI data from satellite observations with different spatial resolutions based on an ensemble multiscale filter (EnMsF). The LAI average values corresponding to the date of satellite observations were calculated from the multi-year Moderate Resolution Imaging Spectroradiometer (MODIS) LAI product and were used as a priori knowledge for LAI in order to construct an initial ensemble multiscale tree (EnMsT). Satellite observations obtained at different spatial resolutions were then applied to update the LAI values at each node of the EnMsT using a two-sweep filtering procedure. Next, the retrieved LAI values at the finest scale were used as a priori knowledge for LAI for the new round of construction and updating of the EnMsT, until the sum of the difference of LAI values at each node of the EnMsT between two adjacent updates is less than a given threshold. The method was tested using Thematic Mapper (TM) or Enhanced Thematic Mapper Plus (ETM+) surface reflectance data and MODIS surface reflectance data from five sites that have different vegetation types. The results demonstrate that the retrieved LAI values for each spatial resolution were in good agreement with the aggregated LAI reference map values for the corresponding spatial resolution. The retrieved LAI values at the coarsest scale provided better accuracy with the aggregated LAI reference map values (root mean square error (RMSE) = 0.45) compared with that obtained from the MODIS LAI values (RMSE = 1.30).

Comparison and combination of EAKF and SIR-PF in the Bayesian filter framework

Bayesian estimation theory provides a general approach for the state estimate of linear or nonlinear and Gaussian or non-Gaussian systems. In this study, we first explore two Bayesian-based methods: ensemble adjustment Kalman filter (EAKF) and sequential importance resampling particle filter (SIR-PF), using a well-known nonlinear and non-Gaussian model (Lorenz '63 model). The EAKF, which is a deterministic scheme of the ensemble Kalman filter (EnKF), performs better than the classical (stochastic) EnKF in a general framework. Comparison between the SIR-PF and the EAKF reveals that the former outperforms the latter if ensemble size is so large that can avoid the filter degeneracy, and vice versa. The impact of the probability density functions and effective ensemble sizes on assimilation performances are also explored. On the basis of comparisons between the SIR-PF and the EAKF, a mixture filter, called ensemble adjustment Kalman particle filter (EAKPF), is proposed to combine their both merits. Similar to the ensemble Kalman particle filter, which combines the stochastic EnKF and SIR-PF analysis schemes with a tuning parameter, the new mixture filter essentially provides a continuous interpolation between the EAKF and SIR-PF. The same Lorenz '63 model is used as a testbed, showing that the EAKPF is able to overcome filter degeneracy while maintaining the non-Gaussian nature, and performs better than the EAKF given limited ensemble size.

Reducing Correlation Sampling Error in Ensemble Kalman Filter Data Assimilation

Abstract Ensemble Kalman filters are widely used for data assimilation in large geophysical models. Good results with affordable ensemble sizes require enhancements to the basic algorithms to deal with insufficient ensemble variance and spurious ensemble correlations between observations and state variables. These challenges are often dealt with by using inflation and localization algorithms. A new method for understanding and reducing some ensemble filter errors is introduced and tested. The method assumes that sampling error due to small ensemble size is the primary source of error. Sampling error in the ensemble correlations between observations and state variables is reduced by estimating the distribution of correlations as part of the ensemble filter algorithm. This correlation error reduction (CER) algorithm can produce high-quality ensemble assimilations in low-order models without using any a priori localization like a specified localization function. The method is also applied in an observing system simulation experiment with a very coarse resolution dry atmospheric general circulation model. This demonstrates that the algorithm provides insight into the need for localization in large geophysical applications, suggesting that sampling error may be a primary cause in some cases.

Nonlinear stability and ergodicity of ensemble based Kalman filters

The ensemble Kalman filter (EnKF) and ensemble square root filter (ESRF) are data assimilation methods used to combine high dimensional, nonlinear dynamical models with observed data. Despite their widespread usage in climate science and oil reservoir simulation, very little is known about the long-time behavior of these methods and why they are effective when applied with modest ensemble sizes in large dimensional turbulent dynamical systems. By following the basic principles of energy dissipation and controllability of filters, this paper establishes a simple, systematic and rigorous framework for the nonlinear analysis of EnKF and ESRF with arbitrary ensemble size, focusing on the dynamical properties of boundedness and geometric ergodicity. The time uniform boundedness guarantees that the filter estimate will not diverge to machine infinity in finite time, which is a potential threat for EnKF and ESQF known as the catastrophic filter divergence. Geometric ergodicity ensures in addition that the filter has a unique invariant measure and that initialization errors will dissipate exponentially in time. We establish these results by introducing a natural notion of observable energy dissipation. The time uniform bound is achieved through a simple Lyapunov function argument, this result applies to systems with complete observations and strong kinetic energy dissipation, but also to concrete examples with incomplete observations. With the Lyapunov function argument established, the geometric ergodicity is obtained by verifying the controllability of the filter processes; in particular, such analysis for ESQF relies on a careful multivariate perturbation analysis of the covariance eigen-structure.

Impact of VORTEX2 Observations on Analyses and Forecasts of the 5 June 2009 Goshen County, Wyoming, Supercell

Abstract Several data assimilation and forecast experiments are undertaken to determine the impact of special observations taken during the second Verification of the Origins of Rotation in Tornadoes Experiment (VORTEX2) on forecasts of the 5 June 2009 Goshen County, Wyoming, supercell. The data used in these experiments are those from the Mobile Weather Radar, 2005 X-band, Phased Array (MWR-05XP); two mobile mesonets (MM); and several mobile sounding units. Data sources are divided into “routine,” including those from operational Weather Surveillance Radar-1988 Dopplers (WSR-88Ds) and the Automated Surface Observing System (ASOS) network, and “special” observations from the VORTEX2 project. VORTEX2 data sources are denied individually from a total of six ensemble square root filter (EnSRF) data assimilation and forecasting experiments. The EnSRF data assimilation uses 40 ensemble members on a 1-km grid nested inside a 3-km grid. Each experiment assimilates data every 5 min for 1 h, followed by a 1-h forecast. All experiments are able to reproduce the basic evolution of the supercell, though the impact of the VORTEX2 observations was mixed. The VORTEX2 sounding data decreased the mesocyclone intensity in the latter stages of the forecast, consistent with observations. The MWR-05XP data increased the forecast vorticity above approximately 1 km AGL in all experiments and had little impact on forecast vorticity below 1 km AGL. The MM data had negative impacts on the intensity of the low-level mesocyclone, by decreasing the vertical vorticity and indirectly by decreasing the buoyancy of the inflow.