Ensemble Member Selection Research Articles

Characterizing Spatial Structure in Climate Model Ensembles

Abstract This paper presents a methodology that is designed for rapid exploratory analysis of the outputs from ensembles of climate models, especially when these outputs consist of maps. The approach formalizes and extends the technique of “intermodel empirical orthogonal function” analysis, combining multivariate analysis of variance techniques with singular value decompositions (SVDs) of structured components of the ensemble data matrix. The SVDs yield spatial patterns associated with these components, which we call ensemble principal patterns (EPPs). A unique hierarchical partitioning of variation is obtained for balanced ensembles in which all combinations of factors, such as GCM and RCM pairs in a regional ensemble, appear with equal frequency: suggestions are also proposed to handle unbalanced ensembles without imputing missing values or discarding runs. Applications include the selection of ensemble members to propagate uncertainty into subsequent analyses, and the diagnosis of modes of variation associated with specific model variants or parameter perturbations. The approach is illustrated using outputs from the EuroCORDEX regional ensemble over the United Kingdom.

Prioritizing the selection of CMIP6 model ensemble members for downscaling projections of CONUS temperature and precipitation

Given the mismatch between the large volume of data archived for the sixth phase of the Coupled Model Intercomparison Project (CMIP6) and limited personnel and computational resources for downscaling, only a small fraction of the CMIP6 archive can be downscaled. In this work, we develop an approach to robustly sample projected hydroclimate states in CMIP6 for downscaling to test whether the selection of a single initial condition (IC) ensemble member from each CMIP6 model is sufficient to span the range of modeled hydroclimate over the conterminous United States (CONUS) and CONUS sub-regions. We calculate the pattern-centered root mean square difference of IC ensemble member anomalies relative to each model’s historical climatology for shared socioeconomic pathway (SSP) projections over 30-year time periods and compare the ratio of inter-model to intra-model variability for this metric. Regardless of SSP, inter-model variability is generally much greater than intra-model variability at the scales of the CONUS as a whole, as well as for most CONUS sub-regions. However for some variables and scenarios, inter- and intra-model variability are similar at sub-CONUS scales, indicating that selecting a single IC ensemble member per model may be sufficient to sample the range of projected hydroclimate states in the 21st Century across CONUS, but for specific regions and variables, more careful selection of ensemble members may be necessary. Regionally-resolved Taylor diagrams identify where more IC ensemble member downscaling efforts should be focused if resources are available to do so. Our results suggest that, with parsimonious sampling, the requisite computational expense of downscaling temperature and precipitation fields over the CONUS for subsequent CMIP activities may increase only marginally despite the great increase in data volumes with each successive CMIP phase.

Structural Ensemble Regression for Cluster-Based Aggregate Electricity Demand Forecasting

Accurate electricity demand forecasting is vital to the development and evolution of smart grids as well as the reinforcement of demand side management strategies in the energy sector. Since this forecasting task requires the efficient processing of load profiles extracted from smart meters for large sets of clients, the challenges of high dimensionality often lead to the adoption of cluster-based aggregation strategies, resulting in scalable estimation models that operate on aggregate times series formed by client groups that share similar load characteristics. However, it is evident that the clustered time series exhibit different patterns that may not be processed efficiently by a single estimator or a fixed hybrid structure. Therefore, ensemble learning methods could provide an additional layer of model fusion, enabling the resulting estimator to adapt to the input series and yield better performance. In this work, we propose an adaptive ensemble member selection approach for stacking and voting regressors in the cluster-based aggregate forecasting framework that focuses on the examination of forecasting performance on peak and non-peak observations for the development of structurally flexible estimators for each cluster. The resulting ensemble models yield better overall performance when compared to the standalone estimators and our experiments indicate that member selection strategies focusing on the influence of non-peak performance lead to more performant ensemble models in this framework.

Hydrologic multi-model ensemble predictions using variational Bayesian deep learning

Multi-model ensembles enable assessment of model structural uncertainty across multiple disciplines. Bayesian Model Averaging (BMA) is one of the most popular ensemble averaging approaches in hydrology but its predictions are easily impacted by the type of ensemble members selected. Here, we propose a regression-based ensemble approach, namely a Variational Bayesian Long Short-Term Memory network (VB-LSTM) to address this issue. In this approach, a state-of-the-art variational inference (VI) algorithm that is faster and more scalable than Bayesian Markov chain Monte Carlo (MCMC) is employed to approximate the posterior distributions of thousands of parameters in the LSTM networks. To interpret the behavior of deep learning methods, the Permutation Feature Importance (PFI) algorithm is introduced to understand the degree to which VB-LSTM relies on each ensemble member. Twenty conceptual hydrological models are considered to evaluate BMA and VB-LSTM in four catchments from China. Four scenarios with different ensemble members are established to investigate the impacts of ensemble members on model results. Our results show that compared with BMA, VB-LSTM improves deterministic and probabilistic predictions by approximately 10%–30% in terms of Mean Absolute Error (MAE), Sharpness and Continous Ranked Probability Score (CRPS). In addition, the VB-LSTM predictions are more robust and less impacted by the selection of ensemble members. Furthermore, our study encourages the use of Bayesian deep learning in hydrology as an alternative to other approaches tackling model structural uncertainty.

Earth System Model Evaluation Tool (ESMValTool) v2.0 – diagnostics for extreme events, regional and impact evaluation, and analysis of Earth system models in CMIP

Abstract. This paper complements a series of now four publications that document the release of the Earth System Model Evaluation Tool (ESMValTool) v2.0. It describes new diagnostics on the hydrological cycle, extreme events, impact assessment, regional evaluations, and ensemble member selection. The diagnostics are developed by a large community of scientists aiming to facilitate the evaluation and comparison of Earth system models (ESMs) which are participating in the Coupled Model Intercomparison Project (CMIP). The second release of this tool aims to support the evaluation of ESMs participating in CMIP Phase 6 (CMIP6). Furthermore, datasets from other models and observations can be analysed. The diagnostics for the hydrological cycle include several precipitation and drought indices, as well as hydroclimatic intensity and indices from the Expert Team on Climate Change Detection and Indices (ETCCDI). The latter are also used for identification of extreme events, for impact assessment, and to project and characterize the risks and impacts of climate change for natural and socio-economic systems. Further impact assessment diagnostics are included to compute daily temperature ranges and capacity factors for wind and solar energy generation. Regional scales can be analysed with new diagnostics implemented for selected regions and stochastic downscaling. ESMValTool v2.0 also includes diagnostics to analyse large multi-model ensembles including grouping and selecting ensemble members by user-specified criteria. Here, we present examples for their capabilities based on the well-established CMIP Phase 5 (CMIP5) dataset.

СТЕКІНГ НЕЙРОПОДІБНОЇ СТРУКТУРИ МПГП З RBF ШАРОМ НА ПІДСТАВІ ГЕНЕРУВАННЯ ВИПАДКОВОГО КОРТЕЖУ ЇЇ ГІПЕРПАРАМЕТРІВ ДЛЯ ЗАВДАНЬ ПРОГНОЗУВАННЯ

Improving prediction accuracy by artificial intelligence tools is an important task in various industries, economics, medicine. Ensemble learning is one of the possible options to solve this task. In particular, the construction of stacking models based on different machine learning methods, or using different parts of the existing data set demonstrates high prediction accuracy of the. However, the need for proper selection of ensemble members, their optimal parameters, etc., necessitates large time costs for the construction of such models. This paper proposes a slightly different approach to building a simple but effective ensemble method. The authors developed a new model of stacking of nonlinear SGTM neural-like structures, which is based on the use of only one type of ANN as an element base of the ensemble and the use of the same training sample for all members of the ensemble. This approach provides a number of advantages over the procedures for building ensembles based on different machine learning methods, at least in the direction of selecting the optimal parameters for each of them. In our case, a tuple of random hyperparameters for each individual member of the ensemble was used as the basis of ensemble. That is, the training of each combined SGTM neural-like structure with an additional RBF layer, as a separate member of the ensemble occurs using different, randomly selected values of RBF centers and centersfof mass. This provides the necessary variety of ensemble elements. Experimental studies on the effectiveness of the developed ensemble were conducted using a real data set. The task is to predict the amount of health insurance costs based on a number of independent attributes. The optimal number of ensemble members is determined experimentally, which provides the highest prediction accuracy. The results of the work of the developed ensemble are compared with the existing methods of this class. The highest prediction accuracy of the developed ensemble at satisfactory duration of procedure of its training is established.

Streamflow-based evaluation of climate model sub-selection methods

The assessment of climate change and its impact relies on the ensemble of models available and/or sub-selected. However, an assessment of the validity of simulated climate change impacts is not straightforward because historical data is commonly used for bias-adjustment, to select ensemble members or to define a baseline against which impacts are compared—and, naturally, there are no observations to evaluate future projections. We hypothesize that historical streamflow observations contain valuable information to investigate practices for the selection of model ensembles. The Danube River at Vienna is used as a case study, with EURO-CORDEX climate simulations driving the COSERO hydrological model. For each selection method, we compare observed to simulated streamflow shift from the reference period (1960–1989) to the evaluation period (1990–2014). Comparison against no selection shows that an informed selection of ensemble members improves the quantification of climate change impacts. However, the selection method matters, with model selection based on hindcasted climate or streamflow alone is misleading, while methods that maintain the diversity and information content of the full ensemble are favorable. Prior to carrying out climate impact assessments, we propose splitting the long-term historical data and using it to test climate model performance, sub-selection methods, and their agreement in reproducing the indicator of interest, which further provide the expectable benchmark of near- and far-future impact assessments. This test is well-suited to be applied in multi-basin experiments to obtain better understanding of uncertainty propagation and more universal recommendations regarding uncertainty reduction in hydrological impact studies.

Ensemble learning by means of a multi-objective optimization design approach for dealing with imbalanced data sets

Ensemble learning methods have already shown to be powerful techniques for creating classifiers. However, when dealing with real-world engineering problems, class imbalance is usually found. In such scenario, canonical machine learning algorithms may not present desirable solutions, and techniques for overcoming this problem must be used. In addition to using learning algorithms that alleviate the imbalance between classes, multi-objective optimization design (MOOD) approaches can be used to improve the prediction performance of ensembles of classifiers. This paper proposes a study of different MOOD approaches for ensemble learning. First, a taxonomy on multi-objective ensemble learning (MOEL) is proposed. In it, four types of existing approaches are defined: multi-objective ensemble member generation, multi-objective ensemble member selection, multi-objective ensemble member combination, and multi-objective ensemble member selection and combination. Additionally, new approaches can be derived by combining the previous ones, such as multi-objective ensemble member generation and selection, multi-objective ensemble member generation and combination and multi-objective ensemble member generation, selection and combination. With the given taxonomy, two experiments are conducted for comparing (1) the performance of the MOEL techniques for generating and aggregating base models on several imbalanced benchmark problems and (2) the performance of MOEL techniques against other machine learning techniques in a real-world imbalanced drinking-water quality anomaly detection problem. Finally, results indicate that MOOD is able to improve the predictive performance of existing ensemble learning techniques.

A Stochastic Data‐Driven Ensemble Forecasting Framework for Water Resources: A Case Study Using Ensemble Members Derived From a Database of Deterministic Wavelet‐Based Models

AbstractIn water resources applications (e.g., streamflow, rainfall‐runoff, urban water demand [UWD], etc.), ensemble member selection and ensemble member weighting are two difficult yet important tasks in the development of ensemble forecasting systems. We propose and test a stochastic data‐driven ensemble forecasting framework that uses archived deterministic forecasts as input and results in probabilistic water resources forecasts. In addition to input data and (ensemble) model output uncertainty, the proposed approach integrates both ensemble member selection and weighting uncertainties, using input variable selection and data‐driven methods, respectively. Therefore, it does not require one to perform ensemble member selection and weighting separately. We applied the proposed forecasting framework to a previous real‐world case study in Montreal, Canada, to forecast daily UWD at multiple lead times. Using wavelet‐based forecasts as input data, we develop the Ensemble Wavelet‐Stochastic Data‐Driven Forecasting Framework, the first multiwavelet ensemble stochastic forecasting framework that produces probabilistic forecasts. For the considered case study, several variants of Ensemble Wavelet‐Stochastic Data‐Driven Forecasting Framework, produced using different input variable selection methods (partial correlation input selection and Edgeworth Approximations‐based conditional mutual information) and data‐driven models (multiple linear regression, extreme learning machines, and second‐order Volterra series models), are shown to outperform wavelet‐ and nonwavelet‐based benchmarks, especially during a heat wave (first time studied in the UWD forecasting literature).

Influence of proxy data uncertainty on data assimilation for the past climate

Abstract. Data assimilation (DA) is an emerging topic in palaeoclimatology and one of the key challenges in this field. Assimilating proxy-based continental mean temperature reconstructions into the MPI-ESM model showed a lack of information propagation to small spatial scales . Here, we investigate whether this lack of regional skill is due to the methodology or to errors in the assimilated reconstructions. Error separation is fundamental, as it can lead to improvements in DA methods. We address the question by performing a new set of simulations, using two different sets of target data; the proxy-based PAGES 2K reconstructions (DA-P scheme), and the HadCRUT3v instrumental observations (DA-I scheme). Again, we employ ensemble-member selection DA using the MPI-ESM model, and assimilate Northern Hemisphere (NH) continental mean temperatures; the simulated period is 1850–1949 AD. Both DA schemes follow the large-scale target and observed climate variations well, but the assimilation of instrumental data improves the performance. This improvement cannot be seen for Asia, where the limited instrumental coverage leads to errors in the target data and low skill for the DA-I scheme. No skill on small spatial scales is found for either of the two DA schemes, demonstrating that errors in the assimilated data are not the main reason for the unrealistic representation of the regional temperature variability in Europe and the NH. It can thus be concluded that assimilating continental mean temperatures is not ideal for providing skill on small spatial scales.

Incremental Semi-Supervised Clustering Ensemble for High Dimensional Data Clustering

Traditional cluster ensemble approaches have three limitations: ( $1$ ) They do not make use of prior knowledge of the datasets given by experts. ( $2$ ) Most of the conventional cluster ensemble methods cannot obtain satisfactory results when handling high dimensional data. ( $3$ ) All the ensemble members are considered, even the ones without positive contributions. In order to address the limitations of conventional cluster ensemble approaches, we first propose an incremental semi-supervised clustering ensemble framework (ISSCE) which makes use of the advantage of the random subspace technique, the constraint propagation approach, the proposed incremental ensemble member selection process, and the normalized cut algorithm to perform high dimensional data clustering. The random subspace technique is effective for handling high dimensional data, while the constraint propagation approach is useful for incorporating prior knowledge. The incremental ensemble member selection process is newly designed to judiciously remove redundant ensemble members based on a newly proposed local cost function and a global cost function, and the normalized cut algorithm is adopted to serve as the consensus function for providing more stable, robust, and accurate results. Then, a measure is proposed to quantify the similarity between two sets of attributes, and is used for computing the local cost function in ISSCE. Next, we analyze the time complexity of ISSCE theoretically. Finally, a set of nonparametric tests are adopted to compare multiple semi-supervised clustering ensemble approaches over different datasets. The experiments on 18 real-world datasets, which include six UCI datasets and 12 cancer gene expression profiles, confirm that ISSCE works well on datasets with very high dimensionality, and outperforms the state-of-the-art semi-supervised clustering ensemble approaches.

Dynamic Selection of Ensemble Members in Multi-model Hydrometeorological Ensemble Forecasting

Abstract Multi-model prediction ensembles show significant ability to improve forecasts. Nevertheless, the set of models in an ensemble is not always optimal. This work proposes a procedure that allows to select dynamically ensemble members for each forecast. Proposed procedure was evaluated for the task of the water level forecasting in the Baltic See. The regression-based estimation of ensemble forecasts errors was used to implement the selection procedure. Improvement of the forecast quality in terms of mean forecast RMS error and mean forecast skill score are demonstrated.

Hierarchical cluster ensemble selection

Clustering ensemble performance is affected by two main factors: diversity and quality. Selection of a subset of available ensemble members based on diversity and quality often leads to a more accurate ensemble solution. However, there is not a certain relationship between diversity and quality in selection of subset of ensemble members. This paper proposes the Hierarchical Cluster Ensemble Selection (HCES) method and diversity measure to explore how diversity and quality affect final results. The HCES uses single-link, average-link, and complete link agglomerative clustering methods for the selection of ensemble members hierarchically. A pair-wise diversity measure from the recent literature and the proposed diversity measure are applied to these agglomerative clustering algorithms. Using the proposed diversity measure in HCES leads to more diverse ensemble members than that of pairwise diversity measure. Cluster-based Similarity Partition Algorithm (CSPA) and Hypergraph-Partitioning Algorithm (HGPA) were employed in HCES method for obtaining the full ensemble and cluster ensemble selection solution. To evaluate the performance of the HCES method, several experiments were conducted on several real data sets and the obtained results were compared to those of full ensembles. The results showed that the HCES method led to a more significant performance improvement compared with full ensembles.

Comparative study of matrix refinement approaches for ensemble clustering

Cluster ensembles or consensus clusterings have been shown to be better than any standard clustering algorithm at improving accuracy and robustness across various sets of data. This meta-learning formalism also helps users to overcome the dilemma of selecting an appropriate technique and the parameters for that technique. Since founded, different research areas have emerged with the common purpose of enhancing the effectiveness and applicability of cluster ensembles. These include the selection of ensemble members, the imputation of missing values, and the summarization of ensemble members. In particular, this paper is set to provide the review of different matrix refinement approaches that have been recently proposed in the literature for summarizing information of multiple clusterings. With various benchmark datasets and quality measures, the comparative study of these novel techniques is carried out to provide empirical findings from which a practical guideline can be drawn.

Impact of Bias-Correction Type and Conditional Training on Bayesian Model Averaging over the Northeast United States

Abstract The performance of a multimodel ensemble over the northeast United States is evaluated before and after applying bias correction and Bayesian model averaging (BMA). The 13-member Stony Brook University (SBU) ensemble at 0000 UTC is combined with the 21-member National Centers for Environmental Prediction (NCEP) Short-Range Ensemble Forecast (SREF) system at 2100 UTC. The ensemble is verified using 2-m temperature and 10-m wind speed for the 2007–09 warm seasons, and for subsets of days with high ozone and high fire threat. The impacts of training period, bias-correction method, and BMA are explored for these potentially hazardous weather events using the most recent consecutive (sequential training) and most recent similar days (conditional training). BMA sensitivity to the selection of ensemble members is explored. A running mean difference between forecasts and observations using the last 14 days is better at removing temperature bias than is a cumulative distribution function (CDF) or linear regression approach. Wind speed bias is better removed by adjusting the modeled CDF to the observation. High fire threat and ozone days exhibit a larger cool bias and a greater negative wind speed bias than the warm-season average. Conditional bias correction is generally better at removing temperature and wind speed biases than sequential training. Greater probabilistic skill is found for temperature using both conditional bias correction and BMA compared to sequential bias correction with or without BMA. Conditional and sequential BMA results are similar for 10-m wind speed, although BMA typically improves probabilistic skill regardless of training.

Using data assimilation to study extratropical Northern Hemisphere climate over the last millennium

Abstract. Climate proxy data provide noisy, and spatially incomplete information on some aspects of past climate states, whereas palaeosimulations with climate models provide global, multi-variable states, which may however differ from the true states due to unpredictable internal variability not related to climate forcings, as well as due to model deficiencies. Using data assimilation for combining the empirical information from proxy data with the physical understanding of the climate system represented by the equations in a climate model is in principle a promising way to obtain better estimates for the climate of the past. Data assimilation has been used for a long time in weather forecasting and atmospheric analyses to control the states in atmospheric General Circulation Models such that they are in agreement with observation from surface, upper air, and satellite measurements. Here we discuss the similarities and the differences between the data assimilation problem in palaeoclimatology and in weather forecasting, and present and conceptually compare three data assimilation methods that have been developed in recent years for applications in palaeoclimatology. All three methods (selection of ensemble members, Forcing Singular Vectors, and Pattern Nudging) are illustrated by examples that are related to climate variability over the extratropical Northern Hemisphere during the last millennium. In particular it is shown that all three methods suggest that the cold period over Scandinavia during 1790–1820 is linked to anomalous northerly or easterly atmospheric flow, which in turn is related to a pressure anomaly that resembles a negative state of the Northern Annular Mode.

Developing an SVM-based ensemble learning system for customer risk identification collaborating with customer relationship management

In this study, we propose a support vector machine (SVM)-based ensemble learning system for customer relationship management (CRM) to help enterprise managers effectively manage customer risks from the risk aversion perspective. This system differs from the classical CRM for retaining and targeting profitable customers; the main focus of the proposed SVM-based ensemble learning system is to identify high-risk customers in CRM for avoiding possible loss. To build an effective SVM-based ensemble learning system, the effects of ensemble members’ diversity, ensemble member selection and different ensemble strategies on the performance of the proposed SVM-based ensemble learning system are each investigated in a practical CRM case. Through experimental analysis, we find that the Bayesian-based SVM ensemble learning system with diverse components and choose from space selection strategy show the best performance over various testing samples.

Overfitting cautious selection of classifier ensembles with genetic algorithms

Information fusion research has recently focused on the characteristics of the decision profiles of ensemble members in order to optimize performance. These characteristics are particularly important in the selection of ensemble members. However, even though the control of overfitting is a challenge in machine learning problems, much less work has been devoted to the control of overfitting in selection tasks. The objectives of this paper are: (1) to show that overfitting can be detected at the selection stage; and (2) to present strategies to control overfitting. Decision trees and k nearest neighbors classifiers are used to create homogeneous ensembles, while single- and multi-objective genetic algorithms are employed as search algorithms at the selection stage. In this study, we use bagging and random subspace methods for ensemble generation. The classification error rate and a set of diversity measures are applied as search criteria. We show experimentally that the selection of classifier ensembles conducted by genetic algorithms is prone to overfitting, especially in the multi-objective case. In this study, the partial validation, backwarding and global validation strategies are tailored for classifier ensemble selection problem and compared. This comparison allows us to show that a global validation strategy should be applied to control overfitting in pattern recognition systems involving an ensemble member selection task. Furthermore, this study has helped us to establish that the global validation strategy can be used to measure the relationship between diversity and classification performance when diversity measures are employed as single-objective functions.

RBF Networks Ensemble Construction based on Evolutionary Multi-objective Optimization

The ensemble learning has attracted much attention over the last decade. While constructing RBF network ensemble the generally encountered problems are how to construct diverse RBF networks and how to combine outputs. The construction of RBF network can be considered as a multi-objective optimization problem regarding model complexity. A set of RBF networks which is multi-objectively optimized can be obtained by solving the above-mentioned problem. In this paper the construction of RBF networks by evolutionary multi-objective optimization method and its ensemble are considered, and it is applied to the pattern classification problem. Also some ensemble member selection methods and output combination methods are considered. Experimental study on the benchmark problem of pattern classification is carried out; then it is illustrated that the RBF network ensemble has a performance, which is comparable to that of other ensemble methods.

Linear Regime Duration: Is 24 Hours a Long Time in Synoptic Weather Forecasting?

Day-to-day variations in the growth of uncertainty in the current state of the atmosphere have led to operational ensemble weather predictions in which an ensemble of different initial conditions, each perturbed from the best estimate of the current state and yet still consistent with the observations, is forecast. Contrasting competing methods for the selection of ensemble members is a subject of active research; the assumption that the ensemble members represent sufficiently small perturbations so as to evolve within the ‘‘linear regime’’ is implicit to several of these methods. This regime, in which the model dynamics are well represented by a linear approximation, is commonly held to extend to 2 or 3 days for operational forecasts. It is shown that this is rarely the case. A new measure, the relative nonlinearity, which quantifies the duration of the linear regime by monitoring the evolution of ‘‘twin’’ pairs of ensemble members, is introduced. Both European and American ensemble prediction systems are examined; in the cases considered for each system (87 and 25, respectively), the duration of the linear regime is often less than a day and never extends to 2 days. The internal consistency of operational ensemble formation schemes is discussed in light of these results. By decreasing the optimization time, a modified singular vector‐based formation scheme is shown to improve consistency while maintaining traditional skill and spread scores in the seven cases considered. The relevance of the linear regime to issues regarding data assimilation, adaptive observations, and model sensitivity is also noted.