Diverse Ensemble Research Articles

Prune and Tune Ensembles: Low-Cost Ensemble Learning with Sparse Independent Subnetworks

Ensemble Learning is an effective method for improving generalization in machine learning. However, as state-of-the-art neural networks grow larger, the computational cost associated with training several independent networks becomes expensive. We introduce a fast, low-cost method for creating diverse ensembles of neural networks without needing to train multiple models from scratch. We do this by first training a single parent network. We then create child networks by cloning the parent and dramatically pruning the parameters of each child to create an ensemble of members with unique and diverse topologies. We then briefly train each child network for a small number of epochs, which now converge significantly faster when compared to training from scratch. We explore various ways to maximize diversity in the child networks, including the use of anti-random pruning and one-cycle tuning. This diversity enables "Prune and Tune" ensembles to achieve results that are competitive with traditional ensembles at a fraction of the training cost. We benchmark our approach against state of the art low-cost ensemble methods and display marked improvement in both accuracy and uncertainty estimation on CIFAR-10 and CIFAR-100.

Knowledge distillation with ensembles of convolutional neural networks for medical image segmentation.

Purpose: Ensembles of convolutional neural networks (CNNs) often outperform a single CNN in medical image segmentation tasks, but inference is computationally more expensive and makes ensembles unattractive for some applications. We compared the performance of differently constructed ensembles with the performance of CNNs derived from these ensembles using knowledge distillation, a technique for reducing the footprint of large models such as ensembles. Approach: We investigated two different types of ensembles, namely, diverse ensembles of networks with three different architectures and two different loss-functions, and uniform ensembles of networks with the same architecture but initialized with different random seeds. For each ensemble, additionally, a single student network was trained to mimic the class probabilities predicted by the teacher model, the ensemble. We evaluated the performance of each network, the ensembles, and the corresponding distilled networks across three different publicly available datasets. These included chest computed tomography scans with four annotated organs of interest, brain magnetic resonance imaging (MRI) with six annotated brain structures, and cardiac cine-MRI with three annotated heart structures. Results: Both uniform and diverse ensembles obtained better results than any of the individual networks in the ensemble. Furthermore, applying knowledge distillation resulted in a single network that was smaller and faster without compromising performance compared with the ensemble it learned from. The distilled networks significantly outperformed the same network trained with reference segmentation instead of knowledge distillation. Conclusion: Knowledge distillation can compress segmentation ensembles of uniform or diverse composition into a single CNN while maintaining the performance of the ensemble.

Enhancing ensemble diversity based on multiscale dilated convolution in image classification

Convolutional neural networks (CNNs) have achieved extraordinary success on many image classification tasks in recent years. The use of dilated convolution in a CNN can increase the network’s receptive field and improve its performance, and dilated convolution can also be used to compress a CNN to realize a lightweight model. In previous studies, multiscale dilated convolution has been adopted with a focus on improving the internal network structure of a specific CNN model. Because they enable the direct use of pretrained models, transfer learning CNNs (TL-CNNs) have been widely applied for image recognition based on small datasets. This paper proposes a novel multiscale dilated-convolution-based ensemble learning (MDCEL) method for effectively improving the performance of a pretrained CNN model. The authors' primary assumption is that semantic representations of different images can be obtained based on multiscale dilated convolution. Therefore, constructing an ensemble of diverse TL-CNN classifiers makes it possible to achieve higher performance than that offered by the traditional TL-CNN methods. The MDCEL method is highly versatile and can be applied to various conventional pretrained CNN models and lightweight CNN models. Moreover, this method does not require the modification of the internal structures of the pretrained CNN models and has high training efficiency. Experimental results on three public image classification datasets demonstrate that the proposed method outperforms the baseline traditional TL-CNN method. Compared with the baseline approach, the MDCEL approach improves the accuracy and F1 values by nearly 1–4%. In addition, an experiment on a real case dataset obtained from a manufacturing enterprise further proves the practicability of the proposed method.

Dynamic selective Gaussian process regression for forecasting temperature of molten steel in ladle furnace

The requirement for intelligent steelmaking has underlined the significance of data-driven predictions of molten steel temperature in ladle furnace. Recently, predictors based on ensemble learning have shown their superiority over single ones. However, the strong reliability on the ensemble diversity can hardly insure their generalization ability. Moreover, most existing predictors cannot provide statistical meaning to their outputs. This has degraded their engineering value. In this paper, we aim to address these two problems in one scheme, where a dynamic regression ensemble of Gaussian process models is built. Our dynamic ensemble will select the most competent individual for each test pattern according to the competence estimated by informative neighbors. To this end, a distance measure based on RReliefF is constructed to search for these neighbors, rather than traditional K-nearest neighbor. Several evaluation indexes are combined by a meta regressor so that more robust estimation of competence can be achieved. A Bayesian nonparametric model is used for ensemble generation in order to obtain statistical predictions. A data set from real-world ladle furnace is used to verify the effectiveness of the proposed predictor. According to the comparative results, we have found the superiority of our dynamic ensemble over static ensembles and single predictors. Furthermore, the improvement over existing dynamic ensembles has also been confirmed.

Critical Concurrent Feature Selection and Enhanced Heterogeneous Ensemble Learning Approach for Fault Detection in Industrial Processes

Machine learning approaches have been successfully applied to fault detection in complex industrial processes, while it is still challenging to design an optimal feature space and a stable model for various faults. To solve this problem, a hybrid approach with critical concurrent feature selection and enhanced heterogeneous ensemble learning is proposed. First, raw process variables from field instruments and slow features obtained by simple linear transformation are ingeniously combined to construct the concurrent feature space so as to contain static and dynamic information. Then, the critical concurrent feature space of each individual learner is automatically obtained by the optimal selection process, and the feature diversity of ensemble is realized simultaneously. Finally, an enhanced heterogeneous ensemble model is constructed by different optimized individual learners, which effectively improves the classification accuracy and stability in fault detection. The performance of the proposed approach is evaluated in a simulated Tennessee Eastman benchmark and a real-word three-phase flow process. The experimental results illustrate that the performance of each optimized individual learner outperforms classic random forest algorithm. The generalization and stability of the ensemble model are further improved. Compared with traditional classification algorithms, the proposed approach achieves superior performance with accuracy that exceeds 89% for the Tennessee Eastman process and exceeds 88% for the three-phase flow process respectively. Additionally, the selected critical concurrent features indicate that both static and dynamic information play important roles in fault detection of industrial processes.

DexDeepFM: Ensemble Diversity Enhanced Extreme Deep Factorization Machine Model

Predicting user positive response (e.g., purchases and clicks) probability is a critical task in Web applications. To identify predictive features from raw data, the state-of-the-art extreme deep factorization machine model (xDeepFM) introduces a new interaction network to leverage feature interactions at the vector-wise level explicitly. However, since each hidden layer in the interaction network is a collection of feature maps, it can be viewed essentially as an ensemble of different feature maps. In this case, only using a single objective to minimize the prediction loss may lead to overfitting and generate correlated errors. In this article, an ensemble diversity enhanced extreme deep factorization machine model (DexDeepFM) is proposed, which designs the ensemble diversity measure in each hidden layer and considers both ensemble diversity and prediction accuracy in the objective function. In addition, the attention mechanism is introduced to discriminate the importance of ensemble diversity measures with different feature interaction orders. Extensive experiments on three public real-world datasets are conducted to show the effectiveness of the proposed model.

Images to Evoke Decision-Making: Building Compelling Representations for Stakeholder-Driven Futures

Contemporary landscape planning challenges require an increasingly diverse ensemble of voices, including regional stakeholders, physical scientists, social scientists, and technical experts, to provide insight into a landscape’s past trends, current uses, and desired future. To impactfully integrate these disparate components, stakeholder-driven research must include clear lines of communication, share data transparently, and slowly develop trust. Alternative future scenario representations aim to generate conversations through discourse, evoke scenario-based stakeholder input, and ensure stakeholder-based revisions to research models. The current literature lacks a metric for gauging effectiveness and a framework for optimal evaluation for future scenario representations. We have developed and applied a metric for a ranked set of compelling scenario representations using stakeholder input from an active research project. Researchers surveyed stakeholders through a case study in Idaho’s Magic Valley to gauge the effectiveness of each representational approach. To improve future stakeholder-driven geodesign projects and gaps in the research literature, this project provides a ranking of graphic strategies based on the stakeholder survey. Additionally, we provide examples and evaluate graphic representation strategies that can stimulate meaningful conversations, create common understandings, and translate research processes and findings to a variety of audiences. The results of this study intend to provide landscape architects, landscape planners, and geodesign specialists with a framework for evaluating compelling future scenario representations for a stakeholder group.

A Diversity Framework for Dealing With Multiple Types of Concept Drift Based on Clustering in the Model Space.

Data stream applications usually suffer from multiple types of concept drift. However, most existing approaches are only able to handle a subset of types of drift well, hindering predictive performance. We propose to use diversity as a framework to handle multiple types of drift. The motivation is that a diverse ensemble can not only contain models representing different concepts, which may be useful to handle recurring concepts, but also accelerate the adaptation to different types of concept drift. Our framework innovatively uses clustering in the model space to build a diverse ensemble and identify recurring concepts. The resulting diversity also accelerates adaptation to different types of drift where the new concept shares similarities with past concepts. Experiments with 20 synthetic and three real-world data streams containing different types of drift show that our diversity framework usually achieves similar or better prequential accuracy than existing approaches, especially when there are recurring concepts or when new concepts share similarities with past concepts.

The Random Plots Graph Generation Model for Studying Systems with Unknown Connection Structures.

We consider the problem of modeling complex systems where little or nothing is known about the structure of the connections between the elements. In particular, when such systems are to be modeled by graphs, it is unclear what vertex degree distributions these graphs should have. We propose that, instead of attempting to guess the appropriate degree distribution for a poorly understood system, one should model the system via a set of sample graphs whose degree distributions cover a representative range of possibilities and account for a variety of possible connection structures. To construct such a representative set of graphs, we propose a new random graph generator, Random Plots, in which we (1) generate a diversified set of vertex degree distributions and (2) target a graph generator at each of the constructed distributions, one-by-one, to obtain the ensemble of graphs. To assess the diversity of the resulting ensembles, we (1) substantialize the vague notion of diversity in a graph ensemble as the diversity of the numeral characteristics of the graphs within this ensemble and (2) compare such formalized diversity for the proposed model with that of three other common models (Erdős–Rényi–Gilbert (ERG), scale-free, and small-world). Computational experiments show that, in most cases, our approach produces more diverse sets of graphs compared with the three other models, including the entropy-maximizing ERG. The corresponding Python code is available at GitHub.

Machine learning classification of schizophrenia patients and healthy controls using diverse neuroanatomical markers and Ensemble methods

Schizophrenia is a major psychiatric disorder that imposes enormous clinical burden on patients and their caregivers. Determining classification biomarkers can complement clinical measures and improve understanding of the neural basis underlying schizophrenia. Using neuroanatomical features, several machine learning based investigations have attempted to classify schizophrenia from healthy controls but the range of neuroanatomical measures employed have been limited in range to date. In this study, we sought to classify schizophrenia and healthy control cohorts using a diverse set of neuroanatomical measures (cortical and subcortical volumes, cortical areas and thickness, cortical mean curvature) and adopted Ensemble methods for better performance. Additionally, we correlated such neuroanatomical features with Quality of Life (QoL) assessment scores within the schizophrenia cohort. With Ensemble methods and diverse neuroanatomical measures, we achieved classification accuracies ranging from 83 to 87%, sensitivities and specificities varying between 90–98% and 65–70% respectively. In addition to lower QoL scores within schizophrenia cohort, significant correlations were found between specific neuroanatomical measures and psychological health, social relationship subscale domains of QoL. Our results suggest the utility of inclusion of subcortical and cortical measures and Ensemble methods to achieve better classification performance and their potential impact of parsing out neurobiological correlates of quality of life in schizophrenia.

Subspace-based decision trees integration

The ensemble methods are meta-algorithms that combine several base machine learning techniques to produce one optimal predictive model. Many existing committees of classifiers use feature space partitioning to determine the decision regions in which the selection of base classifiers occur or to learn a diverse ensemble of base classifiers. Furthermore, the division of the feature space into subspaces does not depend on the predictive model that defines the decision boundaries. Therefore, we propose a novel ensemble learning algorithm based on the feature space partitioning defined by decision tree boundaries. In addition, in a proposed framework, the feature subspace selection occurs taking class label imbalance ratio into account. The proposed method defines the ensemble class label based on selected previously feature subspaces and a neighborhood of feature subspaces defined by its reference point. Experimental results indicate that our proposed method is more effective than state-of-the-art ensemble methods on twenty-seven benchmark datasets regarding seven representative classification performance measures.

A Sampling-Based Stack Framework for Imbalanced Learning in Churn Prediction

Churn prediction is gaining popularity in the research community as a powerful paradigm that supports data-driven operational decisions. Datasets related to churn prediction are often skewed with imbalanced class distribution. Data-level solutions, like over-sampling and under-sampling, have been commonly used by researchers to address this problem. There are limited number of case studies that attempt to evolve these data-level solutions by integrating them with computationally advanced frameworks, like ensembles. Ensembles primarily employ algorithmic diversity using a fixed set of training instances to achieve superior performance. This study aims to introduce algorithmic diversity in ensembles by modifying the fixed set of training instances using diverse sampling strategies to increase predictive performance in imbalanced learning. Data is acquired from the world’s largest open hotel commerce platform company. A four-part series of experiments is conducted to analyze the effectiveness of sampling techniques and ensemble solutions on model performance. A new sampling-based stack framework called “Stacking of Samplers for Imbalanced Learning” is proposed. The framework combines the prediction capabilities of sampling solutions to stimulate the information gain of the meta features in ensemble. It is observed that the proposed framework leads to improvement in model performance with AUC of 86.4% and top-decile lift of 4.7 for customers of the hotel technology provider. Additionally, results show that the framework records a higher information gain for meta features used in a stack, compared to commonly used stack frameworks.

Industrial fault diagnosis based on diverse variable weighted ensemble learning

Industrial fault diagnosis methods based on ensemble learning have been widely investigated in recent years; However, if the weights of each base model in these ensemble methods are kept constant, it can hinder the performance. To solve this problem, a diverse variable weighted model ensemble (DVWME) method has been proposed in this article for industrial fault diagnosis. Multiple diverse training sets were produced from the primary training set by Bootstrap sampling method. Different kinds of base models were used with different hyper parameters to build the ensemble learning method on diverse training sets, which improved the ensemble performance. And then, variable weights were devised for each base model based on its performance on various fault modes in the validation set, which made base models accurately identify different types of fault samples. Several experiments were conducted on two industrial fault datasets for performance verification, the bearing dataset of Case Western Reserve University and subway wheel dataset of Guangzhou city. Experimental results sustainably demonstrated the performance superiority and universality of proposed method. It indicated that the study provided a promising and powerful methodology for industrial fault diagnosis.

HOLP-DF: HOLP Based Screening Ultrahigh Dimensional Subfeatures in Deep Forest for Remote Sensing Image Classification

To overcome the high intramodel dimensionality and low ensemble diversity issues, which limit the classification performance of original deep forest (DF), a new version of DF, the HOLP-DF, was proposed in this article by introducing model-based high-ordinary least square projection (HOLP) feature screening (FS), random subspace propagation, and reduced error pruning (REP) techniques. To evaluate the performance of the proposed HOLP-DF, total eleven popular FS algorithms and total six advanced deep learning (DL) methods are selected. Experimental results on three widely acknowledged hyperspectral and PolSAR image classification benchmarks showed that: 1) HOLP is an optimal choice for FS in contrast with other screeners in terms of high classification accuracy and execution efficiency; 2) HOLP-DF is capable of obtaining better results than the original DF, DF with confidence screening (DF-CS) and feature screening (DF-FS); 2) optimum sets of model depth, propaganda ratio and screening ratio parameters are 30, 40% and 40%, respectively; 3) performance of HOLP-DF can be further boosted by extra usage of patch-based pooling(PP) and morphological profiling(MP) techniques.

Forecasting conflict using a diverse machine-learning ensemble: Ensemble averaging with multiple tree-based algorithms and variance promoting data configurations

ABSTRACT The article examines the potential of multi-model ensemble learning techniques for conflict research and applies an ensemble averaging framework to the prediction task of the 2020 ViEWS armed conflict forecasting competition. The goal is to predict changes of conflict intensity in fifty-four African countries for 6 months into the future. The presented ensemble combines six individual models, using two tree-based learning algorithms, incorporating two distinct data foundations and two geographical selections into a unified forecasting framework. In one of the two applied datasets, the combination of conflict variables from recent months is implemented to train for inter-temporal connections of previous conflict levels. The second dataset contains important structural, economic, political, and social information relevant for each individual case. Both datasets are used to predict changes in conflict levels. The framework structure and the combination method are presented in detail, and the prediction results for both test periods between 2014–2016 and 2016–2019 are evaluated regarding their quality. The effect of the ensemble structure and the performance of all individual components is comprehensively examined. Real forecasts over the timespan of six months into the future are presented for the upcoming months of October 2020 through March 2021 generated from data available until August 2020. The paper concludes with the presentation and examination of three selected case forecasts for Egypt, Cameroon, and Mozambique.

Minimax and Multi-Criteria Selection of Representative Model Portfolios for Complex Systems Analysis

The analysis of complex systems typically requires a large and diverse ensemble of models to confront the richness and complexity of the problem. Nevertheless, a common development and validation process of each individual model can be rigorous and time consuming. Because of limited time and resources, it is sometimes unrealistic to model every individual object in the population to capture the complex phenomena of the system. One such challenge arises in aerospace engineering when only a very limited number of aircraft performance models can be built to cover the aircraft fleet of the world for the environmental impact modeling of air transportation system. This paper considers the problem of selecting a small proportion of representative models to sufficiently cover and represent the population for more efficient and accurate systems analysis. At the intersection of unsupervised data mining and high-dimensional data analysis, the proposed methodology uses minimax and multi-criteria considerations to select a representative model portfolio at each budget level. Applying on the case study of representative aircraft models for environmental impact modeling, the benefits of the proposed method are demonstrated through both data visualization and quantitative metrics in multiple experiments. Potential limitations, extensions, and broader impacts of the method are also thoroughly discussed.

An analysis of heuristic metrics for classifier ensemble pruning based on ordered aggregation

Classifier ensemble pruning is a strategy through which a subensemble can be identified via optimizing a predefined performance criterion. Choosing the optimum or suboptimum subensemble decreases the initial ensemble size and increases its predictive performance. In this article, a set of heuristic metrics will be analyzed to guide the pruning process. The analyzed metrics are based on modifying the order of the classifiers in the bagging algorithm, with selecting the first set in the queue. Some of these criteria include general accuracy, the complementarity of decisions, ensemble diversity, the margin of samples, minimum redundancy, discriminant classifiers, and margin hybrid diversity. The efficacy of those metrics is affected by the original ensemble size, the required subensemble size, the kind of individual classifiers, and the number of classes. While the efficiency is measured in terms of the computational cost and the memory space requirements. The performance of those metrics is assessed over fifteen binary and fifteen multiclass benchmark classification tasks, respectively. In addition, the behavior of those metrics against randomness is measured in terms of the distribution of their accuracy around the median. Results show that ordered aggregation is an efficient strategy to generate subensembles that improve both predictive performance as well as computational and memory complexities of the whole bagging ensemble.

A-iLearn: An adaptive incremental learning model for spoof fingerprint detection

Incremental learning enables the learner to accommodate new knowledge without retraining the existing model. It is a challenging task that requires learning from new data and preserving the knowledge extracted from the previously accessed data. This challenge is known as the stability-plasticity dilemma. We propose A-iLearn, a generic model for incremental learning which overcomes the stability-plasticity dilemma by carefully integrating the ensemble of base classifiers trained on new data with the current ensemble without retraining the model from scratch using entire data. We demonstrate the efficacy of the proposed A-iLearn model on spoof fingerprint detection application. One of the significant challenges associated with spoof fingerprint detection is the performance drop on spoofs generated using new fabrication materials. A-iLearn is an adaptive incremental learning model that adapts to the features of the “live” and “spoof” fingerprint images and efficiently recognizes the new spoof fingerprints and the known spoof fingerprints when the new data is available. To the best of our knowledge, A-iLearn is the first attempt in incremental learning algorithms that adapts to the properties of data for generating a diverse ensemble of base classifiers. From the experiments conducted on standard high-dimensional datasets LivDet 2011, LivDet 2013 and LivDet 2015, we show that the performance gain on new fake materials is significantly high. On average, we achieve 49.57% improvement in accuracy between the consecutive learning phases.

Diversity, Identity and Le Vivre Ensemble in the Northwest Region of Cameroon since Precolonial Times

Le vivre ensemble recently dominated political discourse in Cameroon due to the difficult cohabitation between the more than 250 ethnic groups that make up the Cameroon ‘nation’. The outcome has been inter-ethnic conflicts and the resurgence of identity crisis. If Cameroon is African in miniature because of its ethnic diversities, then the North West Region is Cameroon in miniature because its populations came from almost all the cultural spheres of Cameroon. This paper attempts an investigation into the attractiveness of the destination for settlers, the identity crisis that emerged from the settlement of strangers and the symbiosis and apparent calm that characterise autochthon-settler relationship in the Northwest Region since Precolonial Times. Data was collected from primary and secondary sources and the qualitative method used to analyse it. The findings suggest that the region’s economic potentials, the hospitality of its indigenes and the inter-dependence between the various ethnic groups made for thelevivre ensemble that characterised the peoples. Unlike the economically advanced Southern part of Cameroon where le vivre ensemble was problematic, the Northwest region has experienced some level of integration amongst different communities probably because of the non-politicisation of identity. The Northwest Region can therefore serve as a veritable laboratory for Cameroon at this time that national unity and national integration are threatened by ethnic and other forms of sub nationalisms.

Beware the Black-Box: On the Robustness of Recent Defenses to Adversarial Examples.

Many defenses have recently been proposed at venues like NIPS, ICML, ICLR and CVPR. These defenses are mainly focused on mitigating white-box attacks. They do not properly examine black-box attacks. In this paper, we expand upon the analyses of these defenses to include adaptive black-box adversaries. Our evaluation is done on nine defenses including Barrage of Random Transforms, ComDefend, Ensemble Diversity, Feature Distillation, The Odds are Odd, Error Correcting Codes, Distribution Classifier Defense, K-Winner Take All and Buffer Zones. Our investigation is done using two black-box adversarial models and six widely studied adversarial attacks for CIFAR-10 and Fashion-MNIST datasets. Our analyses show most recent defenses (7 out of 9) provide only marginal improvements in security (<25%), as compared to undefended networks. For every defense, we also show the relationship between the amount of data the adversary has at their disposal, and the effectiveness of adaptive black-box attacks. Overall, our results paint a clear picture: defenses need both thorough white-box and black-box analyses to be considered secure. We provide this large scale study and analyses to motivate the field to move towards the development of more robust black-box defenses.