Ensemble Pruning Research Articles

SWEP-RF: Accuracy sliding window-based ensemble pruning method for latent sector error prediction in cloud storage computing

Latent sector errors (LSEs) in disk drives cause significant outages, data loss, and unreliability in large-scale cloud storage systems, posing not only technical challenges but also environmental concerns in the context of carbon recycling. Predicting LSEs can help avoid these problems and improve cloud reliability, while also contributing to a more sustainable cloud infrastructure. Ensemble classifiers typically outperform individual classifiers for LSE prediction with high accuracy but can lead to underfitting and incurring additional computational cost, complexity, and time and memory consumption. This research addresses this challenge by proposing a twofold solution: optimizing the ensemble diversity of the resulting Random Forest (RF) classifier through accuracy sliding window-based ensemble pruning (SWEP-RF) and using this pruned ensemble to predict LSEs in cloud storage. By effectively predicting and mitigating LSEs, this approach reduces unnecessary energy consumption and carbon emissions associated with data recovery and reprocessing, aligning with carbon recycling goals. SWEP-RF maximizes its lower margin distribution to adapt the RF prediction performance and produce a strong-performing and effective subensemble, further enhancing the overall energy efficiency of cloud systems. Our approach also reduces ensemble size while maintaining high prediction accuracy, leading to more sustainable resource utilization. We evaluate our algorithm using datasets from Baidu Inc and Backblaze datacenters. Experimental results demonstrate that our approach achieves over 98.6% prediction accuracy, a low false alarm rate (FAR) of 0.003%, and extended meantime to data loss (MTTDL) with lead time in advance (LTA) of up to 383.4 Hrs. and 474.3 Hrs., respectively. SWEP-RF outperforms classical models and current state-of-the-art techniques in prediction accuracy, FAR, MTTDL, processing time, memory consumption, and cloud availability, highlighting its significance in not only enhancing cloud storage reliability but also reducing the carbon footprint of cloud services. Our method is a promising solution for enhancing cloud storage reliability through proactive LSE prediction, while addressing the urgent need for sustainable practices and carbon recycling in cloud computing.

Tri-objective optimization-based cascade ensemble pruning for deep forest

Deep forest is a new multi-layer ensemble model, where the high time costs and storage requirements inhibit its large-scale application. However, current deep forest pruning methods used to alleviate these drawbacks do not consider its cascade coupling characteristics. Therefore, we propose a tri-objective optimization-based cascade ensemble pruning (TOOCEP) algorithm for it. Concretely, we first present a tri-objective optimization-based single-layer pruning (TOOSLP) method to prune its single-layer by simultaneously optimizing three objectives, namely accuracy, independent diversity, and coupled diversity. Particularly, the coupled diversity is designed for deep forest to deal with the coupling relationships between its adjacent layers. Then, we perform TOOSLP in a cascade framework to prune the deep forest layer-by-layer. Experimental results on 15 UCI datasets show that TOOCEP outperforms several state-of-the-art methods in accuracy and pruned rate, which significantly reduces the storage space and accelerate the prediction speed of deep forest.

Reliable Identification and Interpretation of Single-Cell Molecular Heterogeneity and Transcriptional Regulation using Dynamic Ensemble Pruning.

Unsupervised clustering is an essential step in identifying cell types from single-cell RNA sequencing (scRNA-seq) data. However, a common issue with unsupervised clustering models is that the optimization direction of the objective function and the final generated clustering labels in the absence of supervised information may be inconsistent or even arbitrary. To address this challenge, a dynamic ensemble pruning framework (DEPF)is proposed to identify and interpret single-cell molecular heterogeneity. In particular, a silhouette coefficient-based indicator is developed to determine the optimization direction of the bi-objective function. In addition, a hierarchical autoencoder is employed to project the high-dimensional data onto multiple low-dimensional latent space sets, and then a clustering ensemble is produced in the latent space by the basic clustering algorithm. Following that, a bi-objective fruit fly optimization algorithm is designed to prune dynamically the low-quality basic clustering in the ensemble. Multiple experiments are conducted on 28 real scRNA-seq datasets and one large real scRNA-seq dataset from diverse platforms and species to validate the effectiveness of the DEPF. In addition, biological interpretability and transcriptional and post-transcriptional regulatory are conducted to explore biological patterns from the cell types identified, which could provide novel insights into characterizing the mechanisms.

Soft sensor modeling for small data scenarios based on data enhancement and selective ensemble

Data-driven soft sensors have been widely used to facilitate real-time estimations of difficult-to-measure variables. However, sufficient high-quality data are often difficult to obtain due to high cost or low sampling rate. Thus, a soft sensor method based on data enhancement and selective ensemble (DESE) is proposed. First, a generative model is proposed for generating virtual labeled samples by combining supervised variational autoencoder (SVAE) and Wasserstein GAN with gradient penalty (WGAN-gp), which is referred to as SV-WGANgp. Then, SV-WGANgp is trained on various resampled training subsets to generate different sets of virtual samples. Next, diverse enhanced base models are constructed with the extended training sets. Subsequently, a multi-objective optimization (MOO) is utilized to achieve ensemble pruning. Finally, the final prediction is obtained by fusing the selected models. The application results on an industrial chlortetracycline fermentation process and a simulated penicillin fermentation process verify the effectiveness and superiority of the proposed methods.

Diversified Kernel Latent Variable Space and Multi-Objective Optimization for Selective Ensemble Learning-Based Soft Sensor

The improvement of data-driven soft sensor modeling methods and techniques for the industrial process has strongly promoted the development of the intelligent process industry. Among them, ensemble learning is an excellent modeling framework. Accuracy and diversity are two key factors that run through the entire stage of building an ensemble learning-based soft sensor. Existing base model generating methods or ensemble pruning methods always consider the two factors separately, which has limited the development of high-performance but low-complexity soft sensors. To work out this issue, a selective ensemble learning-based soft sensor modeling method based on multi-kernel latent variable space and evolutionary multi-objective optimization is proposed, referred to as MOSE-MLV-VSPLS. This method designs a multiple diversity enhancement mechanism in the base model generation stage. Diversified input variable subspaces are first constructed using the maximum information coefficient on the bootstrapping random resampling subset. Then a set of base models that combine accuracy and diversity are generated on supervised latent variable subspaces under multiple kernel function perturbations. Further, two quantifiable parameters are designed for accuracy and diversity, and the multi-objective gray wolf optimization algorithm is used to select the base models that maximize these two important parameters to achieve effective ensemble pruning at the model ensemble stage. The MOSE-MLV-VSPLS method is applied to two typical industry processes, and the experimental results show that the method is effective and superior in selective ensemble-based soft sensor modeling.

Result diversification with negative type distances by multi-objective evolutionary algorithms

The result diversification problem is to select an optimal subset with high “quality” and “diversity” from a given ground set of items, which is popular in various applications such as web-based search, multi-document summarization and ensemble pruning. The diversity relies on the distance between items. Previous works mainly focused on metric distances, and applied a greedy or local search algorithm with theoretical guarantees. As a kind of global search algorithm inspired by Darwin’s theory of evolution, evolutionary algorithms (EAs) can have a better optimization ability than greedy and local search, but often lack theoretical support. Recently, EAs have been introduced to result diversification, achieving good theoretical guarantees besides superior empirical performances. In this paper, we study whether EAs can still achieve good theoretical guarantees for result diversification with negative type distances, which are also a class of important dissimilarity measures, especially in information retrieval and sketching techniques. We propose to reformulate the result diversification problem with negative type distances as a bi-objective maximization problem, and solve it by multi-objective evolutionary algorithms (MOEAs). We prove that a simple MOEA (i.e., GSEMO) can achieve the best-known polynomial-time approximation ratio. Experiments are also performed to examine the performance of different MOEAs on the application of web-based search.

Optimization of the Regression Ensemble Size

Ensemble learning algorithms such as bagging often generate unnecessarily large models, which consume extra computational resources and may degrade the generalization ability. Pruning can potentially reduce ensemble size as well as improve performance; however, researchers have previously focused more on pruning classifiers rather than regressors. This is because, in general, ensemble pruning is based on two metrics: diversity and accuracy. Many diversity metrics are known for problems dealing with a finite set of classes defined by discrete labels. Therefore, most of the work on ensemble pruning is focused on such problems: classification, clustering, and feature selection. For the regression problem, it is much more difficult to introduce a diversity metric. In fact, the only such metric known to date is a correlation matrix based on regressor predictions. This study seeks to address this gap. First, we introduce the mathematical condition that allows checking whether the regression ensemble includes redundant estimators, i.e., estimators, whose removal improves the ensemble performance. Developing this approach, we propose a new ambiguity-based pruning (AP) algorithm that bases on error-ambiguity decomposition formulated for a regression problem. To check the quality of AP, we compare it with the two methods that directly minimize the error by sequentially including and excluding regressors, as well as with the state-of-art Ordered Aggregation algorithm. Experimental studies confirm that the proposed approach allows reducing the size of the regression ensemble with simultaneous improvement in its performance and surpasses all compared methods.

Unsupervised encoding selection through ensemble pruning for biomedical classification

BackgroundOwing to the rising levels of multi-resistant pathogens, antimicrobial peptides, an alternative strategy to classic antibiotics, got more attention. A crucial part is thereby the costly identification and validation. With the ever-growing amount of annotated peptides, researchers leverage artificial intelligence to circumvent the cumbersome, wet-lab-based identification and automate the detection of promising candidates. However, the prediction of a peptide’s function is not limited to antimicrobial efficiency. To date, multiple studies successfully classified additional properties, e.g., antiviral or cell-penetrating effects. In this light, ensemble classifiers are employed aiming to further improve the prediction. Although we recently presented a workflow to significantly diminish the initial encoding choice, an entire unsupervised encoding selection, considering various machine learning models, is still lacking.ResultsWe developed a workflow, automatically selecting encodings and generating classifier ensembles by employing sophisticated pruning methods. We observed that the Pareto frontier pruning is a good method to create encoding ensembles for the datasets at hand. In addition, encodings combined with the Decision Tree classifier as the base model are often superior. However, our results also demonstrate that none of the ensemble building techniques is outstanding for all datasets.ConclusionThe workflow conducts multiple pruning methods to evaluate ensemble classifiers composed from a wide range of peptide encodings and base models. Consequently, researchers can use the workflow for unsupervised encoding selection and ensemble creation. Ultimately, the extensible workflow can be used as a plugin for the PEPTIDE REACToR, further establishing it as a versatile tool in the domain.

Joint leaf-refinement and ensemble pruning through L_1 regularization

Ensembles are among the state-of-the-art in many machine learning applications. With the ongoing integration of ML models into everyday life, e.g., in the form of the Internet of Things, the deployment and continuous application of models become more and more an important issue. Therefore, small models that offer good predictive performance and use small amounts of memory are required. Ensemble pruning is a standard technique for removing unnecessary classifiers from a large ensemble that reduces the overall resource consumption and sometimes improves the performance of the original ensemble. Similarly, leaf-refinement is a technique that improves the performance of a tree ensemble by jointly re-learning the probability estimates in the leaf nodes of the trees, thereby allowing for smaller ensembles while preserving their predictive performance. In this paper, we develop a new method that combines both approaches into a single algorithm. To do so, we introduce L_1 regularization into the leaf-refinement objective, which allows us to jointly prune and refine trees at the same time. In an extensive experimental evaluation, we show that our approach not only offers statistically significantly better performance than the state-of-the-art but also offers a better accuracy-memory trade-off. We conclude our experimental evaluation with a case study showing the effectiveness of our method in a real-world setting.

CSLSEP: an ensemble pruning algorithm based on clustering soft label and sorting for facial expression recognition

CSLSEP: an ensemble pruning algorithm based on clustering soft label and sorting for facial expression recognition

Combining with lab-on-chip technology and multi-organ fusion strategy to estimate post-mortem interval of rat.

The estimation of post-mortem interval (PMI) is one of the most important problems in forensic pathology all the time. Although many classical methods can be used to estimate time since death, accurate and rapid estimation of PMI is still a difficult task in forensic practice, so the estimation of PMI requires a faster, more accurate, and more convenient method. In this study, an experimental method, lab-on-chip, is used to analyze the characterizations of polypeptide fragments of the lung, liver, kidney, and skeletal muscle of rats at defined time points after death (0, 1, 2, 3, 5, 7, 9, 12, 15, 18, 21, 24, 27, and 30 days). Then, machine learning algorithms (base model: LR, SVM, RF, GBDT, and MLPC; ensemble model: stacking, soft voting, and soft-weighted voting) are applied to predict PMI with single organ. Multi-organ fusion strategy is designed to predict PMI based on multiple organs. Then, the ensemble pruning algorithm determines the best combination of multi-organ. The kidney is the best single organ for predicting the time of death, and its internal and external accuracy is 0.808 and 0.714, respectively. Multi-organ fusion strategy dramatically improves the performance of PMI estimation, and its internal and external accuracy is 0.962 and 0.893, respectively. Finally, the best organ combination determined by the ensemble pruning algorithm is all organs, such as lung, liver, kidney, and skeletal muscle. Lab-on-chip is feasible to detect polypeptide fragments and multi-organ fusion is more accurate than single organ for PMI estimation.

Ensemble Learning of Multiple Models Using Deep Learning for Multiclass Classification of Ultrasound Images of Hepatic Masses.

Ultrasound (US) is often used to diagnose liver masses. Ensemble learning has recently been commonly used for image classification, but its detailed methods are not fully optimized. The purpose of this study is to investigate the usefulness and comparison of some ensemble learning and ensemble pruning techniques using multiple convolutional neural network (CNN) trained models for image classification of liver masses in US images. Dataset of the US images were classified into four categories: benign liver tumor (BLT) 6320 images, liver cyst (LCY) 2320 images, metastatic liver cancer (MLC) 9720 images, primary liver cancer (PLC) 7840 images. In this study, 250 test images were randomly selected for each class, for a total of 1000 images, and the remaining images were used as the training. 16 different CNNs were used for training and testing ultrasound images. The ensemble learning used soft voting (SV), weighted average voting (WAV), weighted hard voting (WHV) and stacking (ST). All four types of ensemble learning (SV, ST, WAV, and WHV) showed higher values of accuracy than the single CNN. All four types also showed significantly higher deep learning (DL) performance than ResNeXt101 alone. For image classification of liver masses using US images, ensemble learning improved the performance of DL over a single CNN.

BIOMEDICAL TEXT DOCUMENT CLASSIFICATION

Information extraction, retrieval, and text categorization are only a few of the significant research fields covered by "bio medical text classification." This study examines many text categorization techniques utilised in practise, as well as their strengths and weaknesses, in order to improve knowledge of various information extraction opportunities in the field of data mining. We compiled a dataset with a focus on three categories: "Thyroid Cancer," "Lung Cancer," and "Colon Cancer." This paper presents an empirical study of a classifier. The investigation was carried out using biomedical literature benchmarks. Many metaheuristic algorithms are investigated, including genetic algorithms, particle swarm optimisation, firefly, cuckoo, and bat algorithms. In addition, the proposed multiple classifier system outperforms ensemble learning, ensemble pruning, and traditional classification methods. Based on the data, we forecast if it is Thyroid Cancer, Lung Cancer, or Colon Cancer using basic EDA, text preprocessing, and several models such as Logistic Regression, Decision Tree Classification, and Random Forest Classification.

Subarchitecture Ensemble Pruning in Neural Architecture Search.

Neural architecture search (NAS) is gaining more and more attention in recent years because of its flexibility and remarkable capability to reduce the burden of neural network design. To achieve better performance, however, the searching process usually costs massive computations that might not be affordable for researchers and practitioners. Although recent attempts have employed ensemble learning methods to mitigate the enormous computational cost, however, they neglect a key property of ensemble methods, namely diversity, which leads to collecting more similar subarchitectures with potential redundancy in the final design. To tackle this problem, we propose a pruning method for NAS ensembles called " subarchitecture ensemble pruning in neural architecture search (SAEP)." It targets to leverage diversity and to achieve subensemble architectures at a smaller size with comparable performance to ensemble architectures that are not pruned. Three possible solutions are proposed to decide which subarchitectures to prune during the searching process. Experimental results exhibit the effectiveness of the proposed method by largely reducing the number of subarchitectures without degrading the performance.

A SURVEY ON BIOMEDICAL TEXT DOCUMENT CLASSIFICATION

Information extraction, information retrieval, and text classification are only a few of the important study areas that fall under the heading of "bio medical text classification." In order to increase understanding of various information extraction opportunities in the field of data mining, this study analyses several text categorization approaches used in practise, their strengths and shortcomings. We have gathered a dataset with a strong emphasis on three categories, including "Thyroid Cancer," "Lung Cancer," and "Colon Cancer." This essay offers an empirical investigation of a classifier. Benchmarks for biomedical text were used to conduct the experiment. We study many metaheuristic algorithms, including genetic algorithms, particle swarm optimization, firefly, cuckoo, and bat algorithms. The suggested multiple classifier system also outperforms ensemble learning, ensemble pruning, and conventional classification algorithms. In the data we use predict the Biomedical text document classification is whether it's Thyroid Cancer, Lung Cancer, Colon Cancer based on the performed basic EDA, text pre-processing, build different models, such as LogisticRegression, DecisiontreeClassification, RandomForest Classification

EEG-based covert speech decoding using random rotation extreme learning machine ensemble for intuitive BCI communication

Background and motivation:Brain–computer interfaces (BCI) assist communication for the handicapped and disabled. Previous work has shown that it is usually non-intuitive, and it frequently employs Electroencephalogram (EEG) based motor imagery. Ensemble learning has been shown to be reliable in a variety of BCI classification tasks, including motor imagery and P300 event-related potential. The purpose of ensemble learning, according to past research, is to get trained accurate but diverse base classifiers to increase generalization performance. Extreme Learning Machine (ELM) with fast learning speed has been proven effective in classification applications and suitable for ensemble learning due to its randomness. The suitability of ensemble methods for intuitive BCI communication using EEG data of different covert speech words is investigated in this work. Methods:To achieve this goal, we construct a random rotation based kernel ELM ensemble, resulting in very diverse classifiers which can learn through transformed spaces. Minimize error ensemble pruning is used to prune low complementarity classifiers, and consequently, combining the remaining high complementarity classifiers to get the final ensemble classifier. Eight subjects performed covert speech tasks which involved mentally repeating five words namely; ‘left’, ‘right’, ‘up’, ‘down’, and ‘stop’. Results:We achieved classification accuracy with pruning and without pruning of 63.67% and 57.13% respectively. Experimental results show that, in comparison to other classification techniques, the proposed technique produces competitive results. Conclusion and Significance:Our findings indicate the potential of random rotation kernel ELM ensemble pruning to effectively classify EEG-based covert speech signals. This proposed research involving an ensemble method for the classification of covert speech words can make a significant contribution towards intuitive BCI development using silent speech.

Group Decision Making-Based Fusion for Human Activity Recognition in Body Sensor Networks.

Ensemble learning systems (ELS) have been widely utilized for human activity recognition (HAR) with multiple homogeneous or heterogeneous sensors. However, traditional ensemble approaches for HAR cannot always work well due to insufficient accuracy and diversity of base classifiers, the absence of ensemble pruning, as well as the inefficiency of the fusion strategy. To overcome these problems, this paper proposes a novel selective ensemble approach with group decision-making (GDM) for decision-level fusion in HAR. As a result, the fusion process in the ELS is transformed into an abstract process that includes individual experts (base classifiers) making decisions with the GDM fusion strategy. Firstly, a set of diverse local base classifiers are constructed through the corresponding mechanism of the base classifier and the sensor. Secondly, the pruning methods and the number of selected base classifiers for the fusion phase are determined by considering the diversity among base classifiers and the accuracy of candidate classifiers. Two ensemble pruning methods are utilized: mixed diversity measure and complementarity measure. Thirdly, component decision information from the selected base classifiers is combined by using the GDM fusion strategy and the recognition results of the HAR approach can be obtained. Experimental results on two public activity recognition datasets (The OPPORTUNITY dataset; Daily and Sports Activity Dataset (DSAD)) suggest that the proposed GDM-based approach outperforms the well-known fusion techniques and other state-of-the-art approaches in the literature.

Adaptive forecasting of wind power based on selective ensemble of offline global and online local learning

Wind power has become an important part of clean energy. Reliable wind power forecasting is the key to performing optimal scheduling of wind energy. However, it is challenging to obtain accurate wind power forecasting due to inherent intermittency and randomness of wind energy. In addition, the characteristics of wind power data will inevitably change with time, which will lead to the performance degradation of the forecasting model. To address these problems, an adaptive wind power forecasting method based on selective ensemble of offline global and online local learning (SEOGOL) is proposed. To ensure the effectiveness of the ensemble, SEOGOL employs a multi-modal perturbation mechanism to enhance the diversity of the base models, including perturbation on modeling samples through multi-resolution and multi-interval time-difference resampling, perturbation on learners, and perturbation on model hyperparameters. In addition, with defining the accuracy and diversity of base models as two optimization objectives, a multi-objective ensemble pruning approach is proposed to achieve model selection. Furthermore, the adaptive fusion of the individual models is achieved according to the Bayesian rule. Compared with traditional non-adaptive and simple adaptive forecasting methods, the proposed method successfully combines the advantages of offline global and online local learning. SEOGOL can not only mine large-scale historical wind power data, but also capture the latest wind power status information, thus effectively improving the accuracy and reliability of wind power forecasting. The effectiveness and superiority of the proposed SEOGOL method are verified using an actual wind power data set.

When Does Diversity Help Generalization in Classification Ensembles?

Ensembles, as a widely used and effective technique in the machine learning community, succeed within a key element-"diversity." The relationship between diversity and generalization, unfortunately, is not entirely understood and remains an open research issue. To reveal the effect of diversity on the generalization of classification ensembles, we investigate three issues on diversity, that is, the measurement of diversity, the relationship between the proposed diversity and the generalization error, and the utilization of this relationship for ensemble pruning. In the diversity measurement, we measure diversity by error decomposition inspired by regression ensembles, which decompose the error of classification ensembles into accuracy and diversity. Then, we formulate the relationship between the measured diversity and ensemble performance through the theorem of margin and generalization and observe that the generalization error is reduced effectively only when the measured diversity is increased in a few specific ranges, while in other ranges, larger diversity is less beneficial to increasing the generalization of an ensemble. Besides, we propose two pruning methods based on diversity management to utilize this relationship, which could increase diversity appropriately and shrink the size of the ensemble without much-decreasing performance. The empirical results validate the reasonableness of the proposed relationship between diversity and ensemble generalization error and the effectiveness of the proposed pruning methods.

Explainable online ensemble of deep neural network pruning for time series forecasting

Both the complex and evolving nature of time series data make forecasting among one of the most challenging tasks in machine learning. Typical methods for forecasting are designed to model time-evolving dependencies between data observations. However, it is generally accepted that none of them are universally valid for every application. Therefore, methods for learning heterogeneous ensembles by combining a diverse set of forecasters together appears as a promising solution to tackle this task. While several approaches in the context of time series forecasting have focused on how to combine individual models in an ensemble, ranging from simple and enhanced averaging tactics to applying meta-learning methods, few works have tackled the task of ensemble pruning, i.e. individual model selection to take part in the ensemble. In addition, in classical ML literature, ensemble pruning techniques are mostly restricted to operate in a static manner. To deal with changes in the relative performance of models as well as changes in the data distribution, we employ gradient-based saliency maps for online ensemble pruning of deep neural networks. This method consists of generating individual models’ performance saliency maps that are subsequently used to prune the ensemble by taking into account both aspects of accuracy and diversity. In addition, the saliency maps can be exploited to provide suitable explanations for the reason behind selecting specific models to construct an ensemble that plays the role of a forecaster at a certain time interval or instant. An extensive empirical study on many real-world datasets demonstrates that our method achieves excellent or on par results in comparison to the state-of-the-art approaches as well as several baselines. Our code is available on Github (https://github.com/MatthiasJakobs/os-pgsm/tree/ecml_journal_2022).