Selective Ensemble Method Research Articles

Argo data anomaly detection algorithm based on selective ensemble of fuzzy clustering

Abstract Argo profiling float data is a crucial data source for fundamental research and predictive forecasting operations in oceanography and environmental science. However, compiling and organizing such datasets demands considerable time and human resources. Therefore, the quest for effective methods of detecting anomalies in Argo data is of paramount importance. In this regard, we propose three improvement strategies within the stacking ensemble framework: preserving the original training set, weighting base model outputs, and combining the two former methods. The aim is to explore implicit relationships within the data, enhance model prediction diversity, and improve Accuracy. Additionally, in the selection of base models, to address the challenge of conventional clustering-based ensemble algorithms in achieving high levels of both diversity and accuracy among base learners, we introduce a selective ensemble method based on C-means clustering. This method selects base learners for the ensemble based on weighted scores derived from membership and performance evaluation metrics. Both of these enhancement approaches demonstrate effective application and improved detection performance when applied to Argo data.

Predicting types of human-related maritime accidents with explanations using selective ensemble learning and SHAP method

Maritime accidents frequently lead to severe property damage and casualties, and an accurate and reliable risk prediction model is necessary to help maritime stakeholders assess the current risk situation. Therefore, the present study proposes a hybrid methodology to develop an explainable prediction model for maritime accident types. Based on the advantages of selective ensemble learning method, this study pioneers to introduce a two-stage model selection method, aiming to enhance the predictive accuracy and stability of the model. Then, SHAP (Shapley Additive Explanations) method is integrated to identify effective mapping associations of seafarers’ unsafe acts and their risk factors with the prediction results. The results demonstrate that the model developed achieves good prediction performance with an accuracy of 87.50 % and an F1-score of 84.98 %, which benefits stakeholders in assessing the type of maritime accident in advance, so as to make proactive intervention measures.

Research on soft sensor modeling method for complex chemical processes based on local semi-supervised selective ensemble learning

To address the adverse effects of nonlinearity and dynamic time-varying in complex chemical processes on the accuracy of the soft sensor model, a local-semi-supervised ensemble learning for soft sensor modeling (local semi-supervised-selective ensemble learning-long short term memory, LS-SEL-LSTM) method is proposed in this article. Firstly, a hierarchical clustering method incorporating spatiotemporal criteria is proposed to reduce the influence of nonlinearity in global model prediction accuracy. The method considers the dynamic time-varying characteristics of temporal data and generates multiple local datasets. Then, to address the issue of multi-rate between auxiliary variables and dominant variables, a semi-supervised weight fusion mechanism based on temporal correlation is proposed, which effectively utilizes auxiliary variables to reconstruct local semi-supervised datasets and establishes local soft sensing models using LSTM. Concurrently, the parameters of the established model were optimized using the flower pollination algorithm. Subsequently, a selective ensemble learning method based on sub-model prediction accuracy and an adaptive combination weight calculation method for sub-models were proposed to improve the prediction accuracy. Finally, the effectiveness of the proposed method was verified through the actual dataset of the sulfur recovery process. The results indicate that LS-SEL-LSTM performs well in handling complex chemical processes with nonlinear and dynamic time-varying characteristics.

Selective ensemble method for anomaly detection based on parallel learning

Anomaly detection is a highly important task in the field of data analysis. Traditional anomaly detection approaches often strongly depend on data size, structure and features, while introducing the idea of ensemble into anomaly detection can greatly improve the generalization ability. Ensemble-based anomaly detection methods still face some challenges, however, such as data imbalance, time and space demand and the selection of base detectors. To this end, we propose a selective ensemble method for anomaly detection based on parallel learning (SEAD-PL). First, a differentiated stratified sampling method is designed to alleviate the problem of data imbalance. Then, a distributed parallel training frame is built to address the problem of excessive time and space consumption for base detector training. Finally, a clustering-based ensemble selection strategy is introduced to balance the accuracy and diversity of base detectors. Experiments are performed on six datasets, which demonstrate that the proposed method has obvious advantages over four selected methods.

An adaptive selective ensemble algorithm for fault classification

With the aim of improving the classification performance of a single extreme learning machine (ELM) for fault diagnosis, an adaptive selective ensemble classification algorithm is proposed based on the idea of ensemble learning. The proposed algorithm achieves better classification performance than a single model and the selected ELM subnetworks are selected adaptively and the ensemble weights of selected ELM subnetworks are calculated adaptively, which avoids the complex process of manually selecting subnetworks and calculating ensemble weights. First, a regularized error weight adjustment ELM is constructed by introducing regularization parameters and error weight adjustment matrix to a standard ELM, where the error weight adjustment matrix is obtained by the method of adaptive Gaussian kernel density estimation. Then, discrepancy subnetworks are constructed using six different activation functionsand the ensemble weights of subnetworks are obtained adaptively according to the normalized ratio of mean and variance of subnetwork F-scores to achieve the ensemble of subnetworks. Finally, the adaptive selective ensemble classification algorithm is validated using the UCI dataset and experimental acoustic emission signals of gearbox faults. The results show that the adaptive selective ensemble method can improve the stability and accuracy of classification algorithms and the achieved classification accuracy for experimental acoustic emission was 0.9773.

Bi-level ensemble method for unsupervised feature selection

Unsupervised feature selection is an important machine learning task and thus attracts increasingly more attention. However, due to the absence of labels, unsupervised feature selection often suffers from stability and robustness problems. To tackle these problems, some works try to ensemble multiple feature selection results to obtain a consensus result. Most of the existing methods do the ensemble on the feature level, i.e., they directly ensemble feature selection results by feature ranking or voting aggregation, without paying any attention to the following downstream tasks. In this paper, we take clustering as the downstream task and wish to ensemble the base results to select features which are appropriate for clustering. To this end, we propose a novel bi-level feature selection ensemble method, which ensembles on two levels: the feature level and the clustering level. Together with feature level ensemble, we also learn a consensus clustering result from base feature selection results with self-paced learning. Then, we apply the consensus clustering result to guide the feature selection in turn. Extensive experiments are conducted to demonstrate that the proposed method outperforms other state-of-the-art feature selection and feature selection ensemble methods in the clustering task. The codes of this paper are released in https://doctor-nobody.github.io/codes/BLFSE.zip.

Diversified Kernel Latent Variable Space and Multi-Objective Optimization for Selective Ensemble Learning-Based Soft Sensor

The improvement of data-driven soft sensor modeling methods and techniques for the industrial process has strongly promoted the development of the intelligent process industry. Among them, ensemble learning is an excellent modeling framework. Accuracy and diversity are two key factors that run through the entire stage of building an ensemble learning-based soft sensor. Existing base model generating methods or ensemble pruning methods always consider the two factors separately, which has limited the development of high-performance but low-complexity soft sensors. To work out this issue, a selective ensemble learning-based soft sensor modeling method based on multi-kernel latent variable space and evolutionary multi-objective optimization is proposed, referred to as MOSE-MLV-VSPLS. This method designs a multiple diversity enhancement mechanism in the base model generation stage. Diversified input variable subspaces are first constructed using the maximum information coefficient on the bootstrapping random resampling subset. Then a set of base models that combine accuracy and diversity are generated on supervised latent variable subspaces under multiple kernel function perturbations. Further, two quantifiable parameters are designed for accuracy and diversity, and the multi-objective gray wolf optimization algorithm is used to select the base models that maximize these two important parameters to achieve effective ensemble pruning at the model ensemble stage. The MOSE-MLV-VSPLS method is applied to two typical industry processes, and the experimental results show that the method is effective and superior in selective ensemble-based soft sensor modeling.

SWSEL: Sliding Window-based Selective Ensemble Learning for class-imbalance problems

For class-imbalance problems, traditional supervised learning algorithms tend to favor majority instances (also called negative instances). Therefore, it is difficult for them to accurately identify the minority instances (also called positive instances). Ensemble learning is a common method to solve the class-imbalance problem. They build multiple classifier systems on the training dataset to improve the recognition accuracy of minority instances. Sliding window is a commonly used method for processing data stream. Few researchers have used sliding windows to select majority instances and construct ensemble learning models. Traditional ensemble learning methods use some or all of the majority instances for modeling by oversampling or undersampling. However, they also inherit the drawbacks of the preprocessing methods. Therefore, in this paper, we try to use similarity mapping to construct pseudo-sequences of majority instances. Then, according to the sliding window idea, we fully use all existing majority instances, and a novel sliding window-based selective ensemble learning method (SWSEL) is proposed to deal with the class-imbalance problem. This method uses the idea of distance alignment in multi-view alignment to align the centers of the minority instances with the majority instances, and slide to select the majority instances on the sequence of pseudo-majority instances. In addition, to prevent too many classifiers from leading to long running times, we use distance metric to select a certain number of base classifiers to build the final ensemble learning model. Extensive experimental results on various real-world datasets show that using SVM, MLP and RF as the base classifier, SWSEL achieves a statistically significant performance improvement on two evaluation metrics, AUC and G-mean, compared to state-of-the-art methods.

Load2Load: Day-ahead load forecasting at aggregated level

A reliable and accurate short-term load forecasting (STLF) helps utilities and energy providers deal with the challenges posed by supply and demand balance, higher penetration of renewable energies and the development of electricity markets with increasingly complex pricing strategies in future smart grids. Recent advances in deep learning have been successively utilized to STLF. However, there is no certain study that evaluates the performances of different STLF methods at an aggregated level on different datasets with different numbers of daily measurements.In this study, a deep learning STLF architecture called Load2Load is proposed for day-ahead forecasting. Different forecasting methods have been evaluated and compared on two datasets with different temporal resolutions and features. An additive ensemble method as well as a selective ensemble method that selects the outputs of different forecasters in an hourly manner are proposed. Moreover; a modified sequential forward feature selection algorithm is proposed, resulting in better performance with a much smaller number of features.Numerical results show that the proposed Load2Load architecture has a competing performance compared to other advanced forecasters. When used together with the proposed ensemble methods, Load2Load can significantly improve the forecasting performance. The proposed feature selection algorithm results in better performance for the majority of the cases while reducing the dimensionality. According to the results with two different datasets; the proposed methods are shown to be robust to temporal resolutions, feature types and sequence lengths.

A local semi-supervised ensemble learning strategy for the data‐driven soft sensor of the power prediction in wind power generation

Soft sensors have been widely employed to estimate the power prediction in wind power generation that are difficult to measure in real-time. However, strongly nonlinear, dynamic time-varying, and multi-rate data characteristics are major causes of poor soft sensor performance. To address this problem, a strategy of building a soft sensor model based on local semi-supervised ensemble learning of least squares support vector regression (LS-EL-LSSVR) is proposed in this paper. Firstly, to reduce the influence of nonlinearity, a localization method based on the combination of time and space criteria is proposed, which can overcome the drawback of the clustering method based on a spatial criterion. Subsequently, the self-training semi-supervised classification method is used to achieve semi-supervised clustering of unlabeled samples to improve the generalization performance of the soft sensor model. Afterward, based on the local semi-supervised clustering sample dataset, the soft sensor model is established according to the least squares support vector machine method. Moreover, a selective ensemble learning method based on the prediction accuracy of local models and an adaptive combination weight calculation method of sub-models is proposed to realize selective ensemble learning and improve the prediction accuracy. The parameters of the proposed method are finalized automatically by the particle swarm optimization technique. The proposed method is simulated by using the actual spatial dynamic wind power prediction dataset, the results demonstrate the effectiveness of the proposed method in dealing with nonlinear, dynamic time-varying, and multi-rate data regression problems in wind power generation process.

Heterogeneous selective ensemble learning model for mill load parameters forecasting by using multiscale mechanical frequency spectrum

A ball mill is a heavy mechanical device and its safe operation affects the entire grinding process. Mill load is a key index in the optimum operation of the grinding process, but it cannot be measured directly. In industrial practice, operational experts normally estimate its value based on their experiences and the mechanical signals produced by the ball mill. In this paper, we proposed a heterogeneous selective ensemble method by using a multiscale mechanical frequency spectrum. The multicomponent adaptive decomposition algorithm is first used to decompose the original shell vibration and acoustic signals into sub-signals with different timescales. Then, selective ensemble (SEN) kernel projection to latent structure algorithm is used to model the spectral data of these sub-signals. Furthermore, the latent features of multiscale spectra are extracted to construct SEN models based on fuzzy inference. Finally, the two types of heterogeneous SEN models are fused by using information entropy. The main contribution of this study is that the proposed soft-sensing model has a dual-layer ensemble structure that can fuse multi-source information in different mechanical sub-signals with physical meaning. Moreover, the proposed model can simulate the fuzzy cognitive behavior of domain experts in the mineral grinding process. The effectiveness of the method is verified by the shell vibration and acoustical data of a laboratory-scale ball mill.

Deep machine learning with grey wolf algorithm and central deference Kalman filter based broken rotor bars detection in induction motor

The present study tries to propose a new method which is using Central deference Kalman Filter (CDKF) as input index of deep machine learning (DML), for simulating state estimation and broken rotor bars (BRBs) diagnosis in induction motors (IMs). In addition, an advanced selective ensemble (DML) method applying the grey wolf optimization (GWO) algorithm to diagnose the BRBs in IMs is proposed in the following study. Here, in the initial step, in order to take sufficient datum within the usual efficiency, it is needed to train the DML network. The outcomes indicate that the offered scheme can more accurately and powerfully detect diverse forms of BRBs with an accuracy of more than 98%. And also, Filter precision is enhanced via changing the sigma points of the filter, however, the stability of the filter enhances more because of its utilization, and the CDKF is further stable and precise in comparison to the Unscented Kalman filter (UKF). The CDKF performance is assessed to estimate the speed and is used as input index of DML to diagnose the broken rotor bar in IMs. The obtained outcomes prove the performance of this combined scheme.

Adaptive Classifier Ensemble Method Based on Spatial Perception for High-Dimensional Data Classification

Classifying high-dimensional small-size data is challenging in the field of pattern recognition. Traditional ensemble learning methods have several limitations: 1) sample-space based methods are easily affected by noise and redundant features; 2) feature-space based methods cannot excavate the essential characteristics of features; 3) feature subspaces cause information loss, which leads to a decline in accuracy; 4) most selective ensemble methods only consider the diversity and performance of sub-classifiers and ignore the impact on integration systems. To address the above limitations, we propose an adaptive classifier ensemble learning method (AdaSPEL) based on spatial perception for high-dimensional data. First, we design a local-space perception method for feature transformation, which encourages both high performance and diversity of the ensemble members. Second, we design a cross-space perception method based on the distribution of samples to obtain the cross-space enhanced features to provide a macro analysis for the characteristics of data. Furthermore, an adaptive selective ensemble method based on local and global evaluation mechanisms is proposed, which considers the impact of sub-classifiers on integrated systems. Experimental results on 33 high-dimensional data sets verify that our method outperforms mainstream ensemble learning methods based on feature space and sample space, and neural network-based algorithms.

Prediction of the Solubility of CO2 in Imidazolium Ionic Liquids Based on Selective Ensemble Modeling Method

Solubility data is one of the essential basic data for CO2 capture by ionic liquids. A selective ensemble modeling method, proposed to overcome the shortcomings of current methods, was developed and applied to the prediction of the solubility of CO2 in imidazolium ionic liquids. Firstly, multiple different sub–models were established based on the diversities of data, structural, and parameter design philosophy. Secondly, the fuzzy C–means algorithm was used to cluster the sub–models, and the collinearity detection method was adopted to eliminate the sub–models with high collinearity. Finally, the information entropy method integrated the sub–models into the selective ensemble model. The validation of the CO2 solubility predictions against experimental data showed that the proposed ensemble model had better performance than its previous alternative, because more effective information was extracted from different angles, and the diversity and accuracy among the sub–models were fully integrated. This work not only provided an effective modeling method for the prediction of the solubility of CO2 in ionic liquids, but also provided an effective method for the discrimination of ionic liquids for CO2 capture.

Data-Driven Evolutionary Algorithm With Perturbation-Based Ensemble Surrogates.

Data-driven evolutionary algorithms (DDEAs) aim to utilize data and surrogates to drive optimization, which is useful and efficient when the objective function of the optimization problem is expensive or difficult to access. However, the performance of DDEAs relies on their surrogate quality and often deteriorates if the amount of available data decreases. To solve these problems, this article proposes a new DDEA framework with perturbation-based ensemble surrogates (DDEA-PES), which contain two efficient mechanisms. The first is a diverse surrogate generation method that can generate diverse surrogates through performing data perturbations on the available data. The second is a selective ensemble method that selects some of the prebuilt surrogates to form a final ensemble surrogate model. By combining these two mechanisms, the proposed DDEA-PES framework has three advantages, including larger data quantity, better data utilization, and higher surrogate accuracy. To validate the effectiveness of the proposed framework, this article provides both theoretical and experimental analyses. For the experimental comparisons, a specific DDEA-PES algorithm is developed as an instance by adopting a genetic algorithm as the optimizer and radial basis function neural networks as the base models. The experimental results on widely used benchmarks and an aerodynamic airfoil design real-world optimization problem show that the proposed DDEA-PES algorithm outperforms some state-of-the-art DDEAs. Moreover, when compared with traditional nondata-driven methods, the proposed DDEA-PES algorithm only requires about 2% computational budgets to produce competitive results.

Extraction Algorithm of English Text Information From Color Images Based on Radial Wavelet Transform

The Chinese and English text information of color image reflects some important contents of color image to a certain extent, and these texts are extracted automatically, which has important application value in the fields of information retrieval, digital library, web page retrieval and intelligent transportation. In this paper, various signal extraction techniques based on radial wavelet transform modulus Maxima are analyzed, and it is found that these techniques have poor ability to extract weak signals, and the higher requirements for directionality lead to pseudo-boundary phenomena in two-dimensional image extraction results. Based on the correlation denoising method of radial wavelet coefficients, the radial wavelet entropy is introduced into the field of signal extraction, and the complex Morlet radial wavelet is selected as the basis function. A complex Morlet radial wavelet entropy extraction algorithm suitable for extracting the English text characteristics of weak signal color images is proposed. In addition, a method of scene text recognition is proposed. Based on D-SIFT local features and spatio-temporal histogram, this method vectorizes text samples in the framework of Bo Ws model. Because the spatio-temporal histogram can flexibly model the structural information of the text, this method can effectively describe the scene text. On this basis, the selective ensemble learning method is used to improve the performance of the text extractor. In order to expand the application range of the algorithm and improve the running efficiency of the integrated extractor, a model compression method based on color image English text samples and integrated extractor is proposed. The integrated extractor which takes up more space and runs slowly is compressed into an equivalent and more efficient one. In order to reduce the number of pseudo-samples needed in the process of model compression, a method of training local extractor based on color image English text samples is proposed, which greatly reduces the number of pseudo-samples needed and improves the efficiency of the model compression method.

Selective ensemble learning method for belief-rule-base classification system based on PAES

Traditional Belief-Rule-Based (BRB) ensemble learning methods integrate all of the trained sub-BRB systems to obtain better results than a single belief-rule-based system. However, as the number of BRB systems participating in ensemble learning increases, a large amount of redundant sub-BRB systems are generated because of the diminishing difference between subsystems. This drastically decreases the prediction speed and increases the storage requirements for BRB systems. In order to solve these problems, this paper proposes BRBCS-PAES: a selective ensemble learning approach for BRB Classification Systems (BRBCS) based on Pareto-Archived Evolutionary Strategy (PAES) multi-objective optimization. This system employs the improved Bagging algorithm to train the base classifier. For the purpose of increasing the degree of difference in the integration of the base classifier, the training set is constructed by the repeated sampling of data. In the base classifier selection stage, the trained base classifier is binary coded, and the number of base classifiers participating in integration and generalization error of the base classifier is used as the objective function for multi-objective optimization. Finally, the elite retention strategy and the adaptive mesh algorithm are adopted to produce the PAES optimal solution set. Three experimental studies on classification problems are performed to verify the effectiveness of the proposed method. The comparison results demonstrate that the proposed method can effectively reduce the number of base classifiers participating in the integration and improve the accuracy of BRBCS.

Selective Ensemble Classification of Image Steganalysis Via Deep Q Network

Currently many important advances in digital media field have been achieved through the ensemble method. In image steganalysis, the application of classifiers has evolved from the early single classifier to the ensemble classifiers. The performance of the ensemble classifier is better than that of a single classifier, but the classifiers may have a certain degree of redundancy. Therefore, it is of great significance to study how to reduce the number of the ensemble classifiers under the premise of ensuring the classification performance. In this letter, we propose a selective ensemble method in image steganalysis based on deep Q network, which combines reinforcement learning with convolutional neural network and are seldom seen in ensemble pruning. This method improves the generalization performance of the model, and reduces the size of ensemble as well. The experimental results show that the proposed method has a certain degree of effect on the ensemble classification optimization of image steganalysis in both spatial and frequency domains.

ANID-SEoKELM: Adaptive network intrusion detection based on selective ensemble of kernel ELMs with random features

This paper presents an adaptive network intrusion detection (ANID) method based on the selective ensemble of kernel extreme learning machines (KELMs) with random features (termed ANID-SEoKELM), aiming at identifying various unauthorized uses, misuses and abuses of computer systems in real time. To generate a lightweight intrusion detector, multiple KELMs are learned independently based on the Bagging strategy with sparse random feature representation (SRFR), to reduce noise and redundant or irrelevant information in network connection instances and ensure the diversity of base learners for the effective ensemble of base learners. A marginal distance minimization (MDM)-based selective ensemble (MDMbSE) method is introduced to generate the ultimate intrusion detector. To ensure the adaptability of the intrusion detector, an incremental learning-based detection-model updating procedure is also derived. Extensive validation and comparative experiments on the benchmark KDD99 dataset and a hybrid heterogeneous network simulation platform mixed with wireless networks and Ethernet networks demonstrate that the ANID-SEoKELM is able to adapt to the dynamically changing network environments hence it can achieve higher detection accuracies stably and efficiently than classic single learner-based intrusion detection methods and representative ensemble-based intrusion detection methods.

Hybridizing kernel‐based fuzzy c‐means with hierarchical selective neural network ensemble model for business failure prediction

AbstractMore and more ensemble models are used to forecast business failure. It is generally known that the performance of an ensemble relies heavily on the diversity between each base classifier. To achieve diversity, this study uses kernel‐based fuzzy c‐means (KFCM) to organize firm samples and designs a hierarchical selective ensemble model for business failure prediction (BFP). First, three KFCM methods—Gaussian KFCM (GFCM), polynomial KFCM (PFCM), and Hyper‐tangent KFCM (HFCM)—are employed to partition the financial data set into three data sets. A neural network (NN) is then adopted as a basis classifier for BFP, and three sets, which are derived from three KFCM methods, are used to build three classifier pools. Next, classifiers are fused by the two‐layer hierarchical selective ensemble method. In the first layer, classifiers are ranked based on their prediction accuracy. The stepwise forward selection method is employed to selectively integrate classifiers according to their accuracy. In the second layer, three selective ensembles in the first layer are integrated again to acquire the final verdict. This study employs financial data from Chinese listed companies to conduct empirical research, and makes a comparative analysis with other ensemble models and all its component models. It is the conclusion that the two‐layer hierarchical selective ensemble is good at forecasting business failure.