Heterogeneous Ensemble Method Research Articles

Spatio-Temporal Heterogeneous Ensemble Learning Method for Predicting Land Subsidence

The prediction of land subsidence is of significant value for the early warning and prevention of geological disasters. Although numerous land subsidence prediction methods are currently available, two obstacles still exist: (i) spatio-temporal heterogeneity of land subsidence is not well considered, and (ii) the prediction performance of individual models is unsatisfactory when the data do not meet their assumptions. To address these issues, we developed a spatio-temporal heterogeneous ensemble learning method for predicting land subsidence. Firstly, a two-stage hybrid spatio-temporal clustering method was proposed to divide the dataset into internally homogeneous spatio-temporal clusters. Secondly, within each spatio-temporal cluster, an ensemble learning strategy was employed to combine one time series prediction model and three spatio-temporal prediction models to reduce the prediction uncertainty of an individual model. Experiments on a land subsidence dataset from Cangzhou, China, show that the prediction accuracy of the proposed method is significantly higher than that of four individual prediction models.

Class-overlap detection based on heterogeneous clustering ensemble for multi-class imbalance problem

The class imbalance problem is one of the main challenges that hinders classifiers from identifying unknown instances. When class distribution imbalance and class overlap coexist in the same data set, the recognition of minority instances by traditional classifiers will be seriously affected. In the binary classification problem, the relationship between the majority class and the minority class is clear, and the overlap problem between classes can be well solved by removing the majority instances in the class-overlap area. However, for multi-class imbalance problems, the relationships between classes will become complex. Clustering techniques are effective methods for identifying overlapping instances; however, a single clustering method cannot effectively identify clusters of different shapes. Therefore, in this paper, a Heterogeneous Clustering Ensemble learning method for Multiple Class-overlap Detection (HCE-MCD) is proposed for multi-class imbalance problems. This method uses a genetic algorithm to select a combination of heterogeneous clustering techniques with the best fitness function from a pool of clustering algorithms and uses majority voting to fuse the results of the clustering algorithm. Furthermore, instances located near the decision boundary are more likely to be overlapping instances. Therefore, according to the clustering results of the heterogeneous clustering ensemble method, we can identify the overlapping instances existing in the data set, thereby eliminating the class overlap problem in the multi-class dataset. Experimental results on 19 open-access datasets show that our proposed method outperforms or partially outperforms state-of-the-art research protocols.

Position prediction of underwater gliders based on a new heterogeneous model ensemble method

An underwater glider (UG) is a promising unmanned underwater vehicle driven by buoyancy that can perform long-period and large-scale ocean observations. Due to the lack of inertial sensors and the low-speed relative to the ocean current, it is difficult to determine the exact position of the UG. Besides, the position of the UG is of great significance for acoustic applications. Therefore, this paper proposes a new heterogeneous model ensemble method (HMEM) based on the whale algorithm (WOA) optimized backpropagation neural network with complement ensemble empirical mode decomposition (CEEMD-BPNN) and the physical model. The physical model can deduce the underwater position of the UG considering the ambient environment, and the WOA-CEEMD-BPNN is adopted to predict the surface position with historical data. Then, the new heterogeneous model ensemble method integrates these two with an unscented Kalman filter according to the predicted surface position. To evaluate the performance of the HMEM, the evaluation index is introduced, and the sea trial conducted by the Petrel-II glider is used as a case study. The experiment demonstrates that the MAE and RMSE of the HMEM are 0.51 km and 0.76 km, respectively, and the mean time difference is 3.08 × 102 s, which outperforms the dynamic model and can achieve accurate position prediction for long periods.

An ensemble spatial prediction method considering geospatial heterogeneity

Ensemble learning synthesizes the advantages of different models and has been widely applied in the field of spatial prediction. However, the nonlinear constraints of spatial heterogeneity on the model ensemble process make it difficult to adaptively determine the ensemble weights, greatly limiting the predictive ability of the ensemble learning model. This paper therefore proposes a novel geographical spatial heterogeneous ensemble learning method (GSH-EL). Firstly, the geographically weighted regression model, geographically optimal similarity model, and random forest model are used as three base learners to express local spatial heterogeneity, global feature correlation, and nonlinear relationship of geographic elements, respectively. Then, a spatially weighted ensemble neural network module (SWENN) of GSH-EL is proposed to express spatial heterogeneity by exploring the complex nonlinear relationship between the spatial proximity and ensemble weights. Finally, the outputs of the three base learners are combined with the spatial heterogeneous ensemble weights from SWENN to obtain the spatial prediction results. The proposed method is validated on the PM2.5 air quality and landslide dataset in China, both of which obtain more accurate prediction results than the existing ensemble learning strategies. The results confirm the need to accurately express spatial heterogeneity in the model ensemble process.

Landslide susceptibility assessment based on frequency ratio and semi-supervised heterogeneous ensemble learning model.

Epistemic uncertainty in data-driven landslide susceptibility assessment often tends to be increased by the limited accuracy of an individual model, as well as uncertainties associated with the selection of non-landslide samples. To address these issues, this paper centers on the landslide disaster in Ji'an City, China, and proposes a heterogeneous ensemble learning method incorporating frequency ratio (FR) and semi-supervised sample expansion. Based on the superimposed results of 12 environmental factor frequency ratios (FFR), non-landslide samples were selected and input into light gradient boosting machine (LightGBM), random forest (RF), and convolutional neural network (CNN) models for prediction along with historical landslide samples. The predicted probability values are integrated by four heterogeneous ensemble strategies to expand samples from high-confidence results. The model's performance is evaluated using the area under the receiver operating characteristic curve (AUC), partition frequency ratio (PFR), and other verification methods. The results demonstrate that the negative sample based on FFR sampling is more accurate than the random sampling method, and the FR-SSELR model based on frequency ratio sampling and semi-supervised ensemble strategy exhibits the highest performance (AUC = 0.971, ACC = 0.941). A more reasonable landslide susceptibility map was drawn based on this model, with the lowest percentage of landslides in the low and very low susceptibility zones (sum of PFR = 0.194), as well as the highest percentage of landslides in the high and very high susceptibility zones (sum of PFR = 6.800). Furthermore, the FR-SSELR model improved economic benefits by 3.82-14.2%, offering valuable guidance for decision-making regarding landslide management and the sustainability of Ji'an City.

A Heterogeneous Ensemble Approach for Terrain Classification with Tracked Mobile Robots

Terrain classification plays a vital role in the autonomous driving of tracked mobile robots. This article proposes a heterogeneous ensemble method to enhance the classification performance with weak classifiers. Combined with principal component analysis (PCA) and ReiefF, a feature selection approach aims to find an effective feature set. Based on the proposed method, the framework has been developed to simplify the signal process in feature extraction. Extreme Learning Machine (ELM) is utilized as the individual classifier for three kinds of signal sources. With the learning process experience, ensemble weights have been generated to support the whole ensemble flow, which is closely correlated with terrain categories, classifiers and signal sources. Due to the structure of tracked mobile robots, the application maneuvers are extended to the front and rear drive modes, while the steady-state turning and straight-line maneuvers. The results demonstrate that the proposed method can significantly enhance the performance of terrain classification.

Predicting Students at Risk of Early Dropping Out from Course Using Ensemble Classification Methods

Many research studies confirmed that the application of machine learning methods for predicting students at risk of academic failure can be beneficial for improving student performance in higher education institutions. Simultaneously, these studies emphasised the importance of the input data quality, its precise pre-processing and thoroughly selected features on final performance metrics of the applied classification algorithms. However, the stand-alone application of well-known classification algorithms still has performance limitations, which can be outperformed using ensembles of classifiers. Therefore, this paper aims to evaluate the overall contribution of the selected ensemble methods to the early prediction of students at risk of dropping out of learning courses. All phases of CRISP-DM methodology are described in this study to explain the complexity of the problem of the practical adoption of voting-based heterogeneous ensemble methods in the case of data obtained from the course of the university learning management system compared to traditional classification algorithms enhanced by ensemble learning with AdaBoost and XGBoost methods. As a result, the study confirmed that the features which characterise the students’ performance and their interactions in the course can be considered the most significant for the identification of the students at risk of dropping out. Moreover, in the case of advanced ensemble algorithms, the overall accuracy, recall, precision, and f1 score were improved by 2-4%. The proposed method has demonstrated exceptional performance. Finally, the paper summarises the findings and their implications for effective prediction.

Heterogeneous ensemble learning for enhanced crash forecasts – A frequentist and machine learning based stacking framework

Introduction: This study aims to increase the prediction accuracy of crash frequency on roadway segments that can forecast future safety on roadway facilities. A variety of statistical and machine learning (ML) methods are used to model crash frequency with ML methods generally having a higher prediction accuracy. Recently, heterogeneous ensemble methods (HEM), including “stacking,” have emerged as more accurate and robust intelligent techniques providing more reliable and accurate predictions. Methods: This study applies “Stacking” to model crash frequency on five-lane undivided (5 T) segments of urban and suburban arterials. The prediction performance of “Stacking” is compared with parametric statistical models (Poisson and negative binomial) and three state-of-the-art ML techniques (Decision tree, random forest, and gradient boosting), each of which is termed as the base-learner. By employing an optimal weight scheme to combine individual base-learners through stacking, the problem of biased predictions in individual base-learners due to differences in specifications and prediction accuracies is avoided. Data including crash, traffic, and roadway inventory were collected and integrated from 2013 to 2017. The data are split into training (2013–2015), validation (2016), and testing (2017) datasets. After training five individual base-learners using training data, prediction outcomes are obtained for the five base-learners using validation data that are then used to train a meta-learner. Results: Results of statistical models reveal that crashes increase with the density (number per mile) of commercial driveways whereas decrease with average offset distance to fixed objects. Individual ML methods show similar results – in terms of variable importance. A comparison of out-of-sample predictions of various models or methods confirms the superiority of “Stacking” over the alternative methods considered. Conclusions and practical applications: From a practical standpoint, “stacking” can enhance prediction accuracy (compared to only one base-learner with a particular specification). When applied systemically, stacking can help identify more appropriate countermeasures.

Identifying predictors of analyst rating quality: An ensemble feature selection approach

Forecasting the analyst rating quality (ARQ), defined as whether a stock rating provided by an analyst can correctly foretell the stock movement, is crucial to fully leveraging the value of this information resource. This study develops a two-phase method to identify key predictors for ARQ forecasting. In the first stage, we conduct a thorough literature review to obtain a comprehensive list of candidate features, and organise them under three categories: analyst-related, rating-related, and stock-related. In the second stage, we propose a heterogeneous community-based ensemble feature selection method (ComEFS), with the goal of identifying a subset of relevant predictors to be jointly used for ARQ forecasting. Thorough experiments are conducted on a real dataset to verify the effectiveness of our proposed method. The empirical results show that key predictors identified by ComEFS exhibit stronger predictive power compared to those identified by benchmark methods. This study provides insights about ARQ forecasting by selecting the right input. Selectively utilizing these predictive features can help improve the performance of downstream machine learning models and ultimately help investors avoid unreliable analyst ratings and financial loss.

Heterogeneous sets in dimensionality reduction and ensemble learning

We present a general framework for dealing with set heterogeneity in data and learning problems, which is able to exploit low complexity components. The main ingredients are (i) A definition of complexity for elements of a convex union that takes into account the complexities of their individual composition – this is used to cover the heterogeneous convex union; and (ii) Upper bounds on the complexities of restricted subsets. We demonstrate this approach in two different application areas, highlighting their conceptual connection. (1) In random projection based dimensionality reduction, we obtain improved bounds on the uniform preservation of Euclidean norms and distances when low complexity components are present in the union. (2) In statistical learning, our generalisation bounds justify heterogeneous ensemble learning methods that were incompletely understood before. We exemplify empirical results with boosting type random subspace and random projection ensembles that implement our bounds.

Heterogeneous selective ensemble learning model for mill load parameters forecasting by using multiscale mechanical frequency spectrum

A ball mill is a heavy mechanical device and its safe operation affects the entire grinding process. Mill load is a key index in the optimum operation of the grinding process, but it cannot be measured directly. In industrial practice, operational experts normally estimate its value based on their experiences and the mechanical signals produced by the ball mill. In this paper, we proposed a heterogeneous selective ensemble method by using a multiscale mechanical frequency spectrum. The multicomponent adaptive decomposition algorithm is first used to decompose the original shell vibration and acoustic signals into sub-signals with different timescales. Then, selective ensemble (SEN) kernel projection to latent structure algorithm is used to model the spectral data of these sub-signals. Furthermore, the latent features of multiscale spectra are extracted to construct SEN models based on fuzzy inference. Finally, the two types of heterogeneous SEN models are fused by using information entropy. The main contribution of this study is that the proposed soft-sensing model has a dual-layer ensemble structure that can fuse multi-source information in different mechanical sub-signals with physical meaning. Moreover, the proposed model can simulate the fuzzy cognitive behavior of domain experts in the mineral grinding process. The effectiveness of the method is verified by the shell vibration and acoustical data of a laboratory-scale ball mill.

An Efficient Method Combined Data-Driven for Detecting Electricity Theft with Stacking Structure Based on Grey Relation Analysis

Nowadays, electricity theft has been a major problem worldwide. Although many single-classification algorithms or an ensemble of single learners (i.e., homogeneous ensemble learning) have proven able to automatically identify suspicious customers in recent years, after the accuracy of these methods reaches a certain level, it still cannot be improved even if it continues to be optimized. To break through this bottleneck, a heterogeneous ensemble learning method with stacking integrated structure of different strong individual learners for detection of electricity theft is presented in this paper. Firstly, we use the grey relation analysis (GRA) method to select the heterogeneous strong classifier combination of LG + LSTM + KNN as the base model layer of stacking structure based on the principle of the highest comprehensive evaluation index value. Secondly, the support vector machine (SVM) model with relatively good results of the stacking overall structure experiment is selected as the model of the meta-model layer. In this way, a heterogeneous integrated learning model for electricity theft detection of the stacking structure is constructed. Finally, the experiments of this model are conducted on electricity consumption data from State Grid Corporation of China, and the results show that the detection performance of the proposed method is better than that of the existing state-of-the-art detection method (where the area under receiver operating characteristic curve (AUC) value is 0.98675).

An approach for improved students’ performance prediction using homogeneous and heterogeneous ensemble methods

<span lang="EN-US">Web-based learning technologies of educational institutions store a massive amount of interaction data which can be helpful to predict students’ performance through the aid of machine learning algorithms. With this, various researchers focused on studying ensemble learning methods as it is known to improve the predictive accuracy of traditional classification algorithms. This study proposed an approach for enhancing the performance prediction of different single classification algorithms by using them as base classifiers of homogeneous ensembles (bagging and boosting) and heterogeneous ensembles (voting and stacking). The model utilized various single classifiers such as multilayer perceptron or neural networks (NN), random forest (RF), naïve Bayes (NB), J48, JRip, OneR, logistic regression (LR), k-nearest neighbor (KNN), and support vector machine (SVM) to determine the base classifiers of the ensembles. In addition, the study made use of the University of California Irvine (UCI) open-access student dataset to predict students’ performance. The comparative analysis of the model’s accuracy showed that the best-performing single classifier’s accuracy increased further from 93.10% to 93.68% when used as a base classifier of a voting ensemble method. Moreover, results in this study showed that voting heterogeneous ensemble performed slightly better than bagging and boosting homogeneous ensemble methods.</span>

Approach to Predict Software Vulnerability Based on Multiple-Level N-gram Feature Extraction and Heterogeneous Ensemble Learning

Software vulnerabilities are one of the roots of computer security problems. The traditional static analysis and dynamic analysis methods based on software source code mainly have some deficiencies, such as high false positive rate, high false negative rate and insufficient semantic information captured. Nevertheless, the application of machine learning, Natural Language Processing and other technologies in software vulnerability prediction can effectively mitigate such issues. This paper proposed a vulnerability prediction method based on multiple-level N-gram feature extraction and heterogeneous ensemble learning. First, by code intermediate representation and constructing a multiple-level N-gram feature generation model, two kinds of N-gram semantic features with different window size and different granularity at word and char level were extracted to retain the semantic and structural information of code. Second, TF–IDF was used to construct the vector space model as the input of prediction model. As a single classifier was prone to overfitting and poor generalization, this paper conducted benchmark testing on five classical machine learning algorithms (NB, SVM, DT, LR, RF), and then combined four (SVM, DT, LR, RF) among them, which had better performance as the base classifiers to form the stacking heterogeneous ensemble method to build the vulnerability prediction model. Finally, the proposed method was verified on buffer overflow vulnerability and resource management vulnerability datasets, with a lowest false positive rate and false negative rate which can reach 1.58% and 4.06%, respectively.

Carbon trading price prediction based on a two-stage heterogeneous ensemble method

Carbon trading price prediction based on a two-stage heterogeneous ensemble method

A Novel Approach of Ensemble Methods Using the Stacked Generalization for High-dimensional Datasets

Stacked generalization-based heterogeneous ensemble methods combine the prediction of multiple classifiers to improve overall classification performance. Several stacking methods are available in the literature, but the criteria to select the number and type of classifiers are missing. This work analyzes the performance of stacked generalization-based ensemble machine learning methods for high-dimensional datasets. Also, the impact of the classifier selection for the first level ( L 0 ) of the stacked generalization method has been studied. Based on that, the criteria for selecting classifiers at the first level of the stacked generalization method are proposed. So, six stacked generalization approaches are presented and have been analyzed for the thirty high-dimensional datasets. The experiments and results indicate that the performance of stacking strategies based on proposed selection criteria performs better. Also, a comparative study about the choice of homogeneous ensemble classifiers in stacked generalization concerning the use of basic classifiers and fusion of basic with homogeneous ensemble methods has been made. It has been observed that the use of only homogeneous ensemble classifiers is not beneficial in the stacked generalization methods. Also, the performance of stacked generalization based on basic classifiers or a combination of homogeneous ensemble methods with basic classifiers is better than homogeneous ensemble methods. The proposed stacking approach based on a combination of the basic and ensemble classifiers has improved the accuracy 0.72% to 8.46%. The impact of removing redundant and non-relevant features on the proposed stacking approaches has been evaluated.

Two-stage stacking heterogeneous ensemble learning method for gasoline octane number loss prediction

Gasoline is the main fuel for small vehicles, and the exhaust emissions from its combustion have a major impact on the atmospheric environment. In the cumbersome process of gasoline refining, the aim is to reduce the sulfur and olefin contents within the raw material while maintaining its octane number as much as possible. In general, the research octane number (RON) loss is measured by using an instrument in the laboratory, which is time-consuming and expensive. Therefore, the use of algorithms to build RON loss prediction models has become a hot topic. Considering that machine learning has a good ability in fitting the non-linear complex data, we propose a stacking-based heterogeneous ensemble method for RON prediction. First, we propose a fusion algorithm of sequence forward search (SFS) and feature importance score to reduce the dimension of data set. Later, the data after feature selection will be used to construct a two-stage stacking heterogeneous ensemble learning model. Finally, the differential evolution (DE) algorithm is used to optimize multiple sensitive parameters involved in the model. Experiments with data obtained in the actual gasoline refining process show that the proposed method can accurately predict the RON loss in the product. Compared to the popular machine learning methods such as support vector machines, random forests, and XGBoost, the proposed method achieves the smallest mean square error. Furthermore, we analyze the important features that affect the RON loss to promote the development of the gasoline refining industry.

Heterogeneous Ensemble with Combined Dimensionality Reduction for Social Spam Detection

This study presents a novel framework based on a heterogeneous ensemble method and a hybrid dimensionality reduction technique for spam detection in micro-blogging social networks. A hybrid of Information Gain (IG) and Principal Component Analysis (PCA) (dimensionality reduction) was implemented for the selection of important features and a heterogeneous ensemble consisting of Naïve Bayes (NB), K Nearest Neighbor (KNN), Logistic Regression (LR) and Repeated Incremental Pruning to Produce Error Reduction (RIPPER) classifiers based on Average of Probabilities (AOP) was used for spam detection. The proposed framework was applied on MPI_SWS and SAC’13 Tip spam datasets and the developed models were evaluated based on accuracy, precision, recall, f-measure, and area under the curve (AUC). From the experimental results, the proposed framework (that is, Ensemble + IG + PCA) outperformed other experimented methods on studied spam datasets. Specifically, the proposed method had an average accuracy value of 87.5%, an average precision score of 0.877, an average recall value of 0.845, an average F-measure value of 0.872 and an average AUC value of 0.943. Also, the proposed method had better performance than some existing methods. Consequently, this study has shown that addressing high dimensionality in spam datasets, in this case, a hybrid of IG and PCA with a heterogeneous ensemble method can produce a more effective method for detecting spam contents.

An ensemble learning method for variable selection and its application on Railroad Fatal Accidents.

Scalars based on linear and generalized linear models are commonly used in fatality and disease predictions. Currently, standard approaches for variable selection in linear and generalized linear models are not explored well with high dimensional data. This manuscript proposed a new variable selection method for logistics models with inspiration from heterogeneous ensemble methods. The goal of this study is to extend classical variables selection methods such as stepwise, lasso, or RF in the case of high dimensional accident data with missing values. Real world accidents-collated data sets are collected from the official website of US Railroad Administration and US Police Department from 2000 to 2015. Preprocessing includes log transformation and z score transformation. Simulated sampling was applied due to the imbalanced outcome. The base variables was selected from the Empirical Model (EM) [1], Lasso model[2], Xgboost[3] and the traditional stepwise backward. An intermediate selection was proposed by ensemble training of base variables. Final models built by variables selected improved Area Under the Curve (AUC) from 0.79 to 0.84 and from 0.79 to 0.84 in test data compared to arbitrary APS model. Comparison with the existing model shows that the logistics model based on selected variables has high accuracy.

A Heterogeneous Ensemble Learning Method For Neuroblastoma Survival Prediction.

Neuroblastoma is a pediatric cancer with high morbidity and mortality. Accurate survival prediction of patients with neuroblastoma plays an important role in the formulation of treatment plans. In this study, we proposed a heterogeneous ensemble learning method to predict the survival of neuroblastoma patients and extract decision rules from the proposed method to assist doctors in making decisions. After data preprocessing, five heterogeneous base learners were developed, which consisted of decision tree, random forest, support vector machine based on genetic algorithm, extreme gradient boosting and light gradient boosting machine. Subsequently, a heterogeneous feature selection method was devised to obtain the optimal feature subset of each base learner, and the optimal feature subset of each base learner guided the construction of the base learners as a priori knowledge. Furthermore, an area under curve-based ensemble mechanism was proposed to integrate the five heterogeneous base learners. Finally, the proposed method was compared with mainstream machine learning methods from different indicators, and valuable information was extracted by using the partial dependency plot analysis method and rule-extracted method from the proposed method. Experimental results show that the proposed method achieves an accuracy of 91.64%, recall of 91.14%, and AUC of 91.35% and is significantly better than the mainstream machine learning methods. In addition, interpretable rules with accuracy higher than 0.900 and predicted responses are extracted from the proposed method. Our study can effectively improve the performance of the clinical decision support system to improve the survival of neuroblastoma patients.