Heterogeneous Ensemble Learning Research Articles

Semantic Query-Featured Ensemble Learning Model for SQL-Injection Attack Detection in IoT-Ecosystems

Structured query language (SQL) has emerged as one of the most used databases, serving an array of Internet-of-Things (IoTs)-enabled services including web-transactions, grid networks, industrial activity log and proactive decision systems, smart-home, financial transactions, business communication etc. With high pace increase in SQL-driven IoT applications, the threat of SQL-injection attacks (SQLIAs) at the middleware layer has increased significantly. To address such issues, machine learning-based SQLIA-prediction systems are proposed; however, majority of the existing methods are found limited in terms of intrusion detection accuracy because of their complete-reliance on structural features and inferior learning model(s). On the contrary, intruders these days intrude the system by mimicking the normal queries and hence confuses most of the classical learning-based methods. To alleviate such problems, this article emphasizes on exploiting semantic features along with the state-of-art highly robust computing environment. We proposed a robust semantic query-featured ensemble learning model for SQLIA prediction. Unlike classical (query's) template-matching or term-assessment-based methods, our proposed SQLIA-prediction model exploits latent semantic features from large SQL-queries to train an ensemble learning model that classifies each query as the normal query or the SQLIA query. Functionally, it performs preprocessing over large set of SQL-queries using count-vectorizer and stopping word removal. Subsequently, it applies Word2Vec feature extraction method over each query using continuous bag of words (CBOW) and N-skip gram (SKG) algorithms, which obtained CBOW and SKG semantic features from each SQL-query. The extracted features were processed for data resampling so as to alleviate the problem of class-imbalance and skewness. To alleviate redundant computation, two feature selection algorithms named Mann-Whitney significance predictor test and principal component analysis were applied over the resampled features. Moreover, to eliminate over-fitting and convergence problem, Min-Max normalization was performed over the selected features which were later processed for learning using a state-of-art robust heterogeneous ensemble learning model. Unlike standalone classifier-based SQLIA, the proposed learning-model employed a set of nine base classifiers designed to serve maximum voting ensemble-based prediction. The proposed ensemble-learning method classified each SQL-query as the normal-query or the SQLIA-query. Simulation results affirmed superiority of the proposed SQLIA prediction model in terms of accuracy (98%), F-Score (0.989), AUC (0.999) signifying its efficacy toward real-world SQL-driven IoT-ecosystems.

Critical Concurrent Feature Selection and Enhanced Heterogeneous Ensemble Learning Approach for Fault Detection in Industrial Processes

Machine learning approaches have been successfully applied to fault detection in complex industrial processes, while it is still challenging to design an optimal feature space and a stable model for various faults. To solve this problem, a hybrid approach with critical concurrent feature selection and enhanced heterogeneous ensemble learning is proposed. First, raw process variables from field instruments and slow features obtained by simple linear transformation are ingeniously combined to construct the concurrent feature space so as to contain static and dynamic information. Then, the critical concurrent feature space of each individual learner is automatically obtained by the optimal selection process, and the feature diversity of ensemble is realized simultaneously. Finally, an enhanced heterogeneous ensemble model is constructed by different optimized individual learners, which effectively improves the classification accuracy and stability in fault detection. The performance of the proposed approach is evaluated in a simulated Tennessee Eastman benchmark and a real-word three-phase flow process. The experimental results illustrate that the performance of each optimized individual learner outperforms classic random forest algorithm. The generalization and stability of the ensemble model are further improved. Compared with traditional classification algorithms, the proposed approach achieves superior performance with accuracy that exceeds 89% for the Tennessee Eastman process and exceeds 88% for the three-phase flow process respectively. Additionally, the selected critical concurrent features indicate that both static and dynamic information play important roles in fault detection of industrial processes.

Developing a heterogeneous ensemble learning framework to evaluate Alkali-silica reaction damage in concrete using acoustic emission signals

The monitoring and evaluation of Alkali-silica reaction (ASR) damage in concrete structures are required to ensure the serviceability and integrity of concrete infrastructures such as bridges and dams. The innovation of this paper lies in the development of an automatic ASR monitoring and evaluation approach by leveraging acoustic emission (AE) and a heterogeneous ensemble learning framework. In this paper, ASR was monitored by AE sensors attached to a concrete specimen, which was placed in a chamber with high humidity and temperature. The recorded AE signals were filtered and divided by four ASR phases according to signal strength, crack width and expansion strain. A heterogeneous ensemble network including convolutional neural networks (CNN) and random forest models was employed to learn different features from AE signals and classify the AE signals into their corresponding phases. The results suggest that the proposed model has a high performance and classifies the signals into the assigned phases with high accuracy.

A hybrid ensemble-based deep-learning framework for landslide susceptibility mapping

Landslides are highly hazardous geological disasters that can potentially threaten the safety of human life and property. As a result, landslide susceptibility mapping (LSM) plays an important role in the landslide prevention system. Recently, many deep learning (DL) models have been adopted for LSM, but they also face problems such as sensitivity to overfitting and lower mapping accuracy. In this paper, a novel hybrid LSM framework is proposed based on four heterogeneous ensemble learning (HEL) methods with three single DL models: deep belief network (DBN), convolutional neural network (CNN) and deep residual network (ResNet). The proposed model is tested at the Three Gorges Reservoir area, China. 202 historical landslides and ten conditioning factors were selected to construct a geospatial dataset for LSM. The conditioning factors with high-correlation and low importance were removed from the dataset by using the Spearman Correlation Index and random forests. The geospatial dataset was then divided into two subsets: 70% for training and 30% for testing. Then LSM results was carried out by the single and proposed HEL-based models. The quantitative evaluation of the results showed that the proposed HEL-based models improved the LSM accuracy, and outperformed the single DL LSM models. Stacking model achieved the highest AUC value (0.984), highest Kappa (86.95%), highest overall accuracy (94.17%), highest precision (88.87%), highest Matthews correlation coefficient (87.03%) and highest F1-score (91.34%) among all of models for the testing dataset, while the Boosting model obtained the highest Recall value (96.02%). At the same time, HEL-based models proposed in this study also show better stability and can avoid the overfitting effectively. In addition, the Gini index showed that elevation factor contributes most in LSM in the study area. In general, the proposed framework has promising applicability in improving LSM accuracy.

An Intelligent Ensemble Classification Method For Spam Diagnosis in Social Networks

In recent years, the destructive behavior of social networks spammers has seriously threatened the information security of ordinary users. To reduce this threat, many researchers have extracted the behavioral characteristics of spam and obtained good results based on machine learning algorithms to identify them. However, most of these studies use a single classification technique that often works differently for different spam data. In this paper, an intelligent ensemble classification method for social networks spam detection is introduced. The proposed heterogeneous ensemble learning framework is based on stack generalization and uses an evolutionary algorithm to improve the modeling process and reduce complexity. In particular, particle swarm optimization has been used as an evolutionary algorithm to optimize model parameters to reduce model complexity. These parameters include a subset of effective features and a subset of the most appropriate single classification techniques. The SPAM E-mail dataset used in this article contains the correct and effective features in spam prediction. Experimental results show that the proposed algorithm effectively improves the detection rate of spam and performs better than the methods used.

Using an ensemble learning approach in digital soil mapping of soil pH for the Thompson-Okanagan region of British Columbia

Information on the spatial distribution of soil pH is essential for assessing soil quality and soil productivity. Digital soil mapping (DSM) is commonly used to predict soil characteristics over various types of landscapes. Over the past decade, researchers have made progress using machine learning techniques to provide reliable predictions of soil properties with limited data. DSM studies often use a single learning approach, which is constructed with a machine learner that systematically extracts soil–environment relationships from a large database, whereby a fitted model is used to predict soil information in an unmapped area. The practice of using an ensemble learning approach, especially one that combines several base learners, has rarely been tested in DSM. We developed a workflow for using an ensemble learning algorithm to predict soil properties for the Thompson-Okanagan region of British Columbia, Canada. Here, we focused on soil pH and tested a variety of base learners. Base learners with high prediction accuracies were then used to construct a SuperLearner (SL) to extract the complex relationships between soil properties and environmental variables. The fitted SL was then used to predict soil properties at 25 m spatial resolution at three depth intervals (0–5, 5–15, and 15–30 cm). Prediction accuracies were assessed using an independent test dataset, which indicated that the SL had a similar prediction accuracy to the best individual base learners. Using the heterogeneous ensemble learning approach with a weighted average stacked generalization process eliminated the need to choose the best base learner.

Machine Learning Approaches with Automated Sleep Staging System based on Two-Layer Heterogeneous Ensemble Learning Stacking Model

Machine Learning Approaches with Automated Sleep Staging System based on Two-Layer Heterogeneous Ensemble Learning Stacking Model

Prediction in Traffic Accident Duration Based on Heterogeneous Ensemble Learning

ABSTRACT Based on millions of traffic accident data in the United States, we build an accident duration prediction model based on heterogeneous ensemble learning to study the problem of accident duration prediction in the initial stage of the accident. First, we focus on the earlier stage of the accident development, and select some effective information from five aspects of traffic, location, weather, points of interest and time attribute. Then, we improve data quality by means of data cleaning, outlier processing and missing value processing. In addition, we encode category features for high-frequency category variables and extract deeper information from the limited initial information through feature extraction. A pre-processing scheme of accident duration data is established. Finally, from the perspective of model, sample and parameter diversity, we use XGBoost, LightGBM, CatBoost, stacking and elastic network to build a heterogeneous ensemble learning model to predict the accident duration. The results show that the model not only has good prediction accuracy but can synthesize multiple models to give a comprehensive degree of importance of influencing factors, and the feature importance of the model shows that the time, location, weather and relevant historical statistics of the accident are important to the accident duration.

A Novel Approach for Classifying COVID-19 Based on LBP-DCNN Feature Fusion and Stacking Heterogeneous Ensemble Learning

A Novel Approach for Classifying COVID-19 Based on LBP-DCNN Feature Fusion and Stacking Heterogeneous Ensemble Learning

Leveraging an Advanced Heterogeneous Ensemble Learning for Outcome-Based Predictive Monitoring Using Business Process Event Logs

Leveraging an Advanced Heterogeneous Ensemble Learning for Outcome-Based Predictive Monitoring Using Business Process Event Logs

Two-stage stacking heterogeneous ensemble learning method for gasoline octane number loss prediction

Gasoline is the main fuel for small vehicles, and the exhaust emissions from its combustion have a major impact on the atmospheric environment. In the cumbersome process of gasoline refining, the aim is to reduce the sulfur and olefin contents within the raw material while maintaining its octane number as much as possible. In general, the research octane number (RON) loss is measured by using an instrument in the laboratory, which is time-consuming and expensive. Therefore, the use of algorithms to build RON loss prediction models has become a hot topic. Considering that machine learning has a good ability in fitting the non-linear complex data, we propose a stacking-based heterogeneous ensemble method for RON prediction. First, we propose a fusion algorithm of sequence forward search (SFS) and feature importance score to reduce the dimension of data set. Later, the data after feature selection will be used to construct a two-stage stacking heterogeneous ensemble learning model. Finally, the differential evolution (DE) algorithm is used to optimize multiple sensitive parameters involved in the model. Experiments with data obtained in the actual gasoline refining process show that the proposed method can accurately predict the RON loss in the product. Compared to the popular machine learning methods such as support vector machines, random forests, and XGBoost, the proposed method achieves the smallest mean square error. Furthermore, we analyze the important features that affect the RON loss to promote the development of the gasoline refining industry.

A two-layer nested heterogeneous ensemble learning predictive method for COVID-19 mortality

The COVID-19 epidemic has had a great adverse impact on the world, having taken a heavy toll, killing hundreds of thousands of people. In order to help the world better combat COVID-19 and reduce its death toll, this study focuses on the COVID-19 mortality. First, using the multiple stepwise regression analysis method, the factors from eight aspects (economy, society, climate etc.) that may affect the mortality rates of COVID-19 in various countries is examined. In addition, a two-layer nested heterogeneous ensemble learning-based prediction method that combines linear regression (LR), support vector machine (SVM), and extreme learning machine (ELM) is developed to predict the development trends of COVID-19 mortality in various countries. Based on data from 79 countries, the experiment proves that age structure (proportion of the population over 70 years old) and medical resources (number of beds) are the main factors affecting the mortality of COVID-19 in each country. In addition, it is found that the number of nucleic acid tests and climatic factors are correlated with COVID-19 mortality. At the same time, when predicting COVID-19 mortality, the proposed heterogeneous ensemble learning-based prediction method shows better prediction ability than state-of-the-art machine learning methods such as LR, SVM, ELM, random forest (RF), long short-term memory (LSTM) etc.

No-reference stereoscopic image quality assessment using quaternion wavelet transform and heterogeneous ensemble learning

As the demand for high-quality stereo images has grown in recent years, stereoscopic image quality assessment (SIQA) has become an important research area in modern image processing technology.In this paper, we propose a no-reference stereoscopic image quality assessment (NR-SIQA) model using heterogeneous ensemble learning ‘quality-aware’ features from luminance image, chrominance image, disparity and cyclopean images via quaternion wavelet transform (QWT). Firstly, luminance image and chrominance image are generated by CIELAB color space as monocular perception, and the novel disparity and cyclopean images are utilized to complement with monocular information. Then, a number of ‘quality-aware’ features in the quaternion wavelet domain are discovered, including entropy, texture features, energy features, energy differences features and MSCN coefficients of high frequency sub-band. Finally, a heterogeneous ensemble model via support vector regression (SVR) & extreme learning machine (ELM) & random forest (RF) is proposed to predict quality score, and bootstrap sampling and rotated feature space are used to increase the diversity of data distribution. Comparing with the state-of-the-art NR-SIQA models, experimental results on four public databases prove the accuracy and robustness of the proposed model.

HEAL: Heterogeneous Ensemble and Active Learning Framework

Network Intrusion Detection is a well-known, relevant problem that has gained even more interest due to the exponential growth of technologies, systems and volume of data. The constantly evolving attacks require a continuous effort towards the development of novel and robust detection solutions. In this paper we propose a Heterogeneous Ensemble and Active Learning (HEAL) system, a novel tool that incorporates the implementation of a dynamic heterogeneous ensemble model with active learning capabilities. This ensures a solution that: i) adapts to changes in data through time, ii) remains robust providing good performance, iii) handles a continuous flow of data, and iv) requires less human intervention when compared against pure active learning solutions. HEAL system uses multiple individual base models to build a heterogeneous ensemble learner that adapts to the specific data characteristics. Then, active learning is applied to the ensemble so that it is retrained and re-evaluated with respect to time and new instances. Instances where the model has a low confidence are labeled by a domain expert. A new model is retrained with these instances and its performance is evaluated. The deployed model is replaced when the new model exhibits performance advantages. Finally, an experimental comparison of the performance at different stages is carried out in a case study using the well-known NSL-KDD data set. In this study we show the advantages of using HEAL system.

A Heterogeneous Ensemble Learning Method For Neuroblastoma Survival Prediction.

Neuroblastoma is a pediatric cancer with high morbidity and mortality. Accurate survival prediction of patients with neuroblastoma plays an important role in the formulation of treatment plans. In this study, we proposed a heterogeneous ensemble learning method to predict the survival of neuroblastoma patients and extract decision rules from the proposed method to assist doctors in making decisions. After data preprocessing, five heterogeneous base learners were developed, which consisted of decision tree, random forest, support vector machine based on genetic algorithm, extreme gradient boosting and light gradient boosting machine. Subsequently, a heterogeneous feature selection method was devised to obtain the optimal feature subset of each base learner, and the optimal feature subset of each base learner guided the construction of the base learners as a priori knowledge. Furthermore, an area under curve-based ensemble mechanism was proposed to integrate the five heterogeneous base learners. Finally, the proposed method was compared with mainstream machine learning methods from different indicators, and valuable information was extracted by using the partial dependency plot analysis method and rule-extracted method from the proposed method. Experimental results show that the proposed method achieves an accuracy of 91.64%, recall of 91.14%, and AUC of 91.35% and is significantly better than the mainstream machine learning methods. In addition, interpretable rules with accuracy higher than 0.900 and predicted responses are extracted from the proposed method. Our study can effectively improve the performance of the clinical decision support system to improve the survival of neuroblastoma patients.

Identification and Portrait of Urban Functional Zones Based on Multisource Heterogeneous Data and Ensemble Learning

Urban functional zones are important space carriers for urban economic and social function. The accurate and rapid identification of urban functional zones is of great significance to urban planning and resource allocation. However, the factors considered in the existing functional zone identification methods are not comprehensive enough, and the recognition of functional zones stops at their categories. This paper proposes a framework that combines multisource heterogeneous data to identify the categories of functional zones and draw the portraits of functional zones. The framework comprehensively describes the features of functional zones from four aspects: building-level metrics, landscape metrics, semantic metrics, and human activity metrics, and uses a combination of ensemble learning and active learning to balance the identification accuracy of functional zones and the labeling cost during large-scale generalization. Furthermore, sentiment analysis, word cloud analysis, and land cover proportion maps are added to the portraits of typical functional zones to make the image of functional zones vivid. The experiment carried out within the Fifth Ring Road, Haidian District, Beijing, shows that the overall accuracy of the method reached 82.37% and the portraits of the four typical functional zones are clear. The method in this paper has good repeatability and generalization, which is helpful to carry out quantitative and objective research on urban functional zones.

Distributed heterogeneous ensemble learning on Apache Spark for ligand-based virtual screening

Virtual screening is one of the most common computer-aided drug design techniques that apply computational tools and methods on large libraries of molecules to extract the drugs. Ensemble learning is a recent paradigm launched to improve machine learning results in terms of predictive performance and robustness. It has been successfully applied in ligand-based virtual screening (LBVS) approaches. Applying ensemble learning on huge molecular libraries is computationally expensive. Hence, the distribution and parallelisation of the task have become a significant step by using sophisticated frameworks such as Apache Spark. In this paper, we propose a new approach HEnsL_DLBVS, for heterogeneous ensemble learning, distributed on Spark to improve the large-scale LBVS results. To handle the problem of imbalanced big training datasets, we propose a novel hybrid technique. We generate new training datasets to evaluate the approach. Experimental results confirm the effectiveness of our approach with satisfactory accuracy and its superiority over homogeneous models.

An Adaptive Heterogeneous Online Learning Ensemble Classifier for Nonstationary Environments.

In recent years, the prevalence of technological advances has led to an enormous and ever-increasing amount of data that are now commonly available in a streaming fashion. In such nonstationary environments, the underlying process generating the data stream is characterized by an intrinsic nonstationary or evolving or drifting phenomenon known as concept drift. Given the increasingly common applications whose data generation mechanisms are susceptible to change, the need for effective and efficient algorithms for learning from and adapting to evolving or drifting environments can hardly be overstated. In dynamic environments associated with concept drift, learning models are frequently updated to adapt to changes in the underlying probability distribution of the data. A lot of work in the area of learning in nonstationary environments focuses on updating the learning predictive model to optimize recovery from concept drift and convergence to new concepts by adjusting parameters and discarding poorly performing models while little effort has been dedicated to investigate what type of learning model is suitable at any given time for different types of concept drift. In this paper, we investigate the impact of heterogeneous online ensemble learning based on online model selection for predictive modeling in dynamic environments. We propose a novel heterogeneous ensemble approach based on online dynamic ensemble selection that accurately interchanges between different types of base models in an ensemble to enhance its predictive performance in nonstationary environments. The approach is known as Heterogeneous Dynamic Ensemble Selection based on Accuracy and Diversity (HDES-AD) and makes use of models generated by different base learners to increase diversity to circumvent problems associated with existing dynamic ensemble classifiers that may experience loss of diversity due to the exclusion of base learners generated by different base algorithms. The algorithm is evaluated on artificial and real-world datasets with well-known online homogeneous online ensemble approaches such as DDD, AFWE, and OAUE. The results show that HDES-AD performed significantly better than the other three homogeneous online ensemble approaches in nonstationary environments.

A Heterogeneous Ensemble Learning Model Based on Data Distribution for Credit Card Fraud Detection

Credit card fraud detection (CCFD) is important for protecting the cardholder’s property and the reputation of banks. Class imbalance in credit card transaction data is a primary factor affecting the classification performance of current detection models. However, prior approaches are aimed at improving the prediction accuracy of the minority class samples (fraudulent transactions), but this usually leads to a significant drop in the model’s predictive performance for the majority class samples (legal transactions), which greatly increases the investigation cost for banks. In this paper, we propose a heterogeneous ensemble learning model based on data distribution (HELMDD) to deal with imbalanced data in CCFD. We validate the effectiveness of HELMDD on two real credit card datasets. The experimental results demonstrate that compared with current state‐of‐the‐art models, HELMDD has the best comprehensive performance. HELMDD not only achieves good recall rates for both the minority class and the majority class but also increases the savings rate for banks to 0.8623 and 0.6696, respectively.

Distributed heterogeneous ensemble learning on Apache Spark for ligand-based virtual screening

Distributed heterogeneous ensemble learning on Apache Spark for ligand-based virtual screening