Hybrid Ensemble Method Research Articles

Comparative Analysis of Machine Learning Models and Hybrid Ensemble Approach’s for Landslide Prediction

The NH-44 Jammu Srinagar National Highway in India is susceptible to landslides, rock falls and shooting stones due to its geological characteristics and steep slopes. This study aims to compare the performance of various Machine Learning (ML) algorithms and hybrid models in predicting landslides using historical data. Seven optimized ML approaches Support Vector Classifier (SVC), Logistic Regression (LR), Decision Tree (DT), K Nearest Classifier (KNC), Random Forest (RF), GaussianNB (GNB) and AdaBoost Classifier (ABC) are used. Additionally, two Hybrid Ensemble methods, the Voting Hybrid Model (VHM) and Stacking Hybrid Model (SHM), are introduced. The performance of each model is evaluated using accuracy, precision, recall, F1-score and AUC metrics. Results indicate that all the models performed well, with hybrid ensemble models surpassing all individual algorithms. The Stacking Hybrid Model (SHM) excels achieving 99.4% accuracy and 98.5% AUC, outperforming the Voting Hybrid Model (VHM). Hybrid models consistently outperform individual models in accuracy and AUC. These proposed methods exhibit robustness and enhanced results in addressing this issue. This framework can help to predict the landslides with high accuracy which can save the lives through timely evacuations from high-risk areas.

A new understanding of the alkali-silica reaction expansion in concrete using a hybrid ensemble model

Alkali-Silica Reaction (ASR) is a critical issue affecting the durability of concrete structures, leading to significant long-term maintenance costs and safety concerns. This study addresses the need for accurate prediction of ASR-induced expansion in concrete, leveraging advanced machine learning techniques. Two hybrid ensemble methods, the inverse variance method and the artificial neural network-based ensemble method, were proposed to integrate the individual models to enhance the predictive capabilities. A comprehensive experimental database comprising over 1997 sets of ASR expansion data was collected from relevant literature with the environmental conditions, specimen geometry and material composition as the input variables and specimen strain as the output variable. The findings indicate that the predictive performance of individual models falls short of the performance of the hybrid ensemble machine learning model, with the inverse variance-based ensemble model exhibiting the highest performance (R = 97.2 %, RMSE = 0.066 mm). The Shapley Additive Explanations analysis reveals that the most influential factors include reaction days, alkali content, aggregate particle size, and silica content, contributing to up to 35 % of the variation in ASR expansion. This work offers valuable insights for the efficient assessment of durability of concrete structures concerning ASR expansion.

Effectiveness of an ensemble technique based on the distributivity equation in detecting suspicious network activity

With the growing complexity and frequency of cyber threats, there is a pressing need for more effective defense mechanisms. Machine learning offers the potential to analyze vast amounts of data and identify patterns indicative of malicious activity, enabling faster and more accurate threat detection. Ensemble methods, by incorporating diverse models with varying vulnerabilities, can increase resilience against adversarial attacks. This study covers the usage and evaluation of the relevance of an innovative approach of ensemble classification for identifying intrusion threats on a large CICIDS2017 dataset. The approach is based on the distributivity equation that appropriately aggregates the underlying classifiers. It combines various standard supervised classification algorithms, including Multilayer Perceptron Network, k-Nearest Neighbors, and Naive Bayes, to create an ensemble. Experiments were conducted to evaluate the effectiveness of the proposed hybrid ensemble method. The performance of the ensemble approach was compared with individual classifiers using measures such as accuracy, precision, recall, F-score, and area under the ROC curve. Additionally, comparisons were made with widely used state-of-the-art ensemble models, including the soft voting method (Weighted Average Probabilities), Adaptive Boosting (AdaBoost), and Histogram-based Gradient Boosting Classification Tree (HGBC) and with existing methods in the literature using the same dataset, such as Deep Belief Networks (DBN), Deep Feature Learning via Graph (Deep GFL). Based on these experiments, it was found that some ensemble methods, such as AdaBoost and Histogram-based Gradient Classification Tree, do not perform reliably for the specific task of identifying network attacks. This highlights the importance of understanding the context and requirements of the data and problem domain. The results indicate that the proposed hybrid ensemble method outperforms traditional algorithms in terms of classification precision and accuracy, and offers insights for improving the effectiveness of intrusion detection systems.

Enhanced Classification of Imbalanced Medical Datasets using Hybrid Data-Level, Cost-Sensitive and Ensemble Methods

Addressing the class imbalance in classification problems is particularly challenging, especially in the context of medical datasets where misclassifying minority class samples can have significant repercussions. This study is dedicated to mitigating class imbalance in medical datasets by employing a hybrid approach that combines data-level, cost-sensitive, and ensemble methods. Through an assessment of the performance, measured by AUC-ROC values, Sensitivity, F1-Score, and G-Mean of 20 data-level and four cost-sensitive models on seventeen medical datasets - 12 small and five large, a hybridized model, SMOTE-RF-CS-LR has been devised. This model integrates the Synthetic Minority Oversampling Technique (SMOTE), the ensemble classifier Random Forest (RF), and the Cost-Sensitive Logistic Regression (CS-LR). Upon testing the hybridized model on diverse imbalanced ratios, it demonstrated remarkable performance, achieving outstanding performance values on the majority of the datasets. Further examination of the model's training duration and time complexity revealed its efficiency, taking less than a second to train on each small dataset. Consequently, the proposed hybridized model not only proves to be time-efficient but also exhibits robust capabilities in handling class imbalance, yielding outstanding classification results in the context of medical datasets.

Towards unbalanced multiclass intrusion detection with hybrid sampling methods and ensemble classification

Intrusion Detection Systems (IDS) play a crucial role in securing computer networks against malicious activities. However, their efficacy is consistently hindered by the persistent challenge of class imbalance in real-world datasets. While various methods, such as resampling techniques, ensemble methods, cost-sensitive learning, data augmentation, and so on, have individually addressed imbalance classification issues, there exists a notable gap in the literature for effective hybrid methodologies aimed at enhancing IDS performance. To bridge this gap, our research introduces an innovative methodology that integrates hybrid undersampling and oversampling strategies within an ensemble classification framework. This novel approach is designed to harmonize dataset distributions and optimize IDS performance, particularly in intricate multi-class scenarios. In-depth evaluations were conducted using well-established intrusion detection datasets, including the Car Hacking: Attack and Defense Challenge 2020 (CHADC2020) and IoTID20. Our results showcase the remarkable efficacy of the proposed methodology, revealing significant improvements in precision, recall, and F1-score metrics. Notably, the hybrid-ensemble method demonstrated an exemplary average F1 score exceeding 98% for both datasets, underscoring its exceptional capability to substantially enhance intrusion detection accuracy. In summary, this research represents a significant contribution to the field of IDS, providing a robust solution to the pervasive challenge of class imbalance. The hybrid framework not only strengthens IDS efficacy but also illuminates the seamless integration of undersampling and oversampling within ensemble classifiers, paving the way for fortified network defenses.

AN UNSUPERVISED-SUPERVISED ENSEMBLE TECHNOLOGY WITH NON-ITERATIVE TRAINING ALGORITHM FOR SMALL BIOMEDICAL DATA ANALYSIS

Improving the accuracy of intelligent data analysis is an important task in various application areas. Existing machine learning methods do not always provide a sufficient level of classification accuracy for their use in practice. That is why, in recent years, hybrid ensemble methods of intellectual data analysis have begun to develop. They are based on the combined use of clustering and classification procedures. This approach provides an increase in the accuracy of the classifier based on machine learning due to the expansion of the space of the input data of the task by the results of the clustering.
 In this paper, the tasks of modification and improvement of such technology for small data analysis are considered. The basis of the modification is the use of clustering with output at the first step of the method to increase the accuracy of the entire technology. Despite the high accuracy of the work, this approach requires a significant expansion of the inputs of the final linear classifier (labels of the obtained clusters are added to the initial inputs). To avoid this shortcoming, the paper proposes an improvement based on the introduction of a new classification procedure at the first step of the method and replaces all the initial inputs of the task with the results of its work. In parallel with it, clustering is performed taking into account the original attribute, the results of which are added to the output of the classifier of the first step. In this way, the formation of an expanded set of data of significantly lower dimensionality in comparison with the existing method takes place (here there is no longer a large number of initial features, which is characteristic of biomedical engineering tasks). This reduces the training time of the method and increases its generalization properties.
 Modeling of the method was based on the use of a short dataset contained in an open repository. After the preprocessing procedures, the dataset has only 294 vectors, each of which was characterized by 18 attributes. Data classification was done using an SGTM-based neural-like structure classifier. This linear classifier provides high accuracy of work. In addition, it does not provide for the implementation of an iterative training procedure and additional adjustment of work parameters. Data clustering was performed using the k-means method. This choice is due to both the simplicity and speed of its work. 
 The search for the optimal number of k-means clusters was carried out using 4 different methods. They all showed different results. That is why, some experiments were conducted to assess the influence of different numbers of clusters (from 3 to 7) on the accuracy of all 4 algorithms of the developed technology. The accuracy of the proposed technology has been established experimentally in comparison with the linear classifier and the existing hybrid method. In addition, by reducing the inputs of the final classifier, the developed technology reduces the duration of the training procedure compared to the basic method. All this ensures the possibility of using the proposed technology when solving various applied problems of medical diagnostics, in particular, based on the analysis of small data.
 Keywords: small data approach, non-iterative training, ensemble learning, unsupervised-supervised technology, biomedical engineering.

Highly imbalanced fault classification of wind turbines using data resampling and hybrid ensemble method approach

Deep learning-based incipient fault diagnostic techniques have achieved surprisingly well in wind turbines. Due to component failures, wind turbines must undergo active maintenance, substantially influencing revenue and power generation. Unfortunately, there are consistently uneven data distributions between samples with faults and those without faults, resulting in incorrect fault classification. Wind turbine fault classification has a significant data imbalance problem, compromising learning attention for majority and minority classes. Machine learning methodologies based on Generative Adversarial Networks (GAN), over-sampling, and under-sampling techniques for generating synthetic data have been widely employed to address the imbalance data problem. However, the traditional synthetic minority oversampling technique (SMOTE) accomplishes oversampling using linear interpolation between close minority class samples, which could be confusing, subpar, and indistinguishable from the majority class. This study suggests combining over and under-sampling using adaptive SMOTE and edited nearest neighbors (ASMOTE-ENN) that incorporate over-sampling with adaptive SMOTE and under-sampling with ENN to improve the quality of the generated samples. With this resampling technique, noise in an imbalanced dataset is reduced on three levels by using an adaptive nearest neighbor selection algorithm to find the nearest neighbors that are visible. Then use SMOTE to create samples that precisely fall into the minority class, and later use the ENN technique to eliminate instances that contribute to noise afterwards. Resampling data created by combining over- and under-sampling approaches to match the data distribution over all classes is the foundation of the suggested method’s efficacy. A hybrid ensemble method is used for effective classification, including boosting, bagging, and stacking techniques. The original unbalanced and balanced data using the ASMOTE-ENN algorithm were classified using the proposed hybrid ensemble method. The classification results show that the proposed strategy is more accurate than a few imbalanced fault diagnosis techniques.

A hybrid ensemble method with negative correlation learning for regression

A hybrid ensemble method with negative correlation learning for regression

Development of a new hybrid ensemble method for accurate characterization of future drought using multiple global climate models

Development of a new hybrid ensemble method for accurate characterization of future drought using multiple global climate models

An improved approach for initial stage detection of laryngeal cancer using effective hybrid features and ensemble learning method

An improved approach for initial stage detection of laryngeal cancer using effective hybrid features and ensemble learning method

Predicting carbonation depth of concrete using a hybrid ensemble model

This study aims to develop a robust and accurate machine learning-based model to improve the prediction accuracy of carbonation depth in complex concrete structures. Two hybrid ensemble methods, the inverse variance method and the artificial neural network-based ensemble method, were proposed to integrate multiple algorithms. The models were trained on a dataset of 532 data points with 6 input variables. Performance evaluation metrics and the Taylor diagram were used to compare the models. The results showed that the hybrid ensemble models outperformed the single models, with the inverse variance-based model achieving the highest performance (R = 0.975, RMSE = 2.978 mm). Additionally, the contribution analysis indicated that carbonation time, CO2 concentration, and the amount of binder were the most influential factors. This study provides an efficient prediction model for carbonation depth and valuable insights for carbonation-resistant design in concrete structures.

A hybrid approach using multiple linear regression and random forest regression to predict molten steel temperature in a continuous casting tundish

ABSTRACT The temperature control of molten steel is essential to ensure operational stability in a steelmaking plant. The calculation of thermal losses in the steelmaking plant’s operations depends on highly dynamic variables, which motivates the construction of predictive models for the steel temperature. This paper proposed a hybrid ensemble method using multiple linear and random forest regression to predict the end molten steel temperature at the secondary refining required to achieve a target tundish temperature. Combining these two methods makes it possible to account for the linear and non-linear relationships in the data. The implemented models were trained on industrial data, and their performance was assessed using root mean squared error (RMSE) and a custom accuracy metric. The results showed that the proposed hybrid method achieves up to 5% better accuracy compared to linear regression or random forest regression methods alone, thus can enhance molten steel prediction in steelmaking plants.

New hybrid ensemble method for anomaly detection in data science

Anomaly detection is a significant research area in data science. Anomaly detection is used to find unusual points or uncommon events in data streams. It is gaining popularity not only in the business world but also in different of other fields, such as cyber security, fraud detection for financial systems, and healthcare. Detecting anomalies could be useful to find new knowledge in the data. This study aims to build an effective model to protect the data from these anomalies. We propose a new hyper ensemble machine learning method that combines the predictions from two methodologies the outcomes of isolation forest-k-means and random forest using a voting majority. Several available datasets, including KDD Cup-99, Credit Card, Wisconsin Prognosis Breast Cancer (WPBC), Forest Cover, and Pima, were used to evaluate the proposed method. The experimental results exhibit that our proposed model gives the highest realization in terms of receiver operating characteristic performance, accuracy, precision, and recall. Our approach is more efficient in detecting anomalies than other approaches. The highest accuracy rate achieved is 99.9%, compared to accuracy without a voting method, which achieves 97%.

On Precision Agriculture: Enhanced Automated Fruit Disease Identification and Classification Using a New Ensemble Classification Method

Fruits are considered among the most nutrient-dense cash crops around the globe. Since fruits come in different types, sizes, shapes, colors, and textures, the manual classification and disease identification of a large quantity of fruit is time-consuming and sluggish, requiring massive human intervention. We propose a multilevel fusion method for fruit disease identification and fruit classification that includes intensive fruit image pre-processing, customized image kernels for feature extraction with state-of-the-art (SOTA) deep methods, Gini-index-based controlled feature selection, and a hybrid ensemble method for identification and classification. We noticed certain limitations in the existing literature of adopting a single data source, in terms of limited data sizes, variability in fruit types, variability in quality, and variability in disease type. Therefore, we extensively aggregated and pre-processed multi-fruit data to simulate our proposed ensemble model on comprehensive datasets to cover both fruit classification and disease identification aspects. The multi-fruit imagery data contained regular and augmented images of fruits including apple, apricot, avocado, banana, cherry, fig, grape, guava, kiwi, mango, orange, peach, pear, pineapple, and strawberry. Similarly, we considered normal and augmented images of rotten fruits including beans (two categories), strawberries (seven categories), and tomatoes (three categories). For consistency, we normalized the images and designed an auto-labeling mechanism based on the existing image clusters to label inconsistent data to appropriate classes. Finally, we verified the auto-labeled data with a complete inspection to correctly assign it to the relevant classes. The proposed ensemble classifier outperforms all other classification methods, achieving 100% and 99% accuracy for fruit classification and disease identification. Further, we performed the analysis of variance (ANOVA) test to validate the statistical significance of the classifiers’ outcomes at α = 0.05. We achieved F-values of 32.41 and 11.42 against F-critical values of 2.62 and 2.86, resulting in p-values of 0.00 (<0.05) for fruit classification and disease identification.

Prediction of Complex Stock Market Data Using an Improved Hybrid EMD-LSTM Model

Because of the complexity, nonlinearity, and volatility, stock market forecasting is either highly difficult or yields very unsatisfactory outcomes when utilizing traditional time series or machine learning techniques. To cope with this problem and improve the complex stock market’s prediction accuracy, we propose a new hybrid novel method that is based on a new version of EMD and a deep learning technique known as long-short memory (LSTM) network. The forecasting precision of the proposed hybrid ensemble method is evaluated using the KSE-100 index of the Pakistan Stock Exchange. Using a new version of EMD that uses the Akima spline interpolation technique instead of cubic spline interpolation, the noisy stock data are first divided into multiple components technically known as intrinsic mode functions (IMFs) varying from high to low frequency and a single monotone residue. The highly correlated sub-components are then used to build the LSTM network. By comparing the proposed hybrid model with a single LSTM and other ensemble models such as the support vector machine (SVM), Random Forest, and Decision Tree, its prediction performance is thoroughly evaluated. Three alternative statistical metrics, namely root means square error (RMSE), mean absolute error (MAE) and mean absolute percentage error (MAPE), are used to compare the aforementioned techniques. The empirical results show that the suggested hybrid Akima-EMD-LSTM model beats all other models taken into consideration for this study and is therefore recommended as an effective model for the prediction of non-stationary and nonlinear complex financial time series data.

Cyberbullying Detection Based on Hybrid Ensemble Method using Deep Learning Technique in Bangla Dataset

Cyberbullying Detection Based on Hybrid Ensemble Method using Deep Learning Technique in Bangla Dataset

Credit scoring based on a Bagging-cascading boosted decision tree

Establishing precise credit scoring models to predict the potential default probability is vital for credit risk management. Machine learning models, especially ensemble learning approaches, have shown substantial progress in the performance improvement of credit scoring. The Bagging ensemble approach improves the credit scoring performance by optimizing the prediction variance while boosting ensemble algorithms reduce the prediction error by controlling the prediction bias. In this study, we propose a hybrid ensemble method that combines the advantages of the Bagging ensemble strategy and boosting ensemble optimization pattern, which can well balance the tradeoff of variance-bias optimization. The proposed method considers XGBoost as a base learner, which ensures the low-bias prediction. Moreover, the Bagging strategy is introduced to train the base learner to prevent over-fitting in the proposed method. Besides, the Bagging-boosting ensemble algorithm is further assembled in a cascading way, making the proposed new hybrid ensemble algorithm a good solution to balance the tradeoff of variance bias for credit scoring. Experimental results on the Australian, German, Japanese, and Taiwan datasets show the proposed Bagging-cascading boosted decision tree provides a more accurate credit scoring result.

Image-Processing-Based Intelligent Defect Diagnosis of Rolling Element Bearings Using Spectrogram Images

Due to the excellent image recognition characteristics of convolutional neural networks (CNN), they have gained significant attention among researchers for image-processing-based defect diagnosis tasks. The use of deep CNN models for rolling element bearings’ (REBs’) defect diagnosis may be computationally expensive, and therefore may not be suitable for some applications where hardware and resources limitations exist. However, instead of using CNN models as end-to-end image classifiers, they can also be used to extract the deep features from images and those features can further be used as input to machine learning (ML) models for defect diagnosis tasks. In addition to extracting deep features using CNN models, there are also other methods for feature extraction from vibration characteristic images, such as the extraction of handcrafted features using the histogram of oriented gradients (HOG) and local binary pattern (LBP) descriptors. These features can also be used as input to classical ML models for image classification tasks. In this study, a performance comparison between all these image-processing-based defect diagnosis techniques was carried out in terms of fault detection accuracy and computational expense. Moreover, based upon the detailed comparison, a hybrid-ensemble method involving decision-level fusion is proposed, which is far less computationally expensive compared to CNN models while using them as end-to-end classifiers. The performance of all these models is also compared in the case of minimal training data availability and for diagnosis under slightly different operating conditions to ascertain their generalizability and ability to correctly diagnose despite the minimal availability of training data. The performance of the proposed hybrid-ensemble method remained outstanding for the REBs’ defect diagnosis despite the minimal of availability training data as well as the slight variation under operating conditions.

Proposed Hybrid Ensemble Learning Algorithms for an Efficient Intrusion Detection System

Due to sophisticated cyber-attacks, and to produce false alarms on suspicious or unusual behavior to monitor computer resources, Intrusion Detection Systems (IDSs) are required. Hence, Many Machine Learning (ML) and data mining techniques have been proposed to increase the effectiveness of IDSs, whereas current IDS algorithms are still struggling to perform effectively while many IDSs depend on a single classifier to detect intrusions. Single-classifier IDSs cannot achieve high accuracy and low false alarm rates because of zero-day attacks. In this paper, a hybrid ensemble method using AdaBoosting and Bagging for IDS is proposed. This study aims to identify unknown (zero-day attacks) and known (well-known) attacks. So, the proposed model comprises three stages. The first stage is preprocessing. The second stage involves the application of AdaBoosting and Bagging methods by four different classifiers modifying (i.e., Naïve Bayesian (NB), Support Vector Machine (SVM), random forest (RF), and K_Nearest Neighbor (KNN)). Such a modification is performed for the AdaBoosting methods. The AdaBoosting classifier is then combined to work in the Bagging method. For attack recognition, uses the voting technique as the third stage. Experimental results reveal that using the UNSW BN15 dataset yields testing with 85.49% accuracy, 99.96% detection rate, and 0.006 false alarm rate. Therefore, the proposed Hybrid AdaBoosting and Bagging Method (HABBM) can outperform other comparable and state-of-the-art techniques across a variety of parameters. Index Terms—AdaBoosting method, Bagging method, Cyber Security CS, Ensemble Method, Intrusion Detection Systems IDSs.

Multi-component fault classification of a wind turbine gearbox using integrated condition monitoring and hybrid ensemble method approach

Gearbox has a high failure rate and downtime per failure in a wind turbine. Condition monitoring (CM) helps in diagnosis and prognosis of the faults. CM uses sensors to acquire data from different components and these signals are processed to extract fault features. The extracted features are further used to train different data driven models to identify and quantify the faults. A single decision-making algorithm does not perform well in different scenarios. To overcome this limitation, ensemble methods are used to diagnose the faults. Bagging and boosting are the widely used ensemble methods. However, boosting algorithms performs well on a noise-free data compared to bagging and in noisy settings, bagging methods are much robust than boosting. For this reason, a hybrid ensemble method that combines boosting, bagging, and stacking techniques to identify faults in a gearbox is proposed. Random forest (RF), decision tress (DT), KNN, AdaBoost, and gradient booster were used to compare the fault classification performance of the developed method. Experiments were conducted on the gearbox considering multi-component faults and integrated condition monitoring scheme. Vibration and acoustic signals acquired for different fault cases were processed using discrete wavelet transform (DWT) and the feature extraction was performed to obtain Kurtosis and Shannon entropy. These features were used to classify the faults using aforementioned algorithms. Results indicate that the developed method has the highest fault classification accuracy of 92%. The proposed method can be further used to diagnose similar machine faults in the industries.