Ensemble Classification Method Research Articles

Ensemble methods for testing a global null

Abstract Testing a global null is a canonical problem in statistics and has a wide range of applications. In view of the fact that no uniformly most powerful test exists, prior and/or domain knowledge are commonly used to focus on a certain class of alternatives to improve the testing power. However, it is generally challenging to develop tests that are particularly powerful against a certain class of alternatives. In this paper, motivated by the success of ensemble learning methods for prediction or classification, we propose an ensemble framework for testing that mimics the spirit of random forests to deal with the challenges. Our ensemble testing framework aggregates a collection of weak base tests to form a final ensemble test that maintains strong and robust power for global nulls. We apply the framework to four problems about global testing in different classes of alternatives arising from whole-genome sequencing (WGS) association studies. Specific ensemble tests are proposed for each of these problems, and their theoretical optimality is established in terms of Bahadur efficiency. Extensive simulations and an analysis of a real WGS dataset are conducted to demonstrate the type I error control and/or power gain of the proposed ensemble tests.

Evaluating and comparing bagging and boosting of hybrid learning for breast cancer screening

PurposeIn many fields, including bioinformatics, ensemble methods have been often employed to perform prediction tasks such as classification and regression due to their ability to improve the accuracy and robustness of machine learning models. Given that the most prevalent cancer and the primary cause of death for women is breast cancer, researchers have started using ensemble approaches to advance their studies in this area. Effectively, ensemble methods can be used to develop clinical decision support systems that can help healthcare providers make more accurate judgements. MethodsIn the present study, we construct and evaluate bagging and boosting ensemble methods for the binary classification of breast cancer screening images from the CBIS-DDSM dataset. To build the bagging ensembles, a hybrid architecture consisting of three feature extractors (Inception V3, MobileNet V2, DenseNet 201) and four classifiers (K-nearest neighbors, Multilayer perceptron, Support vector machine, Decision trees) was chosen. As for the boosting ensembles, the method consists of the same three feature extractors with a decision trees-based classifier and the four boosting methods (AdaBoost, GBM, XGboost, LightGBM). ResultsThe evaluation of both ensemble methods over the mammography dataset was made using four performance metrics (accuracy, sensitivity, F1-score, and recall), as well as the Skott–Knott statistical test and the Borda count voting system to rank all the possible model combinations. Results proved that bagging and boosting ensembles generally outperformed both their base learners and single models, and that the best performing ensemble overall was Gradient boosting with DenseNet 201 as feature extractor and 200 trees which scored an accuracy of 85.50 %. ConclusionsThe purpose of this research was to use hybrid learning based ensembles to maximize the potent performance of numerous models. It effectively illustrated the prospective potential of using ML approaches for the binary classification of mammography data used for breast cancer screening.

Ensemble Learning Based Malicious Node Detection in SDN-Based VANETs

Background: The architecture of Software Defined Networking (SDN) integrated with Vehicular Ad-hoc Networks (VANETs) is considered a practical method for handling large-scale, dynamic, heterogeneous vehicular networks, since it offers flexibility, programmability, scalability, and a global understanding. However, the integration with VANETs introduces additional security vulnerabilities due to the deployment of a logically centralized control mechanism. These security attacks are classified as internal and external based on the nature of the attacker. The method adopted in this work facilitated the detection of internal position falsification attacks. Objective: This study aimed to investigate the performance of k-NN, SVM, Naïve Bayes, Logistic Regression, and Random Forest machine learning (ML) algorithms in detecting position falsification attacks using the Vehicular Reference Misbehavior (VeReMi) dataset. It also aimed to conduct a comparative analysis of two ensemble classification models, namely voting and stacking for final decision-making. These ensemble classification methods used the ML algorithms cooperatively to achieve improved classification. Methods: The simulations and evaluations were conducted using the Python programming language. VeReMi dataset was selected since it was an application-specific dataset for VANETs environment. Performance evaluation metrics, such as accuracy, precision, recall, F-measure, and prediction time were also used in the comparative studies. Results: This experimental study showed that Random Forest ML algorithm provided the best performance in detecting attacks among the ML algorithms. Voting and stacking were both used to enhance classification accuracy and reduce time required to identify an attack through predictions generated by k-NN, SVM, Naïve Bayes, Logistic Regression, and Random Forest classifiers. Conclusion: In terms of attack detection accuracy, both methods (voting and stacking) achieved the same level of accuracy as Random Forest. However, the detection of attack using stacking could be achieved in roughly less than half the time required by voting ensemble. Keywords: Machine learning methods, Majority voting ensemble, SDN-based VANETs, Security attacks, Stacking ensemble classifiers, VANETs,

A multi-layer multi-view stacking model for credit risk assessment

Credit risk assessment plays a key role in determining the banking policies and commercial strategies of financial institutions. Ensemble learning approaches have been validated to be more competitive than individual classifiers and statistical techniques for default prediction. However, most researches focused on improving overall prediction accuracy rather than improving the identification of actual defaulted loans. In addition, model interpretability has not been paid enough attention in previous studies. To fill up these gaps, we propose a Multi-layer Multi-view Stacking Integration (MLMVS) approach to predict default risk in the P2P lending scenario. As the main innovation, our proposal explores multi-view learning and soft probability outputs to produce multi-layer integration based on stacking. An interpretable artificial intelligence tool LIME is embedded for interpreting the prediction results. We perform a comprehensive analysis of MLMVS on the Lending Club dataset and conduct comparative experiments to compare it with a number of well-known individual classifiers and ensemble classification methods, which demonstrate the superiority of MLMVS.

An Ensemble Classification Method Based on Deep Neural Networks for Breast Cancer Diagnosis

Advances in technology have led to advances in breast cancer screening by detecting symptoms that doctors have overlooked. In this paper, an automatic detection system for breast cancer cases based on Internet of Things (IoT) is proposed. First, using IoT technology, direct medical images are sent to the data repository after the suspicious person's visit through medical equipment equipped with IoT. Then, in order to help radiologists, interpret medical images as best as possible, we use four pre-trained convolutional neural network models including InceptionResNetV2, InceptionV3, VGG19 and ResNet152. These models are combined by an ensemble classifier. Also, these models are used to accurately predict cases with breast cancer, healthy people, and cases with pneumonia by using two datasets of X-RAY and CT-scan in a three-class classification. Finally, the best result obtained for CT-scan images belongs to InceptionResNetV2 architecture with 99.36% accuracy and for X-RAY images belongs to InceptionV3 architecture with 96.94% accuracy. The results show that this method leads to a reduction in daily visits to medical centers and thus reduces the pressure on the medical care system. It also helps radiologists and medical staff to detect breast cancer in its early stages.

Packet Length Covert Channel Detection: An Ensemble Machine Learning Approach

New undetectable attack methods made possible by advances in covert channel techniques. Due to their nonstandard data transfer methods, common countermeasures are ineffective. The detection, mitigation, and elimination of hidden channels that operate on the packet length level are extremely challenging problems. Differences in packet size can be used as part of a covert communication strategy for the delivery of sensitive information across networks. The use of machine learning techniques for detecting covert channel attacks has been praised in recent academic research. Researchers in this work developed an effective ensemble classification method for spotting such breaches. Our approach relies on an ensemble model, which is a combination of three different machine learning methods. There are several applications for classifiers including the Support Vector Machine (SVM), Random Forest (RF), and Naive Bayes (NB). The logistic regression (LR) classifier served as a meta classifier to aggregate the results of the component classifiers into an ensemble classifier. According to the findings, the proposed ensemble model is effective. Among single-classification algorithms, it is unrivalled in its ability to detect such covert channel attacks.

An Ensemble Classification Method Based on Deep Neural Networks for Breast Cancer Diagnosis

Advances in technology have led to advances in breast cancer screening by detecting symptoms that doctors have overlooked. In this paper, an automatic detection system for breast cancer cases based on Internet of Things (IoT) is proposed. First, using IoT technology, direct medical images are sent to the data repository after the suspicious person's visit through medical equipment equipped with IoT. Then, in order to help radiologists, interpret medical images as best as possible, we use four pre-trained convolutional neural network models including InceptionResNetV2, InceptionV3, VGG19 and ResNet152. These models are combined by an ensemble classifier. Also, these models are used to accurately predict cases with breast cancer, healthy people, and cases with pneumonia by using two datasets of X-RAY and CT-scan in a three-class classification. Finally, the best result obtained for CT-scan images belongs to InceptionResNetV2 architecture with 99.36% accuracy and for X-RAY images belongs to InceptionV3 architecture with 96.94% accuracy. The results show that this method leads to a reduction in daily visits to medical centers and thus reduces the pressure on the medical care system. It also helps radiologists and medical staff to detect breast cancer in its early stages.

Electroencephalograph Emotion Classification Using a Novel Adaptive Ensemble Classifier Considering Personality Traits.

The study explores the use of Electroencephalograph (EEG) signals as a means to uncover various states of the human brain, with a specific focus on emotion classification. Despite the potential of EEG signals in this domain, existing methods face challenges. Features extracted from EEG signals may not accurately represent an individual's emotional patterns due to interference from time-varying factors and noise. Additionally, higher-level cognitive factors, such as personality, mood, and past experiences, further complicate emotion recognition. The dynamic nature of EEG data in terms of time series introduces variability in feature distribution and interclass discrimination across different time stages. To address these challenges, the paper proposes a novel adaptive ensemble classification method. The study introduces a new method for providing emotional stimuli, categorizing them into three groups (sadness, neutral, and happiness) based on their valence-arousal (VA) scores. The experiment involved 60 participants aged 19-30 years, and the proposed method aimed to mitigate the limitations associated with conventional classifiers. The results demonstrate a significant improvement in the performance of emotion classifiers compared to conventional methods. The classification accuracy achieved by the proposed adaptive ensemble classification method is reported at 87.96%. This suggests a promising advancement in the ability to accurately classify emotions using EEG signals, overcoming the limitations outlined in the introduction. In conclusion, the paper introduces an innovative approach to emotion classification based on EEG signals, addressing key challenges associated with existing methods. By employing a new adaptive ensemble classification method and refining the process of providing emotional stimuli, the study achieves a noteworthy improvement in classification accuracy. This advancement is crucial for enhancing our understanding of the complexities of emotion recognition through EEG signals, paving the way for more effective applications in fields such as neuroinformatics and affective computing.

Sentiment analysis of imbalanced datasets using BERT and ensemble stacking for deep learning

The Internet is a crucial way to share information in both personal and professional areas. Sentiment analysis attracts great interest in marketing, research, and business today. The instability faced by imbalanced datasets on sentiment analysis is examined in this research. Balancing the datasets using techniques based on under-sampling and over-sampling is examined to achieve more efficient classification results as the effects of using BERT as word embedding and ensemble learning methods for classification. The effects of the resampling training set algorithms on different deep learning classifiers were investigated using BERT as a word embedding model and Cohen's kappa, accuracy, ROC-AUC curve, and MCC as evaluation metrics with k-fold validation on three sentiment analysis datasets containing English, Arabic, and Moroccan Arabic Dialect texts. Also, we did those performance metrics for all models when scaling the dataset for training and testing, and we calculated the memory and the execution time for each model. Finally, we analyzed the National Office of Railways of Morocco (ONCF) customers' Facebook comments in Modern Standard Arabic (MSA) and MD to determine customer satisfaction as positive, negative, and neutral comments.

New Classifier Ensemble and Fuzzy Community Detection Methods Using POP Choquet-like Integrals

Among various data analysis methods, classifier ensemble (data classification) and community network detection (data clustering) have aroused the interest of many scholars. The maximum operator, as the fusion function, was always used to fuse the results of the base algorithms in the classifier ensemble and the membership degree of nodes to classes in the fuzzy community. It is vital to use generalized fusion functions in ensemble and community applications. Since the Pseudo overlap function and the Choquet-like integrals are two new fusion functions, they can be combined as a more generalized fusion function. Along this line, this paper presents new classifier ensemble and fuzzy community detection methods using a pseudo overlap pair (POP) Choquet-like integral (expressed as a fraction). First, the pseudo overlap function pair is proposed to replace the product operator of the Choquet integral. Then, the POP Choquet-like integrals are defined to perform the combinatorial step of ensembles of classifiers and to generalize the GN modularity for the fuzzy community network. Finally, two new algorithms are designed for experiments, and some computational experiments with other algorithms show the importance of POP Choquet-like integrals. All of the experimental results show that our algorithms are practical.

Assemble the shallow or integrate a deep? Toward a lightweight solution for glyph-aware Chinese text classification.

As hieroglyphic languages, such as Chinese, differ from alphabetic languages, researchers have always been interested in using internal glyph features to enhance semantic representation. However, the models used in such studies are becoming increasingly computationally expensive, even for simple tasks like text classification. In this paper, we aim to balance model performance and computation cost in glyph-aware Chinese text classification tasks. To address this issue, we propose a lightweight ensemble learning method for glyph-aware Chinese text classification (LEGACT) that consists of typical shallow networks as base learners and machine learning classifiers as meta-learners. Through model design and a series of experiments, we demonstrate that an ensemble approach integrating shallow neural networks can achieve comparable results even when compared to large-scale transformer models. The contribution of this paper includes a lightweight yet powerful solution for glyph-aware Chinese text classification and empirical evidence of the significance of glyph features for hieroglyphic language representation. Moreover, this paper emphasizes the importance of assembling shallow neural networks with proper ensemble strategies to reduce computational workload in predictive tasks.

Sequential ensemble learning for next item recommendation

Predicting the next item that users may engage in is a key task of recommender systems, and many methods have been proposed to deal with this task from different aspects. Theoretically, the proper ensemble of multiple different methods (a.k.a. base models) can make more accurate and stable recommendations. However, most of the existing ensemble methods rely on static aggregation strategies, which fail to capture base models’ dynamic predictive abilities for each user over time. In addition, most of the existing diversity measures used in regression or classification ensemble methods rely on a distance metric of base models’ outputs, which makes it intractably apply for next-item recommendation whose base models output sequential ranking lists.To solve the above problems, we propose a Sequential Ensemble Method, named SEM, to aggregate different base models for next-item recommendation. We assume users’ concentration and base models’ expertise can be inferred from users’ sequential behaviors and base models’ prediction results. Therefore, we propose to explicitly model base models’ dynamic predictive abilities on different users over time based on users’ concentration and base models’ expertise. In addition, we propose a new diversity measure for sequential ranking ensemble, which can perform diversity-based learning over time for better performance of next-item recommendation. Extensive experiments on six real-world data sets show that our method consistently outperforms state-of-the-art methods.

Sentiment analysis of sarcasm detection in social media

When people talk or write more especially on social media, they tend to sometimes, flip the polarity of their expressions. Oftentimes, this change of mood affects the accuracy of the sentiment analysis. Sarcasm is a sophisticated form of irony widely used in social networks and micro blogging websites and is usually a type of sentiment where person speaks the contradictory of what individual word means, expressing gloomy feelings applying positive words. However, it is hard even for humans to recognize. Hence, sarcastic expressions play a vital role in the outcome of automatic sentiment detection in sentiment analysis. To improve on the performance of automatic detection of sentiments in social media networks, there is need for sarcastic statements to be recognised and classify correctly. Sentiment Analysis is a method for identifying people’s opinion, attitude, sentiment, and emotion towards any specific target such as individuals, events, topics, product, organizations, services etc. This paper proposed an ensemble classification method having base classifiers as Random Forest, K-Nearest Neighbour and Naive Bayes for detecting sarcasm using lexical, pragmatic, linguistic incongruity and context incongruity features. The same ensemble method and Support Vector Machines (SVM) were used to identify the sarcasm types so as to diagnose the mood of the users. We report the results and present a comparative evaluation of the ensemble method and SVM classifiers for sarcasm and sarcasm type detection indicating their suitability for the task. The results obtained from the set of experiments conducted indicate the reliability of our approach.

A diabetes prediction model based on Boruta feature selection and ensemble learning

Background and objectiveAs a common chronic disease, diabetes is called the “second killer” among modern diseases. Currently, there is no medical cure for diabetes. We can only rely on medication for auxiliary treatment. However, many diabetic patients still die each year. In addition, a considerable number of people do not pay attention to their physical health or opt out of treatment due to lack of money, which eventually leads to various complications. Therefore, diagnosing diabetes at an early stage and intervening early is necessary; thus, developing an early detection method for diabetes is essential.MethodsIn this study, a diabetes prediction model based on Boruta feature selection and ensemble learning is proposed. The model contains the use of Boruta feature selection, the extraction of salient features from datasets, the use of the K-Means++ algorithm for unsupervised clustering of data and stacking of an ensemble learning method for classification. It has been validated on a diabetes dataset.ResultsThe experiments were performed on the PIMA Indian diabetes dataset. The model was evaluated by accuracy, precision and F1 index. The obtained results show that the accuracy rate of the model reaches 98% and achieves good results.ConclusionCompared with other diabetes prediction models, this model achieved better results, and the obtained results indicate that this model is superior to other models in diabetes prediction and has better performance.

An ensemble classifier method based on teaching-learning-based optimization for breast cancer diagnosis.

Epidemiological studies show that breast cancer is the most common cancer in women in the world. Breast cancer treatment can be very effective, especially when the disease is detected in the early stages. The goal can be achieved by using large-scale breast cancer data with the machine learning models METHODS: This paper proposes a new intelligent approach using an optimized ensemble classifier for breast cancer diagnosis. The classification is done by proposing a new intelligent Group Method of Data Handling (GMDH) neural network-based ensemble classifier. This method improves the performance of the machine learning technique by using a Teaching-Learning-Based Optimization (TLBO) algorithm to optimize the hyperparameters of the classifier. Meanwhile, we use TLBO as an evolutionary method to address the problem of appropriate feature selection in breast cancer data. The simulation results show that the proposed method has a better accuracy between 7 and 26% compared to the best results of the existing equivalent algorithms. According to the obtained results, we suggest the proposed algorithm as an intelligent medical assistant system for breast cancer diagnosis.

Adaptive Subspace Optimization Ensemble Method for High-Dimensional Imbalanced Data Classification.

It is hard to construct an optimal classifier for high-dimensional imbalanced data, on which the performance of classifiers is seriously affected and becomes poor. Although many approaches, such as resampling, cost-sensitive, and ensemble learning methods, have been proposed to deal with the skewed data, they are constrained by high-dimensional data with noise and redundancy. In this study, we propose an adaptive subspace optimization ensemble method (ASOEM) for high-dimensional imbalanced data classification to overcome the above limitations. To construct accurate and diverse base classifiers, a novel adaptive subspace optimization (ASO) method based on adaptive subspace generation (ASG) process and rotated subspace optimization (RSO) process is designed to generate multiple robust and discriminative subspaces. Then a resampling scheme is applied on the optimized subspace to build a class-balanced data for each base classifier. To verify the effectiveness, our ASOEM is implemented based on different resampling strategies on 24 real-world high-dimensional imbalanced datasets. Experimental results demonstrate that our proposed methods outperform other mainstream imbalance learning approaches and classifier ensemble methods.

Automatic diagnosis of COVID-19 related respiratory diseases from speech.

In this work, an attempt is made to propose an intelligent and automatic system to recognize COVID-19 related illnesses from mere speech samples by using automatic speech processing techniques. We used a standard crowd-sourced dataset which was collected by the University of Cambridge through a web based application and an android/iPhone app. We worked on cough and breath datasets individually, and also with a combination of both the datasets. We trained the datasets on two sets of features, one consisting of only standard audio features such as spectral and prosodic features and one combining excitation source features with standard audio features extracted, and trained our model on shallow classifiers such as ensemble classifiers and SVM classification methods. Our model has shown better performance on both breath and cough datasets, but the best results in each of the cases was obtained through different combinations of features and classifiers. We got our best result when we used only standard audio features, and combined both cough and breath data. In this case, we achieved an accuracy of 84% and an Area Under Curve (AUC) score of 84%. Intelligent systems have already started to make a mark in medical diagnosis, and this type of study can help better the health system by providing much needed assistance to the health workers.

A Two-Stage Voting-Boosting Technique for Ensemble Learning in Social Network Sentiment Classification.

In recent years, social network sentiment classification has been extensively researched and applied in various fields, such as opinion monitoring, market analysis, and commodity feedback. The ensemble approach has achieved remarkable results in sentiment classification tasks due to its superior performance. The primary reason behind the success of ensemble methods is the enhanced diversity of the base classifiers. The boosting method employs a sequential ensemble structure to construct diverse data while also utilizing erroneous data by assigning higher weights to misclassified samples in the next training round. However, this method tends to use a sequential ensemble structure, resulting in a long computation time. Conversely, the voting method employs a concurrent ensemble structure to reduce computation time but neglects the utilization of erroneous data. To address this issue, this study combines the advantages of voting and boosting methods and proposes a new two-stage voting boosting (2SVB) concurrent ensemble learning method for social network sentiment classification. This novel method not only establishes a concurrent ensemble framework to decrease computation time but also optimizes the utilization of erroneous data and enhances ensemble performance. To optimize the utilization of erroneous data, a two-stage training approach is implemented. Stage-1 training is performed on the datasets by employing a 3-fold cross-segmentation approach. Stage-2 training is carried out on datasets that have been augmented with the erroneous data predicted by stage 1. To augment the diversity of base classifiers, the training stage employs five pre-trained deep learning (PDL) models with heterogeneous pre-training frameworks as base classifiers. To reduce the computation time, a two-stage concurrent ensemble framework was established. The experimental results demonstrate that the proposed method achieves an F1 score of 0.8942 on the coronavirus tweet sentiment dataset, surpassing other comparable ensemble methods.

On Precision Agriculture: Enhanced Automated Fruit Disease Identification and Classification Using a New Ensemble Classification Method

Fruits are considered among the most nutrient-dense cash crops around the globe. Since fruits come in different types, sizes, shapes, colors, and textures, the manual classification and disease identification of a large quantity of fruit is time-consuming and sluggish, requiring massive human intervention. We propose a multilevel fusion method for fruit disease identification and fruit classification that includes intensive fruit image pre-processing, customized image kernels for feature extraction with state-of-the-art (SOTA) deep methods, Gini-index-based controlled feature selection, and a hybrid ensemble method for identification and classification. We noticed certain limitations in the existing literature of adopting a single data source, in terms of limited data sizes, variability in fruit types, variability in quality, and variability in disease type. Therefore, we extensively aggregated and pre-processed multi-fruit data to simulate our proposed ensemble model on comprehensive datasets to cover both fruit classification and disease identification aspects. The multi-fruit imagery data contained regular and augmented images of fruits including apple, apricot, avocado, banana, cherry, fig, grape, guava, kiwi, mango, orange, peach, pear, pineapple, and strawberry. Similarly, we considered normal and augmented images of rotten fruits including beans (two categories), strawberries (seven categories), and tomatoes (three categories). For consistency, we normalized the images and designed an auto-labeling mechanism based on the existing image clusters to label inconsistent data to appropriate classes. Finally, we verified the auto-labeled data with a complete inspection to correctly assign it to the relevant classes. The proposed ensemble classifier outperforms all other classification methods, achieving 100% and 99% accuracy for fruit classification and disease identification. Further, we performed the analysis of variance (ANOVA) test to validate the statistical significance of the classifiers’ outcomes at α = 0.05. We achieved F-values of 32.41 and 11.42 against F-critical values of 2.62 and 2.86, resulting in p-values of 0.00 (<0.05) for fruit classification and disease identification.

An ensemble oversampling method for imbalanced classification with prior knowledge via generative adversarial network

Currently, an increasing number of real-world applications show characteristics of class-imbalance classification suffering from severe class distribution skewing, thus requiring brand new algorithms to learn from imbalanced datasets. In this paper, a novel oversampling method using GAN framework is proposed for numerical imbalanced data, namely G-GAN. In the method, a Gaussian distribution of minority samples is estimated to get prior knowledge of minority class for the latent space of GAN. In order to increase the randomness of the generated samples, noises are obtained by a mixed strategy, that is, some noises of generator obey Gaussian distribution and others obey random distribution. Then G-GAN is trained to generate dispersive positive samples with the idea of Bagging, which could avoid the occurrence of overfitting. G-GAN is different from other literatures in that GAN does not directly generate minority samples, but adds the distribution information of minority samples to the latent space of GAN, and then generates minority samples. Compared with 11 commonly used oversampling methods, G-GAN obtains promising results in terms of G-mean, AUC, F-measure and ROC utilizing three classifiers on 11 benchmark imbalanced datasets. Furthermore, G-GAN is also validated on AUC metrics of a real Diabetes imbalanced dataset. The results demonstrate that G-GAN can provide great potential for imbalanced classification in the two numerical experiments.