Ensemble Voting Research Articles

VoteDroid: a new ensemble voting classifier for malware detection based on fine-tuned deep learning models

AbstractIn this work, VoteDroid a novel fine-tuned deep learning models-based ensemble voting classifier has been proposed for detecting malicious behavior in Android applications. To this end, we proposed adopting the random search optimization algorithm for deciding the structure of the models used as voter classifiers in the ensemble classifier. We specified the potential components that can be used in each model and left the random search algorithm taking a decision about the structure of the model including the number of each component that should be used and its location in the structure. This optimization method has been used to build three different deep learning models namely CNN-ANN, pure CNN, and pure ANN. After selecting the best structure for each DL model, the selected three models have been trained and tested using the constructed image dataset. Afterward, we suggested hybridizing the fine-tuned three deep-learning models to form one ensemble voting classifier with two different working modes namely MMR (Malware Minority Rule) and LMR (Label Majority Rule). To our knowledge, this is the first time that an ensemble classifier has been fine-tuned and hybridized in this way for malware detection. The results showed that the proposed models were promising, where the classification accuracy exceeded 97% in all experiments.

Voting Based VGG-16 for Feature Selection and Classification of The Brain Tumor Types

Brain tumors, especially malignant ones, can be life-threatening if not detected and treated early. Identifying benign tumors, such as Meningiomas or Pituitary tumors, enables preventive measures and long-term monitoring to detect any changes in tumor size or behavior. This ensures timely intervention if needed. ML related methods for analyzing types of brain tumors have made significant advancements but brain tumour records often have a mismatch of classes, which means that some kinds of tumours are much less common than others. This may result in models that are biased and work well for the majority of the class but not so well for the minority class. Latest research techniques have focused on deep learning approaches. Identifying brain tumor types using neural networks is a complex but promising approach in medical image analysis. Combining NN with ML techniques is what the suggested model does to get to the exact type of brain tumour. In this regard, the proposed model has extracted the features from the tuned model of VGG-16 and after extracting the features from the network to further classify the stage, the model has applied ensemble voting mechanism.

TXAI-ADV: Trustworthy XAI for Defending AI Models against Adversarial Attacks in Realistic CIoT

Adversarial attacks are more prevalent in Consumer Internet of Things (CIoT) devices (i.e., smart home devices, cameras, actuators, sensors, and micro-controllers) because of their growing integration into daily activities, which brings attention to their possible shortcomings and usefulness. Keeping protection in the CIoT and countering emerging risks require constant updates and monitoring of these devices. Machine learning (ML), in combination with Explainable Artificial Intelligence (XAI), has become an essential component of the CIoT ecosystem due to its rapid advancement and impressive results across several application domains for attack detection, prevention, mitigation, and providing explanations of such decisions. These attacks exploit and steal sensitive data, disrupt the devices’ functionality, or gain unauthorized access to connected networks. This research generates a novel dataset by injecting adversarial attacks into the CICIoT2023 dataset. It presents an adversarial attack detection approach named TXAI-ADV that utilizes deep learning (Mutli-Layer Perceptron (MLP) and Deep Neural Network (DNN)) and machine learning classifiers (K-Nearest Neighbor (KNN), Support Vector Classifier (SVC), Gaussian Naive Bayes (GNB), ensemble voting, and Meta Classifier) to detect attacks and avert such situations rapidly in a CIoT. This study utilized Shapley Additive Explanations (SHAP) techniques, an XAI technique, to analyze the average impact of each class feature on the proposed models and select optimal features for the adversarial attacks dataset. The results revealed that, with a 96% accuracy rate, the proposed approach effectively detects adversarial attacks in a CIoT.

Ensemble Learning for Higher Diagnostic Precision in Schizophrenia Using Peripheral Blood Gene Expression Profile.

Stigma contributes to a significant part of the burden of schizophrenia (SCZ), therefore reducing false positives from the diagnosis would be liberating for the individuals with SCZ and desirable for the clinicians. The stigmatization associated with schizophrenia advocates the need for high-precision diagnosis. In this study, we present an ensemble learning-based approach for high-precision diagnosis of SCZ using peripheral blood gene expression profiles. The machine learning (ML) models, support vector machines (SVM), and prediction analysis for microarrays (PAM) were developed using differentially expressed genes (DEGs) as features. The SCZ samples were classified based on a voting ensemble classifier of SVM and PAM. Further, microarray-based learning was used to classify RNA sequencing (RNA-Seq) samples from our case-control study (Pune-SCZ) to assess cross-platform compatibility. Ensemble learning using ML models resulted in a significantly higher precision of 80.41% (SD: 0.04) when compared to the individual models (SVM-radial: 71.69%, SD: 0.04 and PAM 77.20%, SD: 0.02). The RNA sequencing samples from our case-control study (Pune-SCZ) resulted in a moderate precision (59.92%, SD: 0.05). The feature genes used for model building were enriched for biological processes such as response to stress, regulation of the immune system, and metabolism of organic nitrogen compounds. The network analysis identified RBX1, CUL4B, DDB1, PRPF19, and COPS4 as hub genes. In summary, this study developed robust models for higher diagnostic precision in psychiatric disorders. Future efforts will be directed towards multi-omic integration and developing "explainable" diagnostic models.

SAR Image Fusion Classification Based on Improved D-S Evidence Theory

Applications for classifying Synthetic Aperture Radar (SAR) images are critical to environmental monitoring, urban planning, and land resource surveying. Fusion approaches work well for increasing SAR image categorization accuracy. However, effective processing of uncertainty information is frequently overlooked by conventional fusion classification systems. This paper presented a fusion classification approach for SAR images based on improved Dempster-Shafer (D-S) evidence theory, taking into account the uncertainty and possible conflicts in the classifiers’ output. First, after adaptively identifying the optimal hyperparameters of models, the SAR data is identified using Gradient Boosting Machine (GBM), Multilayer Perceptron (MLP), and Random Forest (RF). Different weights are then allocated in accordance with the variations in classifier performance. Ultimately, enhanced D-S evidence theory is used to integrate the output of these classifiers. In addition, uncertainty in classification results is also modeled and visualized. Experiments with Flevoland and Oberpfaffenhofen datasets show that the accuracy of this method is 85.21% and 91.88% respectively, outperforming both the soft voting ensemble classifier (SVE) and D-S evidence theory fusion method. This approach enhances the comprehension and capacity to manage uncertainty in the model output in addition to increasing the accuracy and reliability of the classification of SAR data.

Digital Image Object Detection with GLCM Multi-Degrees and Ensemble Learning

Object detection in digital images has been implemented in various fields. Object detection faces challenges, one of which is rotation problems, causing objects to become unknown. We need a method that can extract features that do not affect rotation and reliable ensemble-based classification. The proposal uses the GLCM-MD (Gray-Level Co-occurrence Matrix Multi-Degrees) extraction method with classification using K-Nearest Neighbours (K-NN) and Random Forest (RF) learning as well as Voting Ensemble (VE) from two single classifications. The main goal is to overcome the difficulty of detecting objects when the object experiences rotation which results in significant visualization variations. In this research, the GLCM method is used to produce features that are stable against rotation. Furthermore, classification methods such as K-Nearest Neighbours (KNN), Random Forest (RF), and KNN-RF fusion using the Voting ensemble method are evaluated to improve detection accuracy. The experimental results show that the use of multi-degrees and the use of ensemble voting at all degrees can increase the accuracy value, and the highest accuracy for extraction using multi-degrees is 95.95%. Based on test results which show that the use of features of various degrees and the ensemble voting method can increase accuracy for detecting objects experiencing rotation

Construction of a Cutting-Tool Wear Prediction Model through Ensemble Learning

This study begins by conducting side milling experiments on SKD11 using tungsten carbide TiAlN-coated end mills to compare the surface roughness performance between two combinations of milling process parameters (feed rate and radial depth of cut), along with three ultrasonic-assisted methods (rotary, dual-axis, and rotary combined with dual-axis). The results suggest that the rotary (z-axis oscillation) ultrasonic-assisted method may provide better performance. Subsequently, this superior ultrasonic-assisted method was applied both with and without laser locally preheating assistance, respectively. Using a Taguchi orthogonal array, milling process parameters (spindle speed, feed rate, and radial depth of cut) were planned for experiments with the same cutting tool and the workpiece just mentioned above. The surface roughness serves as the objective function while being constrained by cutting-tool life. The characteristics of the smaller-the-better in the Taguchi method were applied to determine the optimal combination of process parameters. Based on the optimal milling process parameters obtained and the superior hybrid-assisted method adopted, milling experiments were repeatedly performed to collect the data on cutting force and cutting-tool wear. Feature engineering was performed on the cutting force signals, and different domain characteristics from both the time and frequency domains were extracted. Hereafter, feature selection by random forest and data standardization were further applied to feature extractions, and the data processing was thus completed. For the processed data, a cutting-tool wear prediction model was constructed by ensemble learning. This method leverages various machine learning regression models, including decision tree, random forest, extremely randomized tree, light gradient boosting machine, extreme gradient boosting, AdaBoost, stochastic gradient descent, support vector regression, linear support vector regression, and multilayer perceptron. After hyper-parameter tuning, the ensemble voting regression prediction was performed based on these ten mentioned models. The experimental results demonstrate that the ensemble voting regression model surpasses the performance of each individual machine learning regression model. In addition, this regression model achieves a coefficient of determination (R2) of 0.94576, a root mean square error (RMSE) of 0.24348, a mean squared error (MSE) of 0.05928, and a mean absolute error (MAE) of 0.18182. Therefore, the ensemble learning approach has been proven to be a feasible and effective method for monitoring cutting-tool wear.

Ensemble Approach Using k-Partitioned Isolation Forests for the Detection of Stock Market Manipulation

Stock market manipulation, defined as any attempt to artificially influence stock prices, poses significant challenges by causing financial losses and eroding investor trust. The prevalent reliance on supervised learning models for detecting such manipulations, while showing promise, faces notable hurdles due to the dearth of labeled data and the inability to recognize novel manipulation tactics beyond those explicitly labeled. This study ventures into addressing these gaps by proposing a novel detection framework aimed at identifying suspicious hourly manipulation blocks through an unsupervised learning approach, thereby circumventing the limitations of data labeling and enhancing the adaptability to emerging manipulation strategies. Our methodology involves the innovative creation of features reflecting the behavior of stocks across various time windows followed by the segmentation of the dataset into k subsets. This setup facilitates the identification of potential manipulation instances via a voting ensemble composed of k isolation forest models, which have been chosen for their efficiency in pinpointing anomalies and their linear computational complexity—attributes that are critical for analyzing vast datasets. Evaluated against eight real stocks known to have undergone manipulation, our approach demonstrated a remarkable capability to identify up to 89% of manipulated blocks, thus significantly outperforming previous methods that do not utilize a voting ensemble. This finding not only surpasses the detection rates reported in prior studies but also underscores the enhanced robustness and adaptability of our unsupervised model in uncovering varied manipulation schemes. Through this research, we contribute to the field by offering a scalable and efficient unsupervised learning strategy for stock manipulation detection, thereby marking a substantial advancement over traditional supervised methods and paving the way for more resilient financial markets.

Diabetes prediction utilizing soft voting classifier

Diabetes is one of the dangerous diseases that bring about abnormalities in blood sugar levels. Early treatment can mitigate the negative consequences of this disease. Machine learning algorithms can be leveraged to predict this disease at an early stage. In this study, a soft voting ensemble classifier approach combining random forest, AdaBoost and gradient boost algorithms is adopted to predict diabetes with the highest possible accuracy. The proposed method was tested on a publicly available dataset. The proposed approach predicted diabetes with 100% accuracy. As a result of the experiments conducted within the scope of the study, polyuria and polydipsia variables were found to be the most significant risk factors for this disease. The suggested approach outperformed similar studies in the literature.

Improving prediction of maternal health risks using PCA features and TreeNet model.

Maternal healthcare is a critical aspect of public health that focuses on the well-being of pregnant women before, during, and after childbirth. It encompasses a range of services aimed at ensuring the optimal health of both the mother and the developing fetus. During pregnancy and in the postpartum period, the mother's health is susceptible to several complications and risks, and timely detection of such risks can play a vital role in women's safety. This study proposes an approach to predict risks associated with maternal health. The first step of the approach involves utilizing principal component analysis (PCA) to extract significant features from the dataset. Following that, this study employs a stacked ensemble voting classifier which combines one machine learning and one deep learning model to achieve high performance. The performance of the proposed approach is compared to six machine learning algorithms and one deep learning algorithm. Two scenarios are considered for the experiments: one utilizing all features and the other using PCA features. By utilizing PCA-based features, the proposed model achieves an accuracy of 98.25%, precision of 99.17%, recall of 99.16%, and an F1 score of 99.16%. The effectiveness of the proposed model is further confirmed by comparing it to existing state of-the-art approaches.

An Improved Ensemble Machine Learning Approach for Diabetes Diagnosis

Diabetes is recognized as one of the most detrimental diseases worldwide, characterized by elevated levels of blood glucose stemming from either insulin deficiency or decreased insulin efficacy. Early diagnosis of diabetes enables patients to initiate treatment promptly, thereby minimizing or eliminating the risk of severe complications. Although years of research in computational diagnosis have demonstrated that machine learning offers a robust methodology for predicting diabetes, existing models leave considerable room for improvement in terms of accuracy. This paper proposes an improved ensemble machine learning approach using multiple classifiers for diabetes diagnosis based on the Pima Indians Diabetes Dataset (PIDD). The proposed ensemble voting classifier amalgamates five machine learning algorithms: Decision Tree (DT), Logistic Regression (LR), K-Nearest Neighbor (KNN), Random Forests (RF), and XGBoost. We obtained the individual model accuracies and used the ensemble method to improve accuracy. The proposed approach uses a pre-processing stage of standardization and imputation and applies the Local Outlier Factor (LOF) to remove data anomalies. The model was evaluated using sensitivity, specificity, and accuracy criteria. With a reported accuracy of 81%, the proposed approach shows promise compared to prior classification techniques.

Voting Model Strategies for Reliable Categorical IoT-DDoS Attack Prediction

This research focuses on developing reliable categorical IoT-DDoS attack prediction models using ensemble voting strategies. The study explores various machine learning algorithms suitable for categorical data analysis, employing feature engineering techniques to preprocess IoT data. Ensemble learning methodologies, including bagging, boosting, and stacking, are then utilized to build robust prediction models. Evaluation metrics such as precision, recall, F1-score, and AUC-ROC are used to assess model performance, demonstrating the effectiveness of ensemble voting models in reliably predicting IoT-DDoS attacks. Comparative analyses with individual classifiers highlight the advantages of ensemble approaches in terms of predictive accuracy and robustness against data imbalances and noise. This work contributes to advancing IoT security by providing a practical framework for deploying predictive models that aid in early detection and mitigation of DDoS attacks, enhancing overall resilience against cyber threats in IoT ecosystems.

Data-Driven Techniques for Short-Term Electricity Price Forecasting through Novel Deep Learning Approaches with Attention Mechanisms

The electricity market is constantly evolving, being driven by factors such as market liberalization, the increasing use of renewable energy sources (RESs), and various economic and political influences. These dynamics make it challenging to predict wholesale electricity prices. Accurate short-term forecasting is crucial to maintaining system balance and addressing anomalies such as negative prices and deviations from predictions. This paper investigates short-term electricity price forecasting using historical time series data and employs advanced deep learning algorithms. First, four deep learning models are implemented and proposed, which are a convolutional neural network (CNN) with an integrated attention mechanism, a hybrid CNN followed by a gated recurrent unit model (CNN-GRU) with an attention mechanism, and two ensemble learning models, which are a soft voting ensemble and a stacking ensemble model. Also, the optimized version of a transformer model, the Multi-Head Attention model, is introduced. Finally, the perceptron model is used as a benchmark for comparison. Our results show excellent prediction accuracy, particularly in the hybrid CNN-GRU model with attention, thereby achieving a mean absolute percentage error (MAPE) of 6.333%. The soft voting ensemble model and the Multi-Head Attention model also performed well, with MAPEs of 6.125% and 6.889%, respectively. These findings are significant, as previous studies have not shown high performance with transformer models and attention mechanisms. The presented results offer promising insights for future research in this field.

Identification of circRNA-disease associations via multi-model fusion and ensemble learning.

Circular RNA (circRNA) is a common non-coding RNA and plays an important role in the diagnosis and therapy of human diseases, circRNA-disease associations prediction based on computational methods can provide a new way for better clinical diagnosis. In this article, we proposed a novel method for circRNA-disease associations prediction based on ensemble learning, named ELCDA. First, the association heterogeneous network was constructed via collecting multiple information of circRNAs and diseases, and multiple similarity measures are adopted here, then, we use metapath, matrix factorization and GraphSAGE-based models to extract features of nodes from different views, the final comprehensive features of circRNAs and diseases via ensemble learning, finally, a soft voting ensemble strategy is used to integrate the predicted results of all classifier. The performance of ELCDA is evaluated by fivefold cross-validation and compare with other state-of-the-art methods, the experimental results show that ELCDA is outperformance than others. Furthermore, three common diseases are used as case studies, which also demonstrate that ELCDA is an effective method for predicting circRNA-disease associations.

Optimizing decision-making in IoT through ensemble voting and dynamic group formation with DBSCAN

In the era of widespread connectivity, leveraging artificial intelligence models and analyzing the vast datasets generated by smart devices are central points in IoT research. While existing studies mainly focus on improving the decision-making prowess of central systems, the potential for local optimization remains largely unexplored. This paper presents an Ensemble Voting Scheme with Multilayer Dynamic Groups (EVMDS), which assigns decision weights to IoT devices based on their attribute data. By employing the Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm, dynamic clusters among IoT devices can be identified, the application of ensemble voting rules at each stage of group formation, enabling layered computations to ease backend burden and achieve hierarchical decision-making capability, facilitating regional-level decision-making that strikes a balance between local and global optimization. Through simulated decision-making scenarios in a small-scale IoT environment, our experiments demonstrate the superior accuracy and reliability of the proposed approach compared to existing models.

An Improved MSER using Grid Search based PCA and Ensemble Voting Technique

An Improved MSER using Grid Search based PCA and Ensemble Voting Technique

Deep learning-based prediction of post-pancreaticoduodenectomy pancreatic fistula

Postoperative pancreatic fistula is a life-threatening complication with an unmet need for accurate prediction. This study was aimed to develop preoperative artificial intelligence-based prediction models. Patients who underwent pancreaticoduodenectomy were enrolled and stratified into model development and validation sets by surgery between 2016 and 2017 or in 2018, respectively. Machine learning models based on clinical and body composition data, and deep learning models based on computed tomographic data, were developed, combined by ensemble voting, and final models were selected comparison with earlier model. Among the 1333 participants (training, n = 881; test, n = 452), postoperative pancreatic fistula occurred in 421 (47.8%) and 134 (31.8%) and clinically relevant postoperative pancreatic fistula occurred in 59 (6.7%) and 27 (6.0%) participants in the training and test datasets, respectively. In the test dataset, the area under the receiver operating curve [AUC (95% confidence interval)] of the selected preoperative model for predicting all and clinically relevant postoperative pancreatic fistula was 0.75 (0.71–0.80) and 0.68 (0.58–0.78). The ensemble model showed better predictive performance than the individual ML and DL models.

Adaptive regularized ensemble for evolving data stream classification

Extracting knowledge from data streams requires fast incremental algorithms that are able to handle unlimited processing and ever-changing data with finite memory. A strategy for this challenge is the use of ensembles owing to their ability to tackle concept drift and achieve highly accurate predictions. However, ensembles often require a lot of computational resources. In this study, we propose a novel ensemble-based classification algorithm for data streams, Adaptive Regularized Ensemble (ARE), with low demand for computational resources. The algorithm combines strategies that contribute to high prediction accuracy using only incorrectly classified instances into the training step, random-sized feature subspace for each ensemble element and classifier selection for final ensemble voting. After an extensive experimental study, ARE exhibited high predictive performance and outperformed state-of-the-art ensembles on data streams for real and synthetic datasets while requiring a low processing time and memory consumption.

Educational Data Mining: Employing Machine Learning Techniques and Hyperparameter Optimization to Improve Students’ Academic Performance

Educational data mining (EDM) is a specialized field within data mining that focuses on extracting valuable insights from academic data across high school and university levels. A common practice in EDM involves predicting students’ grades to identify at-risk individuals and improve the efficiency of academic tasks. This knowledge benefits students, parents, and institutions equally. Early detection enables interventions that improve student performance. The literature presents various prediction strategies, each with its own unique advantages and disadvantages. This study aims to comprehensively evaluate the methods, tools, and applications of machine learning (ML) and data mining (DM) in education. The main goal is to improve the accuracy of predicting academic achievements by employing eight widely recognized ML algorithms: naïve bayes (NB), k-nearest neighbors (KNN), support vector machine (SVM), random forest (RF), logistic regression (LR), extreme gradient boost (XGBOOST), and ensemble voting classifier (EVC). The focus is on improving data quality by eliminating instances of noise. Performance evaluation involves assessing parameters such as accuracy, precision, F-measure, and recall. Incorporating cross-validation and hyperparameter tuning improves classification accuracy. The ML models outperform other ensemble approaches, providing a valuable tool for predicting student performance and assisting educators in making proactive decisions through timely alerts.

Applying large language models and chain-of-thought for automatic scoring

This study investigates the application of large language models (LLMs), specifically GPT-3.5 and GPT-4, with Chain-of-Though (CoT) in the automatic scoring of student-written responses to science assessments. We focused on overcoming the challenges of accessibility, technical complexity, and lack of explainability that have previously limited the use of artificial intelligence-based automatic scoring tools among researchers and educators. With a testing dataset comprising six assessment tasks (three binomial and three trinomial) with 1,650 student responses, we employed six prompt engineering strategies to automatically score student responses. The six strategies combined zero-shot or few-shot learning with CoT, either alone or alongside item stem and scoring rubrics, developed based on a novel approach, WRVRT (prompt writing, reviewing, validating, revising, and testing). Results indicated that few-shot (acc = 0.67) outperformed zero-shot learning (acc = 0.60), with 12.6% increase. CoT, when used without item stem and scoring rubrics, did not significantly affect scoring accuracy (acc = 0.60). However, CoT prompting paired with contextual item stems and rubrics proved to be a significant contributor to scoring accuracy (13.44% increase for zero-shot; 3.7% increase for few-shot). We found a more balanced accuracy across different proficiency categories when CoT was used with a scoring rubric, highlighting the importance of domain-specific reasoning in enhancing the effectiveness of LLMs in scoring tasks. We also found that GPT-4 demonstrated superior performance over GPT-3.5 in various scoring tasks when combined with the single-call greedy sampling or ensemble voting nucleus sampling strategy, showing 8.64% difference. Particularly, the single-call greedy sampling strategy with GPT-4 outperformed other approaches. This study also demonstrates the potential of LLMs in facilitating explainable and interpretable automatic scoring, emphasizing that CoT enhances accuracy and transparency, particularly when used with item stem and scoring rubrics.