Heterogeneous Ensemble Classifier Research Articles

Classification of Multiple H&E Images via an Ensemble Computational Scheme.

In this work, a computational scheme is proposed to identify the main combinations of handcrafted descriptors and deep-learned features capable of classifying histological images stained with hematoxylin and eosin. The handcrafted descriptors were those representatives of multiscale and multidimensional fractal techniques (fractal dimension, lacunarity and percolation) applied to quantify the histological images with the corresponding representations via explainable artificial intelligence (xAI) approaches. The deep-learned features were obtained from different convolutional neural networks (DenseNet-121, EfficientNet-b2, Inception-V3, ResNet-50 and VGG-19). The descriptors were investigated through different associations. The most relevant combinations, defined through a ranking algorithm, were analyzed via a heterogeneous ensemble of classifiers with the support vector machine, naive Bayes, random forest and K-nearest neighbors algorithms. The proposed scheme was applied to histological samples representative of breast cancer, colorectal cancer, oral dysplasia and liver tissue. The best results were accuracy rates of 94.83% to 100%, with the identification of pattern ensembles for classifying multiple histological images. The computational scheme indicated solutions exploring a reduced number of features (a maximum of 25 descriptors) and with better performance values than those observed in the literature. The presented information in this study is useful to complement and improve the development of computer-aided diagnosis focused on histological images.

APPLICATION OF HETEROGENEOUS ENSEMBLES IN PROBLEMS OF COMPUTER SYSTEM STATE IDENTIFICATION

The object of the study is the process of identifying anomalies in the operation of a computer system (CS). The subject of the study is ensemble methods for identifying the state of the CS. The goal of the study is to improve the performance of ensemble classifiers based on heterogeneous models. Methods used: machine learning methods, homogeneous and heterogeneous ensemble classifiers, Pasting and Bootstrapping technologies. Results obtained: a comparative analysis of the use of homogeneous and heterogeneous bagging ensembles in data classification problems was carried out. The effectiveness of various approaches to the selection of base ensemble classifiers has been studied. A method for identifying the state of a computer system, based on the heterogeneous bagging ensemble was proposed. Experimental studies made it possible to confirm the main theoretical assumptions, as well as evaluate the efficiency of the constructed heterogeneous ensembles. Conclusions. Based on the results of the study, the method for constructing a heterogeneous bagging ensemble classifier, which differs from known methods in the procedure for selecting base models was proposed. It made possible to increase the classification accuracy. Further development of this research could include the creating and integration of dissimilarity metrics as well as other quantitative metrics for a more accurate and balanced base model selection procedure, which would further improve the performance of the computer system state classifier.

Hybrid feature ranking and classifier aggregation based on multi-criteria decision-making

This study introduces an ensemble methodology, namely, hybrid feature ranking and classifier aggregation (HyFraCa), to integrate ensemble feature selection and ensemble classification in a composite framework. The proposed HyFraCa is embedded in a multi-criteria decision-making (MCDM)-based scheme for feature ranking and classifier weighting, with an effective aggregation rule that yields a consensus feature ranking from ensembles of heterogeneous classifiers and feature selectors. Experimental evaluations on 20 public UCI datasets demonstrated the superiority of HyFraCa in producing a more accurate and generalizable classification compared with state-of-the-art benchmark ensemble methods. HyFraCa also provides robust and reliable consensus feature rankings, which are favorable for real-world classification problems in which feature interpretability is emphasized.

Ensemble of Homogenous and Heterogeneous Classifiers using K-Fold Cross Validation with Reduced Entropy

Chronic kidney disease (CKD) affects millions of people worldwide, greatly reducing their quality of life and creating serious economic, social, and medical problems. Some automated diagnosis methods can detect chronic renal disease. In-depth studies on data mining techniques have recently focused on accuracy in the diagnosis of chronic renal illnesses, either by taking advantage of the disease's simplicity or doing feature selection in addition to pre-processing. In order to handle the unbalanced dataset in this work, Synthetic Minority Over Sampling Technique (SMOTE) is used during pre-processing. For this investigation, 400 data from the publicly accessible UCI machine learning (ML) repository are used. For the implementation, both homogeneous and heterogeneous ensemble classifiers which combine two separate classifiers have been used. Different machine learning (ML) techniques, such as the Classification and Regression Tree (CART), Adaboost classifier, Decision Tree (DT), Reduced Error Pruning Tree, Alternating Decision Tree, and Random Forests Algorithm and their ensembles with a significant reduction in entropy, are used to perform the classification. With a 99.12% accuracy rate and a 99.10% f1 score, the homogeneous classifier Adaboost-Random Forest outperforms other models in the prediction of CKD.

Deep feature fusion and optimized feature selection based ensemble classification of liver lesions

ABSTRACT Classification of liver abnormalities is crucial for the early identification of liver cancer. In clinical settings, radiological professionals typically make diagnoses manually which is subjective, time-consuming and vulnerable to error. Therefore, there is still a demand for precise classification of liver diseases. We propose an Ensemble learning-based classification model to classify liver lesions on CT images. To excerpt all the essential facts from the image, deep feature fusion is incorporated by concatenating the features from pre-trained deep CNN models densenet201 and InceptionResnetV2. To minimize the feature space and boost classification accuracy, hybrid optimization methodologies, Genetic Algorithms and Ant Colony Optimization are applied. Finally, a Heterogeneous Ensemble classifier divides the retrieved features into four groups (liver abscess, liver cirrhosis, hepatocellular carcinoma, and metastasis). It is clearly seen and observed that 98.3% accuracy is contributed by ensemble classifier with the support of concatenated deep features and this classifier excels in all other ways and means.

Predicting stock market direction in South African banking sector using ensemble machine learning techniques

<abstract> <p>The ability to accurately predict stock price direction is important for investors and policymakers. We aim to predict the direction of daily stock returns for five major South African banks using ensemble machine learning techniques. Financial ratios were used as predictors in single classifier and ensemble models. The key findings were that the support vector machine performed best among single classifiers, with the highest accuracy for 4 banks ranging from 54% to 99% and produces fewer wrong classifications compared to its peer single classifiers. More importantly, the heterogeneous ensemble classifier, combining support vector machines, decision trees and k- (KNN) nearest neighbors, achieved average accuracy rates above 95% and outperformed all other models. This confirms that ensemble methods that combine multiple models can generate more accurate predictions compared to single classifiers. The results suggest that the heterogeneous ensemble is a suitable approach for predicting stock price direction in the South African banking sector. The findings imply that investing in banks may be a good decision and can assist investors. However, further research could expand the models to incorporate macroeconomic and other external factors that influence stock prices. Overall, we demonstrate the value of ensemble learning for a complex forecasting problem. The heterogeneous ensemble approach achieved high accuracy and outperformed single classifiers. However, future research incorporating additional factors and policy implications could build on these findings.</p> </abstract>

Class-Level Refactoring Prediction by Ensemble Learning with Various Feature Selection Techniques

Background: Refactoring is changing a software system without affecting the software functionality. The current researchers aim i to identify the appropriate method(s) or class(s) that needs to be refactored in object-oriented software. Ensemble learning helps to reduce prediction errors by amalgamating different classifiers and their respective performances over the original feature data. Other motives are added in this paper regarding several ensemble learners, errors, sampling techniques, and feature selection techniques for refactoring prediction at the class level. Objective: This work aims to develop an ensemble-based refactoring prediction model with structural identification of source code metrics using different feature selection techniques and data sampling techniques to distribute the data uniformly. Our model finds the best classifier after achieving fewer errors during refactoring prediction at the class level. Methodology: At first, our proposed model extracts a total of 125 software metrics computed from object-oriented software systems processed for a robust multi-phased feature selection method encompassing Wilcoxon significant text, Pearson correlation test, and principal component analysis (PCA). The proposed multi-phased feature selection method retains the optimal features characterizing inheritance, size, coupling, cohesion, and complexity. After obtaining the optimal set of software metrics, a novel heterogeneous ensemble classifier is developed using techniques such as ANN-Gradient Descent, ANN-Levenberg Marquardt, ANN-GDX, ANN-Radial Basis Function; support vector machine with different kernel functions such as LSSVM-Linear, LSSVM-Polynomial, LSSVM-RBF, Decision Tree algorithm, Logistic Regression algorithm and extreme learning machine (ELM) model are used as the base classifier. In our paper, we have calculated four different errors i.e., Mean Absolute Error (MAE), Mean magnitude of Relative Error (MORE), Root Mean Square Error (RMSE), and Standard Error of Mean (SEM). Result: In our proposed model, the maximum voting ensemble (MVE) achieves better accuracy, recall, precision, and F-measure values (99.76, 99.93, 98.96, 98.44) as compared to the base trained ensemble (BTE) and it experiences less errors (MAE = 0.0057, MORE = 0.0701, RMSE = 0.0068, and SEM = 0.0107) during its implementation to develop the refactoring model. Conclusions: Our experimental result recommends that MVE with upsampling can be implemented to improve the performance of the refactoring prediction model at the class level. Furthermore, the performance of our model with different data sampling techniques and feature selection techniques has been shown in the form boxplot diagram of accuracy, F-measure, precision, recall, and area under the curve (AUC) parameters.

Weighted heterogeneous ensemble for the classification of intrusion detection using ant colony optimization for continuous search spaces

This paper proposes a heterogeneous ensemble classifier configuration for a multiclass intrusion detection problem. The ensemble is composed of k-nearest neighbors, artificial neural networks, and naïve Bayes classifiers. The decisions of these classifiers are combined with weighted majority voting, where optimal weights are generated by ant colony optimization for continuous search spaces. As a comparison basis, we have also implemented the ensemble configuration with the unweighted majority voting or Winner Takes All strategy. To ensure the maximum variety of classifiers, we have implemented three versions of each classification algorithm by varying each classifier’s parameters making a total of nine diverse experts for the ensemble. For our empirical study, we used the full NSL-KDD dataset to classify network traffic into one of five different classes. Our results indicate that the ensemble configuration using ACOR-optimized weights is capable of resolving the conflicts between multiple classifiers and improving the overall classification accuracy of the ensemble.

Automatic detection of Alzheimer’s disease progression: An efficient information fusion approach with heterogeneous ensemble classifiers

Predicting Alzheimer’s disease (AD) progression is crucial for improving the management of this chronic disease. Usually, data from AD patients are multimodal and time series in nature. This study proposes a novel ensemble learning framework for AD progression incorporating heterogeneous base learners into an integrated model using the stacking technique. This framework is used to build a 4-class ensemble classifier, which predicts AD progression 2.5 years in the future based on the multimodal time-series data. Statistical measures have been extracted from the longitudinal data to be used by the conventional machine learning models. The examined ensemble members include k-nearest neighbor, extreme gradient boosting, support vector machine, random forest, decision tree, and multilayer perceptron. We utilize three time-series modalities and one static non-time series modality of 1371 subjects from the Alzheimer’s disease neuroimaging initiative (ADNI) to validate our model. Several homogeneous and heterogeneous combinations of ensemble members were implemented, and their performance compared. The balance between accuracy and diversity when selecting ensemble members was investigated. We found that both accuracy and diversity are equally critical metrics to obtain an optimal ensemble model. Furthermore, our testing showed that the proposed model achieves outstanding progression prediction performance. The proposed model achieved a high performance without using neuroimaging data, which means that the model could be implemented in low-cost healthcare environments. The proposed model has achieved superior results compared with the state-of-the-art techniques in Alzheimer’s and ensemble classifiers domains. The proposed framework can be used to implement efficient information fusion ensembles for other medical and non-medical problems.

Predicting the suitable fertilizer for crop based on soil and environmental factors using various feature selection techniques with classifiers

AbstractAgriculture is an essential part of human life. The crop productivity is based on the soil and environmental factors. Different crops are cultivated in different areas. Nowadays, the crop productivity level is affected by the climate change and diseases in the crops. Due to this pest infestation, the crop growth is heavily affected. To overcome this problem, the right fertilizer for a particular crop has to be chosen and fertilizer helps farmers to improve the crop productivity rate. This process can be done by using various machine learning techniques. In this work, various features selection techniques with classifiers used to predict the suitable fertilizer for a crop. The experimental results show that recursive feature elimination along the proposed Heterogeneous Stacked Ensemble classifier gives better prediction rate than other methods.

A Machine Intelligence Framework for Prediction of Coronary Artery Disease Using Ensemble of Heterogeneous Classifiers

A Machine Intelligence Framework for Prediction of Coronary Artery Disease Using Ensemble of Heterogeneous Classifiers

New Voting Schemas for Heterogeneous Ensemble of Classifiers in the Problem of Football Results Prediction

Sports events attract a lot of attention and are an exciting playground for testing various machine learning methods. However, the non-consistency of data, the necessity of including a chronology of events, and correlation among the data lead to a situation where complex algorithms are not in favor of deriving good classification results.This article focuses on extending our previous works and deriving new voting schemas for the ensemble of classifiers. The ensemble is built based on well-known algorithms from literature and can be considered a heterogeneous approach. Especially, we put a particular focus on improving the poor results achieved for selected decision classes. To do so, we first identify the weak spots for existing methods, extend our approach to the new voting schemas, and eventually improve the classification results.However, in the presented approach, we observe the problem of the trade-of between improving some results while decreasing the overall good results of the remaining decision classes.The whole experiments are performed on the real-world data obtained from websites, preprocessed, and transformed to ft into the format used by the popular classification methods. Obtained results indicate that the analyzed problem is not trivial, and improvements for some results lead to decreasing quality of some others.

Access control system to information resources of swarm robotic systems in the presence of intruded malicious agents

A distributed approach to the creation of robotic swarm systems (SRS) will allow solving a wide range of problems in the fields of environmental protection, medicine, cleaning, surveillance, and many others. On the other hand, this approach allows an attacker to intrude malicious agents whose actions aimed at reducing the quality of the entire system. In this regard, the issue of research, design, and testing of these systems from the point of view of information security becomes relevant. Based on this, the purpose of this study is to develop an access control system (ACS) to SRS information resources in the presence of intruded malicious agents to improve the efficiency of SRS functioning when performing spatially distributed tasks. Within the framework of solving the task set, an approach to the synthesis of ACS is proposed, which makes it possible to ensure the continuous detection of malicious agents at all stages of the life cycle of the SRS operation. An element of the scientific novelty of the presented solution is the algorithm for synthesizing an ensemble of heterogeneous classifiers, the main idea of which is a combination of stacking and boosting methods for preparing a training sample and training a meta-classifier. Distinctive features of the proposed algorithm are the ability to use many different classifiers as basic classifiers, as well as the ability to reduce the size of the training set of the meta-classifier by using boosting when selecting samples in the training set. The proposed approach to the synthesis of ACS to SRS information resources using the presented algorithm improves the accuracy of detecting malicious agents both in the process of task distribution and in the process of task execution. The presented solution is implemented in the form of software in the Python programming language, which can be used in modeling decentralized control systems of SRS. The article refers to the scientific specialty 2.3.6 “Methods and systems of information protection, information security”.

Ensemble Classifier for Stock Trading Recommendation

ABSTRACT This paper presents a heterogeneous ensemble classifier for price trend prediction of a stock, in which the prediction results are subsequently used in trading recommendation. The proposed ensemble model is based on Support vector machine, Artificial neural networks, Random forest, Extreme gradient boosting, and Light gradient boosting machine. A feature selection is performed to choose an optimal set of 45 technical indicators as input attributes of the model. Each base classifier is executed with an extensive hyperparameter tuning to improve performance. The prediction results from five base classifiers are aggregated through a modified majority voting among three classifiers with the highest accuracies, to obtain final prediction result. The performance of proposed ensemble classifier is evaluated using daily historical prices of 20 stocks from Stock Exchange of Thailand, with 3 overlapping datasets of 5-year intervals during 2014–2020 for different market conditions. The experimental results show that the proposed ensemble classifier clearly outperforms buy-and-hold strategy, individual base classifiers, and the ensemble with straightforward majority voting in terms of both trading return and Sharpe ratio.

Big data classification using heterogeneous ensemble classifiers in Apache Spark based on MapReduce paradigm

In this era of big data, processing large scale data efficiently and accurately has become a challenging problem. Ensemble classification is a type of supervised learning that uses multiple experts to generate the final output. It provides a way to classify data more accurately. As a result of using multiple classifiers, they are often more complicated than single classifiers, especially for big data problems. Apache Spark is a unified analytics engine for big data processing which provides a scalable framework to analyze the data. In this paper, we first extend our previous work and design a distributed heterogeneous ensemble classifier inspired by the boosting approach, which is capable of dealing with big datasets. Using heterogeneous classifiers makes it possible to have more diverse classifiers, and consequently, a more accurate classifier is obtained. Then, we present the Spark version of the proposed approach to speed up our heterogeneous ensemble classifier using the MapReduce paradigm. In order to evaluate our approach, we have applied it to seven big datasets. Extensive experimental results indicate the superiority of the proposed method over the existing ensemble algorithms implemented by Spark MLlib in terms of the classification accuracy, performance, and scalability.

Weighted Synthetic Minority Over-Sampling Technique (WSMOTE) Algorithm and Ensemble Classifier for Hepatocellular Carcinoma (HCC) In Liver Disease System

HCC (Hepato Cellular Carcinoma) is a generic liver cancer causing death in people suffering from LC (Liver Cirrhosis). Early prognosis of HCC is a significant factor to the line of LC treatment for clinicians. Though current treatments for HCC have been effective, patients have responded negatively or exhibited aggressive biological behaviours. Identification accuracy reduces when sub-optimal models of MLTs (Machine Learning Techniques) process multiple classes. MLT models base their predictions on their training phases where class imbalances in datasets have retained challenges in terms of unsatisfactory results. This research work attempts to overcome these issues with its proposed WSMOTE (Weighted Synthetic Minority Over-sampling Technique) algorithm which is targeted at dataset imbalances. Missing values are imputed using IFCM (Improved Fuzzy C Means) clustering for enhancing analysis accuracy. Imputed features are chosen selectively using IEFS (Intelligence Ensemble-Based Feature Selection). A heterogeneous ensemble classifier using Bootstrap aggregation is applied for combining predictions of multiple classifiers including KSVMs (Kernel Support Vector Machines) and FCNNs (Fuzzy Convolution Neural Networks) for accuracy in outputs. This work’s schema when tested on MATLAB (Matrix Laboratory) was found to classify better than most other methods in terms of precision, recall, F-measure and accuracy.

Comparing Performance of Machine Learning Algorithms for Default Risk Prediction in Peer to Peer Lending

The purpose of this research is to evaluate several popular machine learning algorithms for credit scoring for peer to peer lending. The dataset to fit the models is extracted from the official site of Lending Club. Several models have been implemented, including single classifiers (logistic regression, decision tree, multilayer perceptron), homogeneous ensembles (XGBoost, GBM, Random Forest) and heterogeneous ensemble classifiers like Stacked Ensembles. Results show that ensemble classifiers outperform single ones with Stacked Ensemble and XGBoost being the leaders.

A Credit Scoring Heterogeneous Ensemble Model Using Stacking and Voting

Background/Objectives: Recent studies emphasized on using ensemble models over single ones to solve credit scoring problems. The objective of this study is to build a heterogeneous ensemble classifier model with an improved classification accuracy. Methods: This study focuses on developing a heterogeneous ensemble classifier using Logistic Regression, K-nearest neighbor, Decision tree, Random Forest, Naïve Base and Support vector machine as base classifiers and Random Forest, Logistic Regression and Support vector machine as meta-classifiers. The proposed model is built using these six base classifiers for ensemble aggregation. A feature selection algorithm based on the random forest technique is used for selecting the best features. A stacking and voting method are used for building ensemble model. Findings: The ensemble classifier gives superior predictive performance than single classifiers SVM, DT, RF, NB, KNN and LR with an accuracy of 91.56% for Australian dataset and 84.35% for German dataset. Novelty: The proposed model uses stacking and majority voting method for ensemble classification. Initially, stacking is applied to the base classifiers. This is done in two levels. First the training dataset is split into 10 folds for cross validation. The output of each classifier is taken, and the dataset is updated with the meta-features. In the second level, three meta-classifiers (MC), namely LR, SVM and RF are used. Majority voting is applied to the output of these meta-classifiers for the prediction. Keywords: Credit scoring; ensemble model; SVM; DT; RF; NB; KNN; LR

Memory-Efficient, Accurate and Early Diagnosis of Diabetes Through a Machine Learning Pipeline Employing Crow Search-Based Feature Engineering and a Stacking Ensemble

The early diagnosis of diabetes helps in avoiding the major risks associated with the disorder. The proposed research involves the design of a machine learning pipeline which generates the most representative feature subset of minimal size that predicts the onset of Diabetes with highest accuracy. It employs a novel diabetes dataset which is gender-neutral and representative enough unlike the well-known PID dataset. The machine learning pipelines involve multiple feature engineering pipelines to generate a reduced feature subset which is fed into multiple heterogeneous classifiers. The feature engineering involves feature selection as well as feature extraction. The former uses the ANOVA filter and Crow Search Optimization algorithm. The latter employs the Singular Value Decomposition. The classification is performed on the preprocessed dataset using a wide range of heterogeneous classifiers like Naive Bayes’, Logistic Regression, K-Nearest Neighbor, Decision Trees, Support Vector Machine, Random Forest, AdaBoost, and GradientBoost as base learners followed by their stacking ensemble. The performance evaluation of each machine learning pipeline is done through Repeated Stratified K-fold Cross Validation using the metrics of accuracy, precision, recall, F1 Score and area under Receiver Operating Characteristic curve. For each pipeline, the number of features in the preprocessed dataset varies and the highest accuracy of 98.4% is achieved with Crow Search algorithm through a stacking ensemble of multiple heterogeneous classifiers. A comparative analysis with a recent related work on the same dataset shows that the proposed feature engineering pipelines with the same set of classifiers outperform with improved accuracy using a feature set of reduced size.

Heterogeneous ensemble spectral classifiers for hyperspectral images

Hyperspectral images (HSIs) have attracted worldwide attention as they make it possible to remotely distinguish between spectrally similar materials. The data of hyperspectral image with high dimensionality, redundant features, unbalanced distribution and so on, however, impose great difficulties for HSI classification. An ensemble of classifiers are usually considered to provide more accurate predictions than a single classifier. In this paper, a heterogeneous ensemble consisting of a support vector machine, a kernel extreme learning machine and a multinomial logistic regression classifier is constructed to increase classification accuracies of HSIs. Furthermore, the feature extraction method is employed to transform the spectral features to promote the diversity of the ensemble, and a voting mechanism is introduced to predict the label on unseen data. Our experimental results with two widely used real hyperspectral datasets demonstrate that the proposed heterogeneous ensemble classifier obtains satisfactory performance with respect to spectral feature sets, even though the number of labelled samples is small.