Individual Base Classifiers Research Articles

MalDetect: A classifier fusion approach for detection of android malware

Android has been a significant target of malware applications due to the exponential growth of mobile devices. This may result in severe threats to Android users such as financial loss, information leakage etc. The security of smartphones has encouraged the researchers in applying machine learning techniques to enhance the Android malware detection rate as the traditional approaches are not effective in distinguishing unknown malware. In recent years, many malicious apps have been detected on Android devices using machine learning based approaches. The problem with these detection systems is that they give a high false positive and false negative rate. Therefore, it is imperative to develop methods for better identifying and classifying malware. The main contribution of this paper is to improve the classification accuracy by fusing the base classifiers. This paper proposes a fusion approach named as MalDetect that enables an effective combination of machine learning techniques to improve the identification rate of malware. The proposed approach is based on the base classifiers and a set of ranking algorithms defined on their predictive error rate. The purpose of this method is to improve the detection rate of Android malware. Two datasets i.e. Drebin and AndroMD are used to illustrate the effectiveness of the proposed approach. Experimental results show that the proposed approach performs better than the individual base classifiers and traditional ensemble learning techniques for identifying Android malware. Further, the proposed approach is compared with the renowned Stacking ensemble technique that uses a meta-classifier. From the experimental outcomes, it is found that the weighted F-measure obtained from the Stacking for Drebin and AndroMD datasets is 0.9811 and 0.9775 respectively whereas the weighted F-measure acquired from the proposed approach is 0.9857 and 0.9790 respectively.

Mitigating anomalous electricity consumption in smart cities using an AI‐based stacked‐generalization technique

AbstractEnergy management and efficient asset utilization play an important role in the economic development of a country. The electricity produced at the power station faces two types of losses from the generation point to the end user. These losses are technical losses (TL) and non‐technical losses (NTL). TLs occurs due to the use of inefficient equipment. While NTLs occur due to the anomalous consumption of electricity by the customers, which happens in many ways; energy theft being one of them. Energy theft majorly happens to cut down on the electricity bills. These losses in the smart grid (SG) are the main issue in maintaining grid stability and cause revenue loss to the utility. The automatic metering infrastructure (AMI) system has reduced grid instability but it has opened up new ways for NTLs in the form of different cyber‐physical theft attacks (CPTA). Machine learning (ML) techniques can be used to detect and minimize CPTA. However, they have certain limitations and cannot capture the energy consumption patterns (ECPs) of all the users, which decreases the performance of ML techniques in detecting malicious users. In this paper, we propose a novel ML‐based stacked generalization method for the cyber‐physical theft issue in the smart grid. The original data obtained from the grid is preprocessed to improve model training and processing. This includes NaN‐imputation, normalization, outliers' capping, support vector machine‐synthetic minority oversampling technique (SVM‐SMOTE) balancing, and principal component analysis (PCA) based data reduction techniques. The pre‐processed dataset is provided to the ML models light gradient boosting (LGB), extra trees (ET), extreme gradient boosting (XGBoost), and random forest (RF), to accurately capture all consumers' overall ECP. The predictions from these base models are fed to a meta‐classifier multi‐layer perceptron (MLP). The MLP combines the learning capability of all the base models and gives an improved final prediction. The proposed structure is implemented and verified on the publicly available real‐time large dataset of the State Grid Corporation of China (SGCC). The proposed model outperformed the individual base classifiers and the existing research in terms of CPTA detection with false positive rate (FPR), false negative rate (FNR), F1‐score, and accuracy values of 0.72%, 2.05%, 97.6%, and 97.69%, respectively.

Heterogeneous learning method of ensemble classifiers for identification and classification of power quality events and fault transients in wind power integrated microgrid

This paper proposes heterogeneous based ensemble Classifiers (voting and stacking method) to identify and classify different power system disturbances (power quality (PQ), faults, transients, and wind power variation) in wind integrated microgrid network. In the pre-processing stage of classification, a Discrete wavelet transform (DWT) technique is applied to extract the features from power system disturbance signals. The classification process for the proposed ensemble models involves two levels of classification. At the first level, the extracted features from the simulated power system events are used to learn the different individual base classifiers (logistic regression (LR), K-Nearest Neighbor (KNN), and J48 Decision tree (JDT)]. In second stage, a Meta-level classification is carried out based on predictions of base classifiers to get final predictions of class labels. First, the proposed ensemble models are utilized to discriminate the power system disturbances under random varying wind power condition and the predictive results (classification accuracy and performance indices) of ensemble models are compared with individual base classifiers (LR, KNN., and JDT). In addition, a sensitivity analysis is carried out under real time varying wind power condition and noisy environment of event signals to verify the effectiveness of ensemble models in further level. Furthermore, the robustness of proposed stacking ensemble model is verified with classification of single and combined PQ events of synthetic data, generated from the mathematical based PQ model software Predictions results under the different conditions show that stacking ensemble model offers substantial performance and discriminates the power disturbances with higher accuracy of classification than base classifiers and voting ensemble model.

Co-occurrence histogram based ensemble of classifiers for classification of cervical cancer cells

To explore the potential of conventional image processing techniques in the classification of cervical cancer cells, in this work, a co-occurrence histogram method was employed for image feature extraction and an ensemble classifier was developed by combining the base classifiers, namely, the artificial neural network (ANN), random forest (RF), and support vector machine (SVM), for image classification. The segmented pap-smear cell image dataset was constructed by the k-means clustering technique and used to evaluate the performance of the ensemble classifier which was formed by the combination of the above considered base classifiers. The result was also compared with that achieved by the individual base classifiers as well as that trained with color, texture, and shape features. The maximum average classification accuracy of 93.44% was obtained when the ensemble classifier was applied and trained with co-occurrence histogram features, which indicates that the ensemble classifier trained with the co-occurrence histogram features is more suitable for the classification of cervical cancer cells.

A Voting Approach of Ensemble Classifier for Detection of Power Quality in Islanded PV Microgrid

In this research, an ensemble classifier using a voting method is proposed to categorize different Power Quality events (PQEs), switching, and fault transients in solar photovoltaic (PV) connected Microgrid (MG) network. For this analysis, an islanded MG network model is developed in the MATLAB-Simulink software framework to study the various power system disturbances (PQEs, switching transients, and Line to Ground (LG) fault). The voltage/current signals of various power disturbances are used for the extraction of features by applying a discrete wavelet transform technique. Then, the individual base classifiers (Bayesian Net, Naive Bayesian, and Logistic Regression) are trained by using the extracted features in the first stage. Subsequently, the voting method of Meta classification is applied to obtain the final predictions of class events in the second level, based on the predictions of base classifiers. The classification and statistical analysis are carried out under standard test conditions (STC) and real-time-varying solar irradiance of PV. The evaluation results of classification and statistical analysis clearly show that the proposed ensemble classifier offers improved classification accuracy with superior performance than individual base classifiers under STC and uncertain conditions of solar PV.

Stacking approach for accurate Invasive Ductal Carcinoma classification

The accurate diagnosis of Breast cancer (BC) requires adequately exploiting Artificial intelligence (AI)-based methods in the diagnosing process. To tackle the issue of accurate BC diagnosis, we have proposed a deep learning-based stacking method (StackBC). In particular, we have incorporated deep learning models including Convolutional Neural Network (CNN), Long Short Term Memory (LSTM), and Gated Recurrent Unit (GRU) for the classification of Invasive Ductal Carcinoma (IDC). Additionally, Transfer Learning (TL) and Data Augmentation (DA) approaches have been incorporated to balance the dataset and adequately train the model. To further improve the predictive outputs of the model, we used the stacking technique. Among the three individual base classifiers, the performance of the GRU model was better. Hence, we selected the GRU as a meta classifier to distinguish between Non-IDC and IDC breast images. The experimental results confirmed that the StackBC method outperformed state-of-the-art methods.

A unified deep neuro-fuzzy approach for COVID-19 twitter sentiment classification

Covid-19 braces serious mental health crisis across the world. Since a vast majority of the population exploit social media platforms such as twitter to exchange information, rapid collecting and analyzing social media data to understand personal well-being and subsequently adopting adequate measures could avoid severe socio-economic damage. Sentiment analysis on twitter data is very useful to understand and identify the mental health issues. In this research, we proposed a unified deep neuro-fuzzy approach for Covid-19 twitter sentiment classification. Fuzzy logic has been a very powerful tool for twitter data analysis where approximate semantic and syntactic analysis is more relevant because correcting spelling and grammar in tweets are merely obnoxious. We conducted the experiment on three challenging COVID-19 twitter sentiment datasets. Experimental results demonstrate that fuzzy Sugeno integral based ensembled classifiers succeed over individual base classifiers.

Vehicle accident sub-classification modeling using stacked generalization: A multisensor fusion approach

Road accidents caused by several factors are responsible for the deaths and injuries of people. The majority of deaths occur within first few hours of an accident. If in case of an accident the victim becomes incapacitated, and there is no onlooker to help, then an accurate automated accident reporting system can significantly reduce the number of such deaths. This paper reports an Android smartphone-based end-to-end Internet of Things (IoT) system that can transmit accident information to emergency services and affected families once a vehicle accident is detected. Along with the detection, the classification of vehicle accidents can be very helpful in identifying suitable life-saving medical-aid and appropriate rescue operation equipment. The main objective of this research work is to develop a machine learning (ML) model that can detect as well as classify vehicle accidents accurately into eight categories. To enhance the classification efficacy of the system, a multi-sensor fusion framework has been proposed that incorporates various sensor-fusion techniques at different levels of its implementation along with several preprocessing methods such as 10-ms moving maximum, complementary filters, and a 2-sec sliding window. The framework uses Logistic Regression (LR) based stacked generalization approach to combine the decisions of three ML classifiers, which are based on Decision Tree (DT), Naïve Bayes (NB), and Random Forest (RF) methods. The LR-based stacking classifier is found to be very accurate with an F1-score of 0.95, whose performance is significantly better than the performance of all individual base-classifiers.

Improving Depression Prediction Accuracy Using Fisher Score-Based Feature Selection and Dynamic Ensemble Selection Approach Based on Acoustic Features of Speech

Depression affects over 322 million people, and it is the most common source of disability worldwide. Literature in speech processing revealed that speech could be used for detecting depression. Depressed individuals exhibit varied acoustic characteristics compared to non-depressed. A four-staged machine learning classification system is developed to investigate the acoustic parameters to detect depression. Stage one uses speech recordings from a publicly available and clinically validated dataset DAIC-WOZ. The baseline acoustic feature vector, eGeMAPS, is extracted from the dataset in stage two. Adaptive synthetic (ADASYN) is performed along with data preprocessing to overcome the class imbalance. In stage three, we conducted feature selection (FS) using three techniques; Boruta FS, recursive feature elimination using support vector machine (SVM-RFE), and the fisher score-based FS. Experimentation with various machine learning base classifiers like gaussian naïve bayes (GNB), support vector machine (SVM), k-nearest neighbors (KNN), logistic regression (LR), and random forest classifier (RF) is performed in stage four. The hyperparameters of the classifiers are tuned using the GridSearchCV technique throughout the 10-fold stratified cross-validation (CV). Then we employed multiple dynamic ensemble selection of classifier algorithms (DES) with k=3 and k=5 utilizing the pool of aforementioned four base classifiers to improve the accuracy. We present a comparative study using eGeMAPS features against the base classifiers and the experimented DES classifiers. Our results on the DAIC-WOZ benchmark dataset suggested that K-Nearest Oracles Union (KNORA-U) DES with k=3 has superior accuracy using a subset of 15 features selected by fisher score-based FS than the individual base classifiers.

Developing a Stacked Ensemble-Based Classification Scheme to Predict Second Primary Cancers in Head and Neck Cancer Survivors.

Despite a considerable expansion in the present therapeutic repertoire for other malignancy managements, mortality from head and neck cancer (HNC) has not significantly improved in recent decades. Moreover, the second primary cancer (SPC) diagnoses increased in patients with HNC, but studies providing evidence to support SPCs prediction in HNC are lacking. Several base classifiers are integrated forming an ensemble meta-classifier using a stacked ensemble method to predict SPCs and find out relevant risk features in patients with HNC. The balanced accuracy and area under the curve (AUC) are over 0.761 and 0.847, with an approximately 2% and 3% increase, respectively, compared to the best individual base classifier. Our study found the top six ensemble risk features, such as body mass index, primary site of HNC, clinical nodal (N) status, primary site surgical margins, sex, and pathologic nodal (N) status. This will help clinicians screen HNC survivors before SPCs occur.

Ensemble Classifier for Stock Trading Recommendation

ABSTRACT This paper presents a heterogeneous ensemble classifier for price trend prediction of a stock, in which the prediction results are subsequently used in trading recommendation. The proposed ensemble model is based on Support vector machine, Artificial neural networks, Random forest, Extreme gradient boosting, and Light gradient boosting machine. A feature selection is performed to choose an optimal set of 45 technical indicators as input attributes of the model. Each base classifier is executed with an extensive hyperparameter tuning to improve performance. The prediction results from five base classifiers are aggregated through a modified majority voting among three classifiers with the highest accuracies, to obtain final prediction result. The performance of proposed ensemble classifier is evaluated using daily historical prices of 20 stocks from Stock Exchange of Thailand, with 3 overlapping datasets of 5-year intervals during 2014–2020 for different market conditions. The experimental results show that the proposed ensemble classifier clearly outperforms buy-and-hold strategy, individual base classifiers, and the ensemble with straightforward majority voting in terms of both trading return and Sharpe ratio.

Credit scoring based on tree-enhanced gradient boosting decision trees

Credit scoring is an important tool for banks and lending companies to realize credit risk exposure management and gain profits. GBDTs, a group of boosting-type ensemble algorithms, have shown promising improvement for credit scoring. However, GBDT improves the credit scoring performance by iteratively modifying only the fitting target for each base classifier and invariably works on the same features, which limits the diversity of individual classifiers in GBDT; Moreover, the performance-interpretability dilemma motivated a large number of works to focus on the pursuit of high-performance ensemble strategies, which leads to the lack of explorations on the interpretability of the credit scoring models. Based on the above limitations, two tree-based augmented GBDTs (AugBoost-RFS and AugBoost-RFU) are proposed in this work for credit scoring. In the proposed methods, a step-wise feature augmentation mechanism is introduced for GBDT to enrich the diversity of individual base classifiers; Tree-based embedding technologies simplify the process of feature augmentation and inherit interpretability of GBDT. Results on 4 large-scale credit scoring datasets show AugBoost-RFS/AugBoost-RFU outperforms GBDT; Besides, supervised tree-based step-wise feature augmentation for GBDT achieves comparable results to neural network-based step-wise feature augmentation while significantly improve the augmentation efficiency. Moreover, the intrinsic global interpreted results and decision rules of tree-enhanced GBDTs, as well as the marginal contributions of features that are visualized by TreeSHAP demonstrate AugBoost-RFS/AugBoost-RFU can be good candidates for interpretable credit scoring.

Ensemble Classifiers Employed for Spam Review Detection

The advancement of technology and the use of internet have changed many aspects of human culture over the years. Today, consumers take confidence in e-commerce platforms like amazon and eBay for comprehensive understanding of products and services when making a purchase decision. Here the web or user-generated content from consumers of such products and services, known as reviews, are exploited by spam reviewers to falsely promote or downgrade some targeted products. Despite potential solutions, Identifying and preventing review spam are still one of the top challenges faced by web search engines today. Therefore, in the quest to provide a more improved and efficient classification of review spam, this research probed different techniques in order to find most effective solution to spam detection. The research employed three base classifiers, Naïve Bayes, Support Vector Machines and Logistic Regression to form ensemble classifiers complimented with Arching classifier. The Arching classifier performs the weighted voting that produces the final class label with performance and accuracy higher than either of the individual base classifiers. Cross-validation is used as evaluation metrics to measure the performance or effectiveness of the ensemble classifiers while the experimental results shows that the ensemble classifiers achieve the best results compared to the single based classifier in terms of Precision, Recall, F1-measure and Accuracy.

Ensemble-based active learning using fuzzy-rough approach for cancer sample classification

Background and Objective: Classification of cancer from gene expression data is one of the major research areas in the field of machine learning and medical science. Generally, conventional supervised methods are not able to produce desired classification accuracy due to inadequate training samples present in gene expression data to train the system. Ensemble-based active learning technique in this situation can be effective as it determines few informative samples by all the base classifiers and ensemble the decisions of all the base classifiers to get the most informative samples. Most informative samples are labeled by the subject experts and those are added to the training set, which can improve the classification accuracy.Method: We propose a novel ensemble-based active learning using fuzzy-rough approach for cancer sample classification from microarray gene expression data. The proposed method is able to deal with the uncertainty, overlap and indiscernibility usually present in the subtype classes of the gene expression data and can improve the accuracy of the individual base classifier in presence of limited training samples.Results: The proposed method is validated using eight microarray gene expression datasets. The performance of the proposed method in terms of classification accuracy, precision, recall, F1-measures and kappa is compared with six other methods. The improvements in accuracy achieved by the proposed method compared to its nearest competitive methods are 2.96%, 9.34%, 0.93%, 3.69%, 7.2% and 4.53% respectively for Colon cancer, Prostate cancer, SRBCT, Ovarian cancer, DLBCL and Central nervous system datasets. Results of the paired t-test justify the statistical relevance of the results in favor of the proposed method for most of the datasets.Conclusion: The proposed method is an effective general purpose ensemble-based active learning adopting the fuzzy-rough concept and therefore can be applied for other classification problem in future.

Combining Linear Classifiers Using Probability-Based Potential Functions

The score function can be used as a measure for evaluating predicted probabilities of the classification models. In multiple classifiers systems, one of the problems is the diversity of the way of determining the scoring function of individual base classifiers. To alleviate this limitation, in this article, we propose a novel concept of calculating a scoring function defined by the probability-based potential function. The proposed potential functions take into account the distance of the recognized object from the decision boundary as well as a prior probability of the class labels. The proposed score function has the same nature for all linear base classifiers, which defined the multiple classifiers model. Additionally, the proposed method is compared with other ensemble algorithms based on homogeneous linear base classifiers. The experiments on seventy databases demonstrate the effectiveness of our method. To discuss the results of our experiments, we use multiple classification performance measures dedicated to standard and imbalanced datasets. The statistical analysis of the experiments is also performed.

Ensemble learning via constraint projection and undersampling technique for class-imbalance problem

Ensemble learning is an effective technique for the class-imbalance problem, and the key for obtaining a successful ensemble is to create individual base classifiers with high accuracy and diversity. In this paper, we propose a novel ensemble learning method via constraint projection and undersampling technique, constructing each base classifier through the following two steps: 1) constructing a set of pairwise constraints by undersampling examples from the minority/majority class set and learning a projection matrix from the pairwise constraint set and 2) undersampling the original training set to obtaining a new training set on which a base classifier is constructed in the new feature space defined by the projection matrix. For the first step, the projection matrix is mainly used to enhance the separability between the diverse class examples and thus to improve the performance of the base classifier, and the undersampling technique is used to create diverse sets of pairwise constraints to train diverse projection matrices, thus introducing diversity to base classifiers. For the second step, the undersampling technique aims to improve the performance of base classifiers on the minority class and further increase the diversity between the individual base classifiers. The experimental results show that the proposed method shows significantly better performance on the measures of recall, g-mean, f-measure and AUC than other state-of-the-art methods for 29 datasets with various data distributions and imbalance ratios.

Intercropping Classification From GF-1 and GF-2 Satellite Imagery Using a Rotation Forest Based on an SVM

Remote sensing has been widely used in vegetation cover research but is rarely used for intercropping area monitoring. To investigate the efficiency of Chinese Gaofen satellite imagery, in this study the GF-1 and GF-2 of Moyu County south of the Tarim Basin were studied. Based on Chinese GF-1 and GF-2 satellite imagery features, this study has developed a comprehensive feature extraction and intercropping classification scheme. Textural features derived from a Gray level co-occurrence matrix (GLCM) and vegetation features derived from multi-temporal GF-1 and GF-2 satellites were introduced and combined into three different groups. The rotation forest method was then adopted based on a Support Vector Machine (RoF-SVM), which offers the advantage of using an SVM algorithm and that boosts the diversity of individual base classifiers by a rotation forest. The combined spectral-textural-multitemporal features achieved the best classification result. The results were compared with those of the maximum likelihood classifier, support vector machine and random forest method. It is shown that the RoF-SVM algorithm for the combined spectral-textural-multitemporal features can effectively classify an intercropping area (overall accuracy of 86.87% and kappa coefficient of 0.78), and the classification result effectively eliminated salt and pepper noise. Furthermore, the GF-1 and GF-2 satellite images combined with spectral, textural, and multi-temporal features can provide sufficient information on vegetation cover located in an extremely complex and diverse intercropping area.

Cascaded classifiers and stacking methods for classification of pulmonary nodule characteristics

Background and ObjectivesDetection and classification of pulmonary nodules are critical tasks in medical image analysis. The Lung Image Database Consortium (LIDC) database is a widely used resource for small pulmonary nodule classification research. This dataset is comprised of nodule characteristic evaluations and CT scans of patients. Although these characteristics are utilized in several studies, they can be used to improve classification performance. MethodsNumerous methods have been proposed to classify malignancy, but there are not many studies that facilitate nodule characteristics in classification steps. In this study, we use information on nodule characteristics and propose cascaded classification schemes. A group of hand-crafted features and deep features are used to define the nodules. In the first step of the classifier, the nodule characteristics are classified based on individual base classifiers. In the second step, the results of the first level classifier are combined for use in malignancy classification. In addition, stacking methods are applied to improve the performance of the cascaded classifiers. ResultsThe results confirmed that combining deep and hand-crafted features contribute to classification performance with an 8% improvement in average classification accuracy, 9% improvement in sensitivity, and 3% in specificity. Deep features from a nodule bounding area are more descriptive than the exact nodule region. The best performing cascaded classifier featured a classification accuracy of 84.70%, sensitivity of 67.37%, and specificity of 95.46%. First level stacking demonstrated similar results on classification accuracy and specificity but sensitivity was measured at 75.59%. Stacking on both levels provided the best classification accuracy and specificity with scores of 86.98% and 96.06%, respectively. When the malignancy ratings were grouped, stacking on both levels demonstrated better performance than other methods with a classification accuracy of 88.80%, sensitivity of 88.41%, and specificity of 94.12%. ConclusionsInformation on cascading characteristics with image features is beneficial for the classification of the malignancy ratings. Stacking approaches on both levels demonstrate better classification accuracy, but in the context of sensitivity, first level stacking performs better. Grouping the malignancy ratings results in better classification outcomes as in the case of similar studies in the literature.

Optimized Classification Predictions with a New Index Combining Machine Learning Algorithms

Voting is a commonly used ensemble method aiming to optimize classification predictions by combining results from individual base classifiers. However, the selection of appropriate classifiers to participate in voting algorithm is currently an open issue. In this study we developed a novel Dissimilarity-Performance (DP) index which incorporates two important criteria for the selection of base classifiers to participate in voting: their differential response in classification (dissimilarity) when combined in triads and their individual performance. To develop this empirical index we firstly used a range of different datasets to evaluate the relationship between voting results and measures of dissimilarity among classifiers of different types (rules, trees, lazy classifiers, functions and Bayes). Secondly, we computed the combined effect on voting performance of classifiers with different individual performance and/or diverse results in the voting performance. Our DP index was able to rank the classifier combinations according to their voting performance and thus to suggest the optimal combination. The proposed index is recommended for individual machine learning users as a preliminary tool to identify which classifiers to combine in order to achieve more accurate classification predictions avoiding computer intensive and time-consuming search.

A DBN-based resampling SVM ensemble learning paradigm for credit classification with imbalanced data

Credit risk assessment is often accompanied with sampling data imbalance. For this reason, this paper tries to propose a deep belief network (DBN) based resampling support vector machine (SVM) ensemble learning paradigm to solve imbalanced data problem in credit classification. In this paradigm, a bagging algorithm is first used to generate variable training subsets to make the subsets rebalanced and suitable in size. Then the SVM model is used as individual base classifier to formulate diverse ensemble input members. Finally, the DBN model is applied as an ensemble method to fuse the input members to aggregate the classification results. In addition, the weights of different classes are changed by introducing a revenue matrix in terms of revenue-sensitive technique, which helps to make the results more reasonable. The experimental results indicate that the classification performance are improved effectively when the DBN-based ensemble strategy is integrated with re-sampling techniques, especially in imbalanced-data problem, implying that the proposed DBN-based resampling SVM ensemble learning paradigm can be used as a promising tool for credit risk classification with imbalanced data.