Binary Classification Datasets Research Articles

An Investigation of SMOTE based Methods for Imbalanced Datasets with Data Complexity Analysis

Many binary class datasets in real-life applications are affected by class imbalance problem. Data complexities like noise examples, class overlap and small disjuncts problems are observed to play a key role in producing poor classification performance. These complexities tend to exist in tandem with class imbalance problem. Synthetic Minority Oversampling Technique (SMOTE) is a well-known method to re-balance the number of examples in imbalanced datasets. However, this technique cannot effectively tackle data complexities and it also has the capability of magnifying the degree of complexities. Also, the performance of the SMOTE is still not satisfactory. Therefore, various SMOTE variants have been proposed to overcome the downsides of SMOTE either by combining SMOTE with other algorithms or modifying the existing SMOTE algorithm. This paper aims to comparatively review the algorithms applied in SMOTE variants and investigate which data complexities are being addressed in what variants. Series of experiments are conducted on 24 binary class imbalanced datasets to observe the changes in the data complexity measures after SMOTE variants were applied in these datasets. The evaluation metrics like G-Mean and F1-Score are also analyzed to investigate the difference in classification performance between SMOTE variants.

MiNB: Minority Sensitive Naïve Bayesian Algorithm for Multi-Class Classification of Unbalanced Data

The unbalanced nature of data makes it tough to achieve the desire performance goal for classification algorithms. The sub-optimal prediction system isn't a viable solution due to the high misclassification cost of minority events. Thus accurate imbalanced data classification could be a path changer for prediction in domains like medical diagnosis, judiciary, and disaster management systems. To date, most of the existing studies of imbalanced data are for the binary class dataset and supported by data sampling techniques that suffer from loss of information and over-fitting. In this paper, we present the modified naïve Bayesian algorithm for unbalanced data classification that eliminates the requirement of data level sampling. We compared our proposed model with the data sampling technique and cost-sensitive techniques. We use minority sensitive TP Rate, class-specific misclassification rate, and overall performance parameters such as accuracy, f-measure and G-mean. The result shows that our proposed algorithm shows a more optimal result for unbalanced data classification. Results shows reduction in misclassification rate and improve predictive performance for the minority class.

Online News Sentiment Classification Using DistilBERT

The ability of pre-trained BERT model to achieve outstanding performances on many Natural Language Processing (NLP) tasks has attracted the attention of researchers in recent times. However, the huge computational and memory requirements have hampered its widespread deployment on devices with limited resources. The concept of knowledge distillation has shown to produce smaller and faster distilled models with less trainable parameters and intended for resource-constrained environments. The distilled models can be fine-tuned with great performance on a wider range of tasks, such as sentiment classification. This paper evaluates the performance of DistilBERT model and other pre-canned text classifiers on a Covid-19 online news binary classification dataset. The analysis shows that despite having fewer trainable parameters than the BERT-based model, the DistilBERT model achieved an accuracy of 0.94 on the validation set after only two training epochs. The paper also highlights the usefulness of the ktrain library in facilitating the building, training, and application of state-of-the-art Machine Learning and Deep Learning models.

TwoClsBalancer: Sınıf Dengesizliği Problemi İçin Makine Öğrenmesine Dayalı Etkileşimli Bir Web Uygulaması

Objective: The main purpose of this research is to develop a novel user-friendly web tool based on machine learning approaches, which applies a variety of techniques to address the class imbalance problem. Material and Methods: Shiny, an opensource R package, was used to develop the proposed web tool. The interactive tool can handle the class imbalance problem for binary classification dataset(s) by implementing sampling-based methods. As a clinical application, the dataset retrospectively obtained from the database of the Cardiovascular Surgery Department of Turgut Özal Medical Center, İnönü University, Malatya, Türkiye was used in this web-based software. To overcome the class imbalance problem, sampling-based methods were implemented on the original dataset. After this process, the classification of hypertension in patients with coronary artery disease was achieved by three classification models. Results: According to the outputs of the developed web application, the best classification performance was obtained by the support vector machines with radial basis function kernel (SVM-RBF) model after applying the density-based synthetic minority over-sampling technique oversampling method. The accuracy, sensitivity, specificity, precision, f-measure, and g-mean metrics of the relevant model were calculated as 0.99, 0.99, 0.99, 0.95, 0.97, and 0.97, respectively. Conclusion: The oversampling methods used in this research indicated a more positive contribution to the classification performance of the models as compared to the undersampling methods. When the undersampling methods were applied, the three classification models did not demonstrate successful classification performance, whereas the SVM-RBF model outperformed the other two models when the oversampling methods were implemented. The designed interactive web application is freely accessible through http://biostatapps.inonu.edu.tr/twoclsbalancer.

On Using Classification Datasets to Evaluate Graph Outlier Detection: Peculiar Observations and New Insights.

It is common practice of the outlier mining community to repurpose classification datasets toward evaluating various detection models. To that end, often a binary classification dataset is used, where samples from one of the classes are designated as the "inlier" samples, and the other class is substantially down-sampled to create the (ground-truth) "outlier" samples. Graph-level outlier detection (GLOD) is rarely studied but has many potentially influential real-world applications. In this study, we identify an intriguing issue with repurposing graph classification datasets for GLOD. We find that ROC-AUC performance of the models changes significantly ("flips" from high to very low, even worse than random) depending on which class is down-sampled. Interestingly, ROC-AUCs on these two variants approximately sum to 1 and their performance gap is amplified with increasing propagations for a certain family of propagation-based outlier detection models. We carefully study the graph embedding space produced by propagation-based models and find two driving factors: (1) disparity between within-class densities, which is amplified by propagation, and (2) overlapping support (mixing of embeddings) across classes. We also study other graph embedding methods and downstream outlier detectors, and we find that the intriguing "performance flip" issue still widely exists but which version of the down-sample achieves higher performance may vary. Thoughtful analysis over comprehensive results further deepens our understanding of the established issue. With this study, we aim at drawing attention to this (to our knowledge) previously unnoticed issue for the rarely studied GLOD problem, and specifically to the following questions: (1) Given the performance flip issue we identified, where one version of the down-sample often yields worse-than-random performance, is it appropriate to evaluate GLOD by average performance across all down-sampled versions when repurposing graph classification datasets? (2) Considering consistently observed performance flip issue across different graph embedding methods we studied, is it possible to design better graph embedding methods to overcome the issue? We conclude the article with our insights to these questions.

Intrusion Detection Model Based on TF.IDF and C4.5 Algorithms

In recent years, the use of machine learning and data mining technologies has drawn researchers’ attention to new ways to improve the performance of Intrusion Detection Systems (IDS). These techniques have proven to be an effective method in distinguishing malicious network packets. One of the most challenging problems that researchers are faced with is the transformation of data into a form that can be handled effectively by Machine Learning Algorithms (MLA). In this paper, we present an IDS model based on the decision tree C4.5 algorithm with transforming simulated UNSW-NB15 dataset as a pre-processing operation. Our model uses Term Frequency.Inverse Document Frequency (TF.IDF) to convert data types to an acceptable and efficient form for machine learning to achieve high detection performance. The model has been tested with randomly selected 250000 records of the UNSW-NB15 dataset. Selected records have been grouped into various segment sizes, like 50, 500, 1000, and 5000 items. Each segment has been, further, grouped into two subsets of multi and binary class datasets. The performance of the Decision Tree C4.5 algorithm with Multilayer Perceptron (MLP) and Naive Bayes (NB) has been compared in Weka software. Our proposed method significantly has improved the accuracy of classifiers and decreased incorrectly detected instances. The increase in accuracy reflects the efficiency of transforming the dataset with TF.IDF of various segment sizes.

Deep/Transfer Learning with Feature Space Ensemble Networks (FeatSpaceEnsNets) and Average Ensemble Networks (AvgEnsNets) for Change Detection Using DInSAR Sentinel-1 and Optical Sentinel-2 Satellite Data Fusion

Differential interferometric synthetic aperture radar (DInSAR), coherence, phase, and displacement are derived from processing SAR images to monitor geological phenomena and urban change. Previously, Sentinel-1 SAR data combined with Sentinel-2 optical imagery has improved classification accuracy in various domains. However, the fusing of Sentinel-1 DInSAR processed imagery with Sentinel-2 optical imagery has not been thoroughly investigated. Thus, we explored this fusion in urban change detection by creating a verified balanced binary classification dataset comprising 1440 blobs. Machine learning models using feature descriptors and non-deep learning classifiers, including a two-layer convolutional neural network (ConvNet2), were used as baselines. Transfer learning by feature extraction (TLFE) using various pre-trained models, deep learning from random initialization, and transfer learning by fine-tuning (TLFT) were all evaluated. We introduce a feature space ensemble family (FeatSpaceEnsNet), an average ensemble family (AvgEnsNet), and a hybrid ensemble family (HybridEnsNet) of TLFE neural networks. The FeatSpaceEnsNets combine TLFE features directly in the feature space using logistic regression. AvgEnsNets combine TLFEs at the decision level by aggregation. HybridEnsNets are a combination of FeatSpaceEnsNets and AvgEnsNets. Several FeatSpaceEnsNets, AvgEnsNets, and HybridEnsNets, comprising a heterogeneous mixture of different depth and architecture models, are defined and evaluated. We show that, in general, TLFE outperforms both TLFT and classic deep learning for the small dataset used and that larger ensembles of TLFE models do not always improve accuracy. The best performing ensemble is an AvgEnsNet (84.862%) comprised of a ResNet50, ResNeXt50, and EfficientNet B4. This was matched by a similarly composed FeatSpaceEnsNet with an F1 score of 0.001 and variance of 0.266 less. The best performing HybridEnsNet had an accuracy of 84.775%. All of the ensembles evaluated outperform the best performing single model, ResNet50 with TLFE (83.751%), except for AvgEnsNet 3, AvgEnsNet 6, and FeatSpaceEnsNet 5. Five of the seven similarly composed FeatSpaceEnsNets outperform the corresponding AvgEnsNet.

Evolutionary multiple instance boosting framework for weakly supervised learning

Multiple instance boosting (MILBoost) is a framework which uses multiple instance learning (MIL) with boosting technique to solve the problems regarding weakly labeled inexact data. This paper proposes an enhanced multiple boosting framework—evolutionary MILBoost (EMILBoost) which utilizes differential evolution (DE) to optimize the combination of weak classifier or weak estimator weights in the framework. A standard MIL dataset MUSK and a binary classification dataset Hastie_10_2 are used to evaluate the results. Results are presented in terms of bag and instance classification error and also confusion matrix of test data.

Metaheuristic Optimized Multi-Level Classification Learning System for Engineering Management

Multi-class classification is one of the major challenges in machine learning and an ongoing research issue. Classification algorithms are generally binary, but they must be extended to multi-class problems for real-world application. Multi-class classification is more complex than binary classification. In binary classification, only the decision boundaries of one class are to be known, whereas in multiclass classification, several boundaries are involved. The objective of this investigation is to propose a metaheuristic, optimized, multi-level classification learning system for forecasting in civil and construction engineering. The proposed system integrates the firefly algorithm (FA), metaheuristic intelligence, decomposition approaches, the one-against-one (OAO) method, and the least squares support vector machine (LSSVM). The enhanced FA automatically fine-tunes the hyperparameters of the LSSVM to construct an optimized LSSVM classification model. Ten benchmark functions are used to evaluate the performance of the enhanced optimization algorithm. Two binary-class datasets related to geotechnical engineering, concerning seismic bumps and soil liquefaction, are then used to clarify the application of the proposed system to binary problems. Further, this investigation uses multi-class cases in civil engineering and construction management to verify the effectiveness of the model in the diagnosis of faults in steel plates, quality of water in a reservoir, and determining urban land cover. The results reveal that the system predicts faults in steel plates with an accuracy of 91.085%, the quality of water in a reservoir with an accuracy of 93.650%, and urban land cover with an accuracy of 87.274%. To demonstrate the effectiveness of the proposed system, its predictive accuracy is compared with that of a non-optimized baseline model, single multi-class classification algorithms (sequential minimal optimization (SMO), the Multiclass Classifier, the Naïve Bayes, the library support vector machine (LibSVM) and logistic regression) and prior studies. The analytical results show that the proposed system is promising project analytics software to help decision makers solve multi-level classification problems in engineering applications.

Feature Selection on Elite Hybrid Binary Cuckoo Search in Binary Label Classification.

For the low optimization accuracy of the cuckoo search algorithm, a new search algorithm, the Elite Hybrid Binary Cuckoo Search (EHBCS) algorithm, is improved by feature weighting and elite strategy. The EHBCS algorithm has been designed for feature selection on a series of binary classification datasets, including low-dimensional and high-dimensional samples by SVM classifier. The experimental results show that the EHBCS algorithm achieves better classification performances compared with binary genetic algorithm and binary particle swarm optimization algorithm. Besides, we explain its superiority in terms of standard deviation, sensitivity, specificity, precision, and F-measure.

Minimum class variance class-specific extreme learning machine for imbalanced classification

Imbalanced problems occur in real-world applications when the number of majority instances far exceeds the number of minority instances. Traditional extreme learning machine (ELM) classifier becomes biased towards the majority class due to imbalanced learning. To handle this inherent drawback, several modifications of ELM have been proposed such as weighted ELM (WELM), variances-constrained WELM (VW-ELM) to tackle the class imbalance problem effectively. One of our recent works class-specific ELM (CSELM) employs class-specific regularization and has been shown to outperform WELM for imbalanced learning. Motivated by CSELM, this work proposes a minimum class variance class-specific extreme learning machine (MCVCSELM), a variant of CSELM for tackling binary class imbalance problems more effectively. MCVCSELM uses the advantages of both the minimum class variance and the class-specific regularization. The proposed work also has lower computational complexity compared to WELM and VW-ELM. In class-specific cost regulation ELM (CCR-ELM), the calculation of the regularization parameters does not consider class distribution and class overlap. However, the performance of the CCR-ELM is comparable to ELM. MCVCSELM utilizes a class-specific regularization parameter whose value is decided by using the class proportion. The experimental results on 38 binary class datasets with different imbalanced ratios demonstrate that the proposed algorithm outperforms several state-of-the-art methods for imbalanced learning.

AdaDT: An adaptive decision tree for addressing local class imbalance based on multiple split criteria

As it is well known, decision tree is a kind of data-driven classification model, and its primary core is the split criterion. Although a great deal of split criteria have been proposed so far, almost all of them focus on the global class distribution of the training data. However, they ignored the local class imbalance problem that commonly appears during the decision tree induction over balanced or roughly balanced binary class data sets. In the present study, this problem is investigated in detail and an adaptive approach based on multiple existing split criteria is proposed. In the proposed scheme, the local class imbalanced ratio is considered as the weight factor to weigh the importance between these split criteria so as to determine the optimal splitting point at each internal node. In order to evaluate the effectiveness of the proposed method, it is applied on twenty roughly balanced real-world binary class data sets. Experimental results show that the proposed method not only outperforms all other methods, but also improves the prediction accuracy of each class.

Utilizing machine learning for detecting flowering in mid-range digital repeat photography

The responses of plants to climate change are typically reflected in the changes in leaf and flowering phenology. By exploiting the strength and simplicity of repeated digital photography and color indices, a majority of the phenological studies have been successful at investigating leaf phenology, while flowering phenology is rarely studied using the automatic capture and analysis of repeated photography. In this study, we trained and tested 5 different pretrained Convolutional Neural Network (CNN) algorithms to detect flowering events from images of white colored flowering trees and analyzed the possible factors that can affect the performance of the models. We collected images from the web and processed the images into a binary classification dataset in which a positive label indicated a tree in bloom. We also installed time-lapse cameras and captured images to validate the performances of the models in the real-world. Regarding the CNN architectures, the VGG16, ResNet50, ResNet101, MobileNet, and NASNet models were adopted, and the model weights were pretrained using the ImageNet-1000 dataset. After 20 epochs of training with 16,005 images, all of the models were successfully trained, reaching over 98% test accuracy, and 4 models reached over 99% test accuracy. All the models also showed accurate and stable performances in detecting flowering in time-series datasets with a minor inconstancy at the beginning of the flowering stages. Overall, the NASNet model showed the best performance in both the test dataset and the time-series datasets. A detailed analysis of the performance revealed that the models were especially prone to misclassify images with small relative flowering areas and were affected by the number of samples in the training dataset. We concluded that the preprocessing of the images and the size of the training dataset are essential for the high performance of the models compared to the architecture of the individual models. Furthermore, in addition to the need for a larger dataset, the proper resolution is required to successfully detect flowering from repeated photography, and most current phenological networks do not meet this condition. We suggest that mid-range photography combined with CNN algorithms can be a legitimate approach to properly accumulate and automatically process the data for studying flowering phenology.

Learning a Deep Similarity Network for Hyperspectral Image Classification

Hyperspectral image (HSI) classification is a challenging task due to subtle interclass difference and large intraclass variability, especially when the available training samples are scarce. To overcome this barrier, this article proposes a novel deep similarity network (DSN) for HSI classification, which not only ensures enough samples for training but also extracts more discriminative features. Unlike other classification methods, our essential idea is to approach the classification task by learning a new similarity measure of pixel pairs under a two-branch neural network. Specifically, a binary classification dataset with same-class and different-class pixel pairs is first constructed, which can significantly increase the number of training samples. Then, the DSN utilizes two subnetworks to extract deep features from the pixel pairs, and computes the similarity between the extracted deep features by a fusion subnetwork. Finally, the output of the DSN is used to measure the similarity to each class and the similarity determines the class label. To make full use of the spatial information, the extended multiattribute profile is incorporated to the DSN. Moreover, a joint loss function is proposed to enhance the discrimination and alleviate the challenge caused by the spatial variability of spectral signatures. Experiments on real HSI datasets verify the superiority of the DSN over several state-of-the-art methods in HSI classification. For instance, the overall accuracy of the DSN on Houston2013 dataset is 89.07%, which achieves a marked improvement of at least 4.2% over all compared methods like convolutional neural network, deep learning with attribute profiles and so on.

DTO-SMOTE: Delaunay Tessellation Oversampling for Imbalanced Data Sets

One of the significant challenges in machine learning is the classification of imbalanced data. In many situations, standard classifiers cannot learn how to distinguish minority class examples from the others. Since many real problems are unbalanced, this problem has become very relevant and deeply studied today. This paper presents a new preprocessing method based on Delaunay tessellation and the preprocessing algorithm SMOTE (Synthetic Minority Over-sampling Technique), which we call DTO-SMOTE (Delaunay Tessellation Oversampling SMOTE). DTO-SMOTE constructs a mesh of simplices (in this paper, we use tetrahedrons) for creating synthetic examples. We compare results with five preprocessing algorithms (GEOMETRIC-SMOTE, SVM-SMOTE, SMOTE-BORDERLINE-1, SMOTE-BORDERLINE-2, and SMOTE), eight classification algorithms, and 61 binary-class data sets. For some classifiers, DTO-SMOTE has higher performance than others in terms of Area Under the ROC curve (AUC), Geometric Mean (GEO), and Generalized Index of Balanced Accuracy (IBA).

Hyperparameter tuning methods in automated machine learning

In this paper, we make a comprehensive review of the challenging task of hyperparameter optimization in automated machine learning. The commonly used hyperparameter optimization methods include single-shot sampling strategies, e.g., grid search, random search and sequential strategies where new trials are gradually augmented based on existing information, including Bayesian optimization, evolutionary algorithms, reinforcement learning-based methods.We find the sequential number-theoretic optimization (SNTO) algorithm proposed by Kai-Tai Fang and Yuan Wang in 1990 can also be applied in hyperparameter optimization, where sequential uniform designs are utilized to search the global optima in complex response surfaces. For illustration, various hyperparameter optimization methods are tested with two widely-used machine learning models including the support vector machine (SVM) and extreme gradient boosting (XGBoost), on two classical binary classification datasets. By analyzing the experimental results, we find a modified SNTO algorithm is quite promising in the hyperparameter optimization task.

An ADMM Based Framework for AutoML Pipeline Configuration

We study the AutoML problem of automatically configuring machine learning pipelines by jointly selecting algorithms and their appropriate hyper-parameters for all steps in supervised learning pipelines. This black-box (gradient-free) optimization with mixed integer & continuous variables is a challenging problem. We propose a novel AutoML scheme by leveraging the alternating direction method of multipliers (ADMM). The proposed framework is able to (i) decompose the optimization problem into easier sub-problems that have a reduced number of variables and circumvent the challenge of mixed variable categories, and (ii) incorporate black-box constraints alongside the black-box optimization objective. We empirically evaluate the flexibility (in utilizing existing AutoML techniques), effectiveness (against open source AutoML toolkits), and unique capability (of executing AutoML with practically motivated black-box constraints) of our proposed scheme on a collection of binary classification data sets from UCI ML & OpenML repositories. We observe that on an average our framework provides significant gains in comparison to other AutoML frameworks (Auto-sklearn & TPOT), highlighting the practical advantages of this framework.

Reduced Dilation-Erosion Perceptron for Binary Classification

Dilation and erosion are two elementary operations from mathematical morphology, a non-linear lattice computing methodology widely used for image processing and analysis. The dilation-erosion perceptron (DEP) is a morphological neural network obtained by a convex combination of a dilation and an erosion followed by the application of a hard-limiter function for binary classification tasks. A DEP classifier can be trained using a convex-concave procedure along with the minimization of the hinge loss function. As a lattice computing model, the DEP classifier assumes the feature and class spaces are partially ordered sets. In many practical situations, however, there is no natural ordering for the feature patterns. Using concepts from multi-valued mathematical morphology, this paper introduces the reduced dilation-erosion (r-DEP) classifier. An r-DEP classifier is obtained by endowing the feature space with an appropriate reduced ordering. Such reduced ordering can be determined using two approaches: one based on an ensemble of support vector classifiers (SVCs) with different kernels and the other based on a bagging of similar SVCs trained using different samples of the training set. Using several binary classification datasets from the OpenML repository, the ensemble and bagging r-DEP classifiers yielded mean higher balanced accuracy scores than the linear, polynomial, and radial basis function (RBF) SVCs as well as their ensemble and a bagging of RBF SVCs.

Improving Classification Accuracy Using Hybrid of Extreme Learning Machine and Artificial Algae Algorithm with Multi-Light Source

Among other machine learning techniques, the extreme learning machine has evidently proved its diagnostic accuracy on many cases in medical domain. Its accuracy mainly depends on the optimal parameters that are used in training. The proposed work is based on optimizing the extreme learning machine using the recently proposed meta-heuristic optimization technique named artificial algae algorithm with multi-light source. In this work, two experiments are conducted using four binary classification datasets related to medical domain. The feasible number of hidden neurons is found from the first experiment using relevant performance parameters. In the second experiment, the classifier with feasible number of hidden neurons is further evaluated with the ten-fold cross-validation method based on its computation time and classification accuracy. In both the experiments, the proposed classifier performance compared with that of other four similar hybrid approaches. It is also statistically compared using Friedman test and Wilcoxon signed rank test based on the area under curve and accuracy values respectively. It is found that the proposed classifier produces better results than the other classifiers.

A NSGA2-LR wrapper approach for feature selection in network intrusion detection

Feature selection is becoming a major preprocessing phase in which irrelevant and redundant features are removed, while the more informative ones are retained. The datasets used in intrusion detection systems contain many features. It is, therefore, necessary to apply a feature selection step to improve the classification performance and reduce the computation time. In this paper, we propose a multi-objective feature selection approach based on NSGA-II and logistic regression in network intrusion detection. The proposed wrapper approach is tested according to two schemes: the first uses binomial logistic regression with many binary-class datasets corresponding to each type of attack, and the second uses multinomial logistic regression with a multi-class dataset. The best obtained subsets are tested using three different decision tree classifiers namely C4.5 decision tree, Random Forest, and Naive Bayes Tree. Three datasets are used in experiments namely NSL-KDD dataset, UNSW-NB15 dataset, and CIC-IDS2017. The obtained results show better accuracy when using binary-class datasets compared to multi-class datasets. Furthermore, the two schemes for feature selection succeed in reducing the features space by removing irrelevant features and keeping only the most informative ones. The obtained results are promising compared to other approaches.