Two-class Problem Research Articles

Multiclass Imbalance Problems: Analysis and Potential Solutions

Class imbalance problems have drawn growing interest recently because of their classification difficulty caused by the imbalanced class distributions. In particular, many ensemble methods have been proposed to deal with such imbalance. However, most efforts so far are only focused on two-class imbalance problems. There are unsolved issues in multiclass imbalance problems, which exist in real-world applications. This paper studies the challenges posed by the multiclass imbalance problems and investigates the generalization ability of some ensemble solutions, including our recently proposed algorithm AdaBoost.NC, with the aim of handling multiclass and imbalance effectively and directly. We first study the impact of multiminority and multimajority on the performance of two basic resampling techniques. They both present strong negative effects. "Multimajority" tends to be more harmful to the generalization performance. Motivated by the results, we then apply AdaBoost.NC to several real-world multiclass imbalance tasks and compare it to other popular ensemble methods. AdaBoost.NC is shown to be better at recognizing minority class examples and balancing the performance among classes in terms of G-mean without using any class decomposition.

Evolutionary response surfaces for classification: an interpretable model

Response surfaces are powerful tools for both classification and regression because they are able to model many different phenomena and construct complex boundaries between classes. With very simple expressions, response surfaces are able to accurately solve difficult problems. Thus, the interpretability of the results is very interesting from the point of view of the expert, which is provided by a classification model from which useful information may be inferred. However, response surfaces suffer from a major problem that limits their applicability. Even with a low degree and a moderate number of features, the number of terms in the surfaces is extremely large. Thus, standard learning algorithms find many problems to efficiently obtain the coefficients of the terms, and the risk of overfitting is high. To overcome this problem we present evolutionary response surfaces for the classification of two-class problems. The use of a fitness function that combines accuracy and interpretability obtains accurate classifiers that are simple and interpretable by the expert. The results obtained for 20 problems from the UCI Machine Learning Repository are comparable with well-known classification algorithms with a more interpretable polynomial function.

Oil spill feature selection and classification using decision tree forest on SAR image data

A novel oil spill feature selection and classification technique is presented, based on a forest of decision trees. The parameters of the two-class classification problem of oil spills and look-alikes are explored. The contribution to the final classification of the 25 most commonly used features in the scientific community was examined. The work is sought in the framework of a multi-objective problem, i.e. the minimization of the used input features and, at the same time, the maximization of the overall testing classification accuracy. Results showed that the optimum forest contains 70 trees and the three most important combinations contain 4, 6 and 9 features. The latter feature combination can be seen as the most appropriate solution of the decision forest study. Examination of the robustness of the above result showed that the proposed combination achieved higher classification accuracy than other well-known statistical separation indexes. Moreover, comparisons with previous findings converge on the classification accuracy (up to 84.5%) and to the number of selected features, but diverge on the actual features. This observation leads to the conclusion that there is not a single optimum feature combination; several sets of combinations exist which contain at least some critical features.

A two-stage evolutionary algorithm based on sensitivity and accuracy for multi-class problems

The machine learning community has traditionally used correct classification rates or accuracy (C) values to measure classifier performance and has generally avoided presenting classification levels of each class in the results, especially for problems with more than two classes. C values alone are insufficient because they cannot capture the myriad of contributing factors that differentiate the performance of two different classifiers. Receiver Operating Characteristic (ROC) analysis is an alternative to solve these difficulties, but it can only be used for two-class problems. For this reason, this paper proposes a new approach for analysing classifiers based on two measures: C and sensitivity (S) (i.e., the minimum of accuracies obtained for each class). These measures are optimised through a two-stage evolutionary process. It was conducted by applying two sequential fitness functions in the evolutionary process, including entropy (E) for the first stage and a new fitness function, area (A), for the second stage. By using these fitness functions, the C level was optimised in the first stage, and the S value of the classifier was generally improved without significantly reducing C in the second stage. This two-stage approach improved S values in the generalisation set (whereas an evolutionary algorithm (EA) based only on the S measure obtains worse S levels) and obtained both high C values and good classification levels for each class. The methodology was applied to solve 16 benchmark classification problems and two complex real-world problems in analytical chemistry and predictive microbiology. It obtained promising results when compared to other competitive multi-class classification algorithms and a multi-objective alternative based on E and S.

An Audio Watermark Algorithm in Wavelet Domain Based on SVM

This paper proposed an audio watermark algorithm with synchronization in wavelet domain based on support vector machine. The synchronization code and encrypted digital watermark are embedded into the low frequency coefficients in DWT domain by using quantization modulation, the image watermark and the synchronization code are extracted through SVM possessing two-class problem. The experimental results show that the audio watermarking scheme is inaudible, but also robust against noise adding, filtering, random cropping attacks.

Extractive speech summarization using evaluation metric-related training criteria

The purpose of extractive speech summarization is to automatically select a number of indicative sentences or paragraphs (or audio segments) from the original spoken document according to a target summarization ratio and then concatenate them to form a concise summary. Much work on extractive summarization has been initiated for developing machine-learning approaches that usually cast important sentence selection as a two-class classification problem and have been applied with some success to a number of speech summarization tasks. However, the imbalanced-data problem sometimes results in a trained speech summarizer with unsatisfactory performance. Furthermore, training the summarizer by improving the associated classification accuracy does not always lead to better summarization evaluation performance. In view of such phenomena, we present in this paper an empirical investigation of the merits of two schools of training criteria to alleviate the negative effects caused by the aforementioned problems, as well as to boost the summarization performance. One is to learn the classification capability of a summarizer on the basis of the pair-wise ordering information of sentences in a training document according to a degree of importance. The other is to train the summarizer by directly maximizing the associated evaluation score or optimizing an objective that is linked to the ultimate evaluation. Experimental results on the broadcast news summarization task suggest that these training criteria can give substantial improvements over a few existing summarization methods.

Impact of class-imbalance on multi-class high-dimensional class prediction

The goal of multi-class supervised classification is to develop a rule that accurately predicts the class membership of new samples when the number of classes is larger than two. In this paper we consider high-dimensional class-imbalanced data: the number of variables greatly exceeds the number of samples and the number of samples in each class is not equal. We focus on Friedman's one-versus-one approach for three-class problems and show how its class probabilities depend on the class probabilities from the binary classification sub-problems. We further explore its performance using diagonal linear discriminant analysis (DLDA) as a base classifier and compare its performance with multi-class DLDA, using simulated and real data. Our results show that the class-imbalance has a significant effect on the classification results: the classification is biased towards the majority class as in the two-class problems and the problem is magnified when the number of variables is large. The amount of the bias depends also, jointly, on the magnitude of the differences between the classes and on the sample size: the bias diminishes when the difference between the classes is larger or the sample size is increased. Also variable selection plays an important role in the class-imbalance problem and the most effective strategy depends on the type of differences that exist between classes. DLDA seems to be among the least sensible classifiers to class-imbalance and its use is recommended also for multi-class problems. Whenever possible the experiments should be planned using balanced data in order to avoid the class-imbalance problem.

Measuring Abnormal Brains: Building Normative Rules in Neuroimaging Using One-Class Support Vector Machines

Pattern recognition methods have demonstrated to be suitable analyses tools to handle the high dimensionality of neuroimaging data. However, most studies combining neuroimaging with pattern recognition methods focus on two-class classification problems, usually aiming to discriminate patients under a specific condition (e.g., Alzheimer’s disease) from healthy controls. In this perspective paper we highlight the potential of the one-class support vector machines (OC-SVM) as an unsupervised or exploratory approach that can be used to create normative rules in a multivariate sense. In contrast with the standard SVM that finds an optimal boundary separating two classes (discriminating boundary), the OC-SVM finds the boundary enclosing a specific class (characteristic boundary). If the OC-SVM is trained with patterns of healthy control subjects, the distance to the boundary can be interpreted as an abnormality score. This score might allow quantification of symptom severity or provide insights about subgroups of patients. We provide an intuitive description of basic concepts in one-class classification, the foundations of OC-SVM, current applications, and discuss how this tool can bring new insights to neuroimaging studies.

Discrimination of plant samples using near-infrared spectroscopy with a principal component accumulation method

Discrimination is a universal problem in various fields. Near-infrared (NIR) spectroscopy combined with pattern recognition technique has shown great power and gained wide acceptance for analyzing complex samples. A principal component accumulation (PCAcc) method was proposed in our previous work for the two-class problem of cancer classification based on microarray gene expression data. In this work, PCAcc is applied to the multiclass problem of plant sample classification based on NIR spectroscopy. A hierarchical way is adopted as in the decision tree methods, each node dividing the samples into two classes. A parameter of resolution (Rs) is used to quantitatively measure the difference between the two classes. To validate the performance of the proposed method, it is applied to the NIR spectral datasets of different parts of tobacco leaves and different brands of cigarettes. The results show that the method can discriminate these samples with a very good performance in terms of classification accuracy.

Multiclass Lagrangian support vector machine

A support vector machine (SVM) has been developed for two-class problems, although its application to multiclass problems is not straightforward. This paper proposes a new Lagrangian SVM (LSVM) for application to multiclass problems. The multiclass Lagrangian SVM is formulated as a single optimization problem considering all the classes together, and a training method tailored to the multiclass problem is presented. A multiclass output representation matrix is defined to simplify the optimization formulation and associated training method. The proposed method is applied to some benchmark datasets in repository, and its effectiveness is demonstrated via simulation.

Data preprocessing techniques for classification without discrimination

Recently, the following Discrimination-Aware Classification Problem was introduced: Suppose we are given training data that exhibit unlawful discrimination; e.g., toward sensitive attributes such as gender or ethnicity. The task is to learn a classifier that optimizes accuracy, but does not have this discrimination in its predictions on test data. This problem is relevant in many settings, such as when the data are generated by a biased decision process or when the sensitive attribute serves as a proxy for unobserved features. In this paper, we concentrate on the case with only one binary sensitive attribute and a two-class classification problem. We first study the theoretically optimal trade-off between accuracy and non-discrimination for pure classifiers. Then, we look at algorithmic solutions that preprocess the data to remove discrimination before a classifier is learned. We survey and extend our existing data preprocessing techniques, being suppression of the sensitive attribute, massaging the dataset by changing class labels, and reweighing or resampling the data to remove discrimination without relabeling instances. These preprocessing techniques have been implemented in a modified version of Weka and we present the results of experiments on real-life data.

Using a Novel AdaBoost Algorithm and Chous Pseudo Amino Acid Composition for Predicting Protein Subcellular Localization

For a protein, an important characteristic is its location or compartment in a cell. This is because a protein has to be located in its proper position in a cell to perform its biological functions. Therefore, predicting protein subcellular location is an important and challenging task in current molecular and cellular biology. In this paper, based on AdaBoost.ME algorithm and Chou's PseAAC (pseudo amino acid composition), a new computational method was developed to identify protein subcellular location. AdaBoost.ME is an improved version of AdaBoost algorithm that can directly extend the original AdaBoost algorithm to deal with multi-class cases without the need to reduce it to multiple two-class problems. In some previous studies the conventional amino acid composition was applied to represent protein samples. In order to take into account the sequence order effects, in this study we use Chou's PseAAC to represent protein samples. To demonstrate that AdaBoost.ME is a robust and efficient model in predicting protein subcellular locations, the same protein dataset used by Cedano et al. (Journal of Molecular Biology, 1997, 266: 594-600) is adopted in this paper. It can be seen from the computed results that the accuracy achieved by our method is better than those by the methods developed by the previous investigators.

Multi-objective hybrid evolutionary algorithms for radial basis function neural network design

This paper presents new multi-objective evolutionary hybrid algorithms for the design of Radial Basis Function Networks (RBFNs) for classification problems. The algorithms are memetic Pareto particle swarm optimization based RBFN (MPPSON), Memetic Elitist Pareto non dominated sorting genetic algorithm based RBFN (MEPGAN) and Memetic Elitist Pareto non dominated sorting differential evolution based RBFN (MEPDEN). The proposed methods integrate accuracy and structure of RBFN simultaneously. These algorithms are implemented on two-class and multiclass pattern classification problems with one complex real problem. The results reveal that the proposed methods are viable, and provide an effective means to solve multi-objective RBFNs with good generalization ability and simple network structure. The accuracy and complexity of the network obtained by the proposed algorithms are compared through statistical tests. This study shows that the proposed methods obtain RBFNs with an appropriate balance between accuracy and simplicity.

Prostate cancer grading: Gland segmentation and structural features

In this paper, we introduce a novel approach to grade prostate malignancy using digitized histopathological specimens of the prostate tissue. Most of the approaches proposed in the literature to address this problem utilize various textural features computed from the prostate tissue image. Our approach differs in that we only focus on the tissue structure and the well-known Gleason grading system specification. The color space representing the tissue image is investigated and basic components of the prostate tissue are detected. The components and their structural relationship constitute a complete gland region. Tissue structural features extracted from gland morphology are used to classify a tissue pattern into three major categories: benign, grade 3 carcinoma and grade 4 carcinoma. Our experiments show that the proposed method outperforms a texture-based method in the three-class classification problem and most of the two-class classification problems except for the grade 3 vs grade 4 classification. Based on these results, we propose a hierarchical (binary) classification scheme which utilizes the two methods and obtains 85.6% accuracy in classifying an input tissue pattern into one of the three classes.

Fuzzy SVM with a New Fuzzy Membership Function to Solve the Two-Class Problems

In dealing with the Two-Class classification problems, the traditional support vector machine (SVM) often cannot achieve good classification accuracy when outliers exist in the training data set. The fuzzy support vector machine (FSVM) can resolve this problem with an appropriate fuzzy membership for each data point. The effect of the outliers can be effectively reduced when the classification problem is solved. In this paper, a new fuzzy membership function is employed in the linear and nonlinear fuzzy support vector machine respectively. The fuzzy membership is calculated based on the structural information of two classes in the input space and in the feature space. This method can distinguish the support vectors and the outliers effectively. Experimental results show that this approach contributes greatly to the reduction of the effect of the outliers and significantly improves the classification accuracy and generalization.

A classification and fuzzy-based approach for digital multi-focus image fusion

This paper presents a new wavelet-based method for fusion of spatially registered multi-focus images. We have formulated the image fusion process as a two-class classification problem: in and out-of-focus classes. First, a 12---dimensional feature vector using dual-tree discrete wavelet transform (DT-DWT) sub-bands of the source images are extracted, and then a trained two-class fisher classifier projects it to the class labels. The classifier output is used as a decision map for fusing high-frequency wavelet coefficients of multi-focus source images in different directions and decomposition levels of the DT-DWT. In addition, there is an uncertainty for selecting high-frequency wavelet coefficients in smooth regions of source images, which causes some misclassified pixels in the classification output or the decision map. In order to solve this uncertainty and integrate as much information as possible from the source images into the fused image, we propose an algorithm based on fuzzy logic, which combines outputs of two different fusion rules based on a dissimilarity measure from the source images: Selection based on the decision map and weighted averaging. An estimation of the decision map is also used for fusing low-frequency wavelet coefficients of the source images instead of simple averaging. After fusing low- and high-frequency wavelet coefficients of the source images, the final fused image is obtained using the inverse DT-DWT. This new method provides improved subjective and objectives results (more than 4.5 dB on average) as compared to previous fusion methods.

Design of reject rules for ECOC classification systems

ECOC is a widely used and successful technique, which implements a multi-class classification system by decomposing the original problem into several two-class problems. In this paper, we study the possibility to provide ECOC systems with a tailored reject option carried out through different schemes that can be grouped under two different categories: an external and an internal approach. The first one is based on the reliability of the entire system output and does not require any change in its structure. The second scheme, instead, estimates the reliability of the internal dichotomizers and implies a slight modification in the decoding stage. Experimental results on popular benchmark data sets are reported to show the behavior of the different schemes.

Run-time malware detection based on positive selection

This paper presents a supervised methodology that detects malware based on positive selection. Malware detection is a challenging problem due to the rapid growth of the number of malware and increasing complexity. Run-time monitoring of program execution behavior is widely used to discriminate between benign and malicious executables due to its effectiveness and robustness. This paper proposes a novel classification algorithm based on the idea of positive selection, which is one of the important algorithms in Artificial Immune Systems (AIS), inspired by positive selection of T-cells. The proposed algorithm is applied to learn and classify program behavior based on I/O Request Packets (IRP). In our experiments, the proposed algorithm outperforms ANSC, Nai ve Bayes, Bayesian Networks, Support Vector Machine, and C4.5 Decision Tree. This algorithm can also be used in general purpose classification problems not just two-class but multi-class problems.

A combined SMOTE and PSO based RBF classifier for two-class imbalanced problems

This contribution proposes a powerful technique for two-class imbalanced classification problems by combining the synthetic minority over-sampling technique (SMOTE) and the particle swarm optimisation (PSO) aided radial basis function (RBF) classifier. In order to enhance the significance of the small and specific region belonging to the positive class in the decision region, the SMOTE is applied to generate synthetic instances for the positive class to balance the training data set. Based on the over-sampled training data, the RBF classifier is constructed by applying the orthogonal forward selection procedure, in which the classifier's structure and the parameters of RBF kernels are determined using a PSO algorithm based on the criterion of minimising the leave-one-out misclassification rate. The experimental results obtained on a simulated imbalanced data set and three real imbalanced data sets are presented to demonstrate the effectiveness of our proposed algorithm.

Memetic Elitist Pareto Differential Evolution algorithm based Radial Basis Function Networks for classification problems

This paper presents a new multi-objective evolutionary hybrid algorithm for the design of Radial Basis Function Networks (RBFNs) for classification problems. The algorithm, MEPDEN, Memetic Elitist Pareto evolutionary approach based on the Non-dominated Sorting Differential Evolution (NSDE) multi-objective evolutionary algorithm which has been adapted to design RBFNs, where the NSDE algorithm is augmented with a local search that uses the Back-propagation algorithm. The MEPDEN is tested on two-class and multiclass pattern classification problems. The results obtained in terms of Mean Square Error (MSE), number of hidden nodes, accuracy (ACC), sensitivity (SEN), specificity (SPE) and Area Under the receiver operating characteristics Curve (AUC), show that the proposed approach is able to produce higher prediction accuracies with much simpler network structures. The accuracy and complexity of the network obtained by the proposed algorithm are compared with Memetic Eilitist Pareto Non-dominated Sorting Genetic Algorithm based RBFN (MEPGAN) through statistical tests. This study showed that MEPDEN obtains RBFNs with an appropriate balance between accuracy and simplicity, outperforming the other method considered.