Multi-class Classification Method Research Articles

A comparison study on multiple binary-class SVM methods for unilabel text categorization

Multiclass support vector machine (SVM) methods are well studied in recent literature. Comparison studies on UCI/statlog multiclass datasets suggest using one-against-one method for multiclass SVM classification. However, in unilabel (multiclass) text categorization with SVMs, no comparison studies exist with one-against-one and other methods, e.g. one-against-all and several well-known improvements to these approaches. In this paper, we bridge this gap by performing empirical comparison of standard one-against-all and one-against-one, together with three improvements to these standard approaches for unilabel text categorization with SVM as base binary learner. We performed all our experiments on three standard text corpuses using two types of document representation. Outcome of our experiments partly support Rifkin and Klautau’s (2004) statement that, for small scale unilabel text categorization tasks, if parameters of the classifiers are well tuned, one-against-all will have better performance than one-against-one and other methods.

Multi-class classifier of non-speech audio based on Fisher kernel

Traditional multi-class classification methods based on Fisher kernel combine generative models such as Gaussian mixture models (GMMs) of all the classes together. However, the combination generates high dimensional feature vectors and leads to large computation. In this paper, a new classification method is proposed. This method adopts an intelligent feature space selection strategy by clustering similar Gaussian mixtures in order to reduce the feature dimensions. Audio classification experiments show that the proposed method is more accurate and effective with less computation compared with traditional methods.

A Multiclass Classification Method Based on Decoding of Binary Classifiers

In this letter, we present new methods of multiclass classification that combine multiple binary classifiers. Misclassification of each binary classifier is formulated as a bit inversion error with probabilistic models by making an analogy to the context of information transmission theory. Dependence between binary classifiers is incorporated into our model, which makes a decoder a type of Boltzmann machine. We performed experimental studies using a synthetic data set, data sets from the UCI repository, and bioinformatics data sets, and the results show that the proposed methods are superior to the existing multiclass classification methods.

New feature extraction method and its application to pattern recognition

Kernel Canonical Correlation Analysis(KCCA) is a recently addressed supervised machine learning methods,which is a powerful approach of extracting nonlinear features.However,the standard KCCA algorithm may suffer from computational problem as the training set increases.To overcome the drawback,an improved KCCA was proposed.Firstly,a scheme based on geometrical consideration was proposed to select a subset of samples that were projected to feature space(Reproducing Kernel Hilbert Space).And then,an efficient algorithm was proposed to enhance the efficiency of the feature extraction,which selected the most contributive eigenvectors for training and classification,and then calculated the corresponding eigenvectors for classification.Finally,the improved KCCA was combined with a multi-class classification method based on Support Vectors Data Description(SVDD) for classification and recognition,which put forward new ideas for pattern recognition based on kernels.The experimental results on ORL face database show that the proposed method reduces the training time and the system storage without deteriorating the recognition accuracy compared with standard KCCA.

Nonlinear Logistic Discrimination Via Regularized Gaussian Basis Expansions

We consider the problem of constructing multi-class classification methods for analyzing data with complex structure. A nonlinear logistic discriminant model is introduced based on Gaussian basis functions constructed by the self-organizing map. In order to select adjusted parameters, we employ model selection criteria derived from information-theoretic and Bayesian approaches. Numerical examples are conducted to investigate the performance of the proposed multi-class discriminant procedure. Our modeling procedure is also applied to protein structure recognition in life science. The results indicate the effectiveness of our strategy in terms of prediction accuracy.

Hierarchical Classification of ECG Beat Using Higher Order Statistics and Hermite Model

Objective: The heartbeat classification of the electrocardiogram is important in cardiac disease diagnosis. For detecting QRS complex, conventional detection algorithm have been designed to detect P, QRS, Twave, first. However, the detection of the P and T wave is difficult because their amplitudes are relatively low, and occasionally they are included in noise. Furthermore the conventional multiclass classification method may have skewed results to the majority class, because of unbalanced data distribution. Methods: The Hermite model of the higher order statistics is good characterization methods for recognizing morphological QRS complex. We applied three morphological feature extraction methods for detecting QRS complex: higher-order statistics, Hermite basis functions and Hermite model of the higher order statistics. Hierarchical scheme tackle the unbalanced data distribution problem. We also employed a hierarchical classification method using support vector machines. Results: We compared classification methods with feature extraction methods. As a result, our mean values of sensitivity for hierarchical classification method (75.47%, 76.16% and 81.21%) give better performance than the conventional multiclass classification method (46.16%). In addition, the Hermite model of the higher order statistics gave the best results compared to the higher order statistics and the Hermite basis functions in the hierarchical classification method. Conclusion: This research suggests that the Hermite model of the higher order statistics is feasible for heartbeat feature extraction. The hierarchical classification is also feasible for heartbeat classification tasks that have the unbalanced data distribution.

Better multiclass classification via a margin-optimized single binary problem

We develop a new multiclass classification method that reduces the multiclass problem to a single binary classifier (SBC). Our method constructs the binary problem by embedding smaller binary problems into a single space. A good embedding will allow for large margin classification. We show that the construction of such an embedding can be reduced to the task of learning linear combinations of kernels. We provide a bound on the generalization error of the multiclass classifier obtained with our construction and outline the conditions for its consistency. Our empirical examination of the new method indicates that it outperforms one-vs.-all, all-pairs and the error-correcting output coding scheme at least when the number of classes is small.

New gene selection method for multiclass tumor classification by class centroid

In the analysis of gene expression profiles, the selection of genetic markers and precise diagnosis of cancer type are crucial for successful treatment. The selection of discriminatory genes is critical to improve the accuracy and decrease computational complexity and cost in microarray analysis. In this paper, we developed a new statistical parameter, the suitability score to filter genes which only utilize sample distances from the class centroid. The filtered genes are employed in the nearest centroid classification to classify cancer. To evaluate the performance of the new statistical parameter, the proposed approach is applied to three publicly available microarray datasets. In this paper we demonstrate that the proposed gene selection method is steady in handling classification tasks and is a useful tool for gene selection and mining high dimension data.

A Fuzzy support vector classifier based on Bayesian optimization

In this paper, we have focused on the use of the support vector data description based on kernel-based possibilistic c-means algorithm (PCM) for solving multi-class classification problems. We propose a weighted support vector data description (SVDD) multi-class classification method, which can be used to deal with the outlier sensitivity problem in traditional multi-class classification problems. The proposed method is the robust version of SVDD by assigning a weight to each data point, which represents fuzzy membership degree of the cluster computed by the kernel-based PCM method. Accordingly, this paper presents the multi classification algorithm and gives the simple classification rule, which satisfies Bayesian optimal decision theory. With a simple classification rule, our experimental results show that the proposed method can reduce the effect of outliers and reduce the rate of classification error.

Accelerated Kernel CCA plus SVDD: A Three-stage Process for Improving Face Recognition

Kernel canonical correlation analysis (KCCA) is a recently addressed supervised machine learning methods, which shows to be a powerful approach of extracting nonlinear features for face classification and other applications. However, the standard KCCA algorithm may suffer from computational problem as the training set increase. To overcome the drawback, we propose a three-stage method to improve the performance of KCCA. Firstly, a scheme based on geometrical consideration is proposed to enhance the extraction efficiency. The algorithm can select a subset of samples whose projections in feature space (Hilbert space) are sufficient to represent all of the data in feature space. Subsequently, an improved algorithm inspired by principal component analysis (PCA) is developed. The algorithm can select the most contributive eigenvectors for training and classification instead of considering all the ones. Finally, a multi-class classification method based on support vectors data description (SVDD) is employed to further enhance the recognition performance as it can avoid the repeated use of training data. The theoretical analysis and the experiment results demonstrate the effectiveness of improvements.

Combining multiple decisions: applications to bioinformatics

Multi-class classification is one of the fundamental tasks in bioinformatics and typically arises in cancer diagnosis studies by gene expression profiling. This article reviews two recent approaches to multi-class classification by combining multiple binary classifiers, which are formulated based on a unified framework of error-correcting output coding (ECOC). The first approach is to construct a multi-class classifier in which each binary classifier to be aggregated has a weight value to be optimally tuned based on the observed data. In the second approach, misclassification of each binary classifier is formulated as a bit inversion error with a probabilistic model by making an analogy to the context of information transmission theory. Experimental studies using various real-world datasets including cancer classification problems reveal that both of the new methods are superior or comparable to other multi-class classification methods.

On the Consistency of Multiclass Classification Methods

Binary classification is a well studied special case of the classification problem. Statistical properties of binary classifiers, such as consistency, have been investigated in a variety of setting...

Support Vector Machine-based classification of protein folds using the structural properties of amino acid residues and amino acid residue pairs

Fold recognition is a key step in the protein structure discovery process, especially when traditional sequence comparison methods fail to yield convincing structural homologies. Although many methods have been developed for protein fold recognition, their accuracies remain low. This can be attributed to insufficient exploitation of fold discriminatory features. We have developed a new method for protein fold recognition using structural information of amino acid residues and amino acid residue pairs. Since protein fold recognition can be treated as a protein fold classification problem, we have developed a Support Vector Machine (SVM) based classifier approach that uses secondary structural state and solvent accessibility state frequencies of amino acids and amino acid pairs as feature vectors. Among the individual properties examined secondary structural state frequencies of amino acids gave an overall accuracy of 65.2% for fold discrimination, which is better than the accuracy by any method reported so far in the literature. Combination of secondary structural state frequencies with solvent accessibility state frequencies of amino acids and amino acid pairs further improved the fold discrimination accuracy to more than 70%, which is approximately 8% higher than the best available method. In this study we have also tested, for the first time, an all-together multi-class method known as Crammer and Singer method for protein fold classification. Our studies reveal that the three multi-class classification methods, namely one versus all, one versus one and Crammer and Singer method, yield similar predictions. Dataset and stand-alone program are available upon request.

Game theoretical analysis of the simple one-vs.-all classifier

One of the popular multi-class classification methods is to combine binary classifiers. As well as the simplest approach, a variety of methods to derive a conclusion from the results of binary classifiers can be created in diverse ways. In this paper, however, we show that the simplest approach by calculating linear combinations of binary classifiers with equal weights has a certain advantage. After introducing the ECOC approach and its extensions, we analyze the problems from a game-theoretical point of view. We show that the simplest approach has the minimax property in the one-vs.-all case.

DECISION TREE SUPPORT VECTOR MACHINE

A new multi-class classifier, decision tree SVM (DTSVM) which is a binary decision tree with a very simple structure is presented in this paper. In DTSVM, a problem of multi-class classification is decomposed into a series of ones of binary classification. Here, the binary decision tree is generated by using kernel clustering algorithm, and each non-leaf node represents one binary classification problem. By compared with the other multi-class classification methods based on the binary classification SVMs, the scale and the complexity of DTSVM are less, smaller number of support vectors are needed, and has faster test speed. The final simulation results confirm the feasibility and the validity of DTSVM.

Second-Order Cone Programming Formulations for Robust Multiclass Classification

Multiclass classification is an important and ongoing research subject in machine learning. Current support vector methods for multiclass classification implicitly assume that the parameters in the optimization problems are known exactly. However, in practice, the parameters have perturbations since they are estimated from the training data, which are usually subject to measurement noise. In this article, we propose linear and nonlinear robust formulations for multiclass classification based on the M-SVM method. The preliminary numerical experiments confirm the robustness of the proposed method.

Assigning Polarity Scores to Reviews Using Machine Learning Techniques

We propose a novel type of document classification task that quantifies how much a given document (review) appreciates the target object by using a continuous measure called sentiment polarity score (SP score) rather than binary polarity (good or bad). An SP score gives a concise summary of a review, and provides more information than binary classification. The difficulty of this task lies in the quantification of polarity. In this paper we use support vector regression (SVR) to tackle this problem. Experiments on book reviews using five-point scales show that SVR outperforms a multi-class classification method using support vector machines, and the results are close to human performance. We also examine the effect of sentence subjectivity detection using a Naive Bayes classifier, and show that this improves the robustness of the classifier.

Masking effects on linear regression in multi-class classification

The linear regression method belongs to the important class of linear methods for multi-class classification. Empirical evidences suggest that a masking problem occurs with the linear regression approach and it is especially prevalent when the number of classes is large. This paper provides an analytical study of this issue and explicitly explains why the linear discriminant analysis procedure removes this problem.

Improving multiclass pattern recognition by the combination of two strategies

We present a new method of multiclass classification based on the combination of one-vs-all method and a modification of one-vs-one method. This combination of one-vs-all and one-vs-one methods proposed enforces the strength of both methods. A study of the behavior of the two methods identifies some of the sources of their failure. The performance of a classifier can be improved if the two methods are combined in one, in such a way that the main sources of their failure are partially avoided.

Data fusion for fault diagnosis using multi-class Support Vector Machines

Multi-source multi-class classification methods based on multi-class Support Vector Machines and data fusion strategies are proposed in this paper. The centralized and distributed fusion schemes are applied to combine information from several data sources. In the centralized scheme, all information from several data sources is centralized to construct an input space. Then a multi-class Support Vector Machine classifier is trained. In the distributed schemes, the individual data sources are processed separately and modelled by using the multi-class Support Vector Machine. Then new data fusion strategies are proposed to combine the information from the individual multi-class Support Vector Machine models. Our proposed fusion strategies take into account that an Support Vector Machine (SVM) classifier achieves classification by finding the optimal classification hyperplane with maximal margin. The proposed methods are applied for fault diagnosis of a diesel engine. The experimental results showed that almost all the proposed approaches can largely improve the diagnostic accuracy. The robustness of diagnosis is also improved because of the implementation of data fusion strategies. The proposed methods can also be applied in other fields.