Single Support Vector Machine Research Articles

Empirical analysis of support vector machine ensemble classifiers

Ensemble classification – combining the results of a set of base learners – has received much attention in the machine learning community and has demonstrated promising capabilities in improving classification accuracy. Compared with neural network or decision tree ensembles, there is no comprehensive empirical research in support vector machine (SVM) ensembles. To fill this void, this paper analyses and compares SVM ensembles with four different ensemble constructing techniques, namely bagging, AdaBoost, Arc-X4 and a modified AdaBoost. Twenty real-world data sets from the UCI repository are used as benchmarks to evaluate and compare the performance of these SVM ensemble classifiers by their classification accuracy. Different kernel functions and different numbers of base SVM learners are tested in the ensembles. The experimental results show that although SVM ensembles are not always better than a single SVM, the SVM bagged ensemble performs as well or better than other methods with a relatively higher generality, particularly SVMs with a polynomial kernel function. Finally, an industrial case study of gear defect detection is conducted to validate the empirical analysis results.

Multi-source land cover classification for forest fire management based on imaging spectrometry and LiDAR data

Forest fire management practices are highly dependent on the proper monitoring of the spatial distribution of the natural and man-made fuel complexes at landscape level. Spatial patterns of fuel types as well as the three-dimensional structure and state of the vegetation are essential for the assessment and prediction of forest fire risk and fire behaviour. A combination of the two remote sensing systems, imaging spectrometry and light detection and ranging (LiDAR), is well suited to map fuel types and properties, especially within the complex wildland–urban interface. LiDAR observations sample the spatial information dimension providing explicit geometric information about the structure of the Earth's surface and super-imposed objects. Imaging spectrometry on the other hand samples the spectral dimension, which is sensitive for discrimination of surface types. As a non-parametric classifier support vector machines (SVM) are particularly well adapted to classify data of high dimensionality and from multiple sources as proposed in this work. The presented approach achieves an improved land cover mapping adapted to forest fire management needs. The map is based on a single SVM classifier combining the spectral and spatial information dimensions provided by imaging spectrometry and LiDAR.

Application of distributed SVM architectures in classifying forest data cover types

In many ‘real-world’ applications, a classification of large data sets, which are often also imbalanced, is difficult due to the small, but usually more interesting classes. In this study, a large data set, forest cover type classes, which is actually multi-class classification defined with seven imbalanced classes and used as a resource inventory information was analyzed and evaluated. The data set was transformed into seven new data sets and a support vector machine (SVM) was employed to solve a binary classification problem of balanced and imbalanced data sets with various sizes. In the two approaches considered, the use of distributed SVM architectures, which basically reduces the complexity of the quadratic optimization problem of very large data sets, and the use of two sampling approaches for classification of imbalanced data sets were combined and results presented. The experimental results of distributed SVM architectures show the improvement of the accuracy for larger data sets in comparison to a single SVM classifier and their ability to improve the correct classification of the minority class.

THE LOCALIZED GENERALIZATION ERROR MODEL FOR SINGLE LAYER PERCEPTRON NEURAL NETWORK AND SIGMOID SUPPORT VECTOR MACHINE

We had developed the localized generalization error model for supervised learning with minimization of Mean Square Error. In this work, we extend the error model to Single Layer Perceptron Neural Network (SLPNN) and Support Vector Machine (SVM) with sigmoid kernel function. For a trained SLPNN or SVM and a given training dataset, the proposed error model bounds above the error for unseen samples which are similar to the training samples. As the major component of the localized generalization error model, the stochastic sensitivity measure formula for perceptron neural network derived in this work has relaxed the assumptions of same distribution for all inputs and each sample perturbed only once in previous works. These make the sensitivity measure applicable to pattern classification problems. The stochastic sensitivity measure of SVM with Sigmoid kernel is also derived in this work as a component of the localized generalization error model. At the end of this paper, we discuss the advantages of the proposed error bound over existing error bound.

ProLoc-GO: Utilizing informative Gene Ontology terms for sequence-based prediction of protein subcellular localization

BackgroundGene Ontology (GO) annotation, which describes the function of genes and gene products across species, has recently been used to predict protein subcellular and subnuclear localization. Existing GO-based prediction methods for protein subcellular localization use the known accession numbers of query proteins to obtain their annotated GO terms. An accurate prediction method for predicting subcellular localization of novel proteins without known accession numbers, using only the input sequence, is worth developing.ResultsThis study proposes an efficient sequence-based method (named ProLoc-GO) by mining informative GO terms for predicting protein subcellular localization. For each protein, BLAST is used to obtain a homology with a known accession number to the protein for retrieving the GO annotation. A large number n of all annotated GO terms that have ever appeared are then obtained from a large set of training proteins. A novel genetic algorithm based method (named GOmining) combined with a classifier of support vector machine (SVM) is proposed to simultaneously identify a small number m out of the n GO terms as input features to SVM, where m <<n. The m informative GO terms contain the essential GO terms annotating subcellular compartments such as GO:0005634 (Nucleus), GO:0005737 (Cytoplasm) and GO:0005856 (Cytoskeleton). Two existing data sets SCL12 (human protein with 12 locations) and SCL16 (Eukaryotic proteins with 16 locations) with <25% sequence identity are used to evaluate ProLoc-GO which has been implemented by using a single SVM classifier with the m = 44 and m = 60 informative GO terms, respectively. ProLoc-GO using input sequences yields test accuracies of 88.1% and 83.3% for SCL12 and SCL16, respectively, which are significantly better than the SVM-based methods, which achieve < 35% test accuracies using amino acid composition (AAC) with acid pairs and AAC with dipedtide composition. For comparison, ProLoc-GO using known accession numbers of query proteins yields test accuracies of 90.6% and 85.7%, which is also better than Hum-PLoc (85.0%) and Euk-OET-PLoc (83.7%) using ensemble classifiers with hybridization of GO terms and amphiphilic pseudo amino acid composition for SCL12 and SCL16, respectively.ConclusionThe growth of Gene Ontology in size and popularity has increased the effectiveness of GO-based features. GOmining can serve as a tool for selecting informative GO terms in solving sequence-based prediction problems. The prediction system using ProLoc-GO with input sequences of query proteins for protein subcellular localization has been implemented (see Availability).

Fusion of Support Vector Machines for Classification of Multisensor Data

The classification of multisensor data sets, consisting of multitemporal synthetic aperture radar data and optical imagery, is addressed. The concept is based on the decision fusion of different outputs. Each data source is treated separately and classified by a support vector machine (SVM). Instead of fusing the final classification outputs (i.e., land cover classes), the original outputs of each SVM discriminant function are used in the subsequent fusion process. This fusion is performed by another SVM, which is trained on the a priori outputs. In addition, two voting schemes are applied to create the final classification results. The results are compared with well-known parametric and nonparametric classifier methods, i.e., decision trees, the maximum-likelihood classifier, and classifier ensembles. The proposed SVM-based fusion approach outperforms all other approaches and significantly improves the results of a single SVM, which is trained on the whole multisensor data set.

Glycosylation site prediction using ensembles of Support Vector Machine classifiers.

BackgroundGlycosylation is one of the most complex post-translational modifications (PTMs) of proteins in eukaryotic cells. Glycosylation plays an important role in biological processes ranging from protein folding and subcellular localization, to ligand recognition and cell-cell interactions. Experimental identification of glycosylation sites is expensive and laborious. Hence, there is significant interest in the development of computational methods for reliable prediction of glycosylation sites from amino acid sequences.ResultsWe explore machine learning methods for training classifiers to predict the amino acid residues that are likely to be glycosylated using information derived from the target amino acid residue and its sequence neighbors. We compare the performance of Support Vector Machine classifiers and ensembles of Support Vector Machine classifiers trained on a dataset of experimentally determined N-linked, O-linked, and C-linked glycosylation sites extracted from O-GlycBase version 6.00, a database of 242 proteins from several different species. The results of our experiments show that the ensembles of Support Vector Machine classifiers outperform single Support Vector Machine classifiers on the problem of predicting glycosylation sites in terms of a range of standard measures for comparing the performance of classifiers. The resulting methods have been implemented in EnsembleGly, a web server for glycosylation site prediction.ConclusionEnsembles of Support Vector Machine classifiers offer an accurate and reliable approach to automated identification of putative glycosylation sites in glycoprotein sequences.

EMD- and SVM-based temperature drift modeling and compensation for a dynamically tuned gyroscope (DTG)

In this paper, support vector machine (SVM) is described and applied in the temperature drift modeling and compensation to reduce the influence of temperature variation on the output of dynamically tuned gyroscope (DTG) and to enhance its precision. To improve the modeling capability, empirical mode decomposition (EMD) is introduced into the SVM model to eliminate any impactive noises. The real temperature drift data set from the long-term measurement system of a certain DTG is employed to validate the effectiveness of the proposed combination model. The modeling and compensation results indicate that the proposed EMD-SVM model outperforms the neural network (NN) and single SVM models, and is feasible and effective in temperature drift modeling and compensation of the DTG.

Temperature drift modelling and compensation for a dynamically tuned gyroscope by combining WT and SVM method

Temperature drift is the main source of errors affecting the precision and performance of a dynamically tuned gyroscope (DTG). In this paper, the support vector machine (SVM), a novel learning machine based on statistical learning theory (SLT), is described and applied in the temperature drift modelling and compensation to reduce the influence of temperature variation on the output of the DTG and to enhance its precision. To improve the modelling and compensation capability, wavelet transform (WT) is introduced into the SVM model to eliminate any impactive noises. The real temperature drift data set from the long-term measurement system of a certain DTG is employed to validate the effectiveness of the proposed combination strategy. Moreover, the traditional neural network (NN) approach is also investigated as a comparison with the SVM based method. The modelling and compensation results indicate that the proposed WT-SVM model outperforms the NN and single SVM models, and is feasible and effective in temperature drift modelling and compensation of the DTG.

CombNET-III: A Support Vector Machine Based Large Scale Classifier with Probabilistic Framework

Several research fields have to deal with very large classification problems, e.g. handwritten character recognition and speech recognition. Many works have proposed methods to address problems with large number of samples, but few works have been done concerning problems with large numbers of classes. CombNET-II was one of the first methods proposed for such a kind of task. It consists of a sequential clustering VQ based gating network (stem network) and several Multilayer Perceptron (MLP) based expert classifiers (branch networks). With the objectives of increasing the classification accuracy and providing a more flexible model, this paper proposes a new model based on the CombNET-II structure, the CombNET-III. The new model, intended for, but not limited to, problems with large number of classes, replaces the branch networks MLP with multiclass Support Vector Machines (SVM). It also introduces a new probabilistic framework that outputs posterior class probabilities, enabling the model to be applied in different scenarios (e.g. together with Hidden Markov Models). These changes permit the use of a larger number of smaller clusters, which reduce the complexity of the final classifiers. Moreover, the use of binary SVM with probabilistic outputs and a probabilistic decoding scheme permit the use of a pairwise output encoding on the branch networks, which reduces the computational complexity of the training stage. The experimental results show that the proposed model outperforms both the previous model CombNET-II and a single multiclass SVM, while presenting considerably smaller complexity than the latter. It is also confirmed that CombNET-III classification accuracy scales better with the increasing number of clusters, in comparison with CombNET-II.

Fault diagnosis of rotating machinery based on improved wavelet package transform and SVMs ensemble

This paper presents a novel method for fault diagnosis based on an improved wavelet package transform (IWPT), a distance evaluation technique and the support vector machines (SVMs) ensemble. The method consists of three stages. Firstly, with investigating the feature of impact fault in vibration signals, a biorthogonal wavelet with impact property is constructed via lifting scheme, and the IWPT is carried out to extract salient frequency-band features from raw vibration signals. Then, the faulty features can be detected by envelope spectrum analysis of wavelet package coefficients of the most salient frequency band. Secondly, with the distance evaluation technique, the optimal features are selected from the statistical characteristics of raw signals and wavelet package coefficients, and the energy characteristics of decomposition frequency band. Finally, the optimal features are input into the SVMs ensemble with AdaBoost algorithm to identify the different abnormal cases. The proposed method is applied to the fault diagnosis of rolling element bearings, and testing results show that the SVMs ensemble can reliably separate different fault conditions and identify the severity of incipient faults, which has a better classification performance compared to the single SVMs.

INTELLIGENT DIAGNOSIS FOR INCIPIENT FAULT BASED ON LIFTING WAVELET PACKAGE TRANSFORM AND SUPPORT VECTOR MACHINES ENSEMBLE

In order to solve the problem of correctly identifying incipient fault for electromechanical equipment and improve classification ability, a novel method of incipient fault intelligent diagnosis based on lifting wavelet package transform (LWPT) and support vector machines (SVMs) ensemble, is proposed. Firstly, a biorthogonal wavelet with impact fault property is constructed via lifting scheme, and the LWPT is carried out to extract sensitive frequency-band features from orginal signals. Then the fault characteristic frequencis can be detected by envelope spectrum analysis of wavelet package coefficients (WPCs) of the most sensitive frequency-band. Secondly, with the distance evaluation technique, the optimal features are obtained from the statistical characteristics of original signals and WPCs. Finally, the optimal features are input into the SVMs ensemble to identify the different fault cases. This method was applied to incipient fault diagnosis of rolling element bearings. Testing results show that the proposed method can effectively extract the fault features, and has better classification performance than the single SVMs, with a high classification success rate.

SENSITIVITY ANALYSIS-BASED FAULT FEATURE SELECTION FOR SVM

Fault features set can be acquired by processing acquired data in machine diagnosis. Unfortunately many of these features are either partially or completely irrelevant or redundant to the fault state usually exist in the fault feature set. These features often decrease diagnosis accuracy and reduce the computational efficiency. Feature selection can remove those redundant features to avoid the influence. A new feature selection method on the basis of feature sensitivity analysis as selection evaluation is designed for support vector machine (SVM). Feature sensitivity is defined as relative feature impor- tance for the classification decision on the basis of a single SVM training run. And genetic algorithm is used for select optimization in the method. According to results of simulated data and diesel engine fault feature selection experiment, it is proved that this method possesses excellent optimization feature selection property, and acquires higher accuracy.

Identification of gas mixtures by a distributed support vector machine network and wavelet decomposition from temperature modulated semiconductor gas sensor

Semiconductor gas sensors are wildly applied in gas identification and temperature-modulated dynamic test method is frequently used to improve sensors’ selectivity and stability. This paper introduces a new strategy combing wavelet decomposition and a distributed support vector machine network to identify gas mixtures. In a previous study concerning identifying gases of a few types, we had demonstrated that wavelet decomposition outperforms FFT in feature extraction from responses of thermally modulated semiconductor gas sensors. Here, we extend this method of recognizing gas mixtures of a large type and generalize the rules of selecting the character variables from decomposed wavelet coefficients as character variables. Besides, a distributed support vector machine network is introduced to discriminate the character variables into gas patterns instead of traditional BP neural network. Experiment results show that the strategy proposed here can perform accurate discrimination between hydrogen (H2), carbon monoxide (CO), ethane (C2H4) and their mixtures (total seven patterns) over a range of 50–1000ppm, by using only one commercial metal oxide semiconductor gas sensor, and the discrimination result has advantages in restraining gas concentration variation, sensor's drift and environmental changes. Experimental result also suggests that a distributed support vector machine outperforms a single support vector machine and a BP neural network in performing gas mixture identification.

Image mining by spectral features: A case study of scenery image classification

Spectral features of images, such as Gabor filters and wavelet transform can be used for texture image classification. That is, a classifier is trained based on some labeled texture features as the training set to classify unlabeled texture features of images into some pre-defined classes. The aim of this paper is twofold. First, it investigates the classification performance of using Gabor filters, wavelet transform, and their combination respectively, as the texture feature representation of scenery images (such as mountain, castle, etc.). A k-nearest neighbor (k-NN) classifier and support vector machine (SVM) are also compared. Second, three k-NN classifiers and three SVMs are combined respectively, in which each of the combined three classifiers uses one of the above three texture feature representations respectively, to see whether combining multiple classifiers can outperform the single classifier in terms of scenery image classification. The result shows that a single SVM using Gabor filters provides the highest classification accuracy than the other two spectral features and the combined three k-NN classifiers and three SVMs.

An Experimental Bias-Variance Analysis of SVM Ensembles Based on Resampling Techniques

Recently, bias-variance decomposition of error has been used as a tool to study the behavior of learning algorithms and to develop new ensemble methods well suited to the bias-variance characteristics of base learners. We propose methods and procedures, based on Domingo's unified bias-variance theory, to evaluate and quantitatively measure the bias-variance decomposition of error in ensembles of learning machines. We apply these methods to study and compare the bias-variance characteristics of single support vector machines (SVMs) and ensembles of SVMs based on resampling techniques, and their relationships with the cardinality of the training samples. In particular, we present an experimental bias-variance analysis of bagged and random aggregated ensembles of SVMs in order to verify their theoretical variance reduction properties. The experimental bias-variance analysis quantitatively characterizes the relationships between bagging and random aggregating, and explains the reasons why ensembles built on small subsamples of the data work with large databases. Our analysis also suggests new directions for research to improve on classical bagging.

FS_SFS: A novel feature selection method for support vector machines

In many pattern recognition applications, high-dimensional feature vectors impose a high computational cost as well as the risk of “overfitting”. Feature Selection addresses the dimensionality reduction problem by determining a subset of available features which is most essential for classification. This paper presents a novel feature selection method named filtered and supported sequential forward search (FS_SFS) in the context of support vector machines (SVM). In comparison with conventional wrapper methods that employ the SFS strategy, FS_SFS has two important properties to reduce the time of computation. First, it dynamically maintains a subset of samples for the training of SVM. Because not all the available samples participate in the training process, the computational cost to obtain a single SVM classifier is decreased. Secondly, a new criterion, which takes into consideration both the discriminant ability of individual features and the correlation between them, is proposed to effectively filter out nonessential features. As a result, the total number of training is significantly reduced and the overfitting problem is alleviated. The proposed approach is tested on both synthetic and real data to demonstrate its effectiveness and efficiency.

Fast Modular Network Implementation for Support Vector Machines

Support vector machines (SVMs) have been extensively used. However, it is known that SVMs face difficulty in solving large complex problems due to the intensive computation involved in their training algorithms, which are at least quadratic with respect to the number of training examples. This paper proposes a new, simple, and efficient network architecture which consists of several SVMs each trained on a small subregion of the whole data sampling space and the same number of simple neural quantizer modules which inhibit the outputs of all the remote SVMs and only allow a single local SVM to fire (produce actual output) at any time. In principle, this region-computing based modular network method can significantly reduce the learning time of SVM algorithms without sacrificing much generalization performance. The experiments on a few real large complex benchmark problems demonstrate that our method can be significantly faster than single SVMs without losing much generalization performance.

Composite Support Vector Machines with Extended Discriminative Features for Accurate Face Detection

This paper describes a pattern classifier for detecting frontal-view faces via learning a decision boundary. The proposed classifier consists of two major parts for improving classification accuracy: the implicit modeling of both the face and the near-face classes resulting in an extended discriminative feature set, and the subsequent composite Support Vector Machines (SVMs) for speeding up the classification. For the extended discriminative feature set, Principal Component Analysis (PCA) or Independent Component Analysis (ICA) is performed for the face and near-face classes separately. The projections and distances to the two different subspaces are complementary, which significantly enhances classification accuracy of SVM. Multiple nonlinear SVMs are trained for the local facial feature spaces considering the general multi-modal characteristic of the face space. Each component SVM has a simpler boundary than that of a single SVM for the whole face space. The most appropriate component SVM is selected by a gating mechanism based on clustering. The classification by utilizing one of the multiple SVMs guarantees good generalization performance and speeds up face detection. The proposed classifier is finally implemented to work in real-time by cascading a boosting based face detector.

Granular support vector machines with association rules mining for protein homology prediction

Protein homology prediction between protein sequences is one of critical problems in computational biology. Such a complex classification problem is common in medical or biological information processing applications. How to build a model with superior generalization capability from training samples is an essential issue for mining knowledge to accurately predict/classify unseen new samples and to effectively support human experts to make correct decisions. A new learning model called granular support vector machines (GSVM) is proposed based on our previous work. GSVM systematically and formally combines the principles from statistical learning theory and granular computing theory and thus provides an interesting new mechanism to address complex classification problems. It works by building a sequence of information granules and then building support vector machines (SVM) in some of these information granules on demand. A good granulation method to find suitable granules is crucial for modeling a GSVM with good performance. In this paper, we also propose an association rules-based granulation method. For the granules induced by association rules with high enough confidence and significant support, we leave them as they are because of their high "purity" and significant effect on simplifying the classification task. For every other granule, a SVM is modeled to discriminate the corresponding data. In this way, a complex classification problem is divided into multiple smaller problems so that the learning task is simplified. The proposed algorithm, here named GSVM-AR, is compared with SVM by KDDCUP04 protein homology prediction data. The experimental results show that finding the splitting hyperplane is not a trivial task (we should be careful to select the association rules to avoid overfitting) and GSVM-AR does show significant improvement compared to building one single SVM in the whole feature space. Another advantage is that the utility of GSVM-AR is very good because it is easy to be implemented. More importantly and more interestingly, GSVM provides a new mechanism to address complex classification problems.