Set Of Support Vectors Research Articles

Support vector machine incremental learning triggered by wrongly predicted samples

According to the classic Karush-Kuhn-Tucker (KKT) theorem, at every step of incremental support vector machine (SVM) learning, the newly adding sample which violates the KKT conditions will be a new support vector (SV) and migrate the old samples between SV set and non-support vector (NSV) set, and at the same time the learning model should be updated based on the SVs. However, it is not exactly clear at this moment that which of the old samples would change between SVs and NSVs. Additionally, the learning model will be unnecessarily updated, which will not greatly increase its accuracy but decrease the training speed. Therefore, how to choose the new SVs from old sets during the incremental stages and when to process incremental steps will greatly influence the accuracy and efficiency of incremental SVM learning. In this work, a new algorithm is proposed to select candidate SVs and use the wrongly predicted sample to trigger the incremental processing simultaneously. Experimental results show that the proposed algorithm can achieve good performance with high efficiency, high speed and good accuracy.

Biased support vector machine and weighted-smote in handling class imbalance problem

Class imbalance occurs when instances in a class are much higher than in other classes. This machine learning major problem can affect the predicted accuracy. Support Vector Machine (SVM) is robust and precise method in handling class imbalance problem but weak in the bias data distribution, Biased Support Vector Machine (BSVM) became popular choice to solve the problem. BSVM provide better control sensitivity yet lack accuracy compared to general SVM. This study proposes the integration of BSVM and SMOTEBoost to handle class imbalance problem. Non Support Vector (NSV) sets from negative samples and Support Vector (SV) sets from positive samples will undergo a Weighted-SMOTE process. The results indicate that implementation of Biased Support Vector Machine and Weighted-SMOTE achieve better accuracy and sensitivity.

A New Technique for Remote Sensing Image Classification Based on Combinatorial Algorithm of SVM and KNN

In remote sensing image classification, distance measurements and classification criteria are equally important; and less accuracy of either would affect classification accuracy. Remote sensing image classification was performed by combining support vector machine (SVM) and [Formula: see text]-nearest neighbor (KNN). This was based on the separability of classes using SVM and the spatial and spectral characteristics of remote sensing data. Moreover, a distance formula is proposed as the measure criterion that considers both luminance and direction of the vectors. First, the SVM is trained and the support vectors (SVs) are obtained for each class. In the testing phase, newly tested samples were entered, and average distance between the test samples and SVs for each class are calculated using the distance formula. Finally, decisions are made to classify the test samples into the class with minimal average distance. This procedure is repeated until all test samples are classified. In the combinatorial algorithm, the nearest neighbor classifier is used to classify a testing sample, i.e. the average Euclidean distance between the testing data point to each set of SVs from different categories is calculated and the nearest neighbor classifier identifies the category with minimum distance. The proposed combinatorial algorithm has advantages over the conventional KNN for eliminating the [Formula: see text] parameter selection problem and reducing heavy learning time. A comparison is carried out with SVM, KNN and Spectral Angle Mapper (SAM) classification using ALOS/PALSAR and PSM images, and the effectiveness of the proposed algorithm is demonstrated.

A fast iterative algorithm for support vector data description

Support vector data description (SVDD) is a well known model for pattern analysis when only positive examples are reliable. SVDD is usually trained by solving a quadratic programming problem, which is time consuming. This paper formulates the Lagrangian of a simply modified SVDD model as a differentiable convex function over the nonnegative orthant. The resulting minimization problem can be solved by a simple iterative algorithm. The proposed algorithm is easy to implement, without requiring any particular optimization toolbox. Theoretical and experimental analysis show that the algorithm converges r-linearly to the unique minimum point. Extensive experiments on pattern classification were conducted, and compared to the quadratic programming based SVDD (QP-SVDD), the proposed approach is much more computationally efficient (hundreds of times faster) and yields similar performance in terms of receiver operating characteristic curve. Furthermore, the proposed method and QP-SVDD extract almost the same set of support vectors.

An enhanced online LS-SVM approach for classification problems

In this paper, two novel approaches are proposed to improve the performance of online least squares support vector machine for classification problem. First, the parameters of support vector classifier model including kernel width parameter are simultaneously updated when a new sample arrives. In that model, kernel width parameter is a nonlinear term which cannot be estimated via least squares solution. Therefore, unscented Kalman filter is adopted to train all the parameters where Karush–Kuhn–Tucker conditions are satisfied. Second, a variable-size moving window, which is updated by an intelligent strategy, is proposed to construct the support vector set. Thus, the proposed model captures the dynamics of data quickly while precluding itself to become clumsy due to big amount of useless data. In addition, adaptive support vector set provides a lower computational load especially for the large data sets. Simultaneous training of the model parameters by unscented Kalman filter and intelligent update of support vector set provides a superior classification performance compared to the online support vector classification approaches in the literature.

A Novel Online LS-SVM Approach for Regression and Classification

In this paper, a novel online least squares support vector machine approach is proposed for classification and regression problems. Gaussian kernel function is used due to its strong generalization capability. The contribution of the paper is twofold. As the first novelty, all parameters of the SVM including the kernel width parameter σ are trained simultaneously when a new sample arrives. Unscented Kalman filter is adopted to train the parameters since it avoids the sub-optimal solutions caused by linearization in contrast to extended Kalman filter. The second novelty is the variable size moving window by an intelligent update strategy for the support vector set. This provides that SVM model captures the dynamics of data quickly while not letting it become clumsy due to the big amount of useless or out-of-date support vector data. Simultaneous training of the kernel parameter by unscented Kalman filter and intelligent update of support vector set provide significant performance using small amount of support vector data for both classification and system identification application results.

A Hybrid Classification Approach Based on Support Vector Machine and K-Nearest Neighbor for Remote Sensing Data

Analysis and classification for remote sensing landscape based on remote sensing imagery is a popular research topic. In this paper, we propose a new remote sensing data classifier by incorporating the support vector machine (SVM) learning information into the K-nearest neighbor (KNN) classifier. The SVM is well known for its extraordinary generalization capability even with limited learning samples, and it is very useful for remote sensing applications as data samples are usually limited. The KNN has been widely used in data classification due to its simplicity and effectiveness. However, the KNN is instance-based and needs to keep all the training samples for classification, which could cause not only high computation complexity but also overfitting problems. Meanwhile, the performance of the KNN classifier is sensitive to the neighborhood size [Formula: see text] and how to select the value of the parameter [Formula: see text] relies heavily on practice and experience. Based on the observations that the SVM can contribute to the KNN on the problems of smaller training samples size as well as the selection of the parameter [Formula: see text], we propose a support vector nearest neighbor (abbreviated as SV-NN) hybrid classification approach which can simplify the parameter selection while maintaining classification accuracy. The proposed approach is consist of two stages. In the first stage, the SVM is performed on the training samples to obtain the reduced support vectors (SVs) for each of the sample categories. In the second stage, a nearest neighbor classifier (NNC) is used to classify a testing sample, i.e. the average Euclidean distance between the testing data point to each set of SVs from different categories is calculated and the NNC identifies the category with minimum distance. To evaluate the effectiveness of the proposed approach, firstly experiments of classification for samples from remote sensing data are evaluated, and then experiments of identifying different land covers regions in the remote sensing images are evaluated. Experimental results show that the SV-NN approach maintains good classification accuracy while reduces the training samples compared with the conventional SVM and KNN classification model.

A fast algorithm for sparse support vector machines for mobile computing applications

This research presents a fast algorithm for projected support vector machines (PSVM) by selecting a basis vector set (BVS) for the kernel-induced feature space, the training points are projected onto the subspace spanned by the selected BVS. A standard linear support vector machine (SVM) is then produced in the subspace with the projected training points. As the dimension of the subspace is determined by the size of the selected basis vector set, the size of the produced SVM expansion can be specified. A two-stage algorithm is derived which selects and refines the basis vector set achieving a locally optimal model. The model expansion coefficients and bias are updated recursively for increase and decrease in the basis set and support vector set. The condition for a point to be classed as outside the current basis vector and selected as a new basis vector is derived and embedded in the recursive procedure. This guarantees the linear independence of the produced basis set. The proposed algorithm is tested and compared with an existing sparse primal SVM (SpSVM) and a standard SVM (LibSVM) on seven public benchmark classification problems. Our new algorithm is designed for use in the application area of human activity recognition using smart devices and embedded sensors where their sometimes limited memory and processing resources must be exploited to the full and the more robust and accurate the classification the more satisfied the user. Experimental results demonstrate the effectiveness and efficiency of the proposed algorithm. This work builds upon a previously published algorithm specifically created for activity recognition within mobile applications for the EU Haptimap project [1]. The algorithms detailed in this paper are more memory and resource efficient making them suitable for use with bigger data sets and more easily trained SVMs.

Incremental and Decremental SVM for Regression

Training a support vector machine (SVM) for regression (function approximation) in an incremental/decremental way consists essentially in migrating the input vectors in and out of the support vector set with specific modification of the associated thresholds. We introduce with full details such a method, which allows for defining the exact increments or decrements associated with the thresholds before vector migrations take place. Two delicate issues are especially addressed: the variation of the regularization parameter (for tuning the model performance) and the extreme situations where the support vector set becomes empty. We experimentally compare our method with several regression methods: the multilayer perceptron, two standard SVM implementations, and two models based on adaptive resonance theory.

An Efficient LS-SVM-Based Method for Fuzzy System Construction

This paper proposes an efficient learning mechanism to build fuzzy rule-based systems through the construction of sparse least-squares support vector machines (LS-SVMs). In addition to the significantly reduced computational complexity in model training, the resultant LS-SVM-based fuzzy system is sparser while offers satisfactory generalization capability over unseen data. It is well known that the LS-SVMs have their computational advantage over conventional SVMs in the model training process; however, the model sparseness is lost, which is the main drawback of LS-SVMs. This is an open problem for the LS-SVMs. To tackle the nonsparseness issue, a new regression alternative to the Lagrangian solution for the LS-SVM is first presented. A novel efficient learning mechanism is then proposed in this paper to extract a sparse set of support vectors for generating fuzzy if–then rules. This novel mechanism works in a stepwise subset selection manner, including a forward expansion phase and a backward exclusion phase in each selection step. The implementation of the algorithm is computationally very efficient due to the introduction of a few key techniques to avoid the matrix inverse operations to accelerate the training process. The computational efficiency is also confirmed by detailed computational complexity analysis. As a result, the proposed approach is not only able to achieve the sparseness of the resultant LS-SVM-based fuzzy systems but significantly reduces the amount of computational effort in model training as well. Three experimental examples are presented to demonstrate the effectiveness and efficiency of the proposed learning mechanism and the sparseness of the obtained LS-SVM-based fuzzy systems, in comparison with other SVM-based learning techniques.

Novel approaches using evolutionary computation for sparse least square support vector machines

This paper introduces two new approaches to building sparse least square support vector machines (LSSVM) based on genetic algorithms (GAs) for classification tasks. LSSVM classifiers are an alternative to SVM ones because the training process of LSSVM classifiers only requires to solve a linear equation system instead of a quadratic programming optimization problem. However, the absence of sparseness in the Lagrange multiplier vector (i.e. the solution) is a significant problem for the effective use of these classifiers. In order to overcome this lack of sparseness, we propose both single and multi-objective GA approaches to leave a few support vectors out of the solution without affecting the classifier׳s accuracy and even improving it. The main idea is to leave out outliers, non-relevant patterns or those ones which can be corrupted with noise and thus prevent classifiers to achieve higher accuracies along with a reduced set of support vectors. Differently from previous works, genetic algorithms are used in this work to obtain sparseness not to find out the optimal values of the LSSVM hyper-parameters.

Novel data condensing method using a prototype׳s front propagation algorithm

This work proposes a method for data condensing. The method is based on the selection of a generator of data prototypes. An algorithm for the front propagation of the prototypes׳ boundaries is performed in order to obtain the class boundaries given by a set of support vectors. The proposed method just has one tuning parameter and presents high classification rates even for complex topological and non-concave classes and low tendency to over-fitting. The most important advantage of the proposed method is its higher condensing rate without a significant detrimental effect on the classification rate. The algorithm is intended to be applied for condensing data in low memory devices and transmission of high-volume of data where data condensing could be crucial.

Support vector set selection using pulse-coupled neural networks

A candidate set of support vectors is selected by using pulse-coupled neural networks to reduce computational cost in learning phase for support vector machines (SVMs). The size of the candidate set of support vectors selected this way is smaller than that of the original training samples so that the computation complexity in learning process for support vectors machines based on this candidate set is reduced and the learning process is accelerated. On the other hand, the candidate set of support vectors includes almost all support vectors, and the performance of the SVM based on this candidate set matches the performance when the full training samples are used.

Feasibility and finite convergence analysis for accurate on-line ν-support vector machine.

The ν-support vector machine ( ν-SVM) for classification has the advantage of using a parameter ν on controlling the number of support vectors and margin errors. Recently, an interesting accurate on-line algorithm accurate on-line ν-SVM algorithm (AONSVM) is proposed for training ν-SVM. AONSVM can be viewed as a special case of parametric quadratic programming techniques. It is demonstrated that AONSVM avoids the infeasible updating path as far as possible, and successfully converges to the optimal solution based on experimental analysis. However, because of the differences between AONSVM and classical parametric quadratic programming techniques, there is no theoretical justification for these conclusions. In this paper, we prove the feasibility and finite convergence of AONSVM under two assumptions. The main results of feasibility analysis include: 1) the inverses of the two key matrices in AONSVM always exist; 2) the rules for updating the two key inverse matrices are reliable; 3) the variable ζ can control the adjustment of the sum of all the weights efficiently; and 4) a sample cannot migrate back and forth in successive adjustment steps among the set of margin support vectors, the set of error support vectors, and the set of the remaining vectors. Moreover, the analyses of AONSVM also provide the proofs of the feasibility and finite convergence for accurate on-line C-SVM learning directly.

Learning Prototype Hyperplanes for Face Verification in the Wild

In this paper, we propose a new scheme called Prototype Hyperplane Learning (PHL) for face verification in the wild using only weakly labeled training samples (i.e., we only know whether each pair of samples are from the same class or different classes without knowing the class label of each sample) by leveraging a large number of unlabeled samples in a generic data set. Our scheme represents each sample in the weakly labeled data set as a mid-level feature with each entry as the corresponding decision value from the classification hyperplane (referred to as the prototype hyperplane) of one Support Vector Machine (SVM) model, in which a sparse set of support vectors is selected from the unlabeled generic data set based on the learnt combination coefficients. To learn the optimal prototype hyperplanes for the extraction of mid-level features, we propose a Fisher’s Linear Discriminant-like (FLD-like) objective function by maximizing the discriminability on the weakly labeled data set with a constraint enforcing sparsity on the combination coefficients of each SVM model, which is solved by using an alternating optimization method. Then, we use the recent work called Side-Information based Linear Discriminant (SILD) analysis for dimensionality reduction and a cosine similarity measure for final face verification. Comprehensive experiments on two data sets, Labeled Faces in the Wild (LFW) and YouTube Faces, demonstrate the effectiveness of our scheme.

Lagrangian methods for approximating the viability kernel in high-dimensional systems

While a number of Lagrangian algorithms to approximate reachability in dozens or even hundreds of dimensions for systems with linear dynamics have recently appeared in the literature, no similarly scalable algorithms for approximating viable sets have been developed. In this paper we describe a connection between reachability and viability that enables us to compute the viability kernel using reach sets. This connection applies to any type of system, such as those with nonlinear dynamics and/or non-convex state constraints; however, here we take advantage of it to construct three viability kernel approximation algorithms for linear systems with convex input and state constraint sets. We compare the performance of the three algorithms and demonstrate that the two based on highly scalable Lagrangian reachability–those using ellipsoidal and support vector set representations–are able to compute the viability kernel for linear systems of larger state dimension than was previously feasible using traditional Eulerian methods. Our results are illustrated on a 6-dimensional pharmacokinetic model and a 20-dimensional model of heat conduction on a lattice.

SIMPLIFICATION OF SUPPORT VECTOR SOLUTIONS USING AN ARTIFICIAL BEE COLONY ALGORITHM

Support vector machines (SVMs) are a relatively recent machine learning technique. One of the SVM problems is that SVM is considerably slower in test phase caused by the large number of support vectors, which limits its practical use. To address this problem, we propose an artificial bee colony (ABC) algorithm to search for an optimal subset of the set of support vectors obtained through the training of the SVM, such that the original discriminant function is best approximated. Experimental results show that the proposed ABC algorithm outperforms some other compared methods in terms of the classification accuracy when the solution is reduced to the same size.

A sparse kernel algorithm for online time series data prediction

Kernel based methods have been widely applied for signal analysis and processing. In this paper, we propose a sparse kernel based algorithm for online time series prediction. In classical kernel methods, the kernel function number is very large which makes them of a high computational cost and only applicable for off-line or batch learning. In online learning settings, the learning system is updated when each training sample is obtained and it requires a higher computational speed. To make the kernel methods suitable for online learning, we propose a sparsification method based on the Hessian matrix of the system loss function to continuously examine the significance of the new training sample in order to select a sparse dictionary (support vector set). The Hessian matrix is equivalent to the correlation matrix of sample inputs in the kernel weight updating using the recursive least square (RLS) algorithm. This makes the algorithm able to be easily implemented with an affordable computational cost for real-time applications. Experimental results show the ability of the proposed algorithm for both real-world and artificial time series data forecasting and prediction.

Exploiting the Classification Performance of Support Vector Machines with Multi-Temporal Moderate-Resolution Imaging Spectroradiometer (MODIS) Data in Areas of Agreement and Disagreement of Existing Land Cover Products

Several studies have focused in the past on global land cover (LC) datasets harmonization and inter-comparison and have found significant inconsistencies. Despite the known discrepancies between existing products derived from medium resolution satellite sensor data, little emphasis has been placed on examining these disagreements to improve the overall classification accuracy of future land cover maps. This work evaluates the classification performance of a least square support vector machine (LS-SVM) algorithm with respect to areas of agreement and disagreement between two existing land cover maps. The approach involves the use of time series of Moderate-resolution Imaging Spectroradiometer (MODIS) 250-m Normalized Difference Vegetation Index (NDVI) (16-day composites) and gridded climatic indicators. LS-SVM is trained on reference samples obtained through visual interpretation of Google Earth (GE) high resolution imagery. The core of the training process is based on repeated random splits of the training dataset to select a small set of suitable support vectors optimizing class separability. A large number of independent validation samples spread over three contrasting regions in Europe (Eastern Austria, Macedonia and Southern France) are used to calculate classification accuracies for the LS-SVM NDVI-derived LC map and for two (globally available) LC products: GLC2000 and GlobCover. The LS-SVM LC map reported an overall accuracy of 70%. Classification accuracies ranged from 71% where GlobCover and GLC2000 agreed to 68% for areas of disagreement. Results indicate that existing LC products are as accurate as the LS-SVM LC map in areas of agreement (with little margin for improvements), while classification accuracy is substantially better for the LS-SVM LC map in areas of disagreement. On average, the LS-SVM LC map was 14% and 18% more accurate compared to GlobCover and GLC2000, respectively.

A sequential algorithm for sparse support vector classifiers

Support vector machines (SVMs), though accurate, are not preferred in applications requiring high classification speed or when deployed in systems of limited computational resources, due to the large number of support vectors involved in the model. To overcome this problem we have devised a primal SVM method with the following properties: (1) it solves for the SVM representation without the need to invoke the representer theorem, (2) forward and backward selections are combined to approach the final globally optimal solution, and (3) a criterion is introduced for identification of support vectors leading to a much reduced support vector set. In addition to introducing this method the paper analyzes the complexity of the algorithm and presents test results on three public benchmark problems and a human activity recognition application. These applications demonstrate the effectiveness and efficiency of the proposed algorithm.