SVM Formulation Research Articles

Transductive polyhedral conic classifiers for machine learning applications

• We introduce new transductive learning methods that use both labeled and unlabeled data samples for learning. • The proposed methods use the polyhedral conic classification function as opposed to the linear SVM formulation. • We use the concave-convex procedure and robust loss functions to solve the resulting optimization problems. • The experimental results show that the proposed methods typically out- perform other transductive learning methods. In this paper, we introduce novel methods called Transductive Polyhedral Conic Classifiers that use both labeled and unlabeled data for classification. The proposed methodology utilizes the concave-convex procedure to solve the resulting optimization problems as in the Robust Transductive Support Vector Machines (RTSVMs). However, unlike RTSVM that uses SVM formulation, our proposed methods use the polyhedral conic classifier formulation that returns tight and closed decision boundaries compared to SVM. We tested the proposed methods on various datasets and experimental results show that our proposed methods yield better results than the existing transductive learning classifiers.

On the optimal separating hyperplane for arbitrary sets: a generalization of the SVM formulation and a convex hull approach

ABSTRACT We generalize the existing formulation and results on linear separability of sets. In order to characterize the solution of the generalized problem, we use the concepts of convex hulls. For finite sets, it is well known the Support Vector Machine technique for finding the optimal separating hyperplane. Here we consider arbitrary sets, allowing infinite, unbounded and nonclosed sets. The problem is formulated as an optimization problem with possibly infinitely many constraints. We prove existence and uniqueness of the solution. Besides, we present some examples and counterexamples to many properties discussed in the text and statements in the literature.

UKF-SVM Based Generalized Predictive Control of Multicompartment Lung Mechanics Model

In this paper, least-squares support vector machine (LS-SVM), whose parameters are updated by unscented Kalman filter (UKF), is adopted in the generalized predictive control (GPC) of a system with general multicompartment lung mechanics. Gaussian kernel function is employed since it presents a good approximation to the inner product of nonlinear mapping possessed in the SVM formulation. In the SVM literature, it is well known that the width parameter σ of the Gaussian kernel function has an important effect on the performance. However, it is not possible to train that parameter together with the other parameters of SVM when using linear least squares. This is why we use UKF for parameter adaptation in the SVM formulation. At each time instant of the control task, all parameters of the LS-SVM model, including σ, are tuned simultaneously. Another reason to employ UKF is; it avoids the suboptimal solutions caused by linearization based filters, e.g., extended Kalman filter. Due to these facts, we train the SVM model using UKF and it will be referred to as the UKF-SVM model. Simulation results concerning the application of UKF-SVM based GPC to a multicompartment lung mechanics model yields plausible performance using small amount of support vectors even when there are time-varying lung parameters and disturbance of high level affecting the system. The adopted approach can also be useful when there is not any knowledge of the system dynamics, i.e., black box. Note that, multicompartment lung mechanics system is a stand-in model that can mimic the behavior of human lung. Thus, it is appropriate for hardware-in-the-loop simulation which opens a path to the real-patient-tests of mechanical respiratory systems in the future.

Accelerated proximal incremental algorithm schemes for non-strongly convex functions

There have been a number of recent advances in accelerated gradient and proximal schemes for optimization of convex finite sum problems. Defazio introduced a simple accelerated scheme for incremental stochastic proximal algorithms inspired by gradient based methods like SAGA. He was able to prove O(1/k) convergence for non-smooth function but only under the assumption of strong convexity of component terms. We introduce a slight modification of his scheme, called MP-SAGA for which we can prove O(1/k) convergence without strong convexity, but for smooth functions. Numerical results show that our method has better or comparable convergence to Defazio's scheme, even for non-strongly convex functions. As important special cases, we also derive an accelerated schemes for a multi–class formulation of SVM as well as clustering based on the SON regularization. Finally, we introduce a simplification of Point–SAGA, called SP–SAGA for problems such as SON with large number of variables and sparse relation between variables and objective terms.

Robust nonparallel support vector machines via second-order cone programming

A novel binary classification approach is proposed in this paper, extending the ideas behind nonparallel support vector machine (NPSVM) to robust machine learning. NPSVM constructs two twin hyperplanes by solving two independent quadratic programming problems and generalizes the well-known twin support vector machine (TWSVM) method. Robustness is conferred on the NPSVM approach by using a probabilistic framework for maximizing model fit, which is cast into two second-order cone programming (SOCP) problems by assuming a worst-case setting for the data distribution of the training patterns. Experiments on benchmark datasets confirmed the theoretical virtues of our approach, showing superior average performance compared with various SVM formulations.

Linear Maximum Margin Classifier for Learning from Uncertain Data.

In this paper, we propose a maximum margin classifier that deals with uncertainty in data input. More specifically, we reformulate the SVM framework such that each training example can be modeled by a multi-dimensional Gaussian distribution described by its mean vector and its covariance matrix-the latter modeling the uncertainty. We address the classification problem and define a cost function that is the expected value of the classical SVM cost when data samples are drawn from the multi-dimensional Gaussian distributions that form the set of the training examples. Our formulation approximates the classical SVM formulation when the training examples are isotropic Gaussians with variance tending to zero. We arrive at a convex optimization problem, which we solve efficiently in the primal form using a stochastic gradient descent approach. The resulting classifier, which we name SVM with Gaussian Sample Uncertainty (SVM-GSU), is tested on synthetic data and five publicly available and popular datasets; namely, the MNIST, WDBC, DEAP, TV News Channel Commercial Detection, and TRECVID MED datasets. Experimental results verify the effectiveness of the proposed method.

Integrated framework for profit-based feature selection and SVM classification in credit scoring

In this paper, we propose a profit-driven approach for classifier construction and simultaneous variable selection based on linear Support Vector Machines. The main goal is to incorporate business-related information such as the variable acquisition costs, the Types I and II error costs, and the profit generated by correctly classified instances, into the modeling process. Our proposal incorporates a group penalty function in the SVM formulation in order to penalize the variables simultaneously that belong to the same group, assuming that companies often acquire groups of related variables for a given cost rather than acquiring them individually. The proposed framework was studied in a credit scoring problem for a Chilean bank, and led to superior performance with respect to business-related goals.

Imbalanced classification in sparse and large behaviour datasets

Recent years have witnessed a growing number of publications dealing with the imbalanced learning issue. While a plethora of techniques have been investigated on traditional low-dimensional data, little is known on the effect thereof on behaviour data. This kind of data reflects fine-grained behaviours of individuals or organisations and is characterized by sparseness and very large dimensions. In this article, we investigate the effects of several over-and undersampling, cost-sensitive learning and boosting techniques on the problem of learning from imbalanced behaviour data. Oversampling techniques show a good overall performance and do not seem to suffer from overfitting as traditional studies report. A variety of undersampling approaches are investigated as well and show the performance degrading effect of instances showing odd behaviour. Furthermore, the boosting process indicates that the regularization parameter in the SVM formulation acts as a weakness indicator and that a combination of weak learners can often achieve better generalization than a single strong learner. Finally, the EasyEnsemble technique is presented as the method outperforming all others. By randomly sampling several balanced subsets, feeding them to a boosting process and subsequently combining their hypotheses, a classifier is obtained that achieves noise/outlier reduction effects and simultaneously explores the majority class space efficiently. Furthermore, the method is very fast since it is parallelizable and each subset is only twice as large as the minority class size.

Distributed optimization of multi-class SVMs.

Training of one-vs.-rest SVMs can be parallelized over the number of classes in a straight forward way. Given enough computational resources, one-vs.-rest SVMs can thus be trained on data involving a large number of classes. The same cannot be stated, however, for the so-called all-in-one SVMs, which require solving a quadratic program of size quadratically in the number of classes. We develop distributed algorithms for two all-in-one SVM formulations (Lee et al. and Weston and Watkins) that parallelize the computation evenly over the number of classes. This allows us to compare these models to one-vs.-rest SVMs on unprecedented scale. The results indicate superior accuracy on text classification data.

Part‐based recognition of vehicle make and model

Fine-grained recognition is a challenge that the computer vision community faces nowadays. The main category of the object is known in this problem and the goal is to determine the subcategory or fine-grained category. Vehicle make and model recognition (VMMR) is a hard fine-grained classification problem, due to the large number of classes, substantial inner-class and small inter-class distance. In this study, a novel approach has been proposed for VMMR based on latent SVM formulation. This approach automatically finds a set of discriminative parts in each class of vehicles by employing a novel greedy parts localisation algorithm, while learning a model per class using both features extracted from these parts and the spatial relationship between them. An effective and practical multi-class data mining method is proposed to filter out hard negative samples in the training procedure. Employing these trained individual models together, the authors’ system can classify vehicles make and model with a high accuracy. For evaluation purposes, a new dataset including more than 5000 vehicles of 28 different makes and models has been collected and fully annotated. The experimental results on this dataset and the CompCars dataset indicate the outstanding performance of the authors’ approach.

An Improved TA-SVM Method Without Matrix Inversion and Its Fast Implementation for Nonstationary Datasets.

Recently, a time-adaptive support vector machine (TA-SVM) is proposed for handling nonstationary datasets. While attractive performance has been reported and the new classifier is distinctive in simultaneously solving several SVM subclassifiers locally and globally by using an elegant SVM formulation in an alternative kernel space, the coupling of subclassifiers brings in the computation of matrix inversion, thus resulting to suffer from high computational burden in large nonstationary dataset applications. To overcome this shortcoming, an improved TA-SVM (ITA-SVM) is proposed using a common vector shared by all the SVM subclassifiers involved. ITA-SVM not only keeps an SVM formulation, but also avoids the computation of matrix inversion. Thus, we can realize its fast version, that is, improved time-adaptive core vector machine (ITA-CVM) for large nonstationary datasets by using the CVM technique. ITA-CVM has the merit of asymptotic linear time complexity for large nonstationary datasets as well as inherits the advantage of TA-SVM. The effectiveness of the proposed classifiers ITA-SVM and ITA-CVM is also experimentally confirmed.

Interactive Phrases: Semantic Descriptions for Human Interaction Recognition.

This paper addresses the problem of recognizing human interactions from videos. We propose a novel approach that recognizes human interactions by the learned high-level descriptions, interactive phrases. Interactive phrases describe motion relationships between interacting people. These phrases naturally exploit human knowledge and allow us to construct a more descriptive model for recognizing human interactions. We propose a discriminative model to encode interactive phrases based on the latent SVM formulation. Interactive phrases are treated as latent variables and are used as mid-level features. To complement manually specified interactive phrases, we also discover data-driven phrases from data in order to find potentially useful and discriminative phrases for differentiating human interactions. An information-theoretic approach is employed to learn the data-driven phrases. The interdependencies between interactive phrases are explicitly captured in the model to deal with motion ambiguity and partial occlusion in the interactions. We evaluate our method on the BIT-Interaction data set, UT-Interaction data set, and Collective Activity data set. Experimental results show that our approach achieves superior performance over previous approaches.

New smoothing SVM algorithm with tight error bound and efficient reduced techniques

The quadratically convergent algorithms for training SVM with smoothing methods are discussed in this paper. By smoothing the objective function of an SVM formulation, Lee and Mangasarian [Comput. Optim. Appl. 20(1):5-22, 2001] presented one such algorithm called SSVM and proved that the error bound between the new smooth problem and the original one was \(O(\frac{1}{p})\) for large positive smoothing parameter p. We derive a new method by smoothing the optimality conditions of the SVM formulation, and we prove that the error bound is \(O(\frac{1}{p^{2}})\), which is better than Lee and Mangasarian’s result. Based on SMW identity and updating Hessian iteratively, some boosting skills are proposed to solve Newton equation with lower computational complexity for reduced smooth SVM algorithms. Many experimental results show that the proposed smoothing method has the same accuracy as SSVM, whose error bound is also tightened to \(O(\frac{1}{p^{2}})\) in this paper, and the proposed boosting skills are efficient for solving large-scale problems by RSVM.

Classification of Clustered Microcalcifications in Mammograms using Particle Swarm Optimization and Least-Squares Support Vector Machine

Feature selection and classifier hyper-parameter optimization are important stages of any computer-aided diagnosis (CADx) system for mammography. The optimal selection for shape features, kernel parameter, and classifier regularization constant is crucial to achieve a good generalization and performance of least-squares support vector machines (LSSVMs). This paper presents a morphology-based CADx that uses a computationally attractive and unified scheme for accomplishing the model selection task. A heuristic parameter search based on particle swarm optimization (PSO) not only reduces the dimensionality of the input feature space but also optimizes hyper-parameters of the classifier. The performance of the proposed shape-based CADx including PSO-LSSVM parameter selection method is examined using 60 microcalcification clusters. Using different cross-validation procedures, the proposed PSO-LSSVM demonstrated a good generalization ability by producing classification accuracies higher than 92%. The best classification accuracy of 97% was obtained using the leave-one-out cross-validation procedure. Comparing the performance of PSO-LSSVM with PSO-SVM method that uses conventional SVM formulation, results demonstrated the attractive computational complexity and classification performance of PSO-LSSVM. General Terms Feature selection, Pattern recognition.

Multiple instance learning of Calmodulin binding sites

Motivation: Calmodulin (CaM) is a ubiquitously conserved protein that acts as a calcium sensor, and interacts with a large number of proteins. Detection of CaM binding proteins and their interaction sites experimentally requires a significant effort, so accurate methods for their prediction are important.Results: We present a novel algorithm (MI-1 SVM) for binding site prediction and evaluate its performance on a set of CaM-binding proteins extracted from the Calmodulin Target Database. Our approach directly models the problem of binding site prediction as a large-margin classification problem, and is able to take into account uncertainty in binding site location. We show that the proposed algorithm performs better than the standard SVM formulation, and illustrate its ability to recover known CaM binding motifs. A highly accurate cascaded classification approach using the proposed binding site prediction method to predict CaM binding proteins in Arabidopsis thaliana is also presented.Availability: Matlab code for training MI-1 SVM and the cascaded classification approach is available on request.Contact: fayyazafsar@gmail.com or asa@cs.colostate.edu

Primal-Dual Instrumental Variable Estimators

This paper gives a primal-dual derivation of the Least Squares Support Vector Machine (LS-SVM) using Instrumental Variables (IVs), denoted simply as the Primal-dual Instrumental Variable Estimator. Then we propose a convex optimization approach for learning the optimal instruments. Besides the traditional argumentation for the use of IVs, the primal-dual derivation gives an interesting other advantage, namely that the complexity of the system to be solved is expressed in the number of instruments, rather than in the number of samples as typically the case for SVM and LS-SVM formulations. This note explores some exciting issues in the design and analysis of such estimator.

Using Sequential Unconstrained Minimization Techniques to simplify SVM solvers

In this paper, we apply Sequential Unconstrained Minimization Techniques (SUMTs) to the classical formulations of both the classical L1 norm SVM and the least squares SVM. We show that each can be solved as a sequence of unconstrained optimization problems with only box constraints. We propose relaxed SVM and relaxed LSSVM formulations that correspond to a single problem in the corresponding SUMT sequence. We also propose a SMO like algorithm to solve the relaxed formulations that works by updating individual Lagrange multipliers. The methods yield comparable or better results on large benchmark datasets than classical SVM and LSSVM formulations, at substantially higher speeds.

Efficient and Effective Visual Codebook Generation Using Additive Kernels

Common visual codebook generation methods used in a bag of visual words model, for example, k-means or Gaussian Mixture Model, use the Euclidean distance to cluster features into visual code words. However, most popular visual descriptors are histograms of image measurements. It has been shown that with histogram features, the Histogram Intersection Kernel (HIK) is more effective than the Euclidean distance in supervised learning tasks. In this paper, we demonstrate that HIK can be used in an unsupervised manner to significantly improve the generation of visual codebooks. We propose a histogram kernel k-means algorithm which is easy to implement and runs almost as fast as the standard k-means. The HIK codebooks have consistently higher recognition accuracy over k-means codebooks by 2-4% in several benchmark object and scene recognition data sets. The algorithm is also generalized to arbitrary additive kernels. Its speed is thousands of times faster than a naive implementation of the kernel k-means algorithm. In addition, we propose a one-class SVM formulation to create more effective visual code words. Finally, we show that the standard k-median clustering method can be used for visual codebook generation and can act as a compromise between the HIK / additive kernel and the k-means approaches.

A new weighted approach to imbalanced data classification problem via support vector machine with quadratic cost function

In this paper, a new weighted approach on Lagrangian support vector machine for imbalanced data classification problem is proposed. The weight parameters are embedded in the Lagrangian SVM formulation. The training method for weighted Lagrangian SVM is presented and its convergence is proven. The weighted Lagrangian SVM classifier is tested and compared with some other SVMs using synthetic and real data to show its effectiveness and feasibility.

Knowledge based Least Squares Twin support vector machines

We propose knowledge based versions of a relatively new family of SVM algorithms based on two non-parallel hyperplanes. Specifically, we consider prior knowledge in the form of multiple polyhedral sets and incorporate the same into the formulation of linear Twin SVM (TWSVM)/Least Squares Twin SVM (LSTWSVM) and term them as knowledge based TWSVM (KBTWSVM)/knowledge based LSTWSVM (KBLSTWSVM). Both of these formulations are capable of generating non-parallel hyperplanes based on real-world data and prior knowledge. We derive the solution of KBLSTWSVM and use it in our computational experiments for comparison against other linear knowledge based SVM formulations. Our experiments show that KBLSTWSVM is a versatile classifier whose solution is extremely simple when compared with other linear knowledge based SVM algorithms.