Solving Support Vector Machines Research Articles

Partial label learning with heterogeneous domain adaptation

Partial label learning (PLL) seeks a classification model with partially labeled (PL) instances. Each PL instance is attached with a candidate label set, with only one being ground-truth. A fundamental assumption of the previous PLL works is that there are sufficient PL instances in the learning process. Nevertheless, this assumption may not always be valid. In real-world scenarios, there may be only a few PL instances available for training. To this end, we introduce a new PLL method with heterogeneous domain adaptation (PLL-HDA). PLL-HDA leverages the knowledge from the source domain to induce a PLL classifier of the target domain, which contains only a few PL instances for training. Firstly, PLL-HDA introduces a common subspace to measure the instances from the source and target domains, where the instances from these domains are described by heterogeneous features of distinct dimensions. Secondly, we augment the features of the common subspace by merging the original features from both domains. Thirdly, a large-margin-based PLL learning system is established on the augmented features. It leads to a simplified dual form, which can be directly resolved by off-the-shelf support vector machine solvers. Lastly, a heuristic framework is proposed to resolve the PLL-HDA classification model. In this framework, the tasks of learning the classifier on the augmented features and identifying the ground-truth labels of target domain are conducted alternately. Extensive experiments on both the controlled PLL and real-world PLL datasets illustrate the superiority of PLL-HDA over the existing PLL methods. Our code is available at: https://github.com/Sux86.

A new method for handling heterogeneous data in bioinformatics

Heterogeneous data, especially a mixture of numerical and categorical data, widely exist in bioinformatics. Most of works focus on defining new distance metrics rather than learning discriminative metrics for mixed data. Here, we create a new support vector heterogeneous metric learning framework for mixed data. A heterogeneous sample pair kernel is defined for mixed data and metric learning is then converted to a sample pair classification problem. The suggested approach lends itself well to effective resolution through conventional support vector machine solvers. Empirical assessments conducted on mixed data benchmarks and cancer datasets affirm the exceptional efficacy demonstrated by the proposed modeling technique.

Fast and Exact Leave-One-Out Analysis of Large-Margin Classifiers

Motivated by the Golub–Heath–Wahba formula for ridge regression, we first present a new leave-one-out lemma for the kernel support vector machines (SVM) and related large-margin classifiers. We then use the lemma to design a novel and efficient algorithm, named “magicsvm,” for training the kernel SVM and related large-margin classifiers and computing the exact leave-one-out cross-validation error. By “magicsvm,” the computational cost of leave-one-out analysis is of the same order of fitting a single SVM on the training data. We show that “magicsvm” is much faster than the state-of-the-art SVM solvers based on extensive simulations and benchmark examples. The same idea is also used to boost the computation speed of the V-fold cross-validation of the kernel classifiers.

On coresets for support vector machines

We present an efficient coreset construction algorithm for large-scale Support Vector Machine (SVM) training in Big Data and streaming applications. A coreset is a small, representative subset of the original data points such that a model trained on the coreset is provably competitive with that trained on the original data set. Since the size of the coreset is generally much smaller than the original set, our preprocess-then-train scheme has potential to lead to significant speedups when training SVM models. We prove lower and upper bounds on the size of the coreset required to obtain small data summaries for the SVM problem. As a corollary, we show that our algorithm can be used to extend the applicability of any off-the-shelf SVM solver to streaming, distributed, and dynamic data settings. We evaluate the performance of our algorithm on real-world and synthetic data sets. Our experimental results reaffirm the favorable theoretical properties of our algorithm and demonstrate its practical effectiveness in accelerating SVM training.

Quantum algorithms for Second-Order Cone Programming and Support Vector Machines

We present a quantum interior-point method (IPM) for second-order cone programming (SOCP) that runs in time O~(nrζκδ2log⁡(1/ϵ)) where r is the rank and n the dimension of the SOCP, δ bounds the distance of intermediate solutions from the cone boundary, ζ is a parameter upper bounded by n, and κ is an upper bound on the condition number of matrices arising in the classical IPM for SOCP. The algorithm takes as its input a suitable quantum description of an arbitrary SOCP and outputs a classical description of a δ-approximate ϵ-optimal solution of the given problem.Furthermore, we perform numerical simulations to determine the values of the aforementioned parameters when solving the SOCP up to a fixed precision ϵ. We present experimental evidence that in this case our quantum algorithm exhibits a polynomial speedup over the best classical algorithms for solving general SOCPs that run in time O(nω+0.5) (here, ω is the matrix multiplication exponent, with a value of roughly 2.37 in theory, and up to 3 in practice). For the case of random SVM (support vector machine) instances of size O(n), the quantum algorithm scales as O(nk), where the exponent k is estimated to be 2.59 using a least-squares power law. On the same family random instances, the estimated scaling exponent for an external SOCP solver is 3.31 while that for a state-of-the-art SVM solver is 3.11.

F-SVM: Combination of Feature Transformation and SVM Learning via Convex Relaxation.

The generalization error bound of the support vector machine (SVM) depends on the ratio of the radius and margin. However, conventional SVM only considers the maximization of the margin but ignores the minimization of the radius, which restricts its performance when applied to joint learning of feature transformation and the SVM classifier. Although several approaches have been proposed to integrate the radius and margin information, most of them either require the form of the transformation matrix to be diagonal, or are nonconvex and computationally expensive. In this paper, we suggest a novel approximation for the radius of the minimum enclosing ball in feature space, and then propose a convex radius-margin-based SVM model for joint learning of feature transformation and the SVM classifier, i.e., F-SVM. A generalized block coordinate descent method is adopted to solve the F-SVM model, where the feature transformation is updated via the gradient descent and the classifier is updated by employing the existing SVM solver. By incorporating with kernel principal component analysis, F-SVM is further extended for joint learning of nonlinear transformation and the classifier. F-SVM can also be incorporated with deep convolutional networks to improve image classification performance. Experiments on the UCI, LFW, MNIST, CIFAR-10, CIFAR-100, and Caltech101 data sets demonstrate the effectiveness of F-SVM.

An efficient multi-feature SVM solver for complex event detection

Multimedia event detection (MED) has become one of the most important visual content analysis tools as the rapid growth of the user generated videos on the Internet. Generally, multimedia data is represented by multiple features and it is difficult to gain better performance for complex event detection with only single feature. However, how to fuse different features effectively is the crucial problem for MED with multiple features. Meanwhile, exploiting multiple features simultaneously in the large-scale scenarios always produces a heavy computational burden. To address these two issues, we propose a self-adaptive multi-feature learning framework with efficient Support Vector Machine (SVM) solver for complex event detection in this paper. Our model is able to utilize multiple features reasonably with an adaptively weighted linear combination manner, which is simple yet effective, according to the various impact that different features on a specific event. In order to mitigate the expensive computational cost, we employ a fast primal SVM solver in the proposed alternating optimization algorithm to obtain the approximate solution with gradient descent method. Extensive experiment results over standard datasets of TRECVID MEDTest 2013 and 2014 demonstrate the effectiveness and superiority of the proposed framework on complex event detection.

Distance Metric Learning via Iterated Support Vector Machines.

Distance metric learning aims to learn from the given training data a valid distance metric, with which the similarity between data samples can be more effectively evaluated for classification. Metric learning is often formulated as a convex or nonconvex optimization problem, while most existing methods are based on customized optimizers and become inefficient for large scale problems. In this paper, we formulate metric learning as a kernel classification problem with the positive semi-definite constraint, and solve it by iterated training of support vector machines (SVMs). The new formulation is easy to implement and efficient in training with the off-the-shelf SVM solvers. Two novel metric learning models, namely positive-semidefinite constrained metric learning (PCML) and nonnegative-coefficient constrained metric learning (NCML), are developed. Both PCML and NCML can guarantee the global optimality of their solutions. Experiments are conducted on general classification, face verification, and person re-identification to evaluate our methods. Compared with the state-of-the-art approaches, our methods can achieve comparable classification accuracy and are efficient in training.

Designing ensemble learning algorithms using kernel methods

Ensemble methods such as boosting combine multiple learners to obtain better prediction than could be obtained from any individual learner. Here we propose a principled framework for directly constructing ensemble learning methods from kernel methods. Unlike previous studies showing the equivalence between boosting and support vector machines (SVMs) which need a translation procedure, we show that it is possible to design boosting-like procedure to solve the SVM optimisation problems. In other words, it is possible to design ensemble methods directly from SVM without any middle procedure. This finding not only enables us to design new ensemble learning methods directly from kernel methods, but also makes it possible to take advantage of those highly-optimised fast linear SVM solvers for ensemble learning. The resulted model is as effective as kernel methods while being as efficient as ensemble methods. We exemplify this framework for designing new binary and multi-class classification ensemble learning as well as a new quantile regression ensemble learning method. Experimental results demonstrate the flexibility and usefulness of the proposed framework.

Designing ensemble learning algorithms using kernel methods

Ensemble methods such as boosting combine multiple learners to obtain better prediction than could be obtained from any individual learner. Here we propose a principled framework for directly constructing ensemble learning methods from kernel methods. Unlike previous studies showing the equivalence between boosting and support vector machines (SVMs) which need a translation procedure, we show that it is possible to design boosting-like procedure to solve the SVM optimisation problems. In other words, it is possible to design ensemble methods directly from SVM without any middle procedure. This finding not only enables us to design new ensemble learning methods directly from kernel methods, but also makes it possible to take advantage of those highly-optimised fast linear SVM solvers for ensemble learning. The resulted model is as effective as kernel methods while being as efficient as ensemble methods. We exemplify this framework for designing new binary and multi-class classification ensemble learning as well as a new quantile regression ensemble learning method. Experimental results demonstrate the flexibility and usefulness of the proposed framework.

A comparative study on large scale kernelized support vector machines

Kernelized support vector machines (SVMs) belong to the most widely used classification methods. However, in contrast to linear SVMs, the computation time required to train such a machine becomes a bottleneck when facing large data sets. In order to mitigate this shortcoming of kernel SVMs, many approximate training algorithms were developed. While most of these methods claim to be much faster than the state-of-the-art solver LIBSVM, a thorough comparative study is missing. We aim to fill this gap. We choose several well-known approximate SVM solvers and compare their performance on a number of large benchmark data sets. Our focus is to analyze the trade-off between prediction error and runtime for different learning and accuracy parameter settings. This includes simple subsampling of the data, the poor-man’s approach to handling large scale problems. We employ model-based multi-objective optimization, which allows us to tune the parameters of learning machine and solver over the full range of accuracy/runtime trade-offs. We analyze (differences between) solvers by studying and comparing the Pareto fronts formed by the two objectives classification error and training time. Unsurprisingly, given more runtime most solvers are able to find more accurate solutions, i.e., achieve a higher prediction accuracy. It turns out that LIBSVM with subsampling of the data is a strong baseline. Some solvers systematically outperform others, which allows us to give concrete recommendations of when to use which solver.

Effective face recognition using bag of features with additive kernels

In past decades, many techniques have been used to improve face recognition performance. The most common and well-studied ways are to use the whole face image to build a subspace based on the reduction of dimensionality. Differing from methods above, we consider face recognition as an image classification problem. The face images of the same person are considered to fall into the same category. Each category and each face image could be both represented by a simple pyramid histogram. Spatial dense scale-invariant feature transform features and bag of features method are used to build categories and face representations. In an effort to make the method more efficient, a linear support vector machine solver, Pegasos, is used for the classification in the kernel space with additive kernels instead of nonlinear SVMs. Our experimental results demonstrate that the proposed method can achieve very high recognition accuracy on the ORL, YALE, and FERET databases.

Multiclass Classification with Cross Entropy-Support Vector Machines

In this paper, an important sampling method – Cross Entropy method is presented to deal with solving support vector machines (SVM) problem for multiclass classification cases. The use of this method is intended to accelarate the process of finding solution without sacrificing its quality. Using one-against-rest (OAR) and one-against-one (OAO) approaches, several binary SVM classifiers are constructed and combined to solve multiclass classification problems. For each binary SVM classifier, the cross entropy method is applied to solve dual SVM problem to find the optimal or at least near optimal solution, in the feature space through kernel map. For the meantime only RBF kernel function is investigated intensively. Experiments were done on four real world data sets. The results show one-against-rest produces better results than one-against-one in terms of computing time and generalization error. In addition, applying cross entropy method on multiclass SVM produces comparable results to the standard quadratic programming SVM in terms of generalization error.

A clipping dual coordinate descent algorithm for solving support vector machines

The dual coordinate descent (DCD) algorithm solves the dual problem of support vector machine (SVM) by minimizing a series of single-variable sub-problems with a random order at inner iterations. Apparently, this DCD algorithm gives a sightless update for all variables at each iteration, which leads to a slow speed. In this paper, we present a clipping dual coordinate descent (clipDCD) algorithm for solving the dual problem of SVM. In each iteration, this clipDCD algorithm only solves one single-variable sub-problem according to the maximal possibility-decrease strategy on objective value. We can easily implement this clipDCD algorithm since it has a much simpler formulation compared with the DCD algorithm. Our experiment results indicate that, if the clipDCD algorithm is employed, SVM, twin SVM (TWSVM) and its extensions not only obtain the same classification accuracies, but also take much faster learning speeds than those classifiers employing the DCD algorithm.

Multi-granularity distance metric learning via neighborhood granule margin maximization

Learning a distance metric from training samples is often a crucial step in machine learning and pattern recognition. Locality, compactness and consistency are considered as the key principles in distance metric learning. However, the existing metric learning methods just consider one or two of them. In this paper, we develop a multi-granularity distance learning technique. First, a new index, neighborhood granule margin, which simultaneously considers locality, compactness and consistency of neighborhood, is introduced to evaluate a distance metric. By maximizing neighborhood granule margin, we formulate the distance metric learning problem as a sample pair classification problem, which can be solved by standard support vector machine solvers. Then a set of distance metrics are learned in different granular spaces. The weights of the granular spaces are learned through optimizing the margin distribution. Finally, the decisions from different granular spaces are combined with weighted voting. Experiments on UCI datasets, gender classification and object categorization tasks show that the proposed method is superior to the state-of-the-art distance metric learning algorithms.

Large-scale linear nonparallel support vector machine solver

Twin support vector machines (TWSVMs), as the representative nonparallel hyperplane classifiers, have shown the effectiveness over standard SVMs from some aspects. However, they still have one serious defect restricting their further study and real applications: they have to compute and store the inverse matrices before training, it is intractable for many applications such as that data appear with a huge number of instances as well as features. This paper proposes a Linear Nonparallel Support Vector Machine, termed as L2-TWSVM, to deal with large-scale data based on an efficient solver – dual coordinate descent (DCD) method. Both theoretical analysis and experiments indicate that our method is not only suitable for large scale problems, but also has better generalization performance than linear TWSVMs and linear SVMs.

Solving support vector machines in reproducing kernel Banach spaces with positive definite functions

In this paper we solve support vector machines in reproducing kernel Banach spaces (RKBSs) instead of the traditional methods in reproducing kernel Hilbert spaces (RKHSs). Using the orthogonality of semi-inner-products of RKBSs, we can obtain the finite-dimensional representations of the dual (normalized-duality-mapping) elements of support vector machine solutions. In addition, we can use Fourier transform techniques to introduce the concept of reproduction in a generalized native space such that it becomes a reproducing kernel Banach space, which can even be embedded into Sobolev spaces. Moreover, its reproducing kernel is associated with a positive definite function. The representations of the optimal solutions of support vector machines (regularized empirical risks) in these reproducing kernel Banach spaces are formulated explicitly and finite-dimensionally in terms of the positive definite functions, and their finite numbers of suitable parameters can be computed by the fixed point iteration. We also give some typical examples of reproducing kernel Banach spaces induced by Matérn functions (Sobolev splines) such that their support vector machine solutions even are computable efficiently. Moreover, each of their reproducing bases includes information from multiple training data points. These kernel-based algorithms give a fresh numerical tool for support vector classifiers.

Kernel-Based Learning From Both Qualitative and Quantitative Labels: Application to Prostate Cancer Diagnosis Based on Multiparametric MR Imaging

Building an accurate training database is challenging in supervised classification. For instance, in medical imaging, radiologists often delineate malignant and benign tissues without access to the histological ground truth, leading to uncertain data sets. This paper addresses the pattern classification problem arising when available target data include some uncertainty information. Target data considered here are both qualitative (a class label) or quantitative (an estimation of the posterior probability). In this context, usual discriminative methods, such as the support vector machine (SVM), fail either to learn a robust classifier or to predict accurate probability estimates. We generalize the regular SVM by introducing a new formulation of the learning problem to take into account class labels as well as class probability estimates. This original reformulation into a probabilistic SVM (P-SVM) can be efficiently solved by adapting existing flexible SVM solvers. Furthermore, this framework allows deriving a unique learned prediction function for both decision and posterior probability estimation providing qualitative and quantitative predictions. The method is first tested on synthetic data sets to evaluate its properties as compared with the classical SVM and fuzzy-SVM. It is then evaluated on a clinical data set of multiparametric prostate magnetic resonance images to assess its performances in discriminating benign from malignant tissues. P-SVM is shown to outperform classical SVM as well as the fuzzy-SVM in terms of probability predictions and classification performances, and demonstrates its potential for the design of an efficient computer-aided decision system for prostate cancer diagnosis based on multiparametric magnetic resonance (MR) imaging.

Large-scale linear nonparallel support vector machine solver

Twin support vector machines (TWSVMs), as the representative nonparallel hyperplane classifiers, have shown the effectiveness over standard SVMs from some aspects. However, they still have some serious defects restricting their further study and real applications: (1) They have to compute and store the inverse matrices before training, it is intractable for many applications where data appear with a huge number of instances as well as features; (2) TWSVMs lost the sparseness by using a quadratic loss function making the proximal hyperplane close enough to the class itself. This paper proposes a Sparse Linear Nonparallel Support Vector Machine, termed as L1-NPSVM, to deal with large-scale data based on an efficient solver—dual coordinate descent (DCD) method. Both theoretical analysis and experiments indicate that our method is not only suitable for large scale problems, but also performs as good as TWSVMs and SVMs.

Efficient HIK SVM Learning for Image Classification

Histograms are used in almost every aspect of image processing and computer vision, from visual descriptors to image representations. Histogram intersection kernel (HIK) and support vector machine (SVM) classifiers are shown to be very effective in dealing with histograms. This paper presents contributions concerning HIK SVM for image classification. First, we propose intersection coordinate descent (ICD), a deterministic and scalable HIK SVM solver. ICD is much faster than, and has similar accuracies to, general purpose SVM solvers and other fast HIK SVM training methods. We also extend ICD to the efficient training of a broader family of kernels. Second, we show an important empirical observation that ICD is not sensitive to the C parameter in SVM, and we provide some theoretical analyses to explain this observation. ICD achieves high accuracies in many problems, using its default parameters. This is an attractive property for practitioners, because many image processing tasks are too large to choose SVM parameters using cross-validation.