Large-scale Support Vector Machines Research Articles

A novel highly efficient alternating direction method of multipliers for large-scale trimmed concave SVM

Support vector machine (SVM) is widely utilized for classification in diverse fields thanks to its superior performance. However, when tackling large-scale SVM problems, it encounters high computational complexity. To address this issue, we develop a novel trimmed concave loss SVM model called Ltri-SVM, which achieves sparsity and robustness simultaneously. The new optimality theory for the nonconvex and nonsmooth Ltri-SVM is developed by our new constructed proximal stationary point. Based on which, a new fast alternating direction method of multipliers (ADMM) with working set and low computational complexity is established for addressing Ltri-SVM. Based on our new established optimality theory, our proposed algorithm will divides the training dataset into two distinct categories: the non-working and the working sets. In every training iteration, the algorithm modifies the parameters associated with the working set, while ensuring that the parameters belonging to the non-working set remain unchanged. This algorithm enables the updating of parameters on smaller datasets, which in turn decreases computational complexity and improves the efficiency of runtime. Further, we prove that our algorithm is global convergence. Numerical experiments have confirmed the excellent performance of our algorithm in terms of accuracy in classification, number of support vector and computational speed, surpassing nine other leading solvers. For instance, when solving the real dataset with over 107 samples, our algorithm is able to complete the task in just 18.92 s, which is much faster than other solvers that take at least 650.4 s.

A New Large Scale SVM for Classification of Imbalanced Evolving Streams

Classification from imbalanced evolving streams possesses a combined challenge of class imbalance and concept drift (CI-CD). However, the state of imbalance is dynamic, a kind of virtual concept drift. The imbalanced distributions and concept drift hinder the online learner’s performance as a combined or individual problem. A weighted hybrid online oversampling approach,”weighted online oversampling large scale support vector machine (WOOLASVM),” is proposed in this work to address this combined problem. The WOOLASVM is an SVM active learning approach with new boundary weighing strategies such as (i) dynamically oversampling the current boundary and (ii) dynamic weighing of the cost parameter of the SVM objective function. Thus at any time step, WOOLASVM maintains balanced class distributions so that the CI-CD problem does not hinder the online learner performance. Over extensive experiments on synthetic and real-world streams with the static and dynamic state of imbalance, the WOOLASVM exhibits better online classification performances than other state-of-the-art methods.

A Fast Algorithm for Training Large Scale Support Vector Machines

The manuscript presents an augmented Lagrangian—fast projected gradient method (ALFPGM) with an improved scheme of working set selection, pWSS, a decomposition based algorithm for training support vector classification machines (SVM). The manuscript describes the ALFPGM algorithm, provides numerical results for training SVM on large data sets, and compares the training times of ALFPGM and Sequential Minimal Minimization algorithms (SMO) from Scikit-learn library. The numerical results demonstrate that ALFPGM with the improved working selection scheme is capable of training SVM with tens of thousands of training examples in a fraction of the training time of some widely adopted SVM tools.

Training very large scale nonlinear SVMs using Alternating Direction Method of Multipliers coupled with the Hierarchically Semi-Separable kernel approximations

• Large scale nonlinear Support Vector Machines can be trained using Alternating Direction Method of Multipliers and Hierarchically-Semi- Separable kernel approximations. • Reduced computational times can be obtained without worsening the classification accuracy. • Grid search for the misclassification parameter selection is extremely efficient in this framework. • Our efficient C++ implementation shows clear advantages over LIBSVM. Typically, nonlinear Support Vector Machines (SVMs) produce significantly higher classification quality when compared to linear ones but, at the same time, their computational complexity is prohibitive for large-scale datasets: this drawback is essentially related to the necessity to store and manipulate large, dense and unstructured kernel matrices. Despite the fact that at the core of training an SVM there is a simple convex optimization problem, the presence of kernel matrices is responsible for dramatic performance reduction, making SVMs unworkably slow for large problems. Aiming [at] an efficient solution of large-scale nonlinear SVM problems, we propose the use of the Alternating Direction Method of Multipliers coupled with Hierarchically Semi-Separable (HSS) kernel approximations. As shown in this work, the detailed analysis of the interaction among their algorithmic components unveils a particularly efficient framework and indeed, the presented experimental results demonstrate[, in the case of Radial Basis Kernels,] a significant speed-up when compared to the state-of-the-art nonlinear SVM libraries (without significantly affecting the classification accuracy).

An Efficient Algorithm for a Class of Large-Scale Support Vector Machines Exploiting Hidden Sparsity

An Efficient Algorithm for a Class of Large-Scale Support Vector Machines Exploiting Hidden Sparsity

Stochastic Subgradient for Large-Scale Support Vector Machine Using the Generalized Pinball Loss Function

In this paper, we propose a stochastic gradient descent algorithm, called stochastic gradient descent method-based generalized pinball support vector machine (SG-GPSVM), to solve data classification problems. This approach was developed by replacing the hinge loss function in the conventional support vector machine (SVM) with a generalized pinball loss function. We show that SG-GPSVM is convergent and that it approximates the conventional generalized pinball support vector machine (GPSVM). Further, the symmetric kernel method was adopted to evaluate the performance of SG-GPSVM as a nonlinear classifier. Our suggested algorithm surpasses existing methods in terms of noise insensitivity, resampling stability, and accuracy for large-scale data scenarios, according to the experimental results.

On coresets for support vector machines

We present an efficient coreset construction algorithm for large-scale Support Vector Machine (SVM) training in Big Data and streaming applications. A coreset is a small, representative subset of the original data points such that a model trained on the coreset is provably competitive with that trained on the original data set. Since the size of the coreset is generally much smaller than the original set, our preprocess-then-train scheme has potential to lead to significant speedups when training SVM models. We prove lower and upper bounds on the size of the coreset required to obtain small data summaries for the SVM problem. As a corollary, we show that our algorithm can be used to extend the applicability of any off-the-shelf SVM solver to streaming, distributed, and dynamic data settings. We evaluate the performance of our algorithm on real-world and synthetic data sets. Our experimental results reaffirm the favorable theoretical properties of our algorithm and demonstrate its practical effectiveness in accelerating SVM training.

Purging of silence for robust speaker identification in colossal database

The aim of this work is to develop an effective speaker recognition system under noisy environments for large data sets. The important phases involved in typical identification systems are feature extraction, training and testing. During the feature extraction phase, the speaker-specific information is processed based on the characteristics of the voice signal. Effective methods have been proposed for the silence removal in order to achieve accurate recognition under noisy environments in this work. Pitch and Pitch-strength parameters are extracted as distinct features from the input speech spectrum. Multi-linear principle component analysis (MPCA) is is utilized to minimize the complexity of the parameter matrix. Silence removal using zero crossing rate (ZCR) and endpoint detection algorithm (EDA) methods are applied on the source utterance during the feature extraction phase. These features are useful in later classification phase, where the identification is made on the basis of support vector machine (SVM) algorithms. Forward loking schostic (FOLOS) is the efficient large-scale SVM algorithm that has been employed for the effective classification among speakers. The evaluation findings indicate that the methods suggested increase the performance for large amounts of data in noise ecosystems.

Large-Scale Twin Parametric Support Vector Machine Using Pinball Loss Function

Traditional hinge loss function-based large-scale support vector machine (SVM) algorithms tend to perform poorly in the presence of noise, especially when the model is trained incrementally. In this paper, we propose an efficient stochastic quasi-Newton method-based twin parametric SVM using the pinball loss function (termed as SQN-PTWSVM), which is efficient and more robust to the presence of noise when compared to conventional hinge loss SVM for large-scale data scenarios. To establish the theoretical convergence of the method, a modified version of SQN-PTWSVM, termed as SQN-SPTWSVM, has also been proposed. It overcomes the poor convergence issue faced by stochastic gradient twin SVM thus resulting in a faster and reliable model. In SQN-SPTWSVM, the hyperplanes obtained are stable enough to handle noise and resampling issues that occur frequently in stochastic learning scenarios, leading to better generalization ability of the classifier. The proposed method has been extended to nonlinear scenarios as well. Moreover, batch versions of the proposed algorithms have also been introduced which significantly reduce the training time and memory requirement of SQN-PTWSVM and SQN-SPTWSVM. The experimental results on several benchmark datasets and activity recognition applications have shown that the performance of our method is better than the existing classifiers in terms of speed and accuracy.

Selection of Support Vector Candidates Using Relative Support Distance for Sustainability in Large-Scale Support Vector Machines

Support vector machines (SVMs) are a well-known classifier due to their superior classification performance. They are defined by a hyperplane, which separates two classes with the largest margin. In the computation of the hyperplane, however, it is necessary to solve a quadratic programming problem. The storage cost of a quadratic programming problem grows with the square of the number of training sample points, and the time complexity is proportional to the cube of the number in general. Thus, it is worth studying how to reduce the training time of SVMs without compromising the performance to prepare for sustainability in large-scale SVM problems. In this paper, we proposed a novel data reduction method for reducing the training time by combining decision trees and relative support distance. We applied a new concept, relative support distance, to select good support vector candidates in each partition generated by the decision trees. The selected support vector candidates improved the training speed for large-scale SVM problems. In experiments, we demonstrated that our approach significantly reduced the training time while maintaining good classification performance in comparison with existing approaches.

Inference in Turbulent Molecular Information Channels Using Support Vector Machine

Inference of transmitter side information is essential to communication. In Molecular Communication (MC), whilst the Bayesian inference of mass diffusion channel parameters is well established, turbulent diffusion (TD) channels are not well-understood. Cascading vortices rapidly transform transmitted momentum (molecular information puffs) into heat, which raises the challenge of receiver inferring transmitter information. Our initial results found that in TD channels, inferring transmitted molecular concentration or timing is challenging. As such, we were motivated to infer transmitter velocity from a flexible receiver sample area. In this paper, we consider an unbounded scenario of Molecular Communication via Turbulent Diffusion (MCvTD) where a transmitter injects several molecular puffs with different velocities. We first developed a time difference concentration (TDC) method based on large-scale support vector machine (SVM) to distinguish the injection velocities. To trade-off the prediction accuracy and number of receiver spatial samples, we propose the stepwise maximum variance (SMV) algorithm to select the limited dominant receiver sampling locations. The overall performance can achieve 100% accuracy in transmitter velocity information recovery, with excellent error vs. receiver size trade-off (e.g., 5% error for 74% area reduction). The research results indicate that velocity modulation at transmitter and TDC with SVM receiver should be used in MCvTD channels.

Optimal feasible step-size based working set selection for large scale SVMs training

Efficient training of support vector machines (SVMs) with large-scale samples is of crucial importance in the era of big data. Sequential minimal optimization (SMO) is considered as an effective solution to this challenging task, and the working set selection is one of the key steps in SMO. Various strategies have been developed and implemented for working set selection in LibSVM and Shark. In this work we point out that the algorithm used in LibSVM does not maintain the box-constraints which, nevertheless, are very important for evaluating the final gain of the selection operation. Here, we propose a new algorithm to address this challenge. The proposed algorithm maintains the box-constraints within a selection procedure using a feasible optional step-size. We systematically study and compare several related algorithms, and derive new theoretical results. Experiments on benchmark data sets show that our algorithm effectively improves the training speed without loss of accuracy.

Hyper-parameter optimization for support vector machines using stochastic gradient descent and dual coordinate descent

We developed a gradient-based method to optimize the regularization hyper-parameter, C, for support vector machines in a bilevel optimization framework. On the upper level, we optimized the hyper-parameter C to minimize the prediction loss on validation data using stochastic gradient descent. On the lower level, we used dual coordinate descent to optimize the parameters of support vector machines to minimize the loss on training data. The gradient of the loss function on the upper level with respect to the hyper-parameter, C, was computed using the implicit function theorem combined with the optimality condition of the lower-level problem, i.e., the dual problem of support vector machines. We compared our method with the existing gradient-based method in the literature on several datasets. Numerical results showed that our method converges faster to the optimal solution and achieves better prediction accuracy for large-scale support vector machine problems.

A Coordinate-Descent Primal-Dual Algorithm with Large Step Size and Possibly Nonseparable Functions

This paper introduces a coordinate descent version of the V\~u-Condat algorithm. By coordinate descent, we mean that only a subset of the coordinates of the primal and dual iterates is updated at each iteration, the other coordinates being maintained to their past value. Our method allows us to solve optimization problems with a combination of differentiable functions, constraints as well as non-separable and non-differentiable regularizers. We show that the sequences generated by our algorithm converge to a saddle point of the problem at stake, for a wider range of parameter values than previous methods. In particular, the condition on the step-sizes depends on the coordinate-wise Lipschitz constant of the differentiable function's gradient, which is a major feature allowing classical coordinate descent to perform so well when it is applicable. We then prove a sublinear rate of convergence in general and a linear rate of convergence if the objective enjoys strong convexity properties. We illustrate the performances of the algorithm on a total-variation regularized least squares regression problem and on large scale support vector machine problems.

Scaling Up Kernel SVM on Limited Resources: A Low-Rank Linearization Approach.

Kernel support vector machines (SVMs) deliver state-of-the-art results in many real-world nonlinear classification problems, but the computational cost can be quite demanding in order to maintain a large number of support vectors. Linear SVM, on the other hand, is highly scalable to large data but only suited for linearly separable problems. In this paper, we propose a novel approach called low-rank linearized SVM to scale up kernel SVM on limited resources. Our approach transforms a nonlinear SVM to a linear one via an approximate empirical kernel map computed from efficient kernel low-rank decompositions. We theoretically analyze the gap between the solutions of the approximate and optimal rank- k kernel map, which in turn provides guidance on the sampling scheme of the Nyström approximation. Furthermore, we extend it to a semisupervised metric learning scenario in which partially labeled samples can be exploited to further improve the quality of the low-rank embedding. Our approach inherits rich representability of kernel SVM and high efficiency of linear SVM. Experimental results demonstrate that our approach is more robust and achieves a better tradeoff between model representability and scalability against state-of-the-art algorithms for large-scale SVMs.

A Matter of Time: Faster Percolator Analysis via Efficient SVM Learning for Large-Scale Proteomics.

Percolator is an important tool for greatly improving the results of a database search and subsequent downstream analysis. Using support vector machines (SVMs), Percolator recalibrates peptide-spectrum matches based on the learned decision boundary between targets and decoys. To improve analysis time for large-scale data sets, we update Percolator's SVM learning engine through software and algorithmic optimizations rather than heuristic approaches that necessitate the careful study of their impact on learned parameters across different search settings and data sets. We show that by optimizing Percolator's original learning algorithm, l2-SVM-MFN, large-scale SVM learning requires nearly only a third of the original runtime. Furthermore, we show that by employing the widely used Trust Region Newton (TRON) algorithm instead of l2-SVM-MFN, large-scale Percolator SVM learning is reduced to nearly only a fifth of the original runtime. Importantly, these speedups only affect the speed at which Percolator converges to a global solution and do not alter recalibration performance. The upgraded versions of both l2-SVM-MFN and TRON are optimized within the Percolator codebase for multithreaded and single-thread use and are available under Apache license at bitbucket.org/jthalloran/percolator_upgrade .

Scalable Gaussian Kernel Support Vector Machines with Sublinear Training Time Complexity

Gaussian kernel Support Vector Machines (SVMs) deliver state-of-the-art generalization performance for non-linear classification, but the time complexity of their training process is at least quadratic w.r.t. the size of training set, preventing them from scaling up on large datasets. To address this issue, we propose a novel approach to large-scale kernel SVMs in sublinear time w.r.t. the size of training set, which combines three well-known and efficient techniques with theoretical guarantees. First, we subsample massive samples to reduce the sample size. Then, we use the random Fourier feature mapping on the subsamples to construct explicit random feature space where we can train a linear SVM to approximate the corresponding Gaussian kernel SVM. Finally, we use parallel algorithms to make our approach more scalable. Deriving the upper bounds of kernel matrix approximation error, hypothesis error and excess risk w.r.t. the size of training set and the dimension of random feature space, we establish the theoretical foundation of our proposed approach. In this way, we can reduce the time complexity of training kernel SVMs without sacrificing much accuracy. Our proposed approach achieves high accuracy and has sublinear training time complexity, which exhibits good scalability theoretically and empirically.

Dual Random Projection for Linear Support Vector Machine

Support Vector Machine (SVM) is a popular machine learning methods in the field of data analysis. The Random Projections (RP) method can solve the problem of dimensionality reduction of high-dimensional data quickly and effectively to reduce the computational cost of the related optimization problem and is widely used in the SVM method. However, the large-scale SVM has the problem of reduced classification accuracy after dimension reduction by random projection feature. Therefore, we propose a linear kernel support vector machine based on dual-random projection (drp-LSVM) for large-scale classification problem, which combines the duality recovery theory. In this paper, we analyze the geometric properties of drp-LSVM, and prove that while dividing the geometric advantage of support vector machine based on random projection (rp-LSVM), the division of hyperplane is closer to that obtained by all data training Original classifier. The experimental results show that the drp-LSVM can reduce the classification error and improve the training precision, and the performance evaluation is closer to the classifier trained with the original data while inheriting the advantages of rp-LSVM.

Uncertainty-safe large scale support vector machines

The issue of large scale binary classification when data is subject to random perturbations is addressed. The proposed model integrates a learning framework that adjusts its robustness to noise during learning. The method avoids over-conservative situations that can be encountered with worst-case robust support vector machine formulations. The algorithm could be seen as a technique to learn the support-set of the noise distribution during the training process. This is achieved by introducing optimization variables that control the magnitude of the noise perturbations that should be taken into account. The magnitude is tuned by optimizing a generalization error. Only rough estimates of perturbations bounds are required. Additionally, a stochastic bi-level optimization technique is proposed to solve the resulting formulation. The algorithm performs very cheap stochastic subgradient moves and is therefore well suited to large datasets. Encouraging experimental results show that the technique outperforms robust second order cone programming formulations.

A matrix-free smoothing algorithm for large-scale support vector machines

This paper is concerned with the iterative solution of sequences of Karush-Kuhn-Tucker (KKT) systems arising from smoothing Newton-type algorithms applied to support vector machines (SVMs). Every perturbed Newton step is crucial for the efficiency of the overall algorithm. However, the KKT systems based on smoothing reformulation become increasingly ill-conditioned as the smoothing parameter approaches to zero. By exploiting preconditioning techniques, we propose a matrix-free smoothing algorithm for solving large-scale support vector machines. Numerical results and comparisons are given to demonstrate the effectiveness and speed of the algorithm.