Linear Programming Support Vector Machines Research Articles

A Mixed Integer Linear Programming Support Vector Machine for Cost-Effective Group Feature Selection: Branch-Cut-and-Price Approach

Recently, cost-based feature selection has received significant attention due to its great ability to achieve promising prediction accuracy at a minimum feature acquisition cost. To further improve its predictive and economic performances, this research proposes a cost-effective 1-norm support vector machine with group feature selection as GFS-CESVM1. Its robust counterpart model, GFS-RCESVM1, is also introduced to address the cost uncertainty of features and feature groups because cost variation commonly exists in real-world problems. The proposed models are formulated as Mixed Integer Linear Programming (MILP). To efficiently solve the proposed SVM MILP models, we develop a Branch-Cut-and-Price (BCP) algorithm that considers only a limited number of variables and/or constraints, which thereby leads to rapid convergence to an optimal solution. Various experimental results on benchmark and synthetic datasets demonstrate that GFS-CESVM1 can achieve competitive outcomes by considering not only individual feature evaluation but also group structural information among features. The GFS-RCESVM1 can identify the subset of features that is immune to cost uncertainty and therefore provide feasible and optimal solutions. Furthermore, our BCP algorithm can dominantly outperform the ordinary BB algorithm for finding better objective value and integrality gap within a short period of time.

Universal Arbitrage-Free Estimation of State Price Density

Given the valuable information content of Arrow-Debreu prices, the recovery of a well-behaved state price density is of considerable importance. However, this is a nontrivial task because of data limitation and the complex arbitrage-free constraints. In this article, we develop a more effective linear programming support vector machine estimator for state price density, which incorporates no-arbitrage restrictions and bid-ask spread. This method does not depend on a particular approximation function and framework and is, therefore, universally applicable. In a parallel empirical study, we apply the method to options on the S&P 500, showing it to be accurate and smooth. TOPICS:Derivatives, options Key Findings ▪ Recovery of a well-behaved state price density is an important but nontrivial because of data limitation and the complex arbitrage-free constraints. ▪ We develop a universally applicable linear programming support vector machine estimator for state price density that incorporates no-arbitrage restrictions and bid-ask spread. ▪ We apply the method empirically to options on the S&P 500, showing it to be accurate and smooth.

Imbalanced Learning with Parametric Linear Programming Support Vector Machine for Weather Data Application

Imbalanced learning is a challenging task in predictive modeling and machine learning that has inspired many researchers to attempt to improve the existing algorithms for more accurate predictions of imbalanced data sets. Due to the nature of rare events, developing reliable and efficient classification models for imbalanced data has not been easily accomplished, and over the past two decades, various methods have been proposed. To this end, we propose a Linear Programming Support Vector Machine (LP-SVM) model to address the issue of imbalanced learning in weather applications. To further improve the model’s predictive accuracy, we have implemented a parameter selection method based on the multi-objective parametric simplex approach for parameter tuning of LP-SVM. For numerical tests, we have used a real data set consisting of weather observations made by the Bureau of Meteorology’s (BM) system in Australia. The results obtained from training and testing the model demonstrate the effectiveness of our proposed model on the tested examples.

A mixed integer linear programming support vector machine for cost-effective feature selection

In the era of big data, feature selection is indispensable as a dimensional reduction technique to lower data complexity and enhance machine learning performances. However, traditional feature selection methods mainly focus on classification performances, while they exclude the impact of associated feature costs; e.g., price, risk, and computational complexity for feature acquisition. In this research, we extend the ℓ1 norm support vector machine (ℓ1-SVM) to address the feature costs, by incorporating a budget constraint to preserve classification accuracy with the least expensive features. Furthermore, we formulate its robust counterpart to address the uncertainty of the feature costs. To enhance computational efficiency, we also develop an algorithm to tighten the bound of the weight vector in the budget constraint. Through the experimental study on a variety of benchmark and synthetic datasets, our proposed mixed integer linear programming (MILP) models show that they can achieve competitive outcomes in terms of predictive and economic performances. Also, the algorithm that tightens the budget constraint helps to curtail computational complexity.

Coordinate Descent Algorithm for Ramp Loss Linear Programming Support Vector Machines

In order to control the effects of outliers in training data and get sparse results, Huang et al. (J Mach Learn Res 15:2185---2211, 2014) proposed the ramp loss linear programming support vector machine. This combination of $$\mathrm {l}_1$$l1 regularization and ramp loss does not only lead to the sparsity of parameters in decision functions, but also limits the effects of outliers with a maximal penalty. However, due to its non-convexity, the computational cost to achieve a satisfying solution is often expensive. In this paper, we propose a modified coordinate descent algorithm, which deals with a series of one-variable piecewise linear subproblems. Considering that the obtained subproblems are DC programming problems, we linearize the concave part of the objective functions and solve the obtained convex problems. To test the performances of the proposed algorithm, numerical experiments have been carried out and analysed on benchmark data sets. To enhance the sparsity and robustness, the experiments are initialized from C-SVM solutions. The results confirm its excellent performances in classification accuracy, robustness and efficiency in computation.

Ramp loss linear programming support vector machine

The ramp loss is a robust but non-convex loss for classification. Compared with other non-convex losses, a local minimum of the ramp loss can be effectively found. The effectiveness of local search comes from the piecewise linearity of the ramp loss. Motivated by the fact that the l1-penalty is piecewise linear as well, the l1-penalty is applied for the ramp loss, resulting in a ramp loss linear programming support vector machine (ramp-LPSVM). The proposed ramp-LPSVM is a piecewise linear minimization problem and the related optimization techniques are applicable. Moreover, the l1-penalty can enhance the sparsity. In this paper, the corresponding misclassification error and convergence behavior are discussed. Generally, the ramp loss is a truncated hinge loss. Therefore ramp-LPSVM possesses some similar properties as hinge loss SVMs. A local minimization algorithm and a global search strategy are discussed. The good optimization capability of the proposed algorithms makes ramp-LPSVM perform well in numerical experiments: the result of ramp-LPSVM is more robust than that of hinge SVMs and is sparser than that of ramp-SVM, which consists of the || ċ || k-penalty and the ramp loss.

Minimizing inter-subject variability in fNIRS-based brain–computer interfaces via multiple-kernel support vector learning

Brain signal variation across different subjects and sessions significantly impairs the accuracy of most brain–computer interface (BCI) systems. Herein, we present a classification algorithm that minimizes such variation, using linear programming support-vector machines (LP-SVM) and their extension to multiple kernel learning methods. The minimization is based on the decision boundaries formed in classifiers’ feature spaces and their relation to BCI variation. Specifically, we estimate subject/session-invariant features in the reproducing kernel Hilbert spaces (RKHS) induced with Gaussian kernels. The idea is to construct multiple subject/session-dependent RKHS and to perform classification with LP-SVMs. To evaluate the performance of the algorithm, we applied it to oxy-hemoglobin data sets acquired from eight sessions and seven subjects as they performed two different mental tasks. Results show that our classifiers maintain good performance when applied to random patterns across varying sessions/subjects.

A fast algorithm for kernel 1-norm support vector machines

This paper presents a fast algorithm called Column Generation Newton (CGN) for kernel 1-norm support vector machines (SVMs). CGN combines the Column Generation (CG) algorithm and the Newton Linear Programming SVM (NLPSVM) method. NLPSVM was proposed for solving 1-norm SVM, and CG is frequently used in large-scale integer and linear programming algorithms. In each iteration of the kernel 1-norm SVM, NLPSVM has a time complexity of O(ℓ3), where ℓ is the sample number, and CG has a time complexity between O(ℓ3) and O(n′3), where n′ is the number of columns of the coefficient matrix in the subproblem. CGN uses CG to generate a sequence of subproblems containing only active constraints and then NLPSVM to solve each subproblem. Since the subproblem in each iteration only consists of n′ unbound constraints, CGN thus has a time complexity of O(n′3), which is smaller than that of NLPSVM and CG. Also, CGN is faster than CG when the solution to 1-norm SVM is sparse. A theorem is given to show a finite step convergence of CGN. Experimental results on the Ringnorm and UCI data sets demonstrate the efficiency of CGN to solve the kernel 1-norm SVM.

Performance analysis of support vector machines classifiers in breast cancer mammography recognition

Support vector machine (SVM) is a supervised machine learning approach that was recognized as a statistical learning apotheosis for the small-sample database. SVM has shown its excellent learning and generalization ability and has been extensively employed in many areas. This paper presents a performance analysis of six types of SVMs for the diagnosis of the classical Wisconsin breast cancer problem from a statistical point of view. The classification performance of standard SVM (St-SVM) is analyzed and compared with those of the other modified classifiers such as proximal support vector machine (PSVM) classifiers, Lagrangian support vector machines (LSVM), finite Newton method for Lagrangian support vector machine (NSVM), Linear programming support vector machines (LPSVM), and smooth support vector machine (SSVM). The experimental results reveal that these SVM classifiers achieve very fast, simple, and efficient breast cancer diagnosis. The training results indicated that LSVM has the lowest accuracy of 95.6107 %, while St-SVM performed better than other methods for all performance indices (accuracy = 97.71 %) and is closely followed by LPSVM (accuracy = 97.3282). However, in the validation phase, the overall accuracies of LPSVM achieved 97.1429 %, which was superior to LSVM (95.4286 %), SSVM (96.5714 %), PSVM (96 %), NSVM (96.5714 %), and St-SVM (94.86 %). Value of ROC and MCC for LPSVM achieved 0.9938 and 0.9369, respectively, which outperformed other classifiers. The results strongly suggest that LPSVM can aid in the diagnosis of breast cancer.

A feature selection method for nonparallel plane support vector machine classification

Over the past decades, 1-norm techniques based on algorithms are widely used to suppress input features. Quite different from traditional 1-norm support vector machine (SVM), direct 1-norm optimization based on the primal problem of nonparallel plane classifiers like generalized proximal support vector machine, twin support vector machine (TWSVM) and least squares twin support vector machine (LSTSVM) are not capable of generating very sparse solutions that are vital for classification and can make them easier to store and faster to compute. To address the issue, in this paper, we develop a feature selection method for LSTSVM, called a feature selection method for nonparallel plane support vector machine classification (FLSTSVM), which is specially designed for strong feature suppression. We incorporate a Tikhonov regularization term to the objective of LSTSVM, and then minimize its 1-norm measure. Solution of FLSTSVM can follow directly from solving two smaller quadratic programming problems (QPPs) arising from two primal QPPs as opposed to two dual ones in TWSVM. FLSTSVM is capable of generating very sparse solutions. This means that FLSTSVM can reduce input features, for the linear case. When a nonlinear classifier is used, few kernel functions determine the classifier. In addition to having strong feature suppression, the edge of our method still lies in its faster computing time compared to that of TWSVM, Newton Method for Linear Programming SVM (NLPSVM) and LPNewton. Lastly, this algorithm is compared on public data sets, as well as an Exclusive Or (XOR) example.

Geochemical Fingerprinting of Coltan Ores by Machine Learning on Uneven Datasets

Two modern machine learning techniques, Linear Programming Boosting (LPBoost) and Support Vector Machines (SVMs), are introduced and applied to a geochemical dataset of niobium–tantalum (“coltan”) ores from Central Africa to demonstrate how such information may be used to distinguish ore provenance, i.e., place of origin. The compositional data used include uni- and multivariate outliers and elemental distributions are not described by parametric frequency distribution functions. The “soft margin” techniques of LPBoost and SVMs can be applied to such data. Optimization of their learning parameters results in an average accuracy of up to c. 92%, if spot measurements are assessed to estimate the provenance of ore samples originating from two geographically defined source areas. A parameterized performance measure, together with common methods for its optimization, was evaluated to account for the presence of uneven datasets. Optimization of the classification function threshold improves the performance, as class importance is shifted towards one of those classes. For this dataset, the average performance of the SVMs is significantly better compared to that of LPBoost.

Support vector machine classification with noisy data: a second order cone programming approach

In this paper, we investigate the stability of linear and quadratic programming support vector machines (SVMs) with bounded noise in the input data using a robust optimisation model. For a linear discriminant function, this model is expressed as a second order cone optimisation problem. Using the concept of the kernel function, we generalise for nonlinear discriminant functions. Intuitively, it looks quite clear that large margin classifiers are robust in terms of bounded input noise. However, there is no theoretical analysis investigating this behaviour. We show that the SVM solution is stable under bounded perturbations of the data both in the linear programming and quadratic programming. Computational results are also presented for toy and real-world data.

Decomposition techniques for training linear programming support vector machines

In this paper, we propose three decomposition techniques for linear programming (LP) problems: (1) Method 1, in which we decompose the variables into the working set and the fixed set, but we do not decompose the constraints, (2) Method 2, in which we decompose only the constraints and (3) Method 3, in which we decompose both the variables and the constraints into two. By Method 1, the value of the objective function is proved to be non-decreasing (non-increasing) for the maximization (minimization) problem and by Method 2, the value is non-increasing (non-decreasing) for the maximization (minimization) problem. Thus, by Method 3, which is a combination of Methods 1 and 2, the value of the objective function is not guaranteed to be monotonic and there is a possibility of infinite loops. We prove that infinite loops are resolved if the variables in an infinite loop are not released from the working set and Method 3 converges in finite steps. We apply Methods 1 and 3 to LP support vector machines (SVMs) and discuss a more efficient method of accelerating training by detecting the increase in the number of violations and restoring variables in the working set that are released at the previous iteration step. By computer experiments for microarray data with huge input variables and a small number of constraints, we demonstrate the effectiveness of Method 1 for training the primal LP SVM with linear kernels. We also demonstrate the effectiveness of Method 3 over Method 1 for the nonlinear LP SVMs.

The hidden neurons selection of the wavelet networks using support vector machines and ridge regression

A 1-norm support vector machine stepwise (SVMS) algorithm is proposed for the hidden neurons selection of wavelet networks (WNs). In this new algorithm, the linear programming support vector machine (LPSVM) is employed to pre-select the hidden neurons, and then a stepwise selection algorithm based on ridge regression is introduced to select hidden neurons from the pre-selection. The main advantages of the new algorithm are that it can get rid of the influence of the ill conditioning of the matrix and deal with the problems that involve a great number of candidate neurons or a large size of samples. Four examples are provided to illustrate the efficiency of the new algorithm.

Feature Selection with Support Vector Machines Applied on Tornado Detection

In this paper, a linear programming support vector machine which is based on L1-norm is applied to do feature selection in the tornado data set. The data is the ouputs of Weather Surveillance Radar 1998 Doppler (WSR-88D). The approach is evaluated based on the indices of probability of detection, false alarm rate, bias and Heidke skill. Tornado circulation attributes/variables derived largely from the National Severe Storms Laboratory Mesocyclone Detection Algorithm (MDA) have been investigated for their efficacy in distinguishing between mesocyclones that become tornadic from those which do not.

A novel approach for accurate prediction of spontaneous passage of ureteral stones: Support vector machines

The objective of this study was to optimally predict the spontaneous passage of ureteral stones in patients with renal colic by applying for the first time support vector machines (SVM), an instance of kernel methods, for classification. After reviewing the results found in the literature, we compared the performances obtained with logistic regression (LR) and accurately trained artificial neural networks (ANN) to those obtained with SVM, that is, the standard SVM, and the linear programming SVM (LP-SVM); the latter techniques show an improved performance. Moreover, we rank the prediction factors according to their importance using Fisher scores and the LP-SVM feature weights. A data set of 1163 patients affected by renal colic has been analyzed and restricted to single out a statistically coherent subset of 402 patients. Nine clinical factors are used as inputs for the classification algorithms, to predict one binary output. The algorithms are cross-validated by training and testing on randomly selected train- and test-set partitions of the data and reporting the average performance on the test sets. The SVM-based approaches obtained a sensitivity of 84.5% and a specificity of 86.9%. The feature ranking based on LP-SVM gives the highest importance to stone size, stone position and symptom duration before check-up. We propose a statistically correct way of employing LR, ANN and SVM for the prediction of spontaneous passage of ureteral stones in patients with renal colic. SVM outperformed ANN, as well as LR. This study will soon be translated into a practical software toolbox for actual clinical usage.

SVM Soft Margin Classifiers: Linear Programming versus Quadratic Programming

Support vector machine (SVM) soft margin classifiers are important learning algorithms for classification problems. They can be stated as convex optimization problems and are suitable for a large data setting. Linear programming SVM classifiers are especially efficient for very large size samples. But little is known about their convergence, compared with the well-understood quadratic programming SVM classifier. In this article, we point out the difficulty and provide an error analysis. Our analysis shows that the convergence behavior of the linear programming SVM is almost the same as that of the quadratic programming SVM. This is implemented by setting a stepping-stone between the linear programming SVM and the classical 1-norm soft margin classifier. An upper bound for the misclassification error is presented for general probability distributions. Explicit learning rates are derived for deterministic and weakly separable distributions, and for distributions satisfying some Tsybakov noise condition.

Linear programming support vector machines

Based on the analysis of the conclusions in the statistical learning theory, especially the VC dimension of linear functions, linear programming support vector machines (or SVMs) are presented including linear programming linear and nonlinear SVMs. In linear programming SVMs, in order to improve the speed of the training time, the bound of the VC dimension is loosened properly. Simulation results for both artificial and real data show that the generalization performance of our method is a good approximation of SVMs and the computation complex is largely reduced by our method.