Support Vector Machine Formulation Research Articles

Adjustable robust optimization approach for SVM under uncertainty

The support vector machine (SVM) is one of the successful approaches to the classification problem. Since the values of features are typically affected by uncertainty, it is important to incorporate uncertainty into the SVM formulation. This paper focuses on developing a robust optimization (RO) model for SVM. A key distinction from existing literature lies in the timing of optimizing decision variables. To the best of our knowledge, in all existing RO models developed for SVM, a common assumption is that all decision variables are decided before the uncertainty realization, which leads to an overly conservative decision boundary. However, this paper adopts a different strategy by determining the variables that assess the misclassification error of data points or their fall within the margin post-realization, resulting in a less conservative model. The RO models where decisions are made in two stages (some before and the rest after the uncertainty resolution), are called adjustable RO models. This adjustment results in a three-level optimization model for which two decomposition-based algorithms are proposed. In these algorithms, after providing a bi-level reformulation, the model is divided into a master-problem (MP) and a sub-problem the interaction of which yields the optimal solution. Acceleration of algorithms via incorporating valid inequalities into MP is another novelty of this paper. Computational results over simulated and real-world datasets confirm the efficiency of the proposed model and algorithms.

Mixed-integer quadratic optimization and iterative clustering techniques for semi-supervised support vector machines

Among the most famous algorithms for solving classification problems are support vector machines (SVMs), which find a separating hyperplane for a set of labeled data points. In some applications, however, labels are only available for a subset of points. Furthermore, this subset can be non-representative, e.g., due to self-selection in a survey. Semi-supervised SVMs tackle the setting of labeled and unlabeled data and can often improve the reliability of the results. Moreover, additional information about the size of the classes can be available from undisclosed sources. We propose a mixed-integer quadratic optimization (MIQP) model that covers the setting of labeled and unlabeled data points as well as the overall number of points in each class. Since the MIQP’s solution time rapidly grows as the number of variables increases, we introduce an iterative clustering approach to reduce the model’s size. Moreover, we present an update rule for the required big-M values, prove the correctness of the iterative clustering method as well as derive tailored dimension-reduction and warm-starting techniques. Our numerical results show that our approach leads to a similar accuracy and precision than the MIQP formulation but at much lower computational cost. Thus, we can solve larger problems. With respect to the original SVM formulation, we observe that our approach has even better accuracy and precision for biased samples.

Generalization of linear and non-linear support vector machine in multiple fields: a review

Support vector machines (SVMs) are a set of related supervised learning methods used for classification and regression. They belong to a family of generalized linear classifiers. In other terms, SVM is a classification and regression prediction tool that uses machine learning theory to maximize predictive accuracy. In this article, the discussion about linear and non-linear SVM classifiers with their functions and parameters is investigated. Due to the equality type of constraints in the formulation, the solution follows from solving a set of linear equations. Besides this, if the under-consideration problem is in the form of a non-linear case, then the problem must convert into linear separable form with the help of kernel trick and solve it according to the methods. Some important algorithms related to sentimental work are also presented in this paper. Generalization of the formulation of linear and non-linear SVMs is also open in this article. In the final section of this paper, the different modified sections of SVM are discussed which are modified by different research for different purposes.

Generalization of linear and non-linear support vector machine in multiple fields: a review

Support vector machines (SVMs) are a set of related supervised learning methods used for classification and regression. They belong to a family of generalized linear classifiers. In other terms, SVM is a classification and regression prediction tool that uses machine learning theory to maximize predictive accuracy. In this article, the discussion about linear and non-linear SVM classifiers with their functions and parameters is investigated. Due to the equality type of constraints in the formulation, the solution follows from solving a set of linear equations. Besides this, if the under-consideration problem is in the form of a non-linear case, then the problem must convert into linear separable form with the help of kernel trick and solve it according to the methods. Some important algorithms related to sentimental work are also presented in this paper. Generalization of the formulation of linear and non-linear SVMs is also open in this article. In the final section of this paper, the different modified sections of SVM are discussed which are modified by different research for different purposes.

A stable variant of linex loss SVM for handling noise with reduced hyperparameters

Support Vector Machine (SVM) primarily uses the hinge loss function with maximum margin. The boundary instances determine the separating hyperplanes. However, in real-world scenarios, noisy instances around the decision boundary can degrade the performance of hinge SVM. To address this issue, researchers introduced the pinball SVM, which uses all the instances to determine the separating hyperplane and has a discontinuous piecewise linear loss function. A more recent variant, the linex loss SVM with NAG, employs a combination of linear and exponential loss functions, resulting in an asymmetric and continuous loss function. Due to the employment of Nesterov accelerated gradient (NAG), the older linex SVM formulation was unstable. This study presents a novel noise-robust variant of linex SVM, termed linexSVM, which is stable in performance compared to the linex SVM with NAG. Additionally, the proposed linexSVM has fewer hyperparameters in the dual formulation than other noise-robust SVM variants, such as pinball SVM, ϵ-insensitive pinball SVM, and ϵ1,ϵ2 pinball SVM. We evaluated the performance of the proposed linexSVM, hinge SVM, and pinball SVM using the UCI dataset repository. The experimental results indicate that the proposed linexSVM outperforms hinge and pinball loss SVMs on datasets with and without noise.

Ensemble Support Vector Recurrent Neural Network for Brain Signal Detection.

The brain-computer interface (BCI) P300 speller analyzes the P300 signals from the brain to achieve direct communication between humans and machines, which can assist patients with severe disabilities to control external machines or robots to complete expected tasks. Therefore, the classification method of P300 signals plays an important role in the development of BCI systems and technologies. In this article, a novel ensemble support vector recurrent neural network (E-SVRNN) framework is proposed and developed to acquire more accurate and efficient electroencephalogram (EEG) signal classification results. First, we construct a support vector machine (SVM) to formulate EEG signals recognizing model. Second, the SVM formulation is transformed into a standard convex quadratic programming (QP) problem. Third, the convex QP problem is solved by combining a varying parameter recurrent neural network (VPRNN) with a penalty function. Experimental results on BCI competition II and BCI competition III datasets demonstrate that the proposed E-SVRNN framework can achieve accuracy rates as high as 100% and 99%, respectively. In addition, the results of comparison experiments verify that the proposed E-SVRNN possesses the best recognition accuracy and information transfer rate (ITR) compared with most of the state-of-the-art algorithms.

Lightweight Anomaly Detection Scheme Using Incremental Principal Component Analysis and Support Vector Machine.

Wireless Sensors Networks have been the focus of significant attention from research and development due to their applications of collecting data from various fields such as smart cities, power grids, transportation systems, medical sectors, military, and rural areas. Accurate and reliable measurements for insightful data analysis and decision-making are the ultimate goals of sensor networks for critical domains. However, the raw data collected by WSNs usually are not reliable and inaccurate due to the imperfect nature of WSNs. Identifying misbehaviours or anomalies in the network is important for providing reliable and secure functioning of the network. However, due to resource constraints, a lightweight detection scheme is a major design challenge in sensor networks. This paper aims at designing and developing a lightweight anomaly detection scheme to improve efficiency in terms of reducing the computational complexity and communication and improving memory utilization overhead while maintaining high accuracy. To achieve this aim, one-class learning and dimension reduction concepts were used in the design. The One-Class Support Vector Machine (OCSVM) with hyper-ellipsoid variance was used for anomaly detection due to its advantage in classifying unlabelled and multivariate data. Various One-Class Support Vector Machine formulations have been investigated and Centred-Ellipsoid has been adopted in this study due to its effectiveness. Centred-Ellipsoid is the most effective kernel among studies formulations. To decrease the computational complexity and improve memory utilization, the dimensions of the data were reduced using the Candid Covariance-Free Incremental Principal Component Analysis (CCIPCA) algorithm. Extensive experiments were conducted to evaluate the proposed lightweight anomaly detection scheme. Results in terms of detection accuracy, memory utilization, computational complexity, and communication overhead show that the proposed scheme is effective and efficient compared few existing schemes evaluated. The proposed anomaly detection scheme achieved the accuracy higher than 98%, with (nd) memory utilization and no communication overhead.

Intuitionistic Fuzzy Laplacian Twin Support Vector Machine for Semi-supervised Classification

In general, data contain noises which come from faulty instruments, flawed measurements or faulty communication. Learning with data in the context of classification or regression is inevitably affected by noises in the data. In order to remove or greatly reduce the impact of noises, we introduce the ideas of fuzzy membership functions and the Laplacian twin support vector machine (Lap-TSVM). A formulation of the linear intuitionistic fuzzy Laplacian twin support vector machine (IFLap-TSVM) is presented. Moreover, we extend the linear IFLap-TSVM to the nonlinear case by kernel function. The proposed IFLap-TSVM resolves the negative impact of noises and outliers by using fuzzy membership functions and is a more accurate reasonable classifier by using the geometric distribution information of labeled data and unlabeled data based on manifold regularization. Experiments with constructed artificial datasets, several UCI benchmark datasets and MNIST dataset show that the IFLap-TSVM has better classification accuracy than other state-of-the-art twin support vector machine (TSVM), intuitionistic fuzzy twin support vector machine (IFTSVM) and Lap-TSVM.

On the goodness of fit of parametric and non-parametric data mining techniques: the case of malaria incidence thresholds in Uganda

To identify which data mining technique (parametric or non-parametric) best fits the predictions on imbalanced malaria incidence dataset. The researchers compared parametric techniques in form of naive Bayes and logistic regression against non-parametric techniques in form of support vector machines and artificial neural networks and their goodness of fit and prediction was assessed using 10-fold and 5-fold cross-validation on an independent validation dataset set to determine which model best fits the predictions on imbalanced malaria incidence dataset. The 10-fold cross-validation outperformed the 5-fold cross-validation in all performance metrics with the naive Bayes classifier attaining accuracy of 69% with a sensitivity of 90.9%, a specificity of 55.6%, a precision of 55.6% and F-measure score of 69.0%, the logistic regression achieved an accuracy of 65.5% with a sensitivity of 83.3%, a specificity of 52.9%, a precision of 55.6% and F-measure score of 66.7%, the support vector machines achieved an accuracy of 82.8% with a sensitivity of 88.2%, a specificity of 75.0%, a precision of 83.3%, and F-measure score of 85.7% whereas the artificial neural networks registered an accuracy of 89.7% with a sensitivity of 94.1%, a specificity of 83.3%, a precision of 88.9%, and F-measure score of 91.4%. Non-parametric data mining techniques in form of artificial neural networks and support vector machines outperformed the parametric data mining technique in form of naive Bayes in making predictions emanating from imbalanced malaria incidence dataset on account of registering higher F-measure values of 91.4% and 85.7% respectively.

On Lagrangian L2-norm pinball twin bounded support vector machine via unconstrained convex minimization

With the introduction of the regularization term in the formulation of the well-known twin support vector machine (TWSVM) for classification , twin bounded support vector machine (TBSVM) method was proposed recently as an improved version by implementing the structural risk minimization principle. However, TBSVM employs hinge loss function and it is sensitive to noise and unstable to re-sampling. Since the pinball loss function related to quantile distance enjoys noise insensitivity property , a novel TBSVM method with squared pinball loss function for classification is proposed. The noise insensitivity and scatter minimization properties are discussed. Our formulation is further simplified as a pair of unconstrained strongly convex minimization problems in the dual space free of matrix inversion terms and having only m variables where m is the number of training examples. As opposed to TWSVM and TBSVM wherein approximate kernel generated surfaces are constructed, kernel trick is applied directly in our formulation and thereby elegant formulation as in the classical support vector machine (SVM) is achieved. Numerical experiments performed on a synthetic and thirteen benchmark datasets with noise where better or comparable generalization performance with faster learning speed by the proposed method confirms its suitability and applicability to problems of interest.

A geometric-based data reduction approach for large low dimensional datasets: Delaunay triangulation in SVM algorithms

Training a support vector machine (SVM) on large datasets is a slow daunting process. Further, SVM becomes slow in the testing phase, due to its large number of support vectors (SVs). This paper proposes an effective geometric algorithm based on construction of Delaunay triangulation (DT) algorithm using Quickhull algorithm with a novel strategy to exactly identify and extract the boundary data points laid between the two classes of a dataset, and later uses these most informative data points as a reduced dataset to solve various SVM algorithms and proposes new DT-SVM algorithms Two synthetic datasets with the size of 1K incrementally up to 500K datasets are generated to extensively verify the effectiveness of the proposed DT-SVM algorithms over various data sizes and for further assessment, the most efficient version of proposed DT-SVM is applied on well-known benchmark datasets from UCI Machine Learning Repository. Two variant of sequential minimization optimization (SMO) decomposition methods, in addition to Least Square form of SVM are implemented to present the scalability of new DT-SVM algorithms in linear/nonlinear separable/non-separable large low dimensional datasets. Moreover, the most efficient version of the proposed algorithm is compared to RCH-SK as a known geometric approach in the SVM literature. The results demonstrate that while the proposed approach improves the scalability of DT-SVM in large low dimensional datasets, it leads SVM algorithms to maintain the accuracy in an acceptable range with considerably lower time in both training and testing phases with using a noticeably fewer number of SVs.

A novel model for anomaly detection in network traffic based on kernel support vector machine

Machine learning models are widely used for anomaly detection in network traffic. Effective transformation of the raw traffic data into mathematical expressions and hyper-parameter adjustment are two important steps before training the machine learning classifier, which is used to predict whether the unknown traffic is normal or abnormal. In this paper, a novel model SVM-L is proposed for anomaly detection in network traffic. In particular, raw URLs are treated as natural language, and then transformed into mathematical vectors via statistical laws and natural language processing technique. They are used as the training data for the traffic classifier, the kernel Support Vector Machine (SVM). Based on the idea of the dual formulation of kernel SVM and Linear Discriminant Analysis (LDA), we propose an optimization model to adjust the hyper-parameter of the classifier. The corresponding problem is simply one-dimensional, and is easily solved by the golden section method. Numerical tests indicate that the proposed model achieves more than 99% accuracy on all tested datasets, and outperforms the state of the arts in terms of standard evaluation measurements.

Spatio-Temporal Planning in Multi-Vehicle Scenarios for Autonomous Vehicle Using Support Vector Machines

Efficient trajectory planning of autonomous vehicles in complex traffic scenarios is of interest both academically and in automotive industry. Time efficiency and safety are of key importance and here a two-step procedure is proposed. First, a convex optimization problem is solved, formulated as a support vector machine (SVM), in order to represent the surrounding environment of the ego vehicle and classify the search space as obstacles or obstacle free. This gives a reduced complexity search space and an A* algorithm is used in a state space lattice in 4 dimensions including position, heading angle and velocity for simultaneous path and velocity planning. Further, a heuristic derived from the SVM formulation is used in the A* search and a pruning technique is introduced to significantly improve search efficiency. Solutions from the proposed planner is compared to optimal solutions computed using optimal control techniques. Three traffic scenarios, a roundabout scenario and two complex takeover maneuvers, with multiple moving obstacles, are used to illustrate the general applicability of the proposed method.

MBSVR: Multiple birth support vector regression

Although the learning speed in twin support vector regression (TWSVR) is four times that in support vector regression (SVR), the computing time and fitting precision of TWSVR are limited. This paper develops multiple birth support vector regression (MBSVR), motivated by the multiple birth support vector machine (MBSVM) formulation. MBSVR constructs the final regressor from K hyperplanes, each of which is obtained by solving a small quadratic programming problem (QPP) with the associated constraints, in which all points in each of the corresponding class should be as far away as possible from its corresponding hyperplane. Since MBSVM can be seen as an extension of the twin support vector machine (TWSVM) and its computing time is less than that of TWSVM, the proposed MBSVR is also faster than TWSVR, especially when the number of classes K is large. To verify the performance of the proposed MBSVR, it is compared with TWSVR, TSVR (another form of twin support vector regression) and SVR on several synthetic datasets and UCI datasets.

Formulating Ensemble Learning of SVMs Into a Single SVM Formulation by Negative Agreement Learning

When a fixed number of support vector machines (SVMs) are taken as the base learners, an attempt to diversify them should be encouraged to achieve a satisfactory ensemble. In this article, by means of a negative agreement learning (NAL) strategy, a new SVM-based ensemble framework is proposed to simultaneously enhance the diversity of SVMs in the ensemble and suppress the training error of the ensemble. The proposed ensemble framework is theoretically derived to have distinctive merits: 1) the ensemble and each of its individual SVM base learner are trained in a joint manner rather than in an independent manner and 2) the NAL strategy facilitates the formulation of the ensemble of SVMs as one single SVM; thus, abundant advances in the training of SVM can be conveniently applied to the proposed ensemble learning of SVMs and there is no need to design special optimization techniques for the involved ensemble learning. Extensive experimental studies demonstrate the effectiveness of the proposed ensemble framework of SVMs.

Sparse pinball twin support vector machines

The original twin support vector machine (TWSVM) formulation works by solving two smaller quadratic programming problems (QPPs) as compared to the traditional hinge-loss SVM (C-SVM) which solves a single large QPP — this makes the TWSVM training and testing process faster than the C-SVM. However, these TWSVM problems are based on the hinge-loss function and, hence, are sensitive to feature noise and unstable for re-sampling. The pinball-loss function, on the other hand, maximizes quantile distances which grants noise insensitivity but this comes at the cost of losing sparsity by penalizing correctly classified samples as well. To overcome the limitations of TWSVM, we propose a novel sparse pinball twin support vector machines (SPTWSVM) based on the ϵ-insensitive zone pinball loss function to rid the original TWSVM of its noise insensitivity and ensure that the resulting TWSVM problems retain sparsity which makes computations relating to predictions just as fast as the original TWSVM. We further investigate the properties of our SPTWSVM including sparsity, noise insensitivity, and time complexity. Exhaustive testing on several benchmark datasets demonstrates that our SPTWSVM is noise insensitive, retains sparsity and, in most cases, outperforms the results obtained by the original TWSVM.

A coarse‐to‐fine two‐step method for semisupervised classification using quasi‐linear Laplacian SVM

This paper proposes a two‐step method to construct a nonlinear classifier based on semisupervised learning in a coarse‐to‐fine way. In the first step, a recursive density‐based spatial clustering of applications with noise clustering algorithm is first introduced to find a group of density clusters, each of which contains only one kind of class labels. An SK algorithm is then applied to pairs of density clusters containing different class labels to find a set of local linear classifiers, which forms a coarse nonlinear separating boundary crossing the low‐density areas by interpolating the local linear classifiers. In the second step, a Laplacian support vector machine (LapSVM) formulation based on graph construction is applied to further implicitly optimize the parameter set of the nonlinear coarse classifier. As a result, the fine‐tuned nonlinear classifier is constructed in exactly the same way as a standard LapSVM, using a special data‐dependent quasi‐linear kernel composed of the interpolation functions and the information of the local linear classifiers obtained in the first step. Moreover, the quasi‐linear kernel is used as a better similarity function for the graph construction. Numerical experiments on various real‐world datasets demonstrate the effectiveness of the proposed method. © 2018 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.

One‐class classification using a support vector machine with a quasi‐linear kernel

This article proposes a novel method for one‐class classification based on a divide‐and‐conquer strategy to improve the one‐class support vector machine (SVM). The idea is to build a piecewise linear separation boundary in the feature space to separate the data points from the origin, which is expected to have a more compact region in the input space. For the purpose, the input space of the dataset is first divided into a group of partitions by using a partitioning mechanism of top s% winner‐take‐all autoencoder. A gated linear network is designed to implement a group of linear classifiers for each partition, in which the gate signals are generated from the autoencoder. By applying a one‐class SVM (OCSVM) formulation to optimize the parameter set of the gated linear network, the one‐class classifier is implemented in an exactly same way as a standard OCSVM with a quasi‐linear kernel composed using a base kernel with the gate signals. The proposed one‐class classification method is applied to different real‐world datasets, and simulation results show that it shows a better performance than a traditional OCSVM. © 2018 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.

Generalized Pinball Loss SVMs

Abstract Support Vector Machine (SVM) is a well-known efficient classification technique which appropriately trade-offs between the training error and the generalization ability of the classifier. But SVM’s are known to be sensitive towards noise and not stable with respect to re-sampling. Recently, Huang et al. (2014) proposed a novel SVM formulation with pinball loss function (Pin-SVM) model to handle noise sensitivity and instability to re-sampling. The Pin-SVM model in its pure form loses sparsity and therefore Huang et al. (2014) proposed a new pinball SVM model termed as ϵ-insensitive zone Pin-SVM. The ϵ-insensitive zone Pin-SVM formulation requires the value of ϵ to be pre-specified. Taking motivation from these developments, we propose a modified (ϵ1, ϵ2)-insensitive zone Pin-SVM ((ϵ1, ϵ2)-Mod-Pin-SVM) model in which the asymmetric spread of insensitive zone is optimized and therefore it is data-driven. An interesting feature of our proposed (ϵ1, ϵ2)-Mod-Pin-SVM model is that it appropriately trade-offs the generalization ability, the sparsity as well as the noise insensitivity. The (ϵ1, ϵ2)-Mod-Pin-SVM model is very general and subsumes a number of popular SVM type models. Its relation with Minimal Complexity Machine (MCM) introduced by Jayadeva (2015) is interesting and novel. The experimental results on several benchmark datasets prove the effectiveness of our proposed (ϵ1, ϵ2)-Mod-Pin-SVM formulation to that of other state-of-the-art classification algorithms.

Maximization of AUC and Buffered AUC in binary classification

In binary classification, performance metrics that are defined as the probability that some error exceeds a threshold are numerically difficult to optimize directly and also hide potentially important information about the magnitude of errors larger than the threshold. Defining similar metrics, instead, using Buffered Probability of Exceedance (bPOE) generates counterpart metrics that resolve both of these issues. We apply this approach to the case of AUC, the Area Under the ROC curve, and define Buffered AUC (bAUC). We show that bAUC can provide insights into classifier performance not revealed by AUC, while being closely related as the tightest concave lower bound and representable as the area under a modified ROC curve. Additionally, while AUC is numerically difficult to optimize directly, we show that bAUC optimization often reduces to convex or linear programming. Extending these results, we show that AUC and bAUC are special cases of Generalized bAUC and that popular Support Vector Machine (SVM) formulations for approximately maximizing AUC are equivalent to direct maximization of Generalized bAUC. We also prove bAUC generalization bounds for these SVM’s. As a central component to these results, we provide an important, novel formula for calculating bPOE, the inverse of Conditional Value-at-Risk. Using this formula, we show that particular bPOE minimization problems reduce to convex and linear programming.