Original Support Vector Machine Research Articles

Mixed-integer quadratic optimization and iterative clustering techniques for semi-supervised support vector machines

AbstractAmong the most famous algorithms for solving classification problems are support vector machines (SVMs), which find a separating hyperplane for a set of labeled data points. In some applications, however, labels are only available for a subset of points. Furthermore, this subset can be non-representative, e.g., due to self-selection in a survey. Semi-supervised SVMs tackle the setting of labeled and unlabeled data and can often improve the reliability of the results. Moreover, additional information about the size of the classes can be available from undisclosed sources. We propose a mixed-integer quadratic optimization (MIQP) model that covers the setting of labeled and unlabeled data points as well as the overall number of points in each class. Since the MIQP’s solution time rapidly grows as the number of variables increases, we introduce an iterative clustering approach to reduce the model’s size. Moreover, we present an update rule for the required big-M values, prove the correctness of the iterative clustering method as well as derive tailored dimension-reduction and warm-starting techniques. Our numerical results show that our approach leads to a similar accuracy and precision than the MIQP formulation but at much lower computational cost. Thus, we can solve larger problems. With respect to the original SVM formulation, we observe that our approach has even better accuracy and precision for biased samples.

A comparative analysis of feature selection models for spatial analysis of floods using hybrid metaheuristic and machine learning models.

The research aims to propose a feature selection model for hydraulic analysis as such a model has not been proposed previously. For this purpose, hybrids of three metaheuristic algorithms, particle swarm optimization (PSO), ant colony optimization (ACO), and genetic algorithm (GA) with two machine learning models which are support vector machine (SVM) and K-nearest neighbor (KNN) are employed. The dataset considered was hydraulic having an association with flood and possessed topographic, geo-environmental, and human-induced variables. The dataset considered had multicollinearity heteroscedasticity and autocorrelation problems. The metaheuristic algorithms were evaluated by varying the number of population size. Among them, PSO performed better by providing an appropriate number of features with a lower number of iterations. We have analyzed the performance of SVM with different kernels; linear, radial basis function (RBF), sigmoid, and polynomial, as the original SVM is designed only for linear datasets but the hydraulic dataset possesses non-linear characteristics as well. The performance of different kernels in terms of their accuracies is evaluated and recorded. This study showed that RBF performed the best and sigmoid showed the least accuracy for GA, PSO, and ACO algorithms. The performance of KNN is evaluated in terms of accuracies by varying the K-values. It was found that KNN shows low accuracy with a small K-value which then attained a maximum level by increasing K-values, and it finally started decreasing, explicitly, by further enhancing K-values. While comparing the performance of hybrids of GA, PSO, and ACO with SVM and KNN, it was analyzed that KNN performed better with these meta-heuristics with PSO-KNN which performed the best among the baseline models. Thus, the study proposes that PSO-KNN can be utilized as a feature selection technique to obtain optimal data subsets for hydraulic modeling and analysis.

Research on Data Classification Method of Optimized Support Vector Machine Based on Gray Wolf Algorithm

The data classification method based on support vector machine (SVM) has been widely used in various studies as a non-linear, high precision, and good generalization ability machine learning method. Among them, the kernel function and its parameters have a great impact on the classification accuracy. In order to find the optimal parameters to improve the classification accuracy of SVM, this paper proposes a data multi-classification method based on gray wolf algorithm optimized SVM(GWO-SVM). In this paper, the iris data set is used to test the performance of GWO-SVM, and the classification result is compared with those based on genetic algorithm (GA), particle swarm optimization (PSO) and the original SVM model. The test results show that the GWO-SVM model has a higher recognition and classification accuracy than the other three models, and has the shortest running time, which has obvious advantages and can effectively improve the classification accuracy of SVM. This method has practical significance in image classification, text classification, and fault detection.

Prediction of the Ultimate Tensile Strength (UTS) of Asymmetric Friction Stir Welding Using Ensemble Machine Learning Methods

This research aims to develop ensemble machine-learning methods for forecasting the ultimate tensile strength (UTS) of friction stir welding (FSW). The substance utilized in the experiment was a mixture of aluminum alloys AA5083 and AA5061. An ensemble machine learning model was created to predict the UTS of the friction stir-welded seam, utilizing 11 FSW parameters as input factors and the UTS as a response variable. The proposed approach used the Gaussian process regression (GPR) and the support vector machine (SVM) model of machine learning to build the ensemble machine learning model. In addition, an efficient technique using a differential evolution algorithm to optimize the weight for the decision fusion was incorporated into the proposed model. The effectiveness of the model was evaluated using three datasets. The first and second datasets were divided into two groups, with 80% for the training dataset and 20% for the testing dataset, while the third dataset comprised the test data to validate the model’s accuracy. The computational results indicated that the proposed model provides more accurate forecasts than existing methods, such as random forest, gradient boosting, ADA boosting, and the original SVM and GPR, by 30.67, 49.18, 16.50, 48.87, and 49.33 %, respectively. In terms of prediction accuracy, the suggested technique for decision fusion surpasses unweighted average ensemble learning (UWE) by 10.32%.

Mid-long term load forecasting model based on support vector machine optimized by improved sparrow search algorithm

The regression model based on the support vector machine (SVM) is a standard mid-term and long-term load forecasting method. However, the hyperparameters of traditional SVM are challenging to determine, leading to its poor forecasting effect. This paper proposes an improved sparrow search algorithm (ISSA) to solve the problem of hyperparameter selection of the SVM model and construct the mid-long term load prediction model by ISSA-SVM. This ISSA is enhanced by a new dynamic adaptive t-distribution mutation. Compared to the SSA, the ISSA has better convergence precision, stability and speed, which was verified by the comparative test based on six benchmark functions. The simulation results also show that the ISSA-SVM can effectively improve the prediction accuracy compared with the original SVM, BP neural network, multiple linear regression, etc.

DPWSS: differentially private working set selection for training support vector machines.

Support vector machine (SVM) is a robust machine learning method and is widely used in classification. However, the traditional SVM training methods may reveal personal privacy when the training data contains sensitive information. In the training process of SVMs, working set selection is a vital step for the sequential minimal optimization-type decomposition methods. To avoid complex sensitivity analysis and the influence of high-dimensional data on the noise of the existing SVM classifiers with privacy protection, we propose a new differentially private working set selection algorithm (DPWSS) in this paper, which utilizes the exponential mechanism to privately select working sets. We theoretically prove that the proposed algorithm satisfies differential privacy. The extended experiments show that the DPWSS algorithm achieves classification capability almost the same as the original non-privacy SVM under different parameters. The errors of optimized objective value between the two algorithms are nearly less than two, meanwhile, the DPWSS algorithm has a higher execution efficiency than the original non-privacy SVM by comparing iterations on different datasets. To the best of our knowledge, DPWSS is the first private working set selection algorithm based on differential privacy.

Evaluation of a calibration transfer between a bench top and portable Mid-InfraRed spectrometer for cocaine classification and quantification

A portable Fourier Transform Mid-InfraRed (FT-MIR) spectrometer using Attenuated Total Reflectance (ATR) sampling is used for daily routine screening of seized powders. Earlier, ATR-FT-MIR combined with Support Vector Machines (SVM) algorithms resulted in a significant improvement of the screening method to a reliable and straightforward classification and quantification tool for both cocaine and levamisole. However, can this tool be transferred to new (hand-held) devices, without loss of the extensive data set? The objective of this study was to perform a calibration transfer between a newly purchased bench top (BT) spectrometer and a portable (P) spectrometer with existing calibration models. Both instruments are from the same brand and have identical characteristics and acquisition parameters (FT instrument, resolution of 4 cm−1 and wavenumber range 4000 to 500 cm−1). The original SVM classification model (n = 515) and SVM quantification model (n = 378) were considered for the transfer trial.Three calibration transfer strategies were assessed: 1) adjustment of slope and bias; 2) correction of spectra from the new instrument BT to P using Piecewise Direct Standardization (PDS) and 3) building a new mixed instrument model with spectra of both instruments. For each approach, additional cocaine powders were measured (n = 682) and the results were compared with GC-MS and GC-FID.The development of a mixed instrument model was the most successful in terms of performance. The future strategy of a mixed model allows applying the models, developed in the laboratory, to portable instruments that are used on-site, and vice versa.The approach offers opportunities to exchange data within a network of forensic laboratories using other FT-MIR spectrometers.

A comparative study of composite kernels for landslide susceptibility mapping: A case study in Yongxin County, China

In this study, an effective kernel-based learning framework for landslide susceptibility mapping (LSM) is presented through an implementation of support vector machines (SVMs) with different composite kernels. Kernel-based classification methods are very popular in statistical classification and regression analysis because they can effectively address intractable issues such as the curse of dimensionality, limited known samples and noise corruption. The most representative of such methods is the SVM technique. Although SVMs have recently been widely used in LSM, they were defined using only the attribute value of each influencing factor and did not consider the high dependency between the adjacent vector-valued grid cells. This caused a labelling uncertainty. To solve this problem, it is necessary to combine both the influencing factor's attribute features and spatial dependency information in the SVM. In this work, we present two forms of composite kernels to combine the two aforementioned types of information: 1) constructed through a single kernel with stacked vectors; 2) built through summation kernels under different restrictions. The main advantages of the proposed framework are twofold. First, the integration of the two types of information can improve the predictive capability of the SVMs by removing the isolated class noise in the LSM results. Second, other useful information can be extracted from the spatial domain, such as the structural features of grid cells within and outside of landslide areas. The SVM comparisons were based on data from Yongxin County, China, containing 364 past landslide occurrences that were separated randomly into a training set (70%) and a validation set (30%). The geo-environmental setting of the study area was analysed and sixteen influencing factors were selected. The validation of these SVMs was performed using the receiver operating characteristic (ROC) and the area under the ROC curve (AUC). Experimental results demonstrate that all the SVM-based landslide susceptibility maps have similar spatial distributions upon visual inspection. Specifically, the mountainous zones in the north and south of the study area are characterized by high and very high susceptibility values, respectively, whereas the central part of the study area is categorized as the least susceptible zone. Meanwhile, the composite kernel-based learning framework can achieve a better prediction accuracy than the original SVM. From quantitative analysis, the four SVMs with a summation kernel obtain the AUC values above 0.8900, which is 0.0117 higher than that of the original SVM. Furthermore, a weighted scheme in the summation kernel can result in AUC values that are at least 0.0014 higher than a directional scheme.

Benchmark Revision for HOG-SVM Pedestrian Detector Through Reinvigorated Training and Evaluation Methodologies

Histogram of oriented gradient (HOG) features in combination with support vector machine (SVM) classification are still widely reported as a de facto standard benchmark to evaluate contemporary pedestrian detectors employing a vast spectrum of sophisticated features and classification schemes. In this paper, however, it has been shown that these benchmark figures for the original HOG-linear SVM detector grossly underestimate its true working potential. An improved SVM training methodology has been proposed that can make this important detector yield up to 14% lower miss rates than being currently reported on standard datasets. The proposed method accomplishes this by considering only a small set of the most relevant training examples and mitigating the class imbalance problem through manipulation of misclassification cost ratios. The second contribution of this work is the investigation of quantization errors which are induced when fast look-up table (LUT)-based kernel SVM classification is used. Based on the findings, a non-linear quantization scheme for the HOG features has been proposed to facilitate fast evaluation of kernel SVM resulting in up to 4% further reduction in miss rates without incurring additional computational cost. Hence, the linear and kernel SVM-based detectors proposed in this paper not only serve as much more stringent benchmarks for pedestrian detection but also provide the practitioners with an improved training framework for similar detection tasks involving other features and objects. The retrained models have been publicly released and provide a much better replacement for those currently being shipped with OpenCV and MATLAB computer vision libraries.

Hyperspectral image classification based on bilateral filter with linear spatial correlation information

ABSTRACT Support Vector Machine (SVM) with the margin theory is widely used for the hyperspectral classification. However, the margin model is a single interval and does not represent the complete distribution of hyperspectral image data sets. In addition, the spatial texture information obtained by filtering in recent years has become a hot research topic for improving classification of hyperspectral images, but the spatial correlation information is often lost in the spatial texture information extraction. To solve this problem, this paper proposed an algorithm with large margin distribution machine (LDM) that combined the spatial information obtained by the bilateral filter and linear spatial correlation information (BFLSCI-LDM). First, spatial features were extracted by bilateral filter from hyperspectral image whose dimensionality was reduced by principal component analysis. Next, the linear spatial correlation information was constructed for hyperspectral images. Finally, the spatial information and original spectral information were combined for LDM. The experimental results of actual hyperspectral images indicated that the proposed BFLSCI-LDM method was superior to other classification methods, including the original SVM with the raw spectral features, the dimensionality reduction features, and spatial-spectral information, the method of edge-preserving filter and recursive filter, and the LDM-based method.

Combination of Support Vector Machine and K-Fold cross validation to predict compressive strength of concrete in marine environment

Abstract Support Vector Machine (SVM) model optimized by K-Fold cross validation was built to predict and evaluate the degradation of concrete strength in a complicated marine environment. Meanwhile, Artificial Neural Network (ANN) and Decision Tree (DT) were built to compare the prediction precision with SVM model. The result shows SVM model has the best prediction performance. For the original SVM model, there is a problem that input parameters of SVM model are incomplete. Based on the analysis of an amount of research papers, number of input parameters increases from original 8 to 19. After the complementation of input parameters, the prediction precision of SVM model gets a good promotion. The average relative error declines from 34.8% to 27.6%, and the median relative error is reduced from 24.70% to 20.8%. The number of data with relative errors less than 40% has reached 396, almost 80% of 500 testing data.

Improved Blind Spectrum Sensing by Covariance Matrix Cholesky Decomposition and RBF-SVM Decision Classification at Low SNRs

An improved blind spectrum sensing scheme is established by the covariance matrix Cholesky decomposition and radial basis function (RBF)-support vector machine (SVM) decision classification at low signal-to-noise ratios (SNRs). Under strong background noises, the proposed scheme improves the recognition rate of primary users (PUs) than that of the current blind spectrum sensing. First, the ratio of the maximum-to-minimum eigenvalue of a covariance matrix obtained by the Cholesky decomposition is used to construct the statistics. Second, the statistics are labeled with “+1” or “−1,” namely, the energy characteristics of the training samples are extracted and marked with “+1” for PUs and “−1” for noises. Finally, an RBF-SVM classification model, with an intelligent RBF as the SVM kernel function, is obtained by training the above-mentioned statistics and the labels. Thus, the received signals are classified as PUs or not be trained in the SVM model. The threshold possesses self-learning ability, and it distinguishes PU signals from noises effectively. The classification among PU signals and noises is implemented by the optimal SVM decision boundary, derived from maximizing the margin of the decision boundary of trained samples for efficient detection. In addition, the complexity of the statistic construction is lower than that of the conventional maximum minimum eigenvalue (MME). The simulation results show that the RBF in our scheme has 77.5% accuracy at −10 dB, and it outperforms linear kernel function significantly by about 27.5% in accuracy at −10 dB. In addition, the average error probability of the proposed scheme is reduced by about 26% when compared with those of original SVM schemes at −20 dB. The proposed scheme also outperforms the current MME detection in detection probability over 10% at −20 dB. Therefore, the proposed blind spectrum sensing scheme can be efficiently used to detect the PUs by the covariance matrix Cholesky decomposition and the RBF-SVM decision classification in the fifth-generation (5G) communications, especially at low SNRs.

Applying genetic algorithms to set the optimal combination of forest fire related variables and model forest fire susceptibility based on data mining models. The case of Dayu County, China

The main objective of the present study was to utilize Genetic Algorithms (GA) in order to obtain the optimal combination of forest fire related variables and apply data mining methods for constructing a forest fire susceptibility map. In the proposed approach, a Random Forest (RF) and a Support Vector Machine (SVM) was used to produce a forest fire susceptibility map for the Dayu County which is located in southwest of Jiangxi Province, China.For this purpose, historic forest fires and thirteen forest fire related variables were analyzed, namely: elevation, slope angle, aspect, curvature, land use, soil cover, heat load index, normalized difference vegetation index, mean annual temperature, mean annual wind speed, mean annual rainfall, distance to river network and distance to road network. The Natural Break and the Certainty Factor method were used to classify and weight the thirteen variables, while a multicollinearity analysis was performed to determine the correlation among the variables and decide about their usability. The optimal set of variables, determined by the GA limited the number of variables into eight excluding from the analysis, aspect, land use, heat load index, distance to river network and mean annual rainfall.The performance of the forest fire models was evaluated by using the area under the Receiver Operating Characteristic curve (ROC-AUC) based on the validation dataset. Overall, the RF models gave higher AUC values. Also the results showed that the proposed optimized models outperform the original models. Specifically, the optimized RF model gave the best results (0.8495), followed by the original RF (0.8169), while the optimized SVM gave lower values (0.7456) than the RF, however higher than the original SVM (0.7148) model. The study highlights the significance of feature selection techniques in forest fire susceptibility, whereas data mining methods could be considered as a valid approach for forest fire susceptibility modeling.

Anomaly Intrusion Detection Based on A hybrid GSA-SVM Classifier

Support vector machine (SVM) is commonly used in IDSs because of its robustness and efficiency in the network classification. However, Parameter optimization influences the classification accuracy of SVM significantly. In order to improve, a hybrid classifier is designed based on a combination of the GSA and SVM algorithms to optimize the accuracy of the SVM classifier. In the GSASVM classifier, the GSA guides the selection of potential subsets that lead to the best detection accuracy. The GSA-SVM algorithm evaluated using KDD CUP 99 data set and compared to the outperformance of the original SVM algorithms. The results show that the performance of GSA-SVM algorithm has higher detection rate with lower false positive rate.
 Keywords: Network Intrusion Detection, Gravitational Search Algorithm (GSA), support vector machines (SVM)

An Exploration Framework for Efficient High-Level Synthesis of Support Vector Machines: Case Study on ECG Arrhythmia Detection for Xilinx Zynq SoC

In recent years, Support Vector Machine (SVM) classifiers have played a crucial role in providing data fusion and high accuracy classification solutions for various, complex, non-linear problems. Their popularity accompanied by the ever-increasing need of implementing it on computationally weak, portable or even wearable systems has refueled the effort to accelerate their execution. In this paper, we explore FPGA-based acceleration to produce efficient SVM hardware co-processors. We propose a systematic two-level approach for SVM acceleration, which first optimizes the global structure of the original SVM's behavioral description to exploit the data- and instruction-level parallelism and then further refines it through a targeted design exploration that matches the accelerator's memory architecture to its computation and memory access patterns. The proposed methodology has been implemented as a framework on top of Vivado High-Level Synthesis (HLS) tool. We evaluate the effectiveness of the methodology through a rich set of analysis and validation results, which show that its adoption delivers SVM accelerator designs achieving latency gains of up to 98.78 % in respect to Vivado-HLS default optimized solution. Finally, using as a case study an ECG analysis and Arrhythmia detection system we show that a target Zynq programmable SoC utilizing the optimized SVM accelerator design outperforms pure software implementations in numerous single or dual core target platforms, achieving speedups, which range from 10 × up to 78 ×.

Dempster-Shafer evidence theory for multi-bearing faults diagnosis

Support vector machines (SVMs) are frequently used in automated machinery faults diagnosis to classify multiple machinery faults by handling a high number of input features with low sampling data sets. SVMs are well known for fault detection that involves binary fault classifications only (i.e., healthy vs. faulty). However, when SVMs are used for multi-faults diagnostics and classification, they result in a drop in classification accuracy; this is because the adaptation of SVMs for multi-faults classifications requires the reduction of the multiple classification problem into multiple subsets of binary classification problems that result in many contradictory results from each individual SVM model. To overcome this problem, a novel SVM-DS (Dempster-Shafer evidence theory) model is proposed to resolve conflicting results generated from each SVM model and thus increase the classification accuracy. The analysis of results shows that the proposed SVM-DS model increased the accuracy of the fault diagnosis model from 76% to 94%, as SVM-DS continuously refines and eliminates all conflicting results from the original SVM model. The proposed SVM-DS model is found to be more accurate and effective in handling multi-faults diagnostic and classification problems commonly faced in the industries, as compared to the original SVM method.

ELECTROCARDIOGRAM ARRHYTHMIA CLASSIFICATION SYSTEM USING SUPPORT VECTOR MACHINE BASED FUZZY LOGIC

Arrhythmia is a cardiovascular disease that can be diagnosed by doctors using an electrocardiogram (ECG). The information contained on the ECG is used by doctors to analyze the electrical activity of the heart and determine the type of arrhythmia suffered by the patient. In this study, ECG arrhythmia classification process was performed using Support Vector Machine based fuzzy logic. In the proposed method, fuzzy membership functions are used to cope with data that are not classifiable in the method of Support Vector Machine (SVM) one-against-one. An early stage of the data processing is the baseline wander removal process on the original ECG signal using Transformation Wavelet Discrete (TWD). Afterwards then the ECG signal is cleaned from the baseline wander segmented into units beat. The next stage is to look for six features of the beat. Every single beat is classified using SVM method based fuzzy logic. Results from this study show that ECG arrhythmia classification using proposed method (SVM based fuzzy logic) gives better results than original SVM method. ECG arrhythmia classification using SVM method based fuzzy logic forms an average value of accuracy level, sensitivity level, and specificity level of 93.5%, 93.5%, and 98.7% respectively. ECG arrhythmia classification using only SVM method forms an average value accuracy level, sensitivity level, and specificity level of 91.83%, 91.83%, and 98.36% respectively.

Fed-batch fermentation penicillin process fault diagnosis and detection based on support vector machine

With the increase of scale and complexity of modern chemical process, fault diagnosis and detection are playing crucial roles in process monitoring. Accidents can be avoided if faults can be detected and excluded in time. In this paper, Principal Components Analysis (PCA) and Recursive Feature Elimination (RFE) are combined with Support Vector Machine (SVM) for fault diagnosis and detection. Specifically, the original SVM, PCA-SVM and SVM-RFE are respectively utilized to identify three faults from the simulation of Fed-Batch Fermentation Penicillin (FBFP) process. Experimental results show that PCA-SVM and SVM-RFE perform better than the original SVM, and the fault detection schemes based on PCA-SVM and SVM-RFE generate satisfactory results.

A Regularized Monotonic Fuzzy Support Vector Machine Model for Data Mining With Prior Knowledge

Incorporating prior knowledge into data mining is an interesting but challenging problem, and this study proposes a novel fuzzy support vector machine (SVM) model to explore this issue. It considers the fact that in many applications, each input point may not be exactly labeled as one particular class, and thus, it applies a fuzzy membership to each input point. It also utilizes expert knowledge concerning the monotonic relations between the response and predictor variables, which is represented in the form of monotonicity constraints. We formulate the classification problem of a monotonically constrained fuzzy SVM, called a monotonic FSVM, derive its dual optimization problem, and theoretically analyze its monotonic property. The Tikhonov regularization method is further applied to ensure that the solution is unique and bounded. A new measure, i.e., the frequency monotonicity rate, is proposed to evaluate the ability of the model to retain the monotonicity. The results of the experiments on real-world and synthetic datasets show that this method, which considers different contributions of each data and the prior knowledge of the monotonicity, has a number of advantages with regard to predictive ability and retaining monotonicity over the original FSVM and SVM models when applied to classification problems.

Proximal gradient method for huberized support vector machine

The Support Vector Machine (SVM) has been used in a wide variety of classification problems. The original SVM uses the hinge loss function, which is non-differentiable and makes the problem difficult to solve in particular for regularized SVMs, such as with $\ell_1$-regularization. This paper considers the Huberized SVM (HSVM), which uses a differentiable approximation of the hinge loss function. We first explore the use of the Proximal Gradient (PG) method to solving binary-class HSVM (B-HSVM) and then generalize it to multi-class HSVM (M-HSVM). Under strong convexity assumptions, we show that our algorithm converges linearly. In addition, we give a finite convergence result about the support of the solution, based on which we further accelerate the algorithm by a two-stage method. We present extensive numerical experiments on both synthetic and real datasets which demonstrate the superiority of our methods over some state-of-the-art methods for both binary- and multi-class SVMs.