Sparse Support Vector Machine Research Articles

Sparse and robust support vector machine with capped squared loss for large-scale pattern classification

Support vector machine (SVM), being considered one of the most efficient tools for classification, has received widespread attention in various fields. However, its performance is hindered when dealing with large-scale pattern classification tasks due to high memory requirements and running very slow. To address this challenge, we construct a novel sparse and robust SVM based on our newly proposed capped squared loss (named as Lcsl-SVM). To solve Lcsl-SVM, we first focus on establishing optimality theory of Lcsl-SVM via our defined proximal stationary point, which is convenient for us to efficiently characterize the Lcsl support vectors of Lcsl-SVM. We subsequently demonstrate that the Lcsl support vectors comprise merely a minor fraction of entire training data. This observation leads us to introduce the concept of the working set. Furthermore, we design a novel subspace fast algorithm with working set (named as Lcsl-ADMM) for solving Lcsl-SVM, which is proven that Lcsl-ADMM has both global convergence and relatively low computational complexity. Finally, numerical experiments show that Lcsl-ADMM has excellent performances in terms of getting the best classification accuracy, using the shortest time and presenting the smallest numbers of support vectors when solving large-scale pattern classification problems.

Voltage Stability Constrained Economic Dispatch for Multi-Infeed HVDC Power Systems

Multi-infeed LCC-HVDC decreases voltage stability margins due to its current source nature. Moreover, the high penetration of variable renewable energy makes it difficult to reserve enough margins for all periods. Therefore, voltage stability constraints need to be considered in the scheduling stage. However, the challenge is that the voltage stability margin is not analytic with respect to state variables. We propose a sparse support vector machine to extract the nonlinear voltage stability rule. To embed the rule into economic dispatch (ED), we apply a convex reformulating technique to the rule and then formulate a voltage stability constrained ED (VSCED) model by semi-definite programming. The model is tested on a modified IEEE-14 system and a real-world system from Jiangsu Province, China; and it is compared with other typical data-driven-based models in the Jiangsu system. The numerical experiments suggest that the stability margins can be effectively reserved above the required threshold within an acceptable computation time by the proposed VSCED model, which outperforms the other models.

Combining a forward supervised filter learning with a sparse NMF for breast cancer histopathological image classification

Histopathological images play a important role in clinical diagnosis, particularly in identifying and assessing the severity of abnormal conditions like benign lesions and malignant tumors. Traditional machine learning techniques for processing histopathology images involve the extraction of manual features from these images, which is typically done with the assistance of industry experts. Recent advancements in Deep Learning (DL), especially with Convolutional Neural Networks (CNN), have enabled the automatic extraction of multi-level abstract features directly from raw data. This capability significantly enhances the performance of complex computer vision tasks. Classic CNN models like AlexNet and VggNet employ back-propagation algorithms to learn filters in the training phase. However, these algorithms demand large labeled datasets, resulting in extensive computational processing. Additionally, they often face the vanishing gradient problem, which can negatively impact the quality of the learning process. Besides, in many domains, acquiring enough labeled images for conducting properly the training phase is a real challenge. To address these challenges, a feed-forward propagation approach was proposed using Non-Negative Matrix Factorization(NMF). The NMF technique factorizes the input data into two latent factors (non-negative matrices). It has been shown that by enforcing constraints such as sparsity on the latent factors, dominant features that are mostly correlated with tumors types can be extracted. In this work, a novel model combining sparse NMF and Support Vector Machine (SVM) was developed for classifying histopathological images. We have derived a mathematical model of a novel feed-forward filter learning approach that combines sparse NMF (SNMF) and Support Vector Machine technique (SVM). The model was used to design and implement a feed-forward CNN classifier to classify histopathology images. This model has been evaluated on the histopathology images from Sultan Qaboos University Hospital (SQUH dataset) and the public BreaKHis dataset. The experiments we have conducted demonstrate the efficiency of the proposed model, especially on small-sized SQUH datasets achieving an AUC of 0.90, 0.89, 0.85, and 0.86 on 4x,10x, 20x, and 40x magnifications, respectively, and achieving an AUC of 0.95 BreaKHis dataset.

Sparse additive support vector machines in bounded variation space

Abstract We propose the total variation penalized sparse additive support vector machine (TVSAM) for performing classification in the high-dimensional settings, using a mixed $l_{1}$-type functional regularization scheme to induce sparsity and smoothness simultaneously. We establish a representer theorem for TVSAM, which turns the infinite-dimensional problem into a finite-dimensional one, thereby providing computational feasibility. Even for the least squares loss, our result fills a gap in the literature when compared with the existing representer theorem. Theoretically, we derive some risk bounds for TVSAM under both exact sparsity and near sparsity, and with arbitrarily specified internal knots. In this process, we develop an important interpolation inequality for the space of functions of bounded variation, relying on analytic techniques such as mollification and partition of unity. An efficient implementation based on the alternating direction method of multipliers is employed.

A new fast ADMM for kernelless SVM classifier with truncated fraction loss

Support vector machine (SVM) is one of well-known supervised machine learning classifier and is used widely in image classification, pattern recognition, disease diagnosis, etc. However, the high computational complexity is a major issue for large-scale classification problems. To address this issue, we propose a new truncated fraction loss function to construct a new sparse and robust SVM model (Ltf-SVM). By newly developed support vectors and a working set based on the proximal operator of truncated fraction loss, we prove the sparsity and robustness to outliers of Ltf-SVM from a theoretical perspective. When it comes to solve Ltf-SVM problems, we develop an alternating direction method of multipliers with a working set for addressing Ltf-SVM. The proposed algorithm is proven to converge globally and enjoys very low computational complexity. Moreover, numerical experiments verify its superior performance with respect to its high accuracy and high computational speed in comparison with other leading solvers.

희박한 최소 절대 편차 지지벡터기계

The support vector machine solves a quadratic programming problem with linear inequality and equality constraints. However, it is not trivial to solve the quadratic problem. The least squares support vector machine(LS-SVM) solves a linear system by equality constraints instead of inequality constraints. LS-SVM is a popular method in regression and classification problems, because it effectively solves simple linear systems. There are two issues with the LS-SVM solution : the lack of robustness to outliers and the absence of sparseness. In this paper, we propose a sparse and robust support vector machine for regression problems using the least absolute deviation support vector machine (LAD-SVM) and recursive reduced LS-SVM (RR-LS-SVM). The split-Bregman iteration gives the exact solution for the LAD-SVM problem, while RR-LS-SVM gives a sparse solution with a much smaller number of all support vectors. Numerical experiments with simulation and benchmark data demonstrate that the proposed algorithm can achieve comparable performance to other methods in terms of robustness and sparseness.

A Duo Autoencoder-SVM based Approach for Secure Performance Monitoring of Industrial Conveyor Belt System

Process industries are fascinated by cyber-physical systems because of the potential to integrate physical systems and the cyber realm, resulting in efficient remote monitoring and control. The conveyor belt system has many critical parameters that require continuous attention, necessitating cyber-physical remote monitoring. Due to cloud-based monitoring of parameters, the system is vulnerable to cyber threats. The proposed technique combines a sparse autoencoder and support vector machine (SVM) to detect false data injection attacks (FDIAs) in the presence of sensor bias fault. The sparse autoencoder extracts sparse features and learns anomaly-free dynamics from the input sensor readings. Then, the trained SVM distinguishes attacks and fault by analysing reconstruction residuals of each measurement reading. The residuals also give an idea about the magnitude of abnormality. The proposed method's efficacy is evaluated in terms of accuracy, precision and false-alarm rate with the help of fault and FDIAs models.

Machine learning models for predicting depression in Korean young employees.

The incidence of depression among employees has gradually risen. Previous studies have focused on predicting the risk of depression, but most studies were conducted using basic statistical methods. This study used machine learning algorithms to build models that detect and identify the important factors associated with depression in the workplace. A total of 503 employees completed an online survey that included questionnaires on general characteristics, physical health, job-related factors, psychosocial protective, and risk factors in the workplace. The dataset contained 27 predictor variables and one dependent variable which referred to the status of employees (normal or at the risk of depression). The prediction accuracy of three machine learning models using sparse logistic regression, support vector machine, and random forest was compared with the accuracy, precision, sensitivity, specificity, and AUC. Additionally, the important factors identified via sparse logistic regression and random forest. All machine learning models demonstrated similar results, with the lowest accuracy obtained from sparse logistic regression and support vector machine (86.8%) and the highest accuracy from random forest (88.7%). The important factors identified in this study were gender, physical health, job, psychosocial protective factors, and psychosocial risk and protective factors in the workplace. The results of this study indicated the potential of machine learning models to accurately predict the risk of depression among employees. The identified factors that influence the risk of depression can contribute to the development of intelligent mental healthcare systems that can detect early signs of depressive symptoms in the workplace.

Sparse and robust SVM classifier for large scale classification

Sparse and robust SVM classifier for large scale classification

Sparse L1-norm quadratic surface support vector machine with Universum data

Sparse L1-norm quadratic surface support vector machine with Universum data

Fast truncated Huber loss SVM for large scale classification

Support vector machine (SVM), as a useful tool of classification, has been widely applied in many fields. However, it may incur computationally infeasibility on very large sample datasets. To handle this problem, this paper presents a novel sparse and robust SVM model with truncated Huber loss, which is known as Lth-SVM. To this end, we establish a first-order optimality condition of Lth-SVM based on the newly introduced P-stationary point, which allows us to define Lth support vectors and working set of Lth-SVM and proves that all of the Lth support vectors are a small portion of the whole training set. Furthermore, we develop a novel efficient alternating direction method of multipliers with working set (dubbed as Lth-ADMM) to address Lth-SVM, which is proven that sequence produced by Lth-ADMM is proved to be a local minimizer of Lth-SVM and enjoys a relatively low computational complexity if the size of the working set is very small. We compare Lth-ADMM with other nine state-of-the-art solvers on both synthetic and real datasets. The extensive numerical experiments demonstrate that Lth-ADMM is capable of delivering higher prediction accuracy, presenting a smaller number of support vectors and running quickly, especially, in large-scale datasets setting.

DCA for Sparse Quadratic Kernel-Free Least Squares Semi-Supervised Support Vector Machine

With the development of science and technology, more and more data have been produced. For many of these datasets, only some of the data have labels. In order to make full use of the information in these data, it is necessary to classify them. In this paper, we propose a strong sparse quadratic kernel-free least squares semi-supervised support vector machine (SSQLSS3VM), in which we add a ℓ0norm regularization term to make it sparse. An NP-hard problem arises since the proposed model contains the ℓ0 norm and another nonconvex term. One important method for solving the nonconvex problem is the DC (difference of convex function) programming. Therefore, we first approximate the ℓ0 norm by a polyhedral DC function. Moreover, due to the existence of the nonsmooth terms, we use the sGS-ADMM to solve the subproblem. Finally, empirical numerical experiments show the efficiency of the proposed algorithm.

A Deep Learning Approach to Network Intrusion Detection Using a Proposed Supervised Sparse Auto-encoder and SVM

A Deep Learning Approach to Network Intrusion Detection Using a Proposed Supervised Sparse Auto-encoder and SVM

Sparse discriminant twin support vector machine for binary classification

Sparse discriminant twin support vector machine for binary classification

Biomarker selection and a prospective metabolite-based machine learning diagnostic for lyme disease

We provide a pipeline for data preprocessing, biomarker selection, and classification of liquid chromatography–mass spectrometry (LCMS) serum samples to generate a prospective diagnostic test for Lyme disease. We utilize tools of machine learning (ML), e.g., sparse support vector machines (SSVM), iterative feature removal (IFR), and k-fold feature ranking to select several biomarkers and build a discriminant model for Lyme disease. We report a 98.13% test balanced success rate (BSR) of our model based on a sequestered test set of LCMS serum samples. The methodology employed is general and can be readily adapted to other LCMS, or metabolomics, data sets.

Numerical and Experimental Evaluation of Structural Changes Using Sparse Auto-Encoders and SVM Applied to Dynamic Responses

The present work evaluates the deep learning algorithm called Sparse Auto-Encoder (SAE) when applied to the characterization of structural anomalies. This study explores the SAE’s performance in a supervised damage detection approach to consolidate its application in the Structural Health Monitoring (SHM) field, especially when dealing with real-case structures. The main idea is to use the SAE to extract relevant features from the monitored signals and the well-known Support Vector Machine (SVM) to classify such characteristics within the context of an SHM problem. Vibration data from a numerical beam model and a highway viaduct in Brazil are considered to assess the proposed approach. In both analyzed examples, the efficiency of the implemented methodology achieved more than 99% of correct damage structural classifications, supporting the conclusion that SAE can extract relevant characteristics from dynamic signals that are useful for SHM applications.

Sparse classification: a scalable discrete optimization perspective

We formulate the sparse classification problem of n samples with p features as a binary convex optimization problem and propose a outer-approximation algorithm to solve it exactly. For sparse logistic regression and sparse SVM, our algorithm finds optimal solutions for n and p in the 10,000 s within minutes. On synthetic data our algorithm achieves perfect support recovery in the large sample regime. Namely, there exists an \(n_0\) such that the algorithm takes a long time to find an optimal solution and does not recover the correct support for \(n<n_0\), while for \(n\geqslant n_0\), the algorithm quickly detects all the true features, and does not return any false features. In contrast, while Lasso accurately detects all the true features, it persistently returns incorrect features, even as the number of observations increases. Consequently, on numerous real-world experiments, our outer-approximation algorithms returns sparser classifiers while achieving similar predictive accuracy as Lasso. To support our observations, we analyze conditions on the sample size needed to ensure full support recovery in classification. For k-sparse classification, and under some assumptions on the data generating process, we prove that information-theoretic limitations impose \(n_0 < C \left( 2 + \sigma ^2\right) k \log (p-k)\), for some constant \(C>0\).

The Sparse Learning of The Support Vector Machine

The sparse model plays an important role in many aeras, such as in the machine learning, image processing and signal processing. The sparse model has the ability of variable selection, so they can solve the over-fitting problem. The sparse model can be introduced into the field of support vector machine in order to get classification of the labels and sparsity of the variables simultaneously. This paper summarizes various sparse support vector machines. Finally, we revealed the research directions of the sparse support vector machines in the future.

Temporal-Spatial Feature Extraction of DSA Video and Its Application in AVM Diagnosis.

Objectives: Brain arteriovenous malformation (AVM) is one of the most common causes of intracranial hemorrhage in young adults, and its expeditious diagnosis on digital subtraction angiography (DSA) is essential for clinical decision-making. This paper firstly proposed a deep learning network to extract vascular time-domain features from DSA videos. Then, the temporal features were combined with spatial radiomics features to build an AVM-assisted diagnosis model.Materials and method: Anteroposterior position (AP) DSA videos from 305 patients, 153 normal and 152 with AVM, were analyzed. A deep learning network based on Faster-RCNN was proposed to track important vascular features in DSA. Then the appearance order of important vascular structures was quantified as the temporal features. The structure distribution and morphological features of vessels were quantified as 1,750 radiomics features. Temporal features and radiomics features were fused in a classifier based on sparse representation and support vector machine. An AVM diagnosis and grading system that combined the temporal and spatial radiomics features of DSA was finally proposed. Accuracy (ACC), sensitivity (SENS), specificity (SPEC), and area under the receiver operating characteristic curve (AUC) were calculated to evaluate the performance of the radiomics model.Results: For cerebrovascular structure detection, the average precision (AP) was 0.922, 0.991, 0.769, 0.899, and 0.929 for internal carotid artery, Willis circle, vessels, large veins, and venous sinuses, respectively. The mean average precision (mAP) of five time phases was 0.902. For AVM diagnosis, the models based on temporal features, radiomics features, and combined features achieved AUC of 0.916, 0.918, and 0.942, respectively. In the AVM grading task, the proposed combined model also achieved AUC of 0.871 in the independent testing set.Conclusion: DSA videos provide rich temporal and spatial distribution characteristics of cerebral blood vessels. Clinicians often interpret these features based on subjective experience. This paper proposes a scheme based on deep learning and traditional machine learning, which effectively integrates the complex spatiotemporal features in DSA, and verifies the value of this scheme in the diagnosis of AVM.

Sparse elastic net multi-label rank support vector machine with pinball loss and its applications

Multi-label rank support vector machine (RankSVM) is an effective technique to deal with multi-label classification problems, which has been widely used in various fields. However, it is sensitive to noise points and cannot delete redundant features for high dimensional problems. Therefore, to address the above two limitations, a sparse elastic net multi-label RankSVM with pinball loss (pin-ENR) is first proposed in this paper. On the one hand, pinball loss is employed to enhance the robustness. On the other hand, it adopts the sparse elastic net regularization, so that it can do variable selection. However, it still has challenges for large-scale problems with a huge number of features, samples, and labels. Therefore, motivated by the sparsity of pin-ENR, a safe simultaneous feature and label-pair elimination rule is further constructed for accelerating pin-ENR, which is termed as FLER-pin-ENR. Its main idea is to delete a large number of inactive features and label-pairs simultaneously before training without sacrificing accuracy. Numerical experiments on four synthetic and seven benchmark datasets demonstrate the feasibility and validity. Moreover, we apply our FLER-pin-ENR to the diagnosis of diabetes complications and the natural scene image classification problems, which further verifies the practicability of our proposed method.