1-norm Support Vector Machine Research Articles

Abstract 7392: Machine learning-based identification of H&E-derived morphologic features associated with CD8+ T cell immune exclusion

Abstract Background: Immune exclusion is characterized by a predominance of CD8+ T cells in the stroma of the tumor microenvironment (TME) which are not in contact with tumor cells. Immune exclusion is a significant obstacle to effective cancer immunotherapy and reversing it has emerged as a new therapeutic opportunity. However, the objective assessment of immune exclusion remains a challenge due to the complexity of the TME and the need for immunostaining to quantify CD8+ cells in tumor and stroma. While artificial intelligence (AI) models applied to H&E images can quantify spatial tissue and cell features at scale, they are unable to identify the molecular subtypes of cells. Here we describe the development of a machine learning model that classifies tumors by their CD8-defined immune phenotypes using features extracted from H&E whole slide images alone. Methods: Al-powered TME models (PathAI PathExplore™) were deployed on H&E images from 39 non-small lung cancer (NSCLC) and triple-negative breast cancer (TNBC) samples. CD8-based assessment of immune phenotypes was done using an adjacent section. 115 H&E-derived features from each of the 25 tumor samples (12 NSCLC, 13 TNBC) classified as immune excluded and 7 tumor samples (3 NSCLC, 4 TNBC) classified as immune infiltrated were used as input into a feature selection process to identify the subset of H&E features predictive of immune phenotype. The feature selection process was driven by an ensemble L1 norm support vector machine (Ens-L1-SVM) model containing 500 L1-SVM base models each trained by 80% of the data. The degree of immune exclusion was determined in the feature space by the distance to the separation hyperplane defined by the model. Out-of-bag error estimation was employed on the remaining 20% of data from each base model to assess model performance. Results: The Ens-L1-SVM model identified 6 H&E features predictive of CD8-defined immune phenotypes, including the density ratio of lymphocytes to macrophages in the cancer epithelium, the density of fibroblasts, and the spatial distributions of cancer cells, macrophages, and lymphocytes. Immune phenotypes were classified by the Ens-L1-SVM model with an estimated accuracy of 97%. Additionally, all selected features and their direction of association aligned with prior knowledge of the mechanisms or manifestations of immune exclusion or infiltration in the TME. Conclusion: This work illustrated that morphologic features derived from H&E images using AI-powered models can be effective predictors of CD8-defined immune exclusion, providing an option for patient stratification by immune phenotype using widely-available H&E images. While the Ens-L1-SVM model showed robust performance on a small number of samples and is expected to have good generalizability due to the linear nature of the model, additional validation across diverse datasets and various tumor indications is needed. Citation Format: Yanchao Wang, Fredrick D. Gootkind, Florent Peyraud, Jean-Philippe Guégan, Antoine Italiano, G. Travis Clifton, Laura A. Dillon, Xinwei Sher. Machine learning-based identification of H&E-derived morphologic features associated with CD8+ T cell immune exclusion [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 7392.

A Mixed Integer Linear Programming Support Vector Machine for Cost-Effective Group Feature Selection: Branch-Cut-and-Price Approach

Recently, cost-based feature selection has received significant attention due to its great ability to achieve promising prediction accuracy at a minimum feature acquisition cost. To further improve its predictive and economic performances, this research proposes a cost-effective 1-norm support vector machine with group feature selection as GFS-CESVM1. Its robust counterpart model, GFS-RCESVM1, is also introduced to address the cost uncertainty of features and feature groups because cost variation commonly exists in real-world problems. The proposed models are formulated as Mixed Integer Linear Programming (MILP). To efficiently solve the proposed SVM MILP models, we develop a Branch-Cut-and-Price (BCP) algorithm that considers only a limited number of variables and/or constraints, which thereby leads to rapid convergence to an optimal solution. Various experimental results on benchmark and synthetic datasets demonstrate that GFS-CESVM1 can achieve competitive outcomes by considering not only individual feature evaluation but also group structural information among features. The GFS-RCESVM1 can identify the subset of features that is immune to cost uncertainty and therefore provide feasible and optimal solutions. Furthermore, our BCP algorithm can dominantly outperform the ordinary BB algorithm for finding better objective value and integrality gap within a short period of time.

Cancer Diagnosis and Disease Gene Identification via Statistical Machine Learning

Diagnosing cancer and identifying the disease gene by using DNA microarray gene expression data are the hot topics in current bioinformatics. This paper is devoted to the latest development in cancer diagnosis and gene selection via statistical machine learning. A support vector machine is firstly introduced for the binary cancer diagnosis. Then, 1-norm support vector machine, doubly regularized support vector machine, adaptive huberized support vector machine and other extensions are presented to improve the performance of gene selection. Lasso, elastic net, partly adaptive elastic net, group lasso, sparse group lasso, adaptive sparse group lasso and other sparse regression methods are also introduced for performing simultaneous binary cancer classification and gene selection. In addition to introducing three strategies for reducing multiclass to binary, methods of directly considering all classes of data in a learning model (multi_class support vector, sparse multinomial regression, adaptive multinomial regression and so on) are presented for performing multiple cancer diagnosis. Limitations and promising directions are also discussed.

A mixed integer linear programming support vector machine for cost-effective feature selection

In the era of big data, feature selection is indispensable as a dimensional reduction technique to lower data complexity and enhance machine learning performances. However, traditional feature selection methods mainly focus on classification performances, while they exclude the impact of associated feature costs; e.g., price, risk, and computational complexity for feature acquisition. In this research, we extend the ℓ1 norm support vector machine (ℓ1-SVM) to address the feature costs, by incorporating a budget constraint to preserve classification accuracy with the least expensive features. Furthermore, we formulate its robust counterpart to address the uncertainty of the feature costs. To enhance computational efficiency, we also develop an algorithm to tighten the bound of the weight vector in the budget constraint. Through the experimental study on a variety of benchmark and synthetic datasets, our proposed mixed integer linear programming (MILP) models show that they can achieve competitive outcomes in terms of predictive and economic performances. Also, the algorithm that tightens the budget constraint helps to curtail computational complexity.

Efficient detailed design optimization of topology optimization concepts by using support vector machines and metamodels

ABSTRACTIn this article, an approach for metamodel-based design optimization (MBDO) of topology optimization (TO) concepts is proposed by using support vector machines (SVMs) as geometric models of the concepts instead of traditional parametric computer aided design (CAD) models. In such a manner, an efficient approach for the MBDO-driven design of TO-based concepts is obtained. An implicit hypersurface representing the TO-based concept is generated by classifying the TO-solutions of zeros and ones by using the 1-norm SVM of Mangasarian. The implicit SVM-based hypersurfaces are then utilized to set up designs of experiments of nonlinear finite element analyses by morphing the TO-based concepts by using Boolean and blending operations. Finally, MBDO is performed by using an ensemble of metamodels consisting of quadratic regression, Kriging, radial basis function networks, polynomial chaos expansion and support vector regression models. The proposed MBDO framework is demonstrated by minimizing the mass of a three-dimensional design domain with a constraint on the plastic limit load. The performance of the approach is most promising.

Statistical margin error bounds for L1-norm support vector machines

Comparing with Lp-norm (1<p<+∞) Support Vector Machines (SVMs), the L1-norm SVM enjoys the nice property of simultaneously performing classification and feature selection. Margin error bounds for SVM on Hilbert spaces (or on more general q-uniformly smooth Banach spaces) have been obtained in the literature to justify the strategy of maximizing the margin in SVM. In this paper, we devote to estimating the margin error bound for L1-norm SVM methods and giving a geometrical interpretation for the result. We show that the fat-shattering dimension of the Banach spaces ℓ1 and ℓ∞ are both infinite. Therefore, we establish margin error bounds for the SVM on finite dimensional spaces with L1-norm, thus supplying statistical justification for the large margin classification of L1-norm SVM on finite dimensional spaces. To complete the theory, corresponding results for the L∞-norm SVM are also presented.

Random Projection for Linear 2-norm Support Vector Machine

The linear 2-norm Support Vector Machine (L2-SVM) builds a hyper-plane which maximizes the 2-norm soft margin. Random projection is a new oblivious feature extraction and dimension reduction method. Both techniques are widely applied in compressed sensing, texture classification, face recognition, and so on. This paper find that random projection can be applied to any input matrix of L2-SVM. Furthermore, we proves that the geometric margin and the minimum enclosing ball in the projected space are almost unchanged with high probability compared with those in the original space. The result is demonstrated by experiments on synthetic and real data. Computational experiments also show that the proposed random projection for L2-SVM has a better performance than random projection for L1-SVM. Besides, the proposed model can solve the large scale classification problems more effectively and efficiently. Moreover, the proposed method is compared with Principal Component Analysis and is applied in Corel images classification problems.

Applying 1-norm SVM with squared loss to gene selection for cancer classification

Gene selection methods available have high computational complexity. This paper applies an 1-norm support vector machine with the squared loss (1-norm SVMSL) to implement fast gene selection for cancer classification. The 1-norm SVMSL, a variant of the 1-norm support vector machine (1-norm SVM) has been proposed. Basically, the 1-norm SVMSL can perform gene selection and classification at the same. However, to improve classification performance, we only use the 1-norm SVMSL as a gene selector, and adopt a subsequent classifier to classify the selected genes. We perform extensive experiments on four DNA microarray data sets. Experimental results indicate that the 1-norm SVMSL has a very fast gene selection speed compared with other methods. For example, the 1-norm SVMSL is almost an order of magnitude faster than the 1-norm SVM, and at least four orders of magnitude faster than SVM-RFE (recursive feature elimination), a state-of-the-art method.

Divide-and-conquer for debiased l1-norm support vector machine in ultra-high dimensions

1-norm support vector machine (SVM) generally has competitive performance compared to standard 2-norm support vector machine in classification problems, with the advantage of automatically selectin...

Stable feature selection based on the ensemble L 1 -norm support vector machine for biomarker discovery.

BackgroundLately, biomarker discovery has become one of the most significant research issues in the biomedical field. Owing to the presence of high-throughput technologies, genomic data, such as microarray data and RNA-seq, have become widely available. Many kinds of feature selection techniques have been applied to retrieve significant biomarkers from these kinds of data. However, they tend to be noisy with high-dimensional features and consist of a small number of samples; thus, conventional feature selection approaches might be problematic in terms of reproducibility.ResultsIn this article, we propose a stable feature selection method for high-dimensional datasets. We apply an ensemble L 1-norm support vector machine to efficiently reduce irrelevant features, considering the stability of features. We define the stability score for each feature by aggregating the ensemble results, and utilize backward feature elimination on a purified feature set based on this score; therefore, it is possible to acquire an optimal set of features for performance without the need to set a specific threshold. The proposed methodology is evaluated by classifying the binary stage of renal clear cell carcinoma with RNA-seq data.ConclusionA comparison with established algorithms, i.e., a fast correlation-based filter, random forest, and an ensemble version of an L 2-norm support vector machine-based recursive feature elimination, enabled us to prove the superior performance of our method in terms of classification as well as stability in general. It is also shown that the proposed approach performs moderately on high-dimensional datasets consisting of a very large number of features and a smaller number of samples. The proposed approach is expected to be applicable to many other researches aimed at biomarker discovery.

Binary classification SVM-based algorithms with interval-valued training data using triangular and Epanechnikov kernels

Classification algorithms based on different forms of support vector machines (SVMs) for dealing with interval-valued training data are proposed in the paper. L2-norm and L∞-norm SVMs are used for constructing the algorithms. The main idea allowing us to represent the complex optimization problems as a set of simple linear or quadratic programming problems is to approximate the Gaussian kernel by the well-known triangular and Epanechnikov kernels. The minimax strategy is used to choose an optimal probability distribution from the set and to construct optimal separating functions. Numerical experiments illustrate the algorithms.

Machine learning identification of EEG features predicting working memory performance in schizophrenia and healthy adults.

BackgroundWith millisecond-level resolution, electroencephalographic (EEG) recording provides a sensitive tool to assay neural dynamics of human cognition. However, selection of EEG features used to answer experimental questions is typically determined a priori. The utility of machine learning was investigated as a computational framework for extracting the most relevant features from EEG data empirically.MethodsSchizophrenia (SZ; n = 40) and healthy community (HC; n = 12) subjects completed a Sternberg Working Memory Task (SWMT) during EEG recording. EEG was analyzed to extract 5 frequency components (theta1, theta2, alpha, beta, gamma) at 4 processing stages (baseline, encoding, retention, retrieval) and 3 scalp sites (frontal-Fz, central-Cz, occipital-Oz) separately for correctly and incorrectly answered trials. The 1-norm support vector machine (SVM) method was used to build EEG classifiers of SWMT trial accuracy (correct vs. incorrect; Model 1) and diagnosis (HC vs. SZ; Model 2). External validity of SVM models was examined in relation to neuropsychological test performance and diagnostic classification using conventional regression-based analyses.ResultsSWMT performance was significantly reduced in SZ (p < .001). Model 1 correctly classified trial accuracy at 84 % in HC, and at 74 % when cross-validated in SZ data. Frontal gamma at encoding and central theta at retention provided highest weightings, accounting for 76 % of variance in SWMT scores and 42 % variance in neuropsychological test performance across samples. Model 2 identified frontal theta at baseline and frontal alpha during retrieval as primary classifiers of diagnosis, providing 87 % classification accuracy as a discriminant function.ConclusionsEEG features derived by SVM are consistent with literature reports of gamma’s role in memory encoding, engagement of theta during memory retention, and elevated resting low-frequency activity in schizophrenia. Tests of model performance and cross-validation support the stability and generalizability of results, and utility of SVM as an analytic approach for EEG feature selection.

Hierarchically penalized support vector machine with grouped variables

When input features are naturally grouped or generated by factors in a linear classification problem, it is more meaningful to identify important groups or factors rather than individual features. The F∞-norm support vector machine (SVM) and the group lasso penalized SVM have been developed to perform simultaneous classification and factor selection. However, these group-wise penalized SVM methods may suffer from estimation inefficiency and model selection inconsistency because they cannot perform feature selection within an identified group. To overcome this limitation, we propose the hierarchically penalized SVM (H-SVM) that not only effectively identifies important groups but also removes irrelevant features within an identified group. Numerical results are presented to demonstrate the competitive performance of the proposed H-SVM over existing SVM methods.

Large margin classification with indefinite similarities

Classification with indefinite similarities has attracted attention in the machine learning community. This is partly due to the fact that many similarity functions that arise in practice are not symmetric positive semidefinite, i.e. the Mercer condition is not satisfied, or the Mercer condition is difficult to verify. Examples of such indefinite similarities in machine learning applications are ample including, for instance, the BLAST similarity score between protein sequences, human-judged similarities between concepts and words, and the tangent distance or the shape matching distance in computer vision. Nevertheless, previous works on classification with indefinite similarities are not fully satisfactory. They have either introduced sources of inconsistency in handling past and future examples using kernel approximation, settled for local-minimum solutions using non-convex optimization, or produced non-sparse solutions by learning in Krein spaces. Despite the large volume of research devoted to this subject lately, we demonstrate in this paper how an old idea, namely the 1-norm support vector machine (SVM) proposed more than 15 years ago, has several advantages over more recent work. In particular, the 1-norm SVM method is conceptually simpler, which makes it easier to implement and maintain. It is competitive, if not superior to, all other methods in terms of predictive accuracy. Moreover, it produces solutions that are often sparser than more recent methods by several orders of magnitude. In addition, we provide various theoretical justifications by relating 1-norm SVM to well-established learning algorithms such as neural networks, SVM, and nearest neighbor classifiers. Finally, we conduct a thorough experimental evaluation, which reveals that the evidence in favor of 1-norm SVM is statistically significant.

SVM–ELM: Pruning of Extreme Learning Machine with Support Vector Machines for Regression

AbstractExtreme Learning Machine provides very competitive performance to other related classical predictive models for solving problems such as regression, clustering, and classification. An ELM possesses the advantage of faster computational time in both training and testing. However, one of the main challenges of an ELM is the selection of the optimal number of hidden nodes. This paper presents a new approach to node selection of an ELM based on a 1-norm support vector machine (SVM). In this method, the targets of SVM yi ∈{+1, –1} are derived using the mean or median of ELM training errors as a threshold for separating the training data, which are projected to SVM dimensions. We present an integrated architecture that exploits the sparseness in solution of SVM to prune out the inactive hidden nodes in ELM. Several experiments are conducted on real-world benchmark datasets, and the results attained attest to the efficiency of the proposed method.

ℓ1-Norm support vector machine for ranking with exponentially strongly mixing sequence

The problem of ranking/ordering instances, instead of simply classifying them, has recently gained much attention in machine learning. Ranking from binary comparisons is a ubiquitous problem in modern machine learning applications. In this paper, we consider ℓ1-norm SVM for ranking. As well known, learning with ℓ1-norm restrictions usually leads to sparsity. Moreover, instead of independently draw sample sequence, we are given sample of exponentially strongly mixing sequence. Under some mild conditions, a learning rate is established.

Robust and Sparse Linear Programming Twin Support Vector Machines

In this paper, we propose a new linear programming formulation of exact 1-norm twin support vector machine (TWSVM) for classification whose solution is obtained by solving a pair of dual exterior penalty problems as unconstrained minimization problems using Newton–Armijo algorithm. The idea of our formulation is to reformulate TWSVM as a strongly convex problem by incorporated regularization techniques and then derive an exact 1-norm linear programming formulation for TWSVM to improve robustness and sparsity. The solution of two modified unconstrained minimization problems reduces to solving just two systems of linear equations as opposed to solving two quadratic programming problems in TWSVM and TBSVM, which leads to extremely simple and fast algorithm. One significant advantage of our proposed method is the implementation of structural risk minimization principle. However, only empirical risk is considered in the primal problems of TWSVM due to its complex structure and thus may incur overfitting and suboptimal in some cases. Our approach has the advantage that a pair of matrix equation of order equals to the number of input examples is solved at each iteration of the algorithm. The algorithm converges from any starting point that can be easily implemented in MATLAB without using any optimization packages. Computational comparisons of our proposed method against original TWSVM, GEPSVM and SVM have been made on both synthetic and benchmark datasets. Experimental results show that our method is better or comparable in both computation time and classification accuracy.

Embedded feature-selection support vector machine for driving pattern recognition

In this work, a more efficient and robust driving pattern recognition technique, extended Support Vector Machine (SVM) with embedded feature selection ability, has been introduced. Besides statistical significance, this proposed SVM also takes into account the accessibility and reliability of features during feature selection, so as to enable the driving condition discrimination system to achieve higher recognition efficiency and robustness. The recognition results of this extended SVM are compared with results from standard 2-norm SVM and linear 1-norm SVM, using representative driving cycle data to demonstrate the function and superiority of the new technique.

The Supervised Normalized Cut Method for Detecting, Classifying, and Identifying Special Nuclear Materials

The detection of illicit nuclear materials is a major tool in preventing and deterring nuclear terrorism. The detection task is extremely difficult because of physical limitations of nuclear radiation detectors, shielding by intervening cargo materials, and the presence of background noise. We aim at enhancing the capabilities of detectors with algorithmic methods specifically tailored for nuclear data. This paper describes a novel graph-theory-based methodology for this task. This research considers for the first time the utilization of supervised normalized cut (SNC) for data mining and classification of measurements obtained from plastic scintillation detectors that are of particularly low resolution. Specifically, the situation considered here is for when both energy spectra and the time dependence of such data are acquired. We present here a computational study, comparing the supervised normalized cut method with alternative classification methods based on support vector machine (SVM), specialized feature-reducing SVMs (i.e., 1-norm SVM, recursive feature elimination SVM, and Newton linear program SVM), and linear discriminant analysis (LDA). The study evaluates the performance of the suggested method in binary and multiple classification problems of nuclear data. The results demonstrate that the new approach is on par or superior in terms of accuracy and much better in computational complexity to SVM (with or without dimension or feature reduction) and LDA with principal components analysis as preprocessing. For binary and multiple classifications, the SNC method is more accurate, more robust, and is computationally more efficient by a factor of 2–80 than the SVM-based and LDA methods.

A fast algorithm for kernel 1-norm support vector machines

This paper presents a fast algorithm called Column Generation Newton (CGN) for kernel 1-norm support vector machines (SVMs). CGN combines the Column Generation (CG) algorithm and the Newton Linear Programming SVM (NLPSVM) method. NLPSVM was proposed for solving 1-norm SVM, and CG is frequently used in large-scale integer and linear programming algorithms. In each iteration of the kernel 1-norm SVM, NLPSVM has a time complexity of O(ℓ3), where ℓ is the sample number, and CG has a time complexity between O(ℓ3) and O(n′3), where n′ is the number of columns of the coefficient matrix in the subproblem. CGN uses CG to generate a sequence of subproblems containing only active constraints and then NLPSVM to solve each subproblem. Since the subproblem in each iteration only consists of n′ unbound constraints, CGN thus has a time complexity of O(n′3), which is smaller than that of NLPSVM and CG. Also, CGN is faster than CG when the solution to 1-norm SVM is sparse. A theorem is given to show a finite step convergence of CGN. Experimental results on the Ringnorm and UCI data sets demonstrate the efficiency of CGN to solve the kernel 1-norm SVM.