Regularized Support Vector Machine Research Articles

Functional Near‐Infrared Spectroscopy‐Based Computer‐Aided Diagnosis of Major Depressive Disorder Using Convolutional Neural Network with a New Channel Embedding Layer Considering Inter‐Hemispheric Asymmetry in Prefrontal Hemodynamic Responses

Background. Functional near‐infrared spectroscopy (fNIRS) is being extensively explored as a potential primary screening tool for major depressive disorder (MDD) because of its portability, cost‐effectiveness, and low susceptibility to motion artifacts. However, the fNIRS‐based computer‐aided diagnosis (CAD) of MDD using deep learning methods has rarely been studied. In this study, we propose a novel deep learning framework based on a convolutional neural network (CNN) for the fNIRS‐based CAD of MDD with high accuracy. Materials and Methods. The fNIRS data of participants—48 patients with MDD and 68 healthy controls (HCs)—were obtained while they performed a Stroop task. The hemodynamic responses calculated from the preprocessed fNIRS data were used as inputs to the proposed CNN model with an ensemble CNN architecture, comprising three 1D depth‐wise convolutional layers specifically designed to reflect interhemispheric asymmetry in hemodynamic responses between patients with MDD and HCs, which is known to be a distinct characteristic in previous MDD studies. The performance of the proposed model was evaluated using a leave‐one‐subject‐out cross‐validation strategy and compared with those of conventional machine learning and CNN models. Results. The proposed model exhibited a high accuracy, sensitivity, and specificity of 84.48%, 83.33%, and 85.29%, respectively. The accuracies of conventional machine learning algorithms—shrinkage linear discriminator analysis, regularized support vector machine, EEGNet, and ShallowConvNet—were 73.28%, 74.14%, 62.93%, and 62.07%, respectively. Conclusions. In conclusion, the proposed deep learning model can differentiate between the patients with MDD and HCs more accurately than the conventional models, demonstrating its applicability in fNIRS‐based CAD systems.

Fuzzy regular least squares twin support vector machine and its application in fault diagnosis

Objective function ignores the generalization error in LSTSVM, and training overfitting results in poor generalization performance. All sample labels are considered deterministic, however, some samples contain outliers affected by noises, which leads to low reliability. A recognition method based on fuzzy regular LSTSVM (FRLSTSVM) is proposed. Firstly, L2 norm regular term is introduced into objective function to improve the generalization performance. Secondly, outliers of samples are detected through support vector domain description (SVDD), which improves outlier detection accuracy. Then, a membership degree S3 is constructed to give the outliers a suitable membership degree, reducing the impact of outliers on results. Finally, FRLSTSVM is extended to a multi-classification model by one versus one (OVO) and binary tree (BT) strategies, and it is combined with an improved multiscale fluctuating Rényi dispersion entropy (IMFRDE) for fault diagnosis. The results show that the method has stronger generalization, lower sensitivity to parameters and higher reliability.

Dissolved Oxygen Prediction Model for the Yangtze River Estuary Basin Using IPSO-LSSVM

Water ecology has always been key to environmental protection, and the combination of human activities and natural factors has caused eutrophication in the Yangtze estuary and adjacent waters. Among them, dissolved oxygen (DO) concentration is the key indicator to judge the quality of water. Firstly, using principal component analysis (PCA) to determine the number of parameters affecting dissolved oxygen concentration, the least squares support vector machine (LSSVM) prediction model with improved particle swarm optimization (IPSO) is proposed to be applied to the dissolved oxygen prediction in Shanghai’s Yangtze River basin through the data-driven modeling approach and the regression prediction capability of the neural network. Eight parameters of water temperature (WT), pH, potassium permanganate (KMnO4), ammonia nitrogen (NH4+-N), total phosphorus (TP), total nitrogen (TN), conductivity (Cond), and nephelometric turbidity unit (NTU) are selected as model inputs in the published public data, and the output is the dissolved oxygen concentration. The optimal combination of model parameters is found according to the IPSO algorithm, which effectively overcomes the parameter selection problem of regular support vector machines (SVM). The mean absolute error (MAE), root mean square error (RMSE), mean absolute percentage error (MAPE), and correlation coefficients of the evaluation indexes of this model (R2) are 0.1702, 0.2221, 0.0267, and 0.9751, respectively. Compared with other similar data driven models, this model has improved model accuracy and stability in predicting DO concentrations in the estuary, and thus it provides technical support for assessing and monitoring offshore water quality.

Over-relaxed multi-block ADMM algorithms for doubly regularized support vector machines

As a classical machine learning model, support vector machine (SVM) has attracted much attention due to its rigorous theoretical foundation and powerful discriminative performance. The doubly regularized SVM (DRSVM) is an important variant of SVM based on elastic-net regularization, which considers both the sparsity and stability of the model. To tackle the problems of explosive increases in data dimensions and data volume, the alternating direction method of multipliers (ADMM) algorithm can be used to train the DRSVM model. ADMM is an effective iterative algorithm for solving convex optimization problems by decomposing a large issue into a series of solvable subproblems, which is also well suited for distributed computing. However, lack of guaranteed convergence and slow convergence rate are two critical limitations of ADMM. In this paper, a 3-block ADMM algorithm based on the over-relaxation technique is proposed to accelerate DRSVM training, namely, the over-relaxed DRSVM (O-RDRSVM). The main strategy of the over-relaxation technique is to further append the information from the previous iteration to the next iteration to improve the convergence of ADMM. We also propose a distributed version of O-RDRSVM to handle parallel and distributed computing faster, termed DO-RDRSVM. Moreover, we develop a fast O-RDRSVM algorithm (FO-RDRSVM) and a fast DO-RDRSVM algorithm (FDO-RDRSVM), which further reduce the computational cost of O-RDRSVM and DO-RDRSVM by employing the matrix inversion lemma. The convergence analyses ensure the effectiveness of our algorithms for DRSVM training. Finally, extensive experiments on public datasets demonstrate the advantages of our algorithms in terms of convergence rate and training time while maintaining accuracy and sparsity comparable to those of previous works.

MLapSVM-LBS: Predicting DNA-binding proteins via a multiple Laplacian regularized support vector machine with local behavior similarity

DNA-binding proteins (DBPs) are of great significance in many basic cellular processes. Experiment-based methods for identifying DBPs are costly and time-consuming. To deal with large-scale DBP identification tasks, a variety of computation-based methods have been developed. Inspired by previous work, we propose a multiple Laplacian regularized support vector machine with local behavior similarity (MLapSVM-LBS) to predict DBP. We serially combine three features that are extracted from protein sequences (including PsePSSM, GE, NMBAC) and feed them into MLapSVM-LBS. Based on human behavior learning theory, MLapSVM-LBS can better represent the relationship between samples through local behavior similarity. We introduce a new edge weight calculation method that takes label information into consideration. In addition, a local distribution parameter reflecting the underlying probability distribution of a sample’s neighborhood is also employed. To further improve the robustness of the model, we utilize multiple Laplacian regularization to build a multigraph model in which five Laplacian graphs are constructed with local behavior similarity by changing the neighborhood size. To appraise the performance of our model, MLapSVM-LBS is trained and tested on the PDB186, PDB1075, PDB2272 and PDB14189 datasets. On two independent testing sets (PDB186 and PDB2272), our method reaches the accuracies of 0.887 and 0.712, respectively. The good results on both datasets demonstrate the reliable performance of our model.

Faster Support Vector Machines

The time complexity of support vector machines (SVMs) prohibits training on huge datasets with millions of data points. Recently, multilevel approaches to train SVMs have been developed to allow for time-efficient training on huge datasets. While regular SVMs perform the entire training in one—time-consuming—optimization step, multilevel SVMs first build a hierarchy of problems decreasing in size that resemble the original problem and then train an SVM model for each hierarchy level, benefiting from the solved models of previous levels. We present a faster multilevel support vector machine that uses a label propagation algorithm to construct the problem hierarchy. Extensive experiments indicate that our approach is up to orders of magnitude faster than the previous fastest algorithm while having comparable classification quality. For example, already one of our sequential solvers is on average a factor 15 faster than the parallel ThunderSVM algorithm, while having similar classification quality. 1

Structure-Based Prediction of hERG-Related Cardiotoxicity: A Benchmark Study.

Drug-induced blockade of the human ether-à-go-go-related gene (hERG) channel is today considered the main cause of cardiotoxicity in postmarketing surveillance. Hence, several ligand-based approaches were developed in the last years and are currently employed in the early stages of a drug discovery process for in silico cardiac safety assessment of drug candidates. Herein, we present the first structure-based classifiers able to discern hERG binders from nonbinders. LASSO regularized support vector machines were applied to integrate docking scores and protein–ligand interaction fingerprints. A total of 396 models were trained and validated based on: (i) high-quality experimental bioactivity information returned by 8337 curated compounds extracted from ChEMBL (version 25) and (ii) structural predictor data. Molecular docking simulations were performed using GLIDE and GOLD software programs and four different hERG structural models, namely, the recently published structures obtained by cryoelectron microscopy (PDB codes: 5VA1 and 7CN1) and two published homology models selected for comparison. Interestingly, some classifiers return performances comparable to ligand-based models in terms of area under the ROC curve (AUCMAX = 0.86 ± 0.01) and negative predictive values (NPVMAX = 0.81 ± 0.01), thus putting forward the herein proposed computational workflow as a valuable tool for predicting hERG-related cardiotoxicity without the limitations of ligand-based models, typically affected by low interpretability and a limited applicability domain. From a methodological point of view, our study represents the first example of a successful integration of docking scores and protein–ligand interaction fingerprints (IFs) through a support vector machine (SVM) LASSO regularized strategy. Finally, the study highlights the importance of using hERG structural models accounting for ligand-induced fit effects and allowed us to select the best-performing protein conformation (made available in the Supporting Information, SI) to be employed for a reliable structure-based prediction of hERG-related cardiotoxicity.

Structural regularized twin support vector machine based on within-class scatter and between-class scatter

Structural regularized twin support vector machine based on within-class scatter and between-class scatter

Prediction of Flood in Barak River using Hybrid Machine Learning Approaches: A Case Study

Flooding causes several threats with outcomes which include peril to human and animal life, damage to property, and adversity to agricultural fields. Therefore, flood prediction is of prime importance for reducing loss of life and devastation to property. To model complex nature of hydrologic processes artificial neural network (ANN) tool is effectively being utilized for modelling different nonlinear relationships, and has proved to be an appropriate method for flood prediction. Present study investigated relative accuracy of radial basis function neural network (RBFNN) and support vector machine (SVM) models combined with Firefly Algorithm (FA) in predicting river flood discharge and contrasted with that of regular ANN, RBFNN and SVM models. Monthly river flow data of Silchar and Dholai stations located in Cachar district of Assam, India are utilized for the present study. For assessing model performance, coefficient of determination (R2), mean square error (MSE) and root mean square error (RMSE) were measured. Evaluation of outcomes shows that both RBF-FA (radial basis function — firefly algorithm) and SVM-FA (support vector machine — firefly algorithm) hybrid models give more precise forecasting results than RBFNN, FFBPNN (feed forward back propagation neural network) and SVM models. Yet, it can be observed that SVM-FA model give better forecasting outputs with R2 value 0.9818 than RBF-FA model. Results also reveal that simple SVM model performs marginally better than simple ANN model.

Valley-loss regular simplex support vector machine for robust multiclass classification

Noise and outlier data processing are important issues to support vector machine (SVM). Although the pinball-loss SVM (Pin-SVM) and ramp-loss SVM (Ramp-SVM) are able to deal with the feature noise and outlier labels respectively, neither can handle both and promoting them from binary-classification to multiclass classification usually requires partitioning strategies. Since regular simplex support vector machine (RSSVM) has been proposed as a novel all-in-one K-classification model with clear advantages over partitioning strategies, developing a novel loss function with feature noise robustness and outlier labels insensitivity meanwhile and embedding it into the framework of RSSVM is potentially promising. In this paper, a newly proposed valley-loss regular simplex support vector machine (V-RSSVM) for robust multiclass classification is presented. Inheriting the merits of both the pinball-type loss and ramp-type loss, valley-loss enjoys not only the robustness to feature noise and outlier labels but also excellent sparseness. To train the V-RSSVM fast, a Concave–Convex Procedure (CCCP) assisted sequential minimization optimization (SMO)-type solver and a speeding up oriented initial solution strategy were developed. We also investigated the robustness, generalization error bound and sparseness of V-RSSVM in theory. Numerical results on twenty-five real-life data sets verify the effectiveness of our proposed V-RSSVM model.

A Label Similarity Probability Filter for Hyperspectral Image Postclassification

This article presents a label similarity probability filter (LSPF) to make hyperspectral image postclassification. The LSPF is inspired by the first law of geography and proposes a class label probability function to quantify the probability of both centered and its neighboring pixels belonging to the same class. It first classifies the hyperspectral data using the regular support vector machine classifier. Then, it binarizes the posterior classification result to obtain the binary label maps of each class. After that, it traverses all spatial windows centered by each pixel and calculates the cumulative probability of all pixels in each class. Finally, the cumulative probabilities are used to make reclassification to obtain the refined classification map. The experiments on Indian Pines, Pavia University, and ZY1-02D Yellow River Estuary data show that LSPF greatly improves the classification accuracy of spectral signatures and outperforms other state-of-the-art spectral-spatial methods.

Selective linearization for multi-block statistical learning

We consider the problem of minimizing a sum of several convex non-smooth functions and discuss the selective linearization method (SLIN), which iteratively linearizes all but one of the functions and employs simple proximal steps. The algorithm is a form of multiple operator splitting in which the order of processing partial functions is not fixed, but rather determined in the course of calculations. SLIN is globally convergent for an arbitrary number of component functions without artificial duplication of variables. We report results from extensive numerical experiments in two statistical learning settings such as large-scale overlapping group Lasso and doubly regularized support vector machine. In each setting, we introduce novel and efficient solutions for solving sub-problems. The numerical results demonstrate the efficacy and accuracy of SLIN.

Cancer Diagnosis and Disease Gene Identification via Statistical Machine Learning

Diagnosing cancer and identifying the disease gene by using DNA microarray gene expression data are the hot topics in current bioinformatics. This paper is devoted to the latest development in cancer diagnosis and gene selection via statistical machine learning. A support vector machine is firstly introduced for the binary cancer diagnosis. Then, 1-norm support vector machine, doubly regularized support vector machine, adaptive huberized support vector machine and other extensions are presented to improve the performance of gene selection. Lasso, elastic net, partly adaptive elastic net, group lasso, sparse group lasso, adaptive sparse group lasso and other sparse regression methods are also introduced for performing simultaneous binary cancer classification and gene selection. In addition to introducing three strategies for reducing multiclass to binary, methods of directly considering all classes of data in a learning model (multi_class support vector, sparse multinomial regression, adaptive multinomial regression and so on) are presented for performing multiple cancer diagnosis. Limitations and promising directions are also discussed.

Predicting future suicidal behaviour in young adults, with different machine learning techniques: A population-based longitudinal study

The predictive accuracy of suicidal behaviour has not improved over the last decades. We aimed to explore the potential of machine learning to predict future suicidal behaviour using population-based longitudinal data. Baseline risk data assessed within the Scottish wellbeing study, in which 3508 young adults (18-34 years) completed a battery of psychological measures, were used to predict both suicide ideation and suicide attempts at one-year follow-up. The performance of the following algorithms was compared: regular logistic regression, K-nearest neighbors, classification tree, random forests, gradient boosting and support vector machine. At one year follow up, 2428 respondents (71%) finished the second assessment. 336 respondents (14%) reported suicide ideation between baseline and follow up, and 50 (2%) reported a suicide attempt. All performance metrics were highly similar across methods. The random forest algorithm was the best algorithm to predict suicide ideation (AUC 0.83, PPV 0.52, BA 0.74) and the gradient boosting to predict suicide attempt (AUC 0.80, PPV 0.10, BA 0.69). The number of respondents with suicidal behaviour at follow up was small. We only had data on psychological risk factors, limiting the potential of the more complex machine learning algorithms to outperform regular logistical regression. When applied to population-based longitudinal data containing multiple psychological measurements, machine learning techniques did not significantly improve the predictive accuracy of suicidal behaviour. Adding more detailed data on for example employment, education or previous health care uptake, might result in better performance of machine learning over regular logistical regression.

Structural improved regular simplex support vector machine for multiclass classification

Although the structural regularized support vector machine (SRSVM) can enhance the generalization capability of the standard support vector machine (SVM), its current version is used only for binary classification. To make SRSVM adapt to the K-class classification, the most direct approach is combining it with partitioning strategies, which may however lead to the following shortcomings: (1) Extracting structural information repeatedly for individual classifiers based on different class partitions increases the computational complexity. (2) Individual classifiers can hardly utilize complete data structural information. Under the basic framework of regular simplex support vector machine (RSSVM), we developed a novel structural improved regular simplex support vector machine (SIRSSVM). SIRSSVM generates only a single primal optimization problem, into which the data structural information within all classes is embedded, rather than using only partial structural information to construct individual classifiers as partitioning strategies do. Additionally, we modified the sequential minimization optimization (SMO)-type solver for RSSVM to adapt the proposed SIRSSVM model. Experimental results verified that our SIRSSVM could achieve excellent performance on both generalization capability and training efficiency.

Robust twin support vector regression based on Huber loss function

Construction of robust regression learning models to fit data with noise is an important and challenging problem of data regression. One of the ways to tackle this problem is the selection of a proper loss function showing insensitivity to noise present in the data. Since Huber function has the property that inputs with large deviations of misfit are penalized linearly and small errors are squared, we present novel robust regularized twin support vector machines for data regression based on Huber and e-insensitive Huber loss functions in this study. The proposed regression models result in solving a pair of strongly convex minimization problems in simple form in primal whose solutions are obtained by functional and Newton–Armijo iterative algorithms. The finite convergence of Newton–Armijo algorithm is proved. Numerical tests are performed on noisy synthetic and benchmark datasets, and their results are compared with few popular regression learning algorithms. The comparative study clearly shows the robustness of the proposed regression methods and further demonstrates their effectiveness and suitability.

Non-interior-point smoothing Newton method for CP revisited and its application to support vector machines

Non-interior-point smoothing Newton method (SNM) for optimization have been widely studied for over three decades. SNM is a popular approach for solving small- and medium-scale complementarity problem (CP) and many optimization problems. The main purpose of this paper is to revisit the SNM and show that the Hessian matrix in SNM becomes increasingly ill-conditioned while smoothing parameter approaches to zero, which leads to their practical use remains limited due to computational difficulties in solving large-scale CP. To tackle this, we redesign a new smoothing method, called accelerated preconditioned smoothing method (APSM) for the efficient solution of regularized support vector machines in machine learning. With the help of suitable preconditioner, we can correct the ill-conditioning of associated smoothing Hessian matrix and thereby the associated smoothing Hessian equation can be solved in a few of iterations by using iterative methods in linear algebra. Two accelerated techniques are designed in the paper to reduce our computation time. Finally we present numerical experiments to support our theoretical guarantees and test accelerated convergence obtained by APSM. The result showed that APSM computes faster than state of art algorithm without reducing classification accuracy.

Automatic detection of small bowel tumors in wireless capsule endoscopy images using ensemble learning.

Wireless Capsule Endoscopy (WCE) is a minimally invasive diagnosis tool for lesion detection in the gastrointestinal tract, reaching places where conventional endoscopy is unable to. However, the significant amount of acquired data leads to difficulties in the diagnosis by the physicians; which can be eased with computer assistance. This paper addresses a method for the automatic detection of tumors in WCE by using a two-step based procedure: region of interest selection and classification. The first step aims to separate abnormal from normal tissue by using automatic segmentation based on a Gaussian Mixture Model (GMM). A modified version of the Anderson method for convergence acceleration of the expectation-maximization (EM) algorithm is proposed. The proposed features for both segmentation and classification are based on the CIELab color space, as a way of bypassing lightness variations, where the L component is discarded. Tissue variability among subjects, light inhomogeneities and even intensity differences among different devices can be overcome by using simultaneously features from both regions. In the second step, an ensemble system with partition of the training data with a new training scheme is proposed. At this stage, the gating network is trained after the experts have been trained decoupling the joint maximization of both modules. The partition module is also used at the test step, leading the incoming data to the most likely expert allowing incremental adaptation by preserving data diversity. This algorithm outperforms others based on texture features selected from Wavelets and Curvelets transforms, classified by a regular support vector machine (SVM) in more than 5%. This work shows that simpler features can outperform more elaborate ones if appropriately designed. In the current case, luminance was discarded to cope with saturated tissue, facilitating the color perception. Ensemble systems remain an open research field. In the current case, changes in both topology and training strategy have led to significant performance improvements. A system with this level of performance can be used in current clinical practice.

A new adaptive weighted imbalanced data classifier via improved support vector machines with high-dimension nature

The standard support vector machine (SVM) models are widely used in various fields, but we show that they are not rationally defined from the perspective of geometric point, which is likely to degrade the models’ performances theoretically, especially under the high-dimensional cases. In this paper, we consider a composite penalty and propose an elastic net support vector machine (ENSVM). Unlike the doubly regularized support vector machine (DrSVM, Wang et al. (2006)), we impose the penalty to the slack variables rather than the normal vectors of the hyperplane. Then, we prove that ENSVM is more rationally defined than standard SVM and DrSVM (in section 3.2.1). Moreover, the ENSVM demonstrates a more stable and high-dimension nature inherently, while the simulation results cogently support these merits. Besides, we also combine fused weights with ENSVM and propose an adaptive weighted elastic net support vector machine (AWENSVM), to make the primal model more adaptive and robust to the imbalanced data. Compared with the other popular SVMs, the AWENSVM model proposed in this paper performs better obviously.

An efficient regularized K-nearest neighbor structural twin support vector machine

K-nearest neighbor based structural twin support vector machine (KNN-STSVM) performs better than structural twin support vector machine (S-TSVM). It applies the intra-class KNN method, and different weights are given to the samples in one class to strengthen the structural information. For the other class, the redundant constraints are deleted by the inter-class KNN method to speed up the training process. However, the empirical risk minimization principle is implemented in the KNN-STSVM, so it easily leads to over-fitting and reduces the prediction accuracy of the classifier. To enhance the generalization ability of the classifier, we propose an efficient regularized K-nearest neighbor structural twin support vector machine, called RKNN-STSVM, by introducing a regularization term into the objective function. So there are two parts in the objective function, one of which is to maximize the margin between the two parallel hyper-planes, and the other one is to minimize the training errors of two classes of samples. Therefore the structural risk minimization principle is implemented in our RKNN-STSVM. Besides, a fast DCDM algorithm is introduced to handle relatively large-scale problems more efficiently. Comprehensive experimental results on twenty-seven benchmark datasets and two popular image datasets demonstrate the efficiency of our proposed RKNN-STSVM.