Regularized Support Vector Machine Research Articles

Testing the Applicability and Transferability of Data-Driven Geospatial Models for Predicting Soil Erosion in Vineyards

Empirically based approaches, like the Universal Soil Loss Equation (USLE), are appropriate for estimating mass movement attributed to rill erosion. USLE and its associates become widespread even in spatially extended studies in spite of its original plot-level concept, as well as with certain constraints on the supply of suitable input spatial data. At the same time, there is a continuously expanding opportunity and offer for the application of remote sensing (RS) imagery together with machine learning (ML) techniques to model and monitor various environmental processes utilizing their versatile benefits. The present study focused on the applicability of data-driven geospatial models for predicting soil erosion in three vineyards in the Upper Pannon Wine Region, Central Europe, considering the seasonal variation in influencing factors. Soil loss was formerly modeled by USLE, thus providing non-observation-based reference datasets for the calibration of parcel-specific prediction models using various ML methods (Random Forest, eXtreme Gradient Boosting, Regularized Support Vector Machine with Linear Kernel), which is a well-established approach in digital soil mapping (DSM). Predictions used spatially exhaustive, auxiliary, and environmental covariables. RS data were represented by multi-temporal Sentinel-2 satellite imagery data, which were supplemented by (i) topographic covariates derived from a UAV-based digital surface model and (ii) digital primary soil property maps. In addition to spatially quantifying soil erosion, the feasibility of transferring the inferred models between nearby vineyards was tested with ambiguous outcomes. Our results indicate that ML models can feasibly replace the empirical USLE model for erosion prediction. However, further research is needed to assess model transferability even to nearby parcels.

Machine learning‐aided prediction of windstorm‐induced vibration responses of long‐span suspension bridges

AbstractLong‐span suspension bridges are significantly susceptible to windstorm‐induced vibrations, leading to critical challenges of field measurements along with multicollinearity and nonlinearity between wind features and bridge dynamic responses. To address these issues, this article proposes an innovative machine learning‐assisted predictive method by integrating a predictor selector developed from regularized neighborhood components analysis and kernel regression modeling through a regularized support vector machine adjusted by Bayesian hyperparameter optimization. The crux of the proposed method lies in advanced machine learning algorithms including metric learning, kernel learning, and hybrid learning integrated in a regularized framework. Utilizing the Hardanger Bridge subjected to different windstorms, the performance of the proposed method is validated and then compared with state‐of‐the‐art regression techniques. Results highlight the effectiveness and practicality of the proposed method with the minimum and maximum R‐squared rates of 89% and 98%, respectively. It also surpasses the state‐of‐the‐art regression techniques in predicting bridge dynamics under different windstorms.

Machine-Learning-Based Prediction of Photobiomodulation Effects on Older Adults With Cognitive Decline Using Functional Near-Infrared Spectroscopy.

Transcranial photobiomodulation (tPBM) has been widely studied for its potential to enhance cognitive functions of the elderly. However, its efficacy varies, with some individuals exhibiting no significant response to the treatment. Considering these inconsistencies, we introduce a machine learning approach aimed at distinguishing between individuals that respond and do not respond to tPBM treatment based on functional near-infrared spectroscopy (fNIRS) acquired before the treatment. We measured nine cognitive scores and recorded fNIRS data from 62 older adults with cognitive decline (43 experimental and 19 control subjects). The experimental group underwent tPBM intervention over a span of 12 weeks. Based on the comparison of the global cognitive score (GCS), merging the nine cognitive scores into a single representation, acquired before and after tPBM treatment, we classified all participants as responders or non-responders to tPBM with a threshold for the GCS change. The fNIRS data were recorded during the resting state, recognition memory task (RMT), Stroop task, and verbal fluency task. A regularized support vector machine was utilized to classify the responders and non-responders to tPBM. The most promising performance of our machine learning model was observed when using the fNIRS data collected during the RMT, which yielded an accuracy of 0.8537, an F1-score of 0.8421, sensitivity of 0.7619, and specificity of 0.95. To the best of our knowledge, this is the first study to demonstrate the feasibility of predicting the tPBM efficacy. Our approach is expected to contribute to more efficient treatment planning by excluding ineffective treatment options.

Functional Near‐Infrared Spectroscopy‐Based Computer‐Aided Diagnosis of Major Depressive Disorder Using Convolutional Neural Network with a New Channel Embedding Layer Considering Inter‐Hemispheric Asymmetry in Prefrontal Hemodynamic Responses

Background. Functional near‐infrared spectroscopy (fNIRS) is being extensively explored as a potential primary screening tool for major depressive disorder (MDD) because of its portability, cost‐effectiveness, and low susceptibility to motion artifacts. However, the fNIRS‐based computer‐aided diagnosis (CAD) of MDD using deep learning methods has rarely been studied. In this study, we propose a novel deep learning framework based on a convolutional neural network (CNN) for the fNIRS‐based CAD of MDD with high accuracy. Materials and Methods. The fNIRS data of participants—48 patients with MDD and 68 healthy controls (HCs)—were obtained while they performed a Stroop task. The hemodynamic responses calculated from the preprocessed fNIRS data were used as inputs to the proposed CNN model with an ensemble CNN architecture, comprising three 1D depth‐wise convolutional layers specifically designed to reflect interhemispheric asymmetry in hemodynamic responses between patients with MDD and HCs, which is known to be a distinct characteristic in previous MDD studies. The performance of the proposed model was evaluated using a leave‐one‐subject‐out cross‐validation strategy and compared with those of conventional machine learning and CNN models. Results. The proposed model exhibited a high accuracy, sensitivity, and specificity of 84.48%, 83.33%, and 85.29%, respectively. The accuracies of conventional machine learning algorithms—shrinkage linear discriminator analysis, regularized support vector machine, EEGNet, and ShallowConvNet—were 73.28%, 74.14%, 62.93%, and 62.07%, respectively. Conclusions. In conclusion, the proposed deep learning model can differentiate between the patients with MDD and HCs more accurately than the conventional models, demonstrating its applicability in fNIRS‐based CAD systems.

Fuzzy regular least squares twin support vector machine and its application in fault diagnosis

Objective function ignores the generalization error in LSTSVM, and training overfitting results in poor generalization performance. All sample labels are considered deterministic, however, some samples contain outliers affected by noises, which leads to low reliability. A recognition method based on fuzzy regular LSTSVM (FRLSTSVM) is proposed. Firstly, L2 norm regular term is introduced into objective function to improve the generalization performance. Secondly, outliers of samples are detected through support vector domain description (SVDD), which improves outlier detection accuracy. Then, a membership degree S3 is constructed to give the outliers a suitable membership degree, reducing the impact of outliers on results. Finally, FRLSTSVM is extended to a multi-classification model by one versus one (OVO) and binary tree (BT) strategies, and it is combined with an improved multiscale fluctuating Rényi dispersion entropy (IMFRDE) for fault diagnosis. The results show that the method has stronger generalization, lower sensitivity to parameters and higher reliability.

Dissolved Oxygen Prediction Model for the Yangtze River Estuary Basin Using IPSO-LSSVM

Water ecology has always been key to environmental protection, and the combination of human activities and natural factors has caused eutrophication in the Yangtze estuary and adjacent waters. Among them, dissolved oxygen (DO) concentration is the key indicator to judge the quality of water. Firstly, using principal component analysis (PCA) to determine the number of parameters affecting dissolved oxygen concentration, the least squares support vector machine (LSSVM) prediction model with improved particle swarm optimization (IPSO) is proposed to be applied to the dissolved oxygen prediction in Shanghai’s Yangtze River basin through the data-driven modeling approach and the regression prediction capability of the neural network. Eight parameters of water temperature (WT), pH, potassium permanganate (KMnO4), ammonia nitrogen (NH4+-N), total phosphorus (TP), total nitrogen (TN), conductivity (Cond), and nephelometric turbidity unit (NTU) are selected as model inputs in the published public data, and the output is the dissolved oxygen concentration. The optimal combination of model parameters is found according to the IPSO algorithm, which effectively overcomes the parameter selection problem of regular support vector machines (SVM). The mean absolute error (MAE), root mean square error (RMSE), mean absolute percentage error (MAPE), and correlation coefficients of the evaluation indexes of this model (R2) are 0.1702, 0.2221, 0.0267, and 0.9751, respectively. Compared with other similar data driven models, this model has improved model accuracy and stability in predicting DO concentrations in the estuary, and thus it provides technical support for assessing and monitoring offshore water quality.

Over-relaxed multi-block ADMM algorithms for doubly regularized support vector machines

As a classical machine learning model, support vector machine (SVM) has attracted much attention due to its rigorous theoretical foundation and powerful discriminative performance. The doubly regularized SVM (DRSVM) is an important variant of SVM based on elastic-net regularization, which considers both the sparsity and stability of the model. To tackle the problems of explosive increases in data dimensions and data volume, the alternating direction method of multipliers (ADMM) algorithm can be used to train the DRSVM model. ADMM is an effective iterative algorithm for solving convex optimization problems by decomposing a large issue into a series of solvable subproblems, which is also well suited for distributed computing. However, lack of guaranteed convergence and slow convergence rate are two critical limitations of ADMM. In this paper, a 3-block ADMM algorithm based on the over-relaxation technique is proposed to accelerate DRSVM training, namely, the over-relaxed DRSVM (O-RDRSVM). The main strategy of the over-relaxation technique is to further append the information from the previous iteration to the next iteration to improve the convergence of ADMM. We also propose a distributed version of O-RDRSVM to handle parallel and distributed computing faster, termed DO-RDRSVM. Moreover, we develop a fast O-RDRSVM algorithm (FO-RDRSVM) and a fast DO-RDRSVM algorithm (FDO-RDRSVM), which further reduce the computational cost of O-RDRSVM and DO-RDRSVM by employing the matrix inversion lemma. The convergence analyses ensure the effectiveness of our algorithms for DRSVM training. Finally, extensive experiments on public datasets demonstrate the advantages of our algorithms in terms of convergence rate and training time while maintaining accuracy and sparsity comparable to those of previous works.

MLapSVM-LBS: Predicting DNA-binding proteins via a multiple Laplacian regularized support vector machine with local behavior similarity

DNA-binding proteins (DBPs) are of great significance in many basic cellular processes. Experiment-based methods for identifying DBPs are costly and time-consuming. To deal with large-scale DBP identification tasks, a variety of computation-based methods have been developed. Inspired by previous work, we propose a multiple Laplacian regularized support vector machine with local behavior similarity (MLapSVM-LBS) to predict DBP. We serially combine three features that are extracted from protein sequences (including PsePSSM, GE, NMBAC) and feed them into MLapSVM-LBS. Based on human behavior learning theory, MLapSVM-LBS can better represent the relationship between samples through local behavior similarity. We introduce a new edge weight calculation method that takes label information into consideration. In addition, a local distribution parameter reflecting the underlying probability distribution of a sample’s neighborhood is also employed. To further improve the robustness of the model, we utilize multiple Laplacian regularization to build a multigraph model in which five Laplacian graphs are constructed with local behavior similarity by changing the neighborhood size. To appraise the performance of our model, MLapSVM-LBS is trained and tested on the PDB186, PDB1075, PDB2272 and PDB14189 datasets. On two independent testing sets (PDB186 and PDB2272), our method reaches the accuracies of 0.887 and 0.712, respectively. The good results on both datasets demonstrate the reliable performance of our model.

Faster Support Vector Machines

The time complexity of support vector machines (SVMs) prohibits training on huge datasets with millions of data points. Recently, multilevel approaches to train SVMs have been developed to allow for time-efficient training on huge datasets. While regular SVMs perform the entire training in one—time-consuming—optimization step, multilevel SVMs first build a hierarchy of problems decreasing in size that resemble the original problem and then train an SVM model for each hierarchy level, benefiting from the solved models of previous levels. We present a faster multilevel support vector machine that uses a label propagation algorithm to construct the problem hierarchy. Extensive experiments indicate that our approach is up to orders of magnitude faster than the previous fastest algorithm while having comparable classification quality. For example, already one of our sequential solvers is on average a factor 15 faster than the parallel ThunderSVM algorithm, while having similar classification quality. 1

Structure-Based Prediction of hERG-Related Cardiotoxicity: A Benchmark Study.

Drug-induced blockade of the human ether-à-go-go-related gene (hERG) channel is today considered the main cause of cardiotoxicity in postmarketing surveillance. Hence, several ligand-based approaches were developed in the last years and are currently employed in the early stages of a drug discovery process for in silico cardiac safety assessment of drug candidates. Herein, we present the first structure-based classifiers able to discern hERG binders from nonbinders. LASSO regularized support vector machines were applied to integrate docking scores and protein–ligand interaction fingerprints. A total of 396 models were trained and validated based on: (i) high-quality experimental bioactivity information returned by 8337 curated compounds extracted from ChEMBL (version 25) and (ii) structural predictor data. Molecular docking simulations were performed using GLIDE and GOLD software programs and four different hERG structural models, namely, the recently published structures obtained by cryoelectron microscopy (PDB codes: 5VA1 and 7CN1) and two published homology models selected for comparison. Interestingly, some classifiers return performances comparable to ligand-based models in terms of area under the ROC curve (AUCMAX = 0.86 ± 0.01) and negative predictive values (NPVMAX = 0.81 ± 0.01), thus putting forward the herein proposed computational workflow as a valuable tool for predicting hERG-related cardiotoxicity without the limitations of ligand-based models, typically affected by low interpretability and a limited applicability domain. From a methodological point of view, our study represents the first example of a successful integration of docking scores and protein–ligand interaction fingerprints (IFs) through a support vector machine (SVM) LASSO regularized strategy. Finally, the study highlights the importance of using hERG structural models accounting for ligand-induced fit effects and allowed us to select the best-performing protein conformation (made available in the Supporting Information, SI) to be employed for a reliable structure-based prediction of hERG-related cardiotoxicity.

Structural regularized twin support vector machine based on within-class scatter and between-class scatter

Structural regularized twin support vector machine based on within-class scatter and between-class scatter

Prediction of Flood in Barak River using Hybrid Machine Learning Approaches: A Case Study

Abstract Flooding causes several threats with outcomes which include peril to human and animal life, damage to property, and adversity to agricultural fields. Therefore, flood prediction is of prime importance for reducing loss of life and devastation to property. To model complex nature of hydrologic processes artificial neural network (ANN) tool is effectively being utilized for modelling different nonlinear relationships, and has proved to be an appropriate method for flood prediction. Present study investigated relative accuracy of radial basis function neural network (RBFNN) and support vector machine (SVM) models combined with Firefly Algorithm (FA) in predicting river flood discharge and contrasted with that of regular ANN, RBFNN and SVM models. Monthly river flow data of Silchar and Dholai stations located in Cachar district of Assam, India are utilized for the present study. For assessing model performance, coefficient of determination (R2), mean square error (MSE) and root mean square error (RMSE) were measured. Evaluation of outcomes shows that both RBF-FA (radial basis function - firefly algorithm) and SVM-FA (support vector machine - firefly algorithm) hybrid models give more precise forecasting results than RBFNN, FFBPNN (feed forward back propagation neural network) and SVM models. Yet, it can be observed that SVM-FA model give better forecasting outputs with R2 value 0.9818 than RBF-FA model. Results also reveal that simple SVM model performs marginally better than simple ANN model.

Valley-loss regular simplex support vector machine for robust multiclass classification

Noise and outlier data processing are important issues to support vector machine (SVM). Although the pinball-loss SVM (Pin-SVM) and ramp-loss SVM (Ramp-SVM) are able to deal with the feature noise and outlier labels respectively, neither can handle both and promoting them from binary-classification to multiclass classification usually requires partitioning strategies. Since regular simplex support vector machine (RSSVM) has been proposed as a novel all-in-one K-classification model with clear advantages over partitioning strategies, developing a novel loss function with feature noise robustness and outlier labels insensitivity meanwhile and embedding it into the framework of RSSVM is potentially promising. In this paper, a newly proposed valley-loss regular simplex support vector machine (V-RSSVM) for robust multiclass classification is presented. Inheriting the merits of both the pinball-type loss and ramp-type loss, valley-loss enjoys not only the robustness to feature noise and outlier labels but also excellent sparseness. To train the V-RSSVM fast, a Concave–Convex Procedure (CCCP) assisted sequential minimization optimization (SMO)-type solver and a speeding up oriented initial solution strategy were developed. We also investigated the robustness, generalization error bound and sparseness of V-RSSVM in theory. Numerical results on twenty-five real-life data sets verify the effectiveness of our proposed V-RSSVM model.

A Label Similarity Probability Filter for Hyperspectral Image Postclassification

This article presents a label similarity probability filter (LSPF) to make hyperspectral image postclassification. The LSPF is inspired by the first law of geography and proposes a class label probability function to quantify the probability of both centered and its neighboring pixels belonging to the same class. It first classifies the hyperspectral data using the regular support vector machine classifier. Then, it binarizes the posterior classification result to obtain the binary label maps of each class. After that, it traverses all spatial windows centered by each pixel and calculates the cumulative probability of all pixels in each class. Finally, the cumulative probabilities are used to make reclassification to obtain the refined classification map. The experiments on Indian Pines, Pavia University, and ZY1-02D Yellow River Estuary data show that LSPF greatly improves the classification accuracy of spectral signatures and outperforms other state-of-the-art spectral-spatial methods.

Selective linearization for multi-block statistical learning

We consider the problem of minimizing a sum of several convex non-smooth functions and discuss the selective linearization method (SLIN), which iteratively linearizes all but one of the functions and employs simple proximal steps. The algorithm is a form of multiple operator splitting in which the order of processing partial functions is not fixed, but rather determined in the course of calculations. SLIN is globally convergent for an arbitrary number of component functions without artificial duplication of variables. We report results from extensive numerical experiments in two statistical learning settings such as large-scale overlapping group Lasso and doubly regularized support vector machine. In each setting, we introduce novel and efficient solutions for solving sub-problems. The numerical results demonstrate the efficacy and accuracy of SLIN.

Cancer Diagnosis and Disease Gene Identification via Statistical Machine Learning

Diagnosing cancer and identifying the disease gene by using DNA microarray gene expression data are the hot topics in current bioinformatics. This paper is devoted to the latest development in cancer diagnosis and gene selection via statistical machine learning. A support vector machine is firstly introduced for the binary cancer diagnosis. Then, 1-norm support vector machine, doubly regularized support vector machine, adaptive huberized support vector machine and other extensions are presented to improve the performance of gene selection. Lasso, elastic net, partly adaptive elastic net, group lasso, sparse group lasso, adaptive sparse group lasso and other sparse regression methods are also introduced for performing simultaneous binary cancer classification and gene selection. In addition to introducing three strategies for reducing multiclass to binary, methods of directly considering all classes of data in a learning model (multi_class support vector, sparse multinomial regression, adaptive multinomial regression and so on) are presented for performing multiple cancer diagnosis. Limitations and promising directions are also discussed.

Predicting future suicidal behaviour in young adults, with different machine learning techniques: A population-based longitudinal study

The predictive accuracy of suicidal behaviour has not improved over the last decades. We aimed to explore the potential of machine learning to predict future suicidal behaviour using population-based longitudinal data. Baseline risk data assessed within the Scottish wellbeing study, in which 3508 young adults (18-34 years) completed a battery of psychological measures, were used to predict both suicide ideation and suicide attempts at one-year follow-up. The performance of the following algorithms was compared: regular logistic regression, K-nearest neighbors, classification tree, random forests, gradient boosting and support vector machine. At one year follow up, 2428 respondents (71%) finished the second assessment. 336 respondents (14%) reported suicide ideation between baseline and follow up, and 50 (2%) reported a suicide attempt. All performance metrics were highly similar across methods. The random forest algorithm was the best algorithm to predict suicide ideation (AUC 0.83, PPV 0.52, BA 0.74) and the gradient boosting to predict suicide attempt (AUC 0.80, PPV 0.10, BA 0.69). The number of respondents with suicidal behaviour at follow up was small. We only had data on psychological risk factors, limiting the potential of the more complex machine learning algorithms to outperform regular logistical regression. When applied to population-based longitudinal data containing multiple psychological measurements, machine learning techniques did not significantly improve the predictive accuracy of suicidal behaviour. Adding more detailed data on for example employment, education or previous health care uptake, might result in better performance of machine learning over regular logistical regression.

Structural improved regular simplex support vector machine for multiclass classification

Although the structural regularized support vector machine (SRSVM) can enhance the generalization capability of the standard support vector machine (SVM), its current version is used only for binary classification. To make SRSVM adapt to the K-class classification, the most direct approach is combining it with partitioning strategies, which may however lead to the following shortcomings: (1) Extracting structural information repeatedly for individual classifiers based on different class partitions increases the computational complexity. (2) Individual classifiers can hardly utilize complete data structural information. Under the basic framework of regular simplex support vector machine (RSSVM), we developed a novel structural improved regular simplex support vector machine (SIRSSVM). SIRSSVM generates only a single primal optimization problem, into which the data structural information within all classes is embedded, rather than using only partial structural information to construct individual classifiers as partitioning strategies do. Additionally, we modified the sequential minimization optimization (SMO)-type solver for RSSVM to adapt the proposed SIRSSVM model. Experimental results verified that our SIRSSVM could achieve excellent performance on both generalization capability and training efficiency.

Robust twin support vector regression based on Huber loss function

Construction of robust regression learning models to fit data with noise is an important and challenging problem of data regression. One of the ways to tackle this problem is the selection of a proper loss function showing insensitivity to noise present in the data. Since Huber function has the property that inputs with large deviations of misfit are penalized linearly and small errors are squared, we present novel robust regularized twin support vector machines for data regression based on Huber and e-insensitive Huber loss functions in this study. The proposed regression models result in solving a pair of strongly convex minimization problems in simple form in primal whose solutions are obtained by functional and Newton–Armijo iterative algorithms. The finite convergence of Newton–Armijo algorithm is proved. Numerical tests are performed on noisy synthetic and benchmark datasets, and their results are compared with few popular regression learning algorithms. The comparative study clearly shows the robustness of the proposed regression methods and further demonstrates their effectiveness and suitability.

Non-interior-point smoothing Newton method for CP revisited and its application to support vector machines

Non-interior-point smoothing Newton method (SNM) for optimization have been widely studied for over three decades. SNM is a popular approach for solving small- and medium-scale complementarity problem (CP) and many optimization problems. The main purpose of this paper is to revisit the SNM and show that the Hessian matrix in SNM becomes increasingly ill-conditioned while smoothing parameter approaches to zero, which leads to their practical use remains limited due to computational difficulties in solving large-scale CP. To tackle this, we redesign a new smoothing method, called accelerated preconditioned smoothing method (APSM) for the efficient solution of regularized support vector machines in machine learning. With the help of suitable preconditioner, we can correct the ill-conditioning of associated smoothing Hessian matrix and thereby the associated smoothing Hessian equation can be solved in a few of iterations by using iterative methods in linear algebra. Two accelerated techniques are designed in the paper to reduce our computation time. Finally we present numerical experiments to support our theoretical guarantees and test accelerated convergence obtained by APSM. The result showed that APSM computes faster than state of art algorithm without reducing classification accuracy.