Nonparallel Classifier Research Articles

Intuitionistic fuzzy twin proximal SVM with fuzzy hyperplane and its application in EEG signal classification

Twin support vector machine (TSVM) is a contemporary machine learning technique to tackle classification and regression problems. However, TSVM lacks the ability to differentiate between support vectors and noises since it neglects the positional information of input data samples, and hence it is sensitive to noises. Additionally, it fails to consider uncertainties associated with the data, which reduces its capacity for generalization. To address these drawbacks, we propose a novel fuzzy hyperplane based intuitionistic fuzzy twin proximal support vector machine. The first significant feature of the proposed approach is that it gives an intuitionistic fuzzy number based on the relevance to each data vector. This efficiently reduces the impact of noise and outliers by leveraging the local neighborhood information within the data points by incorporating membership and non-membership weights. Secondly, all the parameters present in the model are fuzzy variables, including the offset term and the elements of the normal vector. The suggested fuzzy hyperplane successfully reflects the inherent ambiguity prevalent in real-world categorization problems by reflecting vagueness in the input data through the use of fuzzy variables. The model’s efficiency is enhanced by solving two systems of linear equations to obtain two non-parallel classifiers rather than solving two quadratic programming problems as in standard TSVM. Utilizing non-linear kernel functions within the feature space enables the method to effectively identify complex patterns or non-linear relationships within the datasets. In order to demonstrate the effectiveness of the suggested approach, extensive computer experiments have been conducted on a set of eighteen benchmark datasets with both linear and non-linear kernels. In addition, rigorous statistical analysis, including Friedman and post-hoc Nemenyi tests, have been employed to assess the significance of the observed performance differences. Furthermore, we also performed numerical experiments utilizing linear, Gaussian, and polynomial kernels to classify electroencephalogram (EEG) signals. The outcomes of the experiment are analyzed in terms of average accuracy, processing time, and F-measure. The results demonstrate that the proposed method outperforms existing methods and achieves better generalization.

Twin robust matrix machine for intelligent fault identification of outlier samples in roller bearing

In the industrial processes, the intelligent fault diagnosis related to signal analysis and pattern recognition is an important step to ensure the health of mechanical equipment. A popular intelligent monitor method as it has been, support matrix machine (SMM) enables to use the two-dimensional features extracted from vibration signals to build model. The core of the SMM is to extract structure information within matrix by minimizing nuclear norm to approximate the rank of the matrix. However, the nuclear norm has limited performance to eliminate the noise contained in structure information. Furthermore, features extracted from vibration signals often become outliers, and SMM is sensitive to outliers in training data. Therefore, a novel nonparallel classifier called twin robust matrix machine (TRMM) is proposed and applied to roller bearing fault diagnosis. TRMM can not only fully leverage the low-rank structure information, but also has the following novelties. First, TRMM uses the truncated nuclear norm as the low-rank constraint, to pay more attention to the large singular values related to the main structure information. Further, the ramp loss is used in TRMM as the loss function, which reduces the loss penalty for outlier sample and make TRMM insensitive to outlier samples. Finally, the accelerated proximal gradient (APG) is devised to solve the resulting optimization problem. Experimental results show that the proposed method has excellent fault diagnosis performance, especially in the case of existing outlier samples.

A Multiclass Nonparallel Parametric-Margin Support Vector Machine

The twin parametric-margin support vector machine (TPMSVM) is an excellent kernel-based nonparallel classifier. However, TPMSVM was originally designed for binary classification, which is unsuitable for real-world multiclass applications. Therefore, this paper extends TPMSVM for multiclass classification and proposes a novel K multiclass nonparallel parametric-margin support vector machine (MNP-KSVC). Specifically, our MNP-KSVC enjoys the following characteristics. (1) Under the “one-versus-one-versus-rest” multiclass framework, MNP-KSVC encodes the complicated multiclass learning task into a series of subproblems with the ternary output {−1,0,+1}. In contrast to the “one-versus-one” or “one-versus-rest” strategy, each subproblem not only focuses on separating the two selected class instances but also considers the side information of the remaining class instances. (2) MNP-KSVC aims to find a pair of nonparallel parametric-margin hyperplanes for each subproblem. As a result, these hyperplanes are closer to their corresponding class and at least one distance away from the other class. At the same time, they attempt to bound the remaining class instances into an insensitive region. (3) MNP-KSVC utilizes a hybrid classification and regression loss joined with the regularization to formulate its optimization model. Then, the optimal solutions are derived from the corresponding dual problems. Finally, we conduct numerical experiments to compare the proposed method with four state-of-the-art multiclass models: Multi-SVM, MBSVM, MTPMSVM, and Twin-KSVC. Experimental results demonstrate the feasibility and effectiveness of MNP-KSVC in terms of multiclass accuracy and learning time.

NPrSVM: Nonparallel sparse projection support vector machine with efficient algorithm

The recently proposed projection twin support vector machine (PTSVM) is an excellent nonparallel classifier. However, PTSVM employs the least-squares loss function to measure its within-class empirical risk, resulting in several drawbacks, such as non-sparseness for decision, sensitivity to outliers, expensive matrix inversion, and inconsistency in the linear and nonlinear models. To alleviate these issues, in this paper, we propose a novel nonparallel sparse projection support vector machine (NPrSVM). Different from the original PTSVM that squeezes the projected values of within-class instances to its own class center, NPrSVM aims to cluster them as much as possible within an insensitive tube. Specifically, our NPrSVM owns the following attractive merits: (i) Benefiting from the L1-norm symmetric Hinge loss function, NPrSVM not only enjoys sparseness for decision but also improves robustness to outliers. (ii) The elegant formulation of dual problems in NPrSVM no longer involves the matrix inversion during the training procedure. This greatly saves the computing time compared to PTSVM. (iii) While the nonlinear formulation of PTSVM is not the direct extension of linear PTSVM, the linear and nonlinear versions of our NPrSVM are consistent. (iv) An efficient dual coordinate descent algorithm is further designed for NPrSVM to handle large-scale classification. Finally, the feasibility and effectiveness of NPrSVM are validated by extensive experiments on both synthetic and real-world datasets.

P.374 Cortisol and DHEAS related to metabolic side effects in patients with schizophrenia

The recently proposed projection twin support vector machine (PTSVM) is an excellent nonparallel classifier. However, PTSVM employs the least-squares loss function to measure its within-class empirical risk, resulting in several drawbacks, such as non-sparseness for decision, sensitivity to outliers, expensive matrix inversion, and inconsistency in the linear and nonlinear models. To alleviate these issues, in this paper, we propose a novel nonparallel sparse projection support vector machine (NPrSVM). Different from the original PTSVM that squeezes the projected values of within-class instances to its own class center, NPrSVM aims to cluster them as much as possible within an insensitive tube. Specifically, our NPrSVM owns the following attractive merits: (i) Benefiting from the L1-norm symmetric Hinge loss function, NPrSVM not only enjoys sparseness for decision but also improves robustness to outliers. (ii) The elegant formulation of dual problems in NPrSVM no longer involves the matrix inversion during the training procedure. This greatly saves the computing time compared to PTSVM. (iii) While the nonlinear formulation of PTSVM is not the direct extension of linear PTSVM, the linear and nonlinear versions of our NPrSVM are consistent. (iv) An efficient dual coordinate descent algorithm is further designed for NPrSVM to handle large-scale classification. Finally, the feasibility and effectiveness of NPrSVM are validated by extensive experiments on both synthetic and real-world datasets.

ν-projection twin support vector machine for pattern classification

In this paper, we improve the projection twin support vector machine (PTSVM) to a novel nonparallel classifier, termed as ν-PTSVM. Specifically, our ν-PTSVM aims to seek an optimal projection for each class such that, in each projection direction, instances of their own class are clustered around their class center while keep instances of the other class at least one distance away from such center. Different from PTSVM, our ν-PTSVM enjoys the following characteristics: (i) ν-PTSVM is equipped by a more theoretically sound parameter ν, which can be used to control the bounds of fraction of both support vectors and margin-error instances. (ii) By reformulating the least-square loss of within-class instances in primal problems of ν-PTSVM, its dual problems no longer involve the time-costly matrix inversion. (iii) ν-PTSVM behaves consistent between its linear and nonlinear cases. Namely, the kernel trick can be applied directly to ν-PTSVM for its nonlinear extension. Experimental evaluations on both synthetic and real-world datasets demonstrate the feasibility and effectiveness of the proposed approach.

A novel projection nonparallel support vector machine for pattern classification

In this paper, we propose a novel nonparallel classifier termed as projection nonparallel support vector machine (PNPSVM), which is fully different from the existing nonparallel classifiers. The new classifier needs two steps to obtain the optimal proximal hyperplanes. The first step is to obtain two projection directions, which could achieve maximum class separability by minimizing the within-class distance and maximizing the between-class distance simultaneously, to be treated as the normal vectors of the optimal proximal hyperplanes, and the second step is to determine the specific locations of the optimal proximal hyperplanes based on an appropriate central sample. Furthermore, the improved successive overrelaxation (SOR) algorithm is applied to solve our PNPSVM. The incomparable advantages of this paper can be summarized as follows: (1) implementing the structural risk minimization principle in the primal problems; (2) utilizing the potential structural information of data by considering both the tightness between the similar patterns and the discrepancy between the dissimilar pairs; (3) the kernel trick can be applied directly since the dual problems have the similar elegant formulation as that of standard support vector machine; (4) SOR technique is introduced to solve our optimization problems. The comprehensive experimental results on an artificial dataset and twenty-four UCI datasets demonstrate the effectiveness of our method in classification accuracy.

Learning with label proportions based on nonparallel support vector machines

Learning a classifier from groups of unlabeled data, only knowing, for each group, the proportions of data with particular labels, is an important branch of classification tasks that are conceivable in many practical applications. In this paper, we proposed a novel solution for the problem of learning with label proportions (LLP) based on nonparallel support vector machines, termed as proportion-NPSVM, which can improve the classifiers to be a pair of nonparallel classification hyperplanes. The unique property of our method is that it only needs to solve a pair of smaller quadratic programming problems. Moreover, it can efficiently incorporate the known group label proportions with the latent unknown observation labels into one optimization model under a large-margin framework. Compared to the existing approaches, there are several advantages shown as follows: 1) it does not need to make restrictive assumptions on the training data; 2) nonparallel classifiers can be achieved without computing the large inverse matrices; 3) the optimization model can be effectively solved by using the alternative strategy with SMO technique or SOR method; 4) proportion-NPSVM has better generalization ability. Sufficient experimental results on both binary-classes and multi-classes data sets show the efficiency of our proposed method in classification accuracy, which prove the state-of-the-art method for LLP problems compared with competing algorithms.

Weighted multicategory nonparallel planes SVM classifiers

We formulate K-category non-parallel classifiers by using “One-Versus-All” (OVA) approach: Multicategory Generalized Eigenvalue Proximal Support Vector Machine (MGEPSVM) and Multicategory Improved Generalized Eigenvalue Proximal Support Vector Machine (MIGEPSVM). These classifiers generate K nonparallel decision surfaces for K classes where each surface is closest to its corresponding class and the farthest from the rest of the classes. However, in some cases, this approach leads to poor performance due to the resultant of two unbalanced classes. To minimize the effect of unbalanced classes, we proposed Weighted MGEPSVM (WMGEPSVM) and Weighted MIGEPSVM (WMIGEPSVM) where the weight factor is determined by using proposed modified balancing technique. In this paper, we also propose K-category non-parallel classifiers by using “One-Versus-One” (OVO) approach: Multicategory Generalized Eigenvalue Proximal Support Vector Machine (OVO-MGEPSVM) and Multicategory Improved Generalized Eigenvalue Proximal Support Vector Machine (OVO-MIGEPSVM). To check the robustness of the model numerical experiments have been carried out on twelve different benchmark datasets. Experimental results indicate that WMGEPSVM and WMIGEPSVM improve the testing accuracy of MGEPSVM and MIGEPSVM, while maintaining the computation time. WMGEPSVM and MGEPSVM are superior to multi-class SVM, MLSTSVM and comparable with multicategory Proximal Support Vector Machine (PSVM).

Regularization feature selection projection twin support vector machine via exterior penalty

In the past years, non-parallel plane classifiers that seek projection direction instead of hyperplane for each class have attracted much attention, such as the multi-weight vector projection support vector machine (MVSVM) and the projection twin support vector machine (PTSVM). Instead of solving two generalized eigenvalue problems in MVSVM, PTSVM solves two related SVM-type problems to obtain the two projection directions by solving two smaller quadratic programming problems, similar to twin support vector machine. In order to suppress input space features, we propose a novel non-parallel classifier to automatically select significant features, called regularization feature selection projection twin support vector machine (RFSPTSVM). In contrast to the PTSVM, we first incorporate a regularization term to ensure the optimization problems are convex, and then replace all the terms with L1-norm ones. By minimizing an exterior penalty function of the linear programming problem and using a fast generalized Newton algorithm, our RFSPTSVM obtains very sparse solutions. For nonlinear case, the method utilizes minimal number of kernel functions. The experimental results on toy datasets, Myeloma dataset, several UCI benchmark datasets, and NDCC generated datasets show the feasibility and effectiveness of the proposed method.

A second-order cone programming formulation for twin support vector machines

Second-order cone programming (SOCP) formulations have received increasing attention as robust optimization schemes for Support Vector Machine (SVM) classification. These formulations study the worst-case setting for class-conditional densities, leading to potentially more effective classifiers in terms of performance compared to the standard SVM formulation. In this work we propose an SOCP extension for Twin SVM, a recently developed classification approach that constructs two nonparallel classifiers. The linear and kernel-based SOCP formulations for Twin SVM are derived, while the duality analysis provides interesting geometrical properties of the proposed method. Experiments on benchmark datasets demonstrate the virtues of our approach in terms of classification performance compared to alternative SVM methods.

Nonparallel Support Vector Machine Based on One Optimization Problem for Pattern Recognition

In this paper, we present a novel nonparallel support vector machine based on one optimization problem (NSVMOOP) for binary classification. Our NSVMOOP is formulated aiming to separate classes from the largest possible angle between the normal vectors and the decision hyperplanes in the feature space, at the same time implementing the structural risk minimization principle. Different from other nonparallel classifiers, such as the representative twin support vector machine, it constructs two nonparallel hyperplanes simultaneously by solving a single quadratic programming problem, on which a modified sequential minimization optimization algorithm is explored. The NSVMOOP is analyzed theoretically and implemented experimentally. Experimental results on both artificial and publicly available benchmark datasets show its feasibility and effectiveness.

Non-parallel support vector classifiers with different loss functions

This paper introduces a general framework of non-parallel support vector machines, which involves a regularization term, a scatter loss and a misclassification loss. When dealing with binary problems, the framework with proper losses covers some existing non-parallel classifiers, such as multisurface proximal support vector machine via generalized eigenvalues, twin support vector machines, and its least squares version. The possibility of incorporating different existing scatter and misclassification loss functions into the general framework is discussed. Moreover, in contrast with the mentioned methods, which applies kernel-generated surface, we directly apply the kernel trick in the dual and then obtain nonparametric models. Therefore, one does not need to formulate two different primal problems for the linear and nonlinear kernel respectively. In addition, experimental results are given to illustrate the performance of different loss functions.

A multi-instance learning algorithm based on nonparallel classifier

In this paper, we proposed a new Multiple-Instance Learning (MIL) method based on nonparallel classifier (called MI-NSVM). The method is mainly divided into two steps. The first step is to generate a spare hyperplane and estimate the score of each instance in positive bags. For the second step, MI-NSVM seeks the “most positive” instance of each positive bag by the information obtained in the first step, and then generates the second hyperplane. MI-NSVM is a useful extension of twin SVM and has the same advantages as it. All experiments show that our method is superior to the traditional MI-SVM and MI-TSVM in both computation time and classification accuracy.

Nonparallel Support Vector Machines for Pattern Classification

We propose a novel nonparallel classifier, called nonparallel support vector machine (NPSVM), for binary classification. Our NPSVM that is fully different from the existing nonparallel classifiers, such as the generalized eigenvalue proximal support vector machine (GEPSVM) and the twin support vector machine (TWSVM), has several incomparable advantages: 1) two primal problems are constructed implementing the structural risk minimization principle; 2) the dual problems of these two primal problems have the same advantages as that of the standard SVMs, so that the kernel trick can be applied directly, while existing TWSVMs have to construct another two primal problems for nonlinear cases based on the approximate kernel-generated surfaces, furthermore, their nonlinear problems cannot degenerate to the linear case even the linear kernel is used; 3) the dual problems have the same elegant formulation with that of standard SVMs and can certainly be solved efficiently by sequential minimization optimization algorithm, while existing GEPSVM or TWSVMs are not suitable for large scale problems; 4) it has the inherent sparseness as standard SVMs; 5) existing TWSVMs are only the special cases of the NPSVM when the parameters of which are appropriately chosen. Experimental results on lots of datasets show the effectiveness of our method in both sparseness and classification accuracy, and therefore, confirm the above conclusion further. In some sense, our NPSVM is a new starting point of nonparallel classifiers.

Efficient sparse nonparallel support vector machines for classification

In this paper, we propose a novel nonparallel classifier, named sparse nonparallel support vector machine (SNSVM), for binary classification. Different with the existing nonparallel classifiers, such as the twin support vector machines (TWSVMs), SNSVM has several advantages: It constructs two convex quadratic programming problems for both linear and nonlinear cases, which can be solved efficiently by successive overrelaxation technique; it does not need to compute the inverse matrices any more before training; it has the similar sparseness with standard SVMs; it degenerates to the TWSVMs when the parameters are appropriately chosen. Therefore, SNSVM is certainly superior to them theoretically. Experimental results on lots of data sets show the effectiveness of our method in both sparseness and classification accuracy and, therefore, confirm the above conclusions further.

Local and Global Regularized Twin SVM

The generalized eigenvalue proximal support vector machine (GEPSVM) and twin support vector machine (TWSVM) was proposed by Mangasarian and Jayadeva respectively, which aroused the interest of academia for its less computation cost and better generalization ability. They use the nonparallel hyperplane classifiers to solve the classification problem. Different from traditionally local or global TWSVM methods, a new Twin SVM algorithm called Local and Global Regularized Twin SVM (TWSVMLG) is proposed in this paper. A global regularizer was imposed across local models to smooth the data labels predicted by those local classifiers and avoid overfitting risk for the local classifiers. The classifier could get stronger discriminating ability when exploring local and global information than traditional algorithms. Finally some experimental results are presented to show the effectiveness of our algorithm.

A feature selection method for nonparallel plane support vector machine classification

Over the past decades, 1-norm techniques based on algorithms are widely used to suppress input features. Quite different from traditional 1-norm support vector machine (SVM), direct 1-norm optimization based on the primal problem of nonparallel plane classifiers like generalized proximal support vector machine, twin support vector machine (TWSVM) and least squares twin support vector machine (LSTSVM) are not capable of generating very sparse solutions that are vital for classification and can make them easier to store and faster to compute. To address the issue, in this paper, we develop a feature selection method for LSTSVM, called a feature selection method for nonparallel plane support vector machine classification (FLSTSVM), which is specially designed for strong feature suppression. We incorporate a Tikhonov regularization term to the objective of LSTSVM, and then minimize its 1-norm measure. Solution of FLSTSVM can follow directly from solving two smaller quadratic programming problems (QPPs) arising from two primal QPPs as opposed to two dual ones in TWSVM. FLSTSVM is capable of generating very sparse solutions. This means that FLSTSVM can reduce input features, for the linear case. When a nonlinear classifier is used, few kernel functions determine the classifier. In addition to having strong feature suppression, the edge of our method still lies in its faster computing time compared to that of TWSVM, Newton Method for Linear Programming SVM (NLPSVM) and LPNewton. Lastly, this algorithm is compared on public data sets, as well as an Exclusive Or (XOR) example.