Universum Support Vector Machine Research Articles

Feature selection by Universum embedding

Feature selection in classification is an important task in machine learning. Inspired by the success of Universum support vector machine proposed by Weston et al. on improving the classification ability of classical support vector machine, this paper considers a special type of Universum and further lets it play its role in both useful feature identification and separating hyperplane construction, aiming to improve both the feature selection ability and classification performance of Universum support vector machine. By introducing this special Universum, a redundant feature can be identified by observing whether some Universum sample is useful. In fact, we prove that by observing the dual solution of the optimization problem, useful features can be selected from a set satisfying some properties. Due to the introduction of these extra Universum samples, it needs to cope with a large-scale optimization problem. To improve the training efficiency, we modify the sequential minimal optimization algorithm and further combine it with the coordinate descent technique to solve the proposed model. Experimental results on artificial datasets, benchmark datasets, and text classification datasets demonstrate that the proposed method improves the classification performance of support vector machine and Universum support vector machine, and also has good feature selection ability.

Multi-view universum support vector machines with insensitive pinball loss

Multi-view learning (MVL) is a promising field that seeks to make the most of information shared across different views, ultimately improving generalization performance. Numerous multi-view support vector machines (SVMs) have been proposed and shown great success. However, hinge loss is widely used by existing methods, which makes these SVMs sensitive to noise and unstable when re-sampling. Furthermore, many researchers have demonstrated the effectiveness of universum data for supervised learning. This paper proposes a novel MVL method called multi-view universum support vector machine with insensitive pinball loss (Pin-MvUSVM). Specifically, we first integrate the pinball loss into the multi-view learning framework, which is dependent on the quantization distance and is robust to noise. Then, we utilize universum data to explore prior knowledge for MVL, which improves the generalization performance. Moreover, we develop multi-view universum twin support vector machine with insensitive pinball loss (Pin-MvUTSVM) as a low-complexity alternative to improve computational efficiency. It solves a pair of smaller-scale quadratic programming problems (QPPs) and utilizes the dual form of the model to solve two convex optimization problems for obtaining the prediction classifier. We conduct classification experiments on four multi-view datasets, and our results show that the proposed convex models outperform several state-of-the-art models.

1-Norm twin random vector functional link networks based on Universum data for leaf disease detection

Due to rapid climate change and man-made activities, the types of leaf diseases are gradually increasing. As a result, taking the essential measures to recognize and diagnose the problem becomes a top priority. Though manual techniques for identifying leaf disease have been used in the past, research-based studies may produce a better result and overcome the disadvantages of the manual process. Nowadays, the random vector functional link (RVFL) network gaining popularity day by day due to the better generalization ability for various applications. In the spirit of the popular twin support vector machine (TSVM) and the twin variant of the random vector functional link (TRVFL) network, a novel twin RVFL (TRVFL1norm) is suggested in this work. TRVFL1norm overcomes the inherent limitation of TRVFL, which loses the sparsity due to the use of 2 norm and also gives a high penalty to outlier points which leads to degrade the generalization performance. Moreover, it is well known that Universum data can be used as prior knowledge in the training model. Therefore, to make the model more robust and generalized, we propose another model, TRVFL1norm based on the Universum data (UTRVFL1norm). UTRVFL1norm possesses the advantages of both 1-norm as well as Universum data. For both TRVFL1norm and UTRVFL1norm, few of the components in the output matrix will be zero due to sparsity hence, the resultant classifier must depend on the lesser output nodes compared to TRVFL. To verify model’s performance, we have compared the classification performance of proposed TRVFL1norm and UTRVFL1norm models with support vector machine (SVM), Universum SVM (USVM), twin support vector machine (TSVM), Universum TSVM (UTSVM), random vector functional link (RVFL), multilayer perceptron (MLP), convolutional neural network (CNN) and recurrent neural network (RNN) on few interesting benchmark datasets. Further, we have also checked the performance of TRVFL1norm and UTRVFL1norm for leaf disease detection as one of the possible applications. The experimental results indicate the efficiency and applicability of the proposed models.

Breast Cancer Classification by Gene Expression Analysis using Hybrid Feature Selection and Hyper-heuristic Adaptive Universum Support Vector Machine

Comprehensive assessments of the molecular characteristics of breast cancer from gene expression patterns can aid in the early identification and treatment of tumor patients. The enormous scale of gene expression data obtained through microarray sequencing increases the difficulty of training the classifier due to large-scale features. Selecting pivotal gene features can minimize high dimensionality and the classifier complexity with improved breast cancer detection accuracy. However, traditional filter and wrapper-based selection methods have scalability and adaptability issues in handling complex gene features. This paper presents a hybrid feature selection method of Mutual Information Maximization - Improved Moth Flame Optimization (MIM-IMFO) for gene selection along with an advanced Hyper-heuristic Adaptive Universum Support classification model Vector Machine (HH-AUSVM) to improve cancer detection rates. The hybrid gene selection method is developed by performing filter-based selection using MIM in the first stage followed by the wrapper method in the second stage, to obtain the pivotal features and remove the inappropriate ones. This method improves standard MFO by a hybrid exploration/exploitation phase to accomplish a better trade-off between exploration and exploitation phases. The classifier HH-AUSVM is formulated by integrating the Adaptive Universum learning approach to the hyper- heuristics-based parameter optimized SVM to tackle the class samples imbalance problem. Evaluated on breast cancer gene expression datasets from Mendeley Data Repository, this proposed MIM-IMFO gene selection-based HH-AUSVM classification approach provided better breast cancer detection with high accuracies of 95.67%, 96.52%, 97.97% and 95.5% and less processing time of 4.28, 3.17, 9.45 and 6.31 seconds, respectively.

Reduced-Kernel Weighted Extreme Learning Machine Using Universum Data in Feature Space (RKWELM-UFS) to Handle Binary Class Imbalanced Dataset Classification

Class imbalance is a phenomenon of asymmetry that degrades the performance of traditional classification algorithms such as the Support Vector Machine (SVM) and Extreme Learning Machine (ELM). Various modifications of SVM and ELM have been proposed to handle the class imbalance problem, which focus on different aspects to resolve the class imbalance. The Universum Support Vector Machine (USVM) incorporates the prior information in the classification model by adding Universum data to the training data to handle the class imbalance problem. Various other modifications of SVM have been proposed which use Universum data in the classification model generation. Moreover, the existing ELM-based classification models intended to handle class imbalance do not consider the prior information about the data distribution for training. An ELM-based classification model creates two symmetry planes, one for each class. The Universum-based ELM classification model tries to create a third plane between the two symmetric planes using Universum data. This paper proposes a novel hybrid framework called Reduced-Kernel Weighted Extreme Learning Machine Using Universum Data in Feature Space (RKWELM-UFS) to handle the classification of binary class-imbalanced problems. The proposed RKWELM-UFS combines the Universum learning method with a Reduced-Kernelized Weighted Extreme Learning Machine (RKWELM) for the first time to inherit the advantages of both techniques. To generate efficient Universum samples in the feature space, this work uses the kernel trick. The performance of the proposed method is evaluated using 44 benchmark binary class-imbalanced datasets. The proposed method is compared with 10 state-of-the-art classifiers using AUC and G-mean. The statistical t-test and Wilcoxon signed-rank test are used to quantify the performance enhancement of the proposed RKWELM-UFS compared to other evaluated classifiers.

Universum based Lagrangian twin bounded support vector machine to classify EEG signals

Background and objectiveThe detection of brain-related problems and neurological disorders like epilepsy, sleep disorder, and so on is done by using electroencephalogram (EEG) signals which contain noisy signals and outliers. Universum data contains a set of a sample that does not belong to any of the concerned classes and serves as the advanced knowledge about the data distribution. Earlier information has been utilized viably in improving classification performance. Recently a novel universum support vector machine (USVM) was proposed for EEG signal classification and further, a universum twin support vector machine (UTWSVM) was proposed based on USVM to improve the performance. Inspired by USVM and UTWSVM, this paper suggests a novel method called universum based Lagrangian twin bounded support vector machine (ULTBSVM), where universum data is utilized to incorporate the prior information about the data distribution to classify healthy and seizure EEG signals. MethodsIn the proposed ULTBSVM the square of the 2-norm of the slack variables is used to formulate the objective function strongly convex; hence it always gives unique solutions. Unlike twin support vector machine (TWSVM) and universum twin support vector machine (UTWSVM), the proposed ULTBSVM is having regularization terms that follow the structural risk minimization (SRM) principle and enhance the stability in the dual formulations, make the model well-posed and prevents the overfitting problem. Here, interracial EEG data have been considered as universum data to classify healthy and seizure signals. Several feature extraction techniques have been implemented to get important noiseless features. ResultsSeveral EEG datasets, as well as publicly available UCI datasets, are utilized to assess the performance of the proposed method. An analytical comparison has been performed of the proposed method with USVM and UTWSVM to detect seizure and healthy signals and for real-world data, the ULTBSVM is compared with the universum based models as well as TWSVM and the proposed method gives better results in most of the cases as compared to the other methods. ConclusionThe results clearly show that ULTBSVM is a potential method for the classification of EEG signals as well as real-world datasets having interracial data as universum data. Here we have used universum points for the binary class classification problem, but one can extend and use it for multi-class classification problems as well.

Robust twin bounded support vector machines for outliers and imbalanced data

Truncated loss functions are robust to class noise and outliers. A robust twin bounded support vector machine is proposed in this paper that truncates the growth of its loss functions at a pre-specified point, thus, flattens the function that pre-specified score afterwards. Moreover, to make the proposed method capable of handling datasets with different imbalance ratio, cost-sensitive learning is implemented by scaling the total error of the classes based on the number of samples of each class. However, the adopted loss functions take a non-convex structure which does not always assure global optimum. To handle this issue, we suggest concave-convex procedure (CCCP) to ensure global convergence by decomposing the cost functions into additions of one convex and one concave part. The behaviour of the proposed method for varied imbalance ratio is analysed experimentally and depicted graphically. Classification performance of the proposed method in terms of AUC, F-Measure and G-Mean is compared with other related methods, viz. Hinge loss support vector machine (SVM), Ramp loss SVM (RSVM), twin SVM (TWSVM), twin bounded SVM (TBSVM), Pinball loss SVM (pin-SVM), entropy-based fuzzy SVM (EFSVM), non-parallel hyperplane Universum SVM (U-NHSVM), stochastic gradient twin support vector machine (SGTSVM), k-nearest neighbor (KNN)-based maximum margin and minimum volume hyper-sphere machine (KNN-M3VHM) and affinity and class probability-based fuzzy SVM (ACFSVM) on several real-world datasets with imbalance ratio raging from low to high. Further, to establish the significance of the proposed method in pattern classification, pair-wise statistical comparison of the methods is performed based on their average ranks on AUC. Experimental results are convincing.

Diagnosis of Alzheimer's disease using universum support vector machine based recursive feature elimination (USVM-RFE)

Alzheimer's disease is one of the most common causes of death in today's world. Magnetic resonance imaging (MRI) provides an efficient and non-invasive approach for diagnosis of Alzheimer's disease. Efficient feature extraction techniques are needed for accurate classification of MRI images. Motivated by the work on support vector machine based recursive feature elimination (SVM-RFE) [16], we propose a novel feature selection technique to incorporate prior information about data distribution in the recursive feature elimination process. Our method is termed as universum support vector machine based recursive feature elimination (USVM-RFE). The proposed method provides global information about data in the RFE process as compared to the local approach of feature selection in SVM-RFE. We also present the application of feature selection and classification algorithms on both voxel based as well as volume based morphometry analysis of structural MRI images (ADNI database). Feature selection is performed using MRI data of brain tissues such as gray matter, white matter, and cerebrospinal fluid. USVM-RFE provides improvement over SVM-RFE in classification of control normal (CN), mild cognitive impairment (MCI), and Alzheimer's disease (AD) subjects. Moreover, better accuracy is obtained by USVM-RFE with lesser number of features in comparison to SVM-RFE. This leads to identification of prominent brain regions for feature selection and classification of MRI images. The highest accuracies obtained by our method for classification of CN vs AD, CN vs MCI, and MCI vs AD are 100%, 90%, and 73.68%, respectively.

An improved non-parallel Universum support vector machine and its safe sample screening rule

A novel non-parallel hyperplane Universum support vector machine (U-NHSVM) is proposed in this paper. Universum data with ensconced prior knowledge are exploited by a non-parallel hyperplane support vector machine. In contrast to other algorithms, the proposed U-NHSVM shows flexibility by exploiting the prior knowledge ensconced in Universum and provides consistency by constructing two non-parallel hyperplanes simultaneously. With Universum, U-NHSVM is clearly effective but also time consuming. Therefore, a safe sample screening rule (SSSR) for U-NHSVM is also proposed based on its sparsity, termed SSSR-U-NHSVM. Because only the non-SVs are excluded from both labelled and Universum samples, the efficiency of SSSR-U-NHSVM is extremely improved while the accuracy is completely conserved. Numerical experiments on seventeen benchmark datasets and a Chinese wine dataset are carried out to demonstrate the validity of the proposed U-NHSVM and SSSR-U-NHSVM.

EEG signal classification using universum support vector machine

Support vector machine (SVM) has been used widely for classification of electroencephalogram (EEG) signals for the diagnosis of neurological disorders such as epilepsy and sleep disorders. SVM shows good generalization performance for high dimensional data due to its convex optimization problem. The incorporation of prior knowledge about the data leads to a better optimized classifier. Different types of EEG signals provide information about the distribution of EEG data. To include prior information in the classification of EEG signals, we propose a novel machine learning approach based on universum support vector machine (USVM) for classification. In our approach, the universum data points are generated by selecting universum from the EEG dataset itself which are the interictal EEG signals. This removes the effect of outliers on the generation of universum data. Further, to reduce the computation time, we use our approach of universum selection with universum twin support vector machine (UTSVM) which has less computational cost in comparison to traditional SVM. For checking the validity of our proposed methods, we use various feature extraction techniques for different datasets consisting of healthy and seizure signals. Several numerical experiments are performed on the generated datasets and the results of our proposed approach are compared with other baseline methods. Our proposed USVM and proposed UTSVM show better generalization performance compared to SVM, USVM, Twin SVM (TWSVM) and UTSVM. The proposed UTSVM has achieved highest classification accuracy of 99% for the healthy and seizure EEG signals.

A safe sample screening rule for Universum support vector machines

Universum support vector machine (U-SVM), due to its tremendous accuracy improvements, has been expanded and applied in all kinds of fields. Universum encodes related prior knowledge but does not belong to any class of interest. With Universum, the number of training samples and computational complexity are clearly increased. Inspired by the sparsity of SVMs, a safe sample screening rule (SSSR) for U-SVM is proposed in this paper. Our SSSR eliminates not only the labelled samples but also the Universum samples before training process, then the computational cost is dramatically reduced. Moreover, the same solution as the original problem can be obtained by utilizing our SSSR, that is, the training process is guaranteed to be accelerated safely. Besides, we extend our rule to the Universum twin support vector machine (U-TSVM), and the SSSR for U-TSVM is also discussed in this paper. To the best of our knowledge, SSSR is the only existing safe screening method for U-SVMs. Numerical experiments on seventeen benchmark datasets, ABCDETC dataset and Chinese wine dataset demonstrate that the computational cost can be dramatically reduced without sacrificing the optimality of the final solution by our SSSR.

Investor sentiment identification based on the universum SVM

Universum refers to additional samples which contain priori knowledge for classification but belonging to none of the class. It has been proved that universum positioned “in between” the two classes obtain better results. Since opinions on stock market defined as investor sentiment involve quite a number of neutral views, these neutral views can be used as universum samples to better identify investor sentiment. With universum samples, this paper uses support vector machine (SVM) to classify posts on stock forum. We define bullish views as positive samples, define bearish views as negative samples, and also further discuss the situation of a 3-class problem with neutral views. Compared with standard SVM, the empirical studies with universum samples in this paper show better performance for both 2- and 3-class classifications.

Semi-supervised support vector classification with self-constructed Universum

In this paper, we propose a strategy dealing with the semi-supervised classification problem, in which the support vector machine with self-constructed Universum is iteratively solved. Universum data, which do not belong to either class of interest, have been illustrated to encode some prior knowledge by representing meaningful concepts in the same domain as the problem at hand. Our new method is applied to seek more reliable positive and negative examples from the unlabeled dataset step by step, and the Universum support vector machine(U-SVM) is used iteratively. Different Universum data will result in different performance, so several effective approaches are explored to construct Universum datasets. Experimental results demonstrate that appropriately constructed Universum will improve the accuracy and reduce the number of iterations.

Development and evaluation of cost-sensitive universum-SVM.

Many machine learning applications involve analysis of high-dimensional data, where the number of input features is larger than/comparable to the number of data samples. Standard classification methods may not be sufficient for such data, and this provides motivation for nonstandard learning settings. One such new learning methodology is called learning through contradiction or Universum-support vector machine (U-SVM). Recent studies have shown U-SVM to be quite effective for sparse high-dimensional data sets. However, all these earlier studies have used balanced data sets with equal misclassification costs. This paper extends the U-SVM formulation to problems with different misclassification costs, and presents practical conditions for the effectiveness of this cost-sensitive U-SVM. Several empirical comparisons are presented to validate the proposed approach.

Practical Conditions for Effectiveness of the Universum Learning

Many applications of machine learning involve analysis of sparse high-dimensional data, in which the number of input features is larger than the number of data samples. Standard inductive learning methods may not be sufficient for such data, and this provides motivation for nonstandard learning settings. This paper investigates a new learning methodology called learning through contradictions or Universum support vector machine (U-SVM). U-SVM incorporates a priori knowledge about application data, in the form of additional Universum samples, into the learning process. This paper investigates possible advantages of U-SVM versus standard SVM, and describes the practical conditions necessary for the effectiveness of the U-SVM. These conditions are based on the analysis of the univariate histograms of projections of training samples onto the normal direction vector of (standard) SVM decision boundary. Several empirical comparisons are presented to illustrate the practical utility of the proposed approach.