Universum Samples Research Articles

Boosting Weakly Supervised Object Localization and Segmentation With Domain Adaption.

Weakly supervised object localization (WSOL), adopting only image-level annotations to learn the pixel-level localization model, can release human resources in the annotation process. Most one-stage WSOL methods learn the localization model with multi-instance learning, making them only activate discriminative object parts rather than the whole object. In our work, we attribute this problem to the domain shift between the training and test process of WSOL and provide a novel perspective that views WSOL as a domain adaption (DA) task. Under this perspective, a DA-WSOL pipeline is elaborated to better assist WSOL with DA approaches by considering the specificities for the adaption of WSOL. Our DA-WSOL pipeline can discern the source-related and the Universum samples from other target samples based on a proposed target sampling strategy and then utilize them to solve the sample unbalancing and label unmatching between the source and target domain of WSOL. Experiments show that our pipeline outperforms SOTA methods on three WSOL benchmarks and can improve the performance of downstream weakly supervised semantic segmentation tasks.

Feature selection by Universum embedding

Feature selection in classification is an important task in machine learning. Inspired by the success of Universum support vector machine proposed by Weston et al. on improving the classification ability of classical support vector machine, this paper considers a special type of Universum and further lets it play its role in both useful feature identification and separating hyperplane construction, aiming to improve both the feature selection ability and classification performance of Universum support vector machine. By introducing this special Universum, a redundant feature can be identified by observing whether some Universum sample is useful. In fact, we prove that by observing the dual solution of the optimization problem, useful features can be selected from a set satisfying some properties. Due to the introduction of these extra Universum samples, it needs to cope with a large-scale optimization problem. To improve the training efficiency, we modify the sequential minimal optimization algorithm and further combine it with the coordinate descent technique to solve the proposed model. Experimental results on artificial datasets, benchmark datasets, and text classification datasets demonstrate that the proposed method improves the classification performance of support vector machine and Universum support vector machine, and also has good feature selection ability.

Semi-supervised Multi-task Learning with Auxiliary data

Compared with single-task learning, multi-tasks can obtain better classifiers by the information provided by each task. In the process of multi-task data collection, we always focus on the target task data in the training process, and ignore the non-target task data and unlabeled data that may be contained in the target task. In response to this issue, this paper introduces auxiliary or Universum into semi-supervised multi-task problem, and proposes a multi-task support vector machine (SU-MTLSVM) method based on semi-supervised learning to handle the case where each task contains the labeled, unlabeled, and Universum samples in the training set. This method introduces Universum as prior knowledge and provides high-dimensional information for semi-supervised learning, and builds a unique classifier from a large amount of unlabeled data. We then use KKT conditions and Lagrangian method to optimize the formulation of the model, and get the model parameters. Finally, we collect different data sets in the experiment part, and compare the performance of multiple baselines with the proposed method. Experiments prove that the method proposed in this paper is more effective for multi-task applications.

Reduced-Kernel Weighted Extreme Learning Machine Using Universum Data in Feature Space (RKWELM-UFS) to Handle Binary Class Imbalanced Dataset Classification

Class imbalance is a phenomenon of asymmetry that degrades the performance of traditional classification algorithms such as the Support Vector Machine (SVM) and Extreme Learning Machine (ELM). Various modifications of SVM and ELM have been proposed to handle the class imbalance problem, which focus on different aspects to resolve the class imbalance. The Universum Support Vector Machine (USVM) incorporates the prior information in the classification model by adding Universum data to the training data to handle the class imbalance problem. Various other modifications of SVM have been proposed which use Universum data in the classification model generation. Moreover, the existing ELM-based classification models intended to handle class imbalance do not consider the prior information about the data distribution for training. An ELM-based classification model creates two symmetry planes, one for each class. The Universum-based ELM classification model tries to create a third plane between the two symmetric planes using Universum data. This paper proposes a novel hybrid framework called Reduced-Kernel Weighted Extreme Learning Machine Using Universum Data in Feature Space (RKWELM-UFS) to handle the classification of binary class-imbalanced problems. The proposed RKWELM-UFS combines the Universum learning method with a Reduced-Kernelized Weighted Extreme Learning Machine (RKWELM) for the first time to inherit the advantages of both techniques. To generate efficient Universum samples in the feature space, this work uses the kernel trick. The performance of the proposed method is evaluated using 44 benchmark binary class-imbalanced datasets. The proposed method is compared with 10 state-of-the-art classifiers using AUC and G-mean. The statistical t-test and Wilcoxon signed-rank test are used to quantify the performance enhancement of the proposed RKWELM-UFS compared to other evaluated classifiers.

Multiple Universum Empirical Kernel Learning

This paper proposes a novel framework called Multiple Universum Empirical Kernel Learning (MUEKL) that combines the Universum learning with Multiple Empirical Kernel Learning (MEKL) for the first time to inherit the advantages of both techniques. The proposed MUEKL not only obtained supplementary information of multiple feature spaces through MEKL, but also obtained priori information of samples by Universum learning. MUEKL incorporates a novel method, Imbalanced Modified Universum (IMU), to generate more efficient Universum samples by introducing the imbalanced ratio of data. MUEKL develops the basic multiple kernel learning framework by introducing a regularization of Universum data. The function of the introduced regularization is to adjust the classifier boundary closer to the Universum data to alleviate the influence of the imbalanced data. Moreover, MUEKL performs excellent generalization for both the imbalanced and balanced problems. Extensive experiments verify the effectiveness of the MUEKL and IMU.

An improved non-parallel Universum support vector machine and its safe sample screening rule

A novel non-parallel hyperplane Universum support vector machine (U-NHSVM) is proposed in this paper. Universum data with ensconced prior knowledge are exploited by a non-parallel hyperplane support vector machine. In contrast to other algorithms, the proposed U-NHSVM shows flexibility by exploiting the prior knowledge ensconced in Universum and provides consistency by constructing two non-parallel hyperplanes simultaneously. With Universum, U-NHSVM is clearly effective but also time consuming. Therefore, a safe sample screening rule (SSSR) for U-NHSVM is also proposed based on its sparsity, termed SSSR-U-NHSVM. Because only the non-SVs are excluded from both labelled and Universum samples, the efficiency of SSSR-U-NHSVM is extremely improved while the accuracy is completely conserved. Numerical experiments on seventeen benchmark datasets and a Chinese wine dataset are carried out to demonstrate the validity of the proposed U-NHSVM and SSSR-U-NHSVM.

Universum Selection for Boosting the Performance of Multiclass Support Vector Machines Based on One-versus-One Strategy

We propose a novel framework to enhance the performance of the one-versus-one support vector machine by using Universum. For solving a multiclass classification problem, one-versus-one is one of the state-of-the-art algorithms, which constructs N(N−1)/2 binary classifiers for an N-class problem. Each binary classifier is originally learned by two classes of data as positive and negative classes while the other N−2 remaining classes are ignored, even if they might also represent a hidden concept of the application domain and can help to boost the performance of the classifier. Vapnik et al. [20, 21] introduced Universum binary support vector machines to enable the use of samples that do not belong to positive and negative classes and called these samples Universum samples. However, not all Universum samples can be helpful; moreover, improper selection of Universum samples can prevent the construction of an effective binary classifier. For the construction of a Universum binary classifier in the one-versus-one strategy, there are 2N−2 candidate subsets of classes of Universum data; a proper selection of them can be difficult, based on the number of classes. We design an algorithm to obtain a suitable subset of classes of Universum data by applying the proposed performance measure that reflects the properties of Universum data relative to labeled training data. This measure is based on the analysis of the projection of Universum data onto the normal direction vector of the standard binary SVM hyperplane. We demonstrate experimentally that our proposed strategy outperforms existing methods.

A safe sample screening rule for Universum support vector machines

Universum support vector machine (U-SVM), due to its tremendous accuracy improvements, has been expanded and applied in all kinds of fields. Universum encodes related prior knowledge but does not belong to any class of interest. With Universum, the number of training samples and computational complexity are clearly increased. Inspired by the sparsity of SVMs, a safe sample screening rule (SSSR) for U-SVM is proposed in this paper. Our SSSR eliminates not only the labelled samples but also the Universum samples before training process, then the computational cost is dramatically reduced. Moreover, the same solution as the original problem can be obtained by utilizing our SSSR, that is, the training process is guaranteed to be accelerated safely. Besides, we extend our rule to the Universum twin support vector machine (U-TSVM), and the SSSR for U-TSVM is also discussed in this paper. To the best of our knowledge, SSSR is the only existing safe screening method for U-SVMs. Numerical experiments on seventeen benchmark datasets, ABCDETC dataset and Chinese wine dataset demonstrate that the computational cost can be dramatically reduced without sacrificing the optimality of the final solution by our SSSR.

Investor sentiment identification based on the universum SVM

Universum refers to additional samples which contain priori knowledge for classification but belonging to none of the class. It has been proved that universum positioned “in between” the two classes obtain better results. Since opinions on stock market defined as investor sentiment involve quite a number of neutral views, these neutral views can be used as universum samples to better identify investor sentiment. With universum samples, this paper uses support vector machine (SVM) to classify posts on stock forum. We define bullish views as positive samples, define bearish views as negative samples, and also further discuss the situation of a 3-class problem with neutral views. Compared with standard SVM, the empirical studies with universum samples in this paper show better performance for both 2- and 3-class classifications.

Exploiting Universum data in AdaBoost using gradient descent

Recently, Universum data that does not belong to any class of the training data, has been applied for training better classifiers. In this paper, we address a novel boosting algorithm called UAdaBoost that can improve the classification performance of AdaBoost with Universum data. UAdaBoost chooses a function by minimizing the loss for labeled data and Universum data. The cost function is minimized by a greedy, stagewise, functional gradient procedure. Each training stage of UAdaBoost is fast and efficient. The standard AdaBoost weights labeled samples during training iterations while UAdaBoost gives an explicit weighting scheme for Universum samples as well. In addition, this paper describes the practical conditions for the effectiveness of Universum learning. These conditions are based on the analysis of the distribution of ensemble predictions over training samples. Experiments on handwritten digits classification and gender classification problems are presented. As exhibited by our experimental results, the proposed method can obtain superior performances over the standard AdaBoost by selecting proper Universum data.

A nonparallel support vector machine for a classification problem with universum learning

Universum samples, defined as samples not belonging to any class for a classification problem of interest, have been useful in supervised learning. Here we design a new nonparallel support vector machine (U-NSVM) that can exploit prior knowledge embedded in the universum to construct a more robust classifier for training. To this end, U-NSVM maximizes the two margins associated with the two closest neighboring classes, which is combined by two nonparallel hyperplanes. Therefore, U-NSVM has better flexibility and can yield a more reasonable classifier in most cases. In addition, our method includes fewer parameters than U-SVM, so is easier to implement. Experiments demonstrate that U-NSVM outperforms the traditional SVM and U-SVM.

Practical Conditions for Effectiveness of the Universum Learning

Many applications of machine learning involve analysis of sparse high-dimensional data, in which the number of input features is larger than the number of data samples. Standard inductive learning methods may not be sufficient for such data, and this provides motivation for nonstandard learning settings. This paper investigates a new learning methodology called learning through contradictions or Universum support vector machine (U-SVM). U-SVM incorporates a priori knowledge about application data, in the form of additional Universum samples, into the learning process. This paper investigates possible advantages of U-SVM versus standard SVM, and describes the practical conditions necessary for the effectiveness of the U-SVM. These conditions are based on the analysis of the univariate histograms of projections of training samples onto the normal direction vector of (standard) SVM decision boundary. Several empirical comparisons are presented to illustrate the practical utility of the proposed approach.