Bayesian semi-supervised learning with support vector machine

Abstract

This paper introduces a Bayesian semi-supervised support vector machine (Semi-BSVM) model for binary classification. Our semi-supervised learning has a distinct advantage over supervised or inductive learning since by design it reduces the problem of overfitting. While a traditional support vector machine (SVM) has the widest margin based on the labeled data only, our semi-supervised form of SVM attempts to find the widest margin in both the labeled and unlabeled data space. This enables us to use some information from the unlabeled data and improve the overall prediction performance. The likelihood is constructed using a special type of hinge loss function which also involves the unlabeled data. A penalty term is added for the likelihood part constructed from the unlabeled data. The parameters and penalties are controlled through nearly diffuse priors for objectivity of the analysis. The rate of learning from the unlabeled data is reflected through the posterior distribution of the penalty parameter from the unlabeled data. This formulation provides us with a control on how much information should be extracted from the unlabeled data without hurting the overall performance of our model. We have applied our model on three simulation data sets and five real life data sets. Our simulation study and real life data analysis show considerable improvement in prediction quality for our semi-supervised learning over supervised learning methods when we have a high learning rate from the unlabeled data. This phenomenon is particularly evident in cases when the amount of unlabeled data is very large compared to the available labeled data.