Sparse Bayesian approach for feature selection

Abstract

This paper employs sparse Bayesian approach to enable the Probabilistic Classification Vector Machine (PCVM) to select a relevant subset of features. Because of probabilistic outputs and the ability to automatically optimize the regularization items, the sparse Bayesian framework has shown great advantages in real-world applications. However, the Gaussian priors that introduce the same prior to different classes may lead to instability in the classifications. An improved Gaussian prior, whose sign is determined by the class label, is adopt in PCVM. In this paper, we present a joint classifier and feature learning algorithm: Feature Selection Probabilistic Classification Vector Machine (FPCVM). The improved Gaussian priors, named as truncated Gaussian prior, are introduced into the feature space for feature selection, and into the sample space to generate sparsity to the weight parameters, respectively. The expectation-maximization (EM) algorithm is employed to obtain a maximum a posteriori (MAP) estimation of these parameters. In experiments, both the accuracy of classification and performance of feature selection are evaluated on synthetic datasets, benchmark datasets and high-dimensional gene expression datasets.