Stochastic feature mapping for PAC-Bayes classification

Abstract

Hidden information derived from probabilistic generative models of data distributions can be used to construct features for discriminative classifiers. This observation has motivated the development of approaches that attempt to couple generative and discriminative models together for classification. However, existing approaches typically feed features derived from generative models to discriminative classifiers, and do not refine the generative models or the feature mapping functions based on classification results. In this paper, we propose a coupling mechanism developed under the PAC-Bayes framework that can fine-tune the generative models and the feature mapping functions iteratively to improve the classifier's performance. In our approach, a stochastic feature mapping, which is a function over the random variables of a generative model, is derived to generate feature vectors for a stochastic classifier. We construct a stochastic classifier over the feature mapping and derive the PAC-Bayes generalization bound for the classifier, for both supervised and semi-supervised learning. This allows us to jointly learn the feature mapping and the classifier by minimizing the bound with an EM-like iterative algorithm using labeled and unlabeled data. The resulting framework integrates the learning of the discriminative classifier and the generative model and allows iterative fine-tuning of the generative models, and the feedforward feature mappings based on task performance feedback. Our experiments show, in three distinct applications, this new framework produces a general classification tool with state-of-the-art performance.