Combining deep generative and discriminative models for Bayesian semi-supervised learning

Jonathan Gordon,José Miguel Hernández-Lobato

doi:10.1016/j.patcog.2019.107156

Abstract

Generative models can be used for a wide range of tasks, and have the appealing ability to learn from both labelled and unlabelled data. In contrast, discriminative models cannot learn from unlabelled data, but tend to outperform their generative counterparts in supervised tasks. We develop a framework to jointly train deep generative and discriminative models, enjoying the benefits of both. The framework allows models to learn from labelled and unlabelled data, as well as naturally account for uncertainty in predictive distributions, providing the first Bayesian approach to semi-supervised learning with deep generative models. We demonstrate that our blended discriminative and generative models outperform purely generative models in both predictive performance and uncertainty calibration in a number of semi-supervised learning tasks.

Highlights

Much of the recent work in machine learning has focused on the development of neural network based models
Our contributions can be summarized as follows: (i) we propose a general purpose framework for combining deep generative and discriminative models in a principled manner, (ii) we provide the first semi-supervised learning method with DGMs that incorporates model uncertainty in its predictions, and (iii) we demonstrate that jointly trained deep generative and discriminative models outperform their generative counterparts in both predictive performance and uncertainty calibration in a number of semi-supervised learning tasks
Semi-supervised learning with generative adversarial networks. Another line of research that is similar in spirit to ours and has shown empirical success is that of semi-supervised learning with Generative Adverserial Networks (GANs; [13])

Summary

Introduction

Much of the recent work in machine learning has focused on the development of neural network based models. Supervised training of neural networks typically (i) assumes access to a dataset D = {xn, yn}Nn=1 of input-output pairs, (ii) models the conditional distribution p(y|xn) of the outputs given the inputs with a neural network with parameters θ, and (iii) optimizes the likelihood of the data with respect to θ This approach has resulted in an astonishing array of successes on AI and pattern recognition tasks such as object recognition [17], remote sensing [39], finegrained action segmentation [10], and game-playing [45]. Achieve efficient and useful approximations to the parameter posterior distributions in discriminative networks [2,18,19,41] Another drawback of discriminative neural networks is that they require massive labelled data sets for training. Our contributions can be summarized as follows: (i) we propose a general purpose framework for combining deep generative and discriminative models in a principled manner, (ii) we provide the first (to the best of our knowledge) semi-supervised learning method with DGMs that incorporates model uncertainty in its predictions, and (iii) we demonstrate that jointly trained deep generative and discriminative models outperform their generative counterparts in both predictive performance and uncertainty calibration in a number of semi-supervised learning tasks

Background

Joint training of deep generative and discriminative models

Model definition and likelihood

Evidence lower bound and amortized variatonal training

Tying the components via a conditional prior

A remark on the conditional prior

Reparameterizing to accelerate training

Approximate inference for the discriminative component parameters

Related work

Interpolating generative and discriminative models

Semi-supervised learning with generative adversarial networks

Causality and semi-supervised learning