Memorized Variational Continual Learning for Dirichlet Process Mixtures

Yang Yang,Bo Chen,Hongwei Liu

doi:10.1109/access.2019.2947722

Abstract

Bayesian nonparametric models are theoretically suitable for streaming data due to their ability to adapt model complexity with the observed data. However, very limited work has addressed posterior inference in a streaming fashion, and most of the existing variational inference algorithms require truncation on variational distributions which cannot vary with the data. In this paper, we focus Dirichlet process mixture models and develop the corresponding variational continual learning approach by maintaining memorized sufficient statistics for previous tasks, called memorized variational continual learning (MVCL), which is able to handle both the posterior update and data in a continual learning setting. Furthermore, we extend MVCL for two cases of mixture models which can handle different data types. The experiments demonstrate the comparable inference capability of our MVCL for both discrete and real-valued datasets with automatically inferring the number of mixture components.

Highlights

In this Information Age, data are being generated by every animate and inanimate object on Earth at any time [1]
MEMORIZED VARIATIONAL CONTINUAL LEARNING we describe our memorized variational continual learning (MVCL) algorithm with birth and merge moves for Dirichlet process mixture (DPM) models in which no truncation is needed
MEMORIZED SUFFICIENT STATISTICS In terms of the incremental variants of the expectation maximization (EM) algorithm [25] and recent VCL algorithm [26], we develop the memorized sufficient statistics denoted as Sk0 = [Nk0, s0k (x)] for our algorithm, which can increase with the coming new task

Summary

Introduction

In this Information Age, data are being generated by every animate and inanimate object on Earth at any time [1]. Often, these data arrive sequentially in time and we are tasked with performing unsupervised learning as the data stream in, without revisiting past data. Dealing with streaming data requires flexible models that can expand with data size and complexity. Bayesian nonparametric (BNP) models are natural to fit this purpose since it can vary the number of mixture components as new data appear. The challenge is that BNP models lack efficient inference methods to deal with large scale and streaming data.

Objectives

Methods

Findings

Conclusion