Fast approximation of variational Bayes Dirichlet process mixture using the maximization–maximization algorithm

Abstract

In Bayesian nonparametrics model such as Dirichlet process mixture (DPM), learning is almost exclusive to either variational inference or Gibbs sampling. Yet variational inference is seldom mainstream in fast algorithms for DPM mainly due to high computational cost. Instead, most fast algorithms are largely based on MAP estimation of Gibbs sampling probabilities. However, they usually face intractable posterior and typically degenerate the conditional likelihood to overcome the inefficiency. Scalable variational inference such as stochastic variational inference exist but these works rely on the same two-step learning approach that involves hyperparameters and expectations update. This constitutes to the high cost often associated with variational inference. Inspired by fast DPM algorithms, we propose using MAP estimation of variational posteriors for approximating expectations. As a result, learning can be completed in a single step. However, we encounter undefined variational posteriors of log expectation. We overcome this problem by the use of lower bounds. When our cluster assignment also uses a MAP estimation, we have a global objective known as the maximization–maximization algorithm. We revisit the concepts of variational inference and observe that some of the analytical solutions obtained by our proposed method are very similar to variational inference. Lastly, we compare our fast approach to variational inference and fast DPM algorithms on some UCI and real datasets. Experimental results showed that our proposed method obtained comparable clustering accuracy and model selection but significantly faster convergence than variational inference.