AmLDA: A Non-VAE Neural Topic Model

Abstract

AbstractThe variational inference (VI) for Bayesian models approximates the true posterior by maximizing a variational lower bound called ELBO. This paper considers the VI for the latent Dirichlet allocation (LDA), a well-studied Bayesian model. The VI for LDA was originally proposed as a variational expectation-maximization (VEM), where we obtain the update equations by setting the derivatives of the ELBO equal to zero. However, its M step requires the analysis of the whole training set, and its E step needs to run the update dozens of times. The stochastic VI (SVI) proposed later has improved the VEM by replacing the M step with a minibatch gradient ascent. Further, a variational autoencoder for LDA called ProdLDA in turn has replaced the E step with a minibatch gradient ascent by using an amortized encoder. Now we can train LDA like a deep neural network. However, ProdLDA marginalizes out the discrete latent variables and thus maximizes an ELBO formulated differently from both the VEM and the SVI for LDA. As a result, ProdLDA suffers from the problem of component collapse. Therefore, we propose a new VI for LDA called AmLDA. As AmLDA maximizes the same ELBO as that which both the VEM and the SVI maximize, it does not suffer from component collapse. Only the parameterization differs because AmLDA uses an amortized network for parameterizing unknown variables. The evaluation was performed over five large datasets. The experimental results show that AmLDA is as effective as the SVI.KeywordsTopic modelingDeep learningVariational autoencoder