Short Text Topic Model with Word Embeddings and Context Information

Abstract

Due to the length limitation of short texts, classical topic models based on document-level word co-occurrence information fail to distill semantically coherent topics from short text collections. Word embeddings are trained from large corpus and inherently encoded with general word semantic information, hence they can be supplemental knowledge to guide topic modeling. General Polya Urn Dirichlet Multinomial Mixture (GPU-DMM) model is the first attempt which leverages word embeddings as external knowledge to enhance topic coherence for short text. However, word embeddings are usually trained on large corpus, the encoded semantic information is not necessarily suitable for training dataset. In this work, we improve the GPU-DMM model by leveraging both context information and word embeddings to distill semantic relatedness of word pairs, which can be further leveraged in model inference process to improve topic coherence. Experimental results of two tasks on two real world short text collections show that our model gains comparable or better performance than GPU-DMM model and other state-of-art short text topic models.