Robustness, replicability and scalability in topic modelling

Abstract

• We identify three key properties a topic model must exhibit for effective application in the social sciences: statistical robustness, descriptive power across all dimensions and reflection of reality. • We propose a simple approach for estimating the statistical robustness of topic models that is based on pairwise similarity scores between documents. • Applying that approach we find that the neural network-based Doc2Vec is more stable than the other topic models tested: Latent Dirichlet Allocation and Non-negative Matrix Factorisation. • We further propose a principal component analysis based approach for assessing the descriptive power of topic models. In applying that approach we find that Doc2Vec performs the best, but LDA does also perform well. • We provide grounds for the application of neural embeddings approaches in the social sciences and also how traditional visualisation techniques can be applied directly to dense vector representations. Approaches for estimating the similarity between individual publications are an area of long-standing interest in the scientometrics and informetrics communities. Traditional techniques have generally relied on references and other metadata, while text mining approaches based on title and abstract text have appeared more frequently in recent years. In principle, topic models have great potential in this domain. But, in practice, they are often difficult to employ successfully, and are notoriously inconsistent as latent space dimension grows. In this manuscript we identify the three properties all usable topic models should have: robustness, descriptive power and reflection of reality. We develop a novel method for evaluating the robustness of topic models and suggest a metric to assess and benchmark descriptive power as number of topics scale. Employing that procedure, we find that the neural-network-based paragraph embedding approach seems capable of providing statistically robust estimates of the document–document similarities, even for topic spaces far larger than what is usually considered prudent for the most common topic model approaches.