Parallel inference for cross-collection latent generalized Dirichlet allocation model and applications

Abstract

Existing cross-collection topic models with document-topic representation encounter performance bottlenecks in large-scale datasets due to their reliance on Dirichlet priors and conventional inference schemes. These constraints become noticeable in models derived from the Latent Dirichlet Allocation (LDA) framework. To address these challenges, this paper introduces the GPU-accelerated cross-collection latent generalized Dirichlet allocation (gccLGDA) model. This innovative approach integrates the benefits of generalized Dirichlet (GD) distribution with the computational prowess of GPU-based parallel inference, offering enhanced cross-collection topic modeling. The gccLGDA employs the GD distribution presenting a more flexible prior with a comprehensive covariance structure, enabling a more nuanced capture of relationships between latent topics across different collections. Leveraging GPU for parallel inference, our model promises scalable and efficient training for expansive datasets, making it apt for large-scale data challenges. Through empirical evaluations in comparative text mining and document classification, we demonstrate the enhanced performance of the gccLGDA, highlighting its advantages over existing cross-collection topic models.