Bayesian Allocation Model: Marginal Likelihood-Based Model Selection for Count Tensors

Abstract

In this article, we introduce a dynamic generative model, the Bayesian allocation model (BAM), for modeling count data. BAM covers various probabilistic nonnegative tensor factorization (NTF) and topic models under one general framework. In BAM, allocations are made using a Bayesian network, whose conditional probability tables can be integrated out analytically. We show that, when allocations are viewed as sequential, the resulting marginal process is a special type of Polya urn process, which we name as Polya-Bayes process, an integer valued self-reinforcing process. Exploiting the Polya urn construction, we develop a novel sequential Monte Carlo (SMC) algorithm for marginal likelihood estimation in BAM, leading to a unified scoring method for discrete variable Bayesian networks with hidden nodes, including various NTF and topic models. The SMC estimator for marginal likelihood has the remarkable property of being unbiased in contrast to variational algorithms which are generally biased. We also demonstrate how our novel SMC-based likelihood estimation can be integrated within a Markov chain Monte Carlo algorithm for a principled and correct (in terms of respecting the true posterior distribution) Bayesian model selection and hyperparameter estimation for BAM. We provide several numerical examples, both on artificial and real datasets, that demonstrate the performance of the algorithms for various data regimes.