Stochastic Neural Variational Learning of Noisy-OR Bayesian Networks for Images

Abstract

Bayesian networks are not only useful as causal probabilistic models but also promising as models of the cerebral cortex. This paper addresses the problem of unsupervised learning in Bayesian networks with noisy-OR conditional probability tables. We employ neural variational inference and learning (NVIL), in which intractable posterior distribution is approximated by the output of a neural network. Both the noisy-OR Bayesian network and the posterior distribution neural network are optimized to maximize the variational lower bound of the true log-likelihood. To examine the effectiveness of the proposed method, we used the MNIST handwritten digit dataset for unsupervised learning of noisy-OR Bayesian networks. We confirmed that noisy-OR Bayesian networks with up to 128 latent variables can learn the given dataset using NVIL. Interestingly, the latent variables of the noisy-OR Bayesian networks learned fragments of digit images as their representation. These representations are easier to interpret than the representations acquired by sigmoid belief networks.