Mean Field Theory based on Belief Networks for Approximate Inference

Abstract

Exact inference in large, densely connected belief networks is computationally intractable, and approximate schemes are therefore of great importance. In the context of approximate inference in sigmoid belief networks, mean field theory has received much interest. In this method the exact log-likelihood is bounded from below using a mean field approximating distribution. In the standard mean field theory, the approximating distribution is assumed to be factorial. In this paper we propose to use a (tractable) belief network as an approximating distribution. We show that belief networks fit very well into mean field theory, and no additional bounds are required. We derive mean field equations which provide an efficient iterative algorithm to optimize the parameters of the approximating distribution. Simulation results on an inference problem indicates a considerable improvement over existing mean field methods.