Naive Mean-field Approximation Research Articles

Lower tails via relative entropy

We show that the naive mean-field approximation correctly predicts the leading term of the logarithmic lower tail probabilities for the number of copies of a given subgraph in G(n,p) and of arithmetic progressions of a given length in random subsets of the integers in the entire range of densities where the mean-field approximation is viable. Our main technical result provides sufficient conditions on the maximum degrees of a uniform hypergraph H that guarantee that the logarithmic lower tail probabilities for the number of edges, induced by a binomial random subset of the vertices of H, can be well approximated by considering only product distributions. This may be interpreted as a weak, probabilistic version of the hypergraph container lemma that is applicable to all sparser-than-average (and not only independent) sets.

Adaptive Thouless–Anderson–Palmer Equation for Higher-order Markov Random Fields

The adaptive Thouless--Anderson--Palmer (TAP) mean-field approximation is one of the advanced mean-field approaches, and it is known as a powerful accurate method for Markov random fields (MRFs) with quadratic interactions (pairwise MRFs). In this study, an extension of the adaptive TAP approximation for MRFs with many-body interactions (higher-order MRFs) is developed. We show that the adaptive TAP equation for pairwise MRFs is derived by naive mean-field approximation with diagonal consistency. Based on the equivalence of the approximate equation obtained from the naive mean-field approximation with diagonal consistency and the adaptive TAP equation in pairwise MRFs, we formulate approximate equations for higher-order Boltzmann machines, which is one of simplest higher-order MRFs, via the naive mean-field approximation with diagonal consistency.

Gaussian-width gradient complexity, reverse log-Sobolev inequalities and nonlinear large deviations

We prove structure theorems for measures on the discrete cube and on Gaussian space, which provide sufficient conditions for mean-field behavior. These conditions rely on a new notion of complexity for such measures, namely the Gaussian-width of the gradient of the log-density. On the cube {−1, 1}n, we show that a measure $${\nu}$$ which exhibits low complexity can be written as a mixtureof measures $${\{\nu_\theta\}_{\theta\in\mathcal{I}}}$$ such that: (i) for each $${\theta}$$ , the measure $${\nu_\theta}$$ is a small perturbation of $${\nu}$$ such that log $$\frac{{d \nu_\theta}}{{d \nu}}$$ is a linear function whose gradient is small and, (ii) $${\nu_\theta}$$ is close to some product measure, in Wasserstein distance, for most $${\theta}$$ . Thus, our framework can be used to study the behavior of low-complexity measures beyond approximation of the partition function, showing that those measures are roughly mixtures of product measures whose entropy is close to that of the original measure. In particular, as a corollary of our theorems, we derive a bound for the naive mean-field approximation of the log-partition function which improves the nonlinear large deviation framework of Chatterjee and Dembo (Adv Math, 319:313–347, 2017. ISSN 0001-8708. https://dx.doi.org/10.1016/j.aim.2017.08.003 ) in several ways: (1) It does not require any bounds on second derivatives. (2) The covering number is replaced by the weaker notion of Gaussian-width. (3) We obtain stronger asymptotics with respect to the dimension. Two other corollaries are decomposition theorems for exponential random graphs and large-degree Ising models. In the Gaussian case, we show that measures of low-complexity exhibit an almost-tight reverse log-Sobolev inequality.

Mean-Field Inference in Gaussian Restricted Boltzmann Machine

A Gaussian restricted Boltzmann machine (GRBM) is a Boltzmann machine defined on a bipartite graph and is an extension of usual restricted Boltzmann machines. A GRBM consists of two different layers: a visible layer composed of continuous visible variables and a hidden layer composed of discrete hidden variables. In this paper, we derive two different inference algorithms for GRBMs based on the naive mean-field approximation (NMFA). One is an inference algorithm for whole variables in a GRBM, and the other is an inference algorithm for partial variables in a GBRBM. We compare the two methods analytically and numerically and show that the latter method is better.

The Variational Garrote

We analyze the variational method for sparse regression using l0 regularization. The variational approximation results in a model that is similar to Breiman's Garrote model. We refer to this method as the Variational Garrote (VG). The VG has the effect of making the problem effectively of maximal rank even when the number of samples is small compared to the number of variables. We propose a naive mean field approximation combined with a maximum a posteriori (MAP) approach to estimate the model parameters and use an annealing and reheating schedule of the sparsity hyper-parameter to avoid local minima. The hyper-parameter is set by cross-validation. We compare the VG with the lasso, ridge regression and the recently introduced Bayesian paired mean field method (PMF) (Titsias and Lazaro-Gredilla in Advances in neural information processing systems, vol. 24, pp. 2339---2347, 2011). For fair comparison, we implemented a similar annealing-reheating schedule for the PMF sparsity parameter. Numerical results show that the VG and PMF yield more accurate predictions and more accurately reconstruct the true model than the other methods. The VG finds correct solutions when the lasso solution is inconsistent due to large input correlations. In the experiments that we consider we find that the VG, although based on a simpler approximation than the PMF, yields qualitatively similar or better results and is computationally more efficient. The naive implementation of the VG scales cubic with the number of features. By introducing Lagrange multipliers we obtain a dual formulation of the problem that scales cubic in the number of samples, but close to linear in the number of features.

TAP equation for non-negative Boltzmann machine

Mean-field methods for spin systems are frequently used in not only statistical physics but also information sciences. We focus on the Plefka expansion method for spin systems with two-body interactions. The Plefka expansion is a useful perturbative expansion of the Gibbs free energy, and it can systematically provide the naive mean-field approximation, the Thouless–Anderson–Palmer (TAP) equation and higher-order approximations. In the first part of this paper, using the linear response relation, we derive a recurrence formula for perturbative coefficients in the Plefka expansion. Our recurrence formula enables us to systematically derive general order coefficients. In the latter part of the paper, we apply our recurrence formula to the non-negative Boltzmann machine in which all spin variables are constrained to have non-negative real values, and we obtain the TAP equation for this model. We verify the performance of our TAP equation by conducting some numerical experiments.

Graphical Model Inference in Optimal Control of Stochastic Multi-Agent Systems

In this article we consider the issue of optimal control in collaborative multi-agent systems with stochastic dynamics. The agents have a joint task in which they have to reach a number of target states. The dynamics of the agents contains additive control and additive noise, and the autonomous part factorizes over the agents. Full observation of the global state is assumed. The goal is to minimize the accumulated joint cost, which consists of integrated instantaneous costs and a joint end cost. The joint end cost expresses the joint task of the agents. The instantaneous costs are quadratic in the control and factorize over the agents. The optimal control is given as a weighted linear combination of single-agent to single-target controls. The single-agent to single-target controls are expressed in terms of diffusion processes. These controls, when not closed form expressions, are formulated in terms of path integrals, which are calculated approximately by Metropolis-Hastings sampling. The weights in the control are interpreted as marginals of a joint distribution over agent to target assignments. The structure of the latter is represented by a graphical model, and the marginals are obtained by graphical model inference. Exact inference of the graphical model will break down in large systems, and so approximate inference methods are needed. We use naive mean field approximation and belief propagation to approximate the optimal control in systems with linear dynamics. We compare the approximate inference methods with the exact solution, and we show that they can accurately compute the optimal control. Finally, we demonstrate the control method in multi-agent systems with nonlinear dynamics consisting of up to 80 agents that have to reach an equal number of target states.

An information geometrical approach to the mean-field approximation for quantum Ising spin models

We study the mean-field approximation for a general class of quantum Ising spin states from an information geometrical point of view. The states we consider are assumed to have at most second-order interactions with arbitrary but deterministic coupling coefficients. We call such a state a quantum Boltzmann machine (QBM) for the reason that it can be regarded as a quantum extension of the equilibrium distribution of a (classical) Boltzmann machine (CBM), which is a well-known stochastic neural network model. The totality of QBMs is then shown to form a quantum exponential family and thus can be considered as a smooth manifold having similar geometrical structures to those of CBMs. We elaborate on the significance and usefulness of information geometrical concepts, in particular the e- and m-projections, in studying the naive mean-field approximation for QBMs. We also discuss the higher-order corrections to the naive mean-field approximation based on the idea of the Plefka expansion in statistical physics. We elucidate the geometrical essence of the corrections and provide the expansion coefficients with expressions in terms of information geometrical quantities. Here, one may note this work as the information geometrical interpretation of (Plefka T 2006 Phys. Rev. E 73 016129) and as the quantum extension of (Tanaka T 2000 Neural Comput. 12 1951–68).

Naive Mean Field Approximation for Image Restoration

We attempt image restoration in the framework of the Baysian inference. Recently, it has been shown that under a certain criterion the MAP (Maximum A Posterior) estimate, which corresponds to the minimization of energy, can be outperformed by the MPM (Maximizer of the Posterior Marginals) estimate, which is equivalent to a finite-temperature decoding method. Since a lot of computational time is needed for the MPM estimate to calculate the thermal averages, the mean field method, which is a deterministic algorithm, is often utilized to avoid this difficulty. We present a statistical-mechanical analysis of naive mean field approximation in the framework of image restoration. We compare our theoretical results with those of computer simulation, and investigate the potential of naive mean field approximation.