Higher-order Boltzmann machines and entropy bounds

Abstract

We examine some aspects of the interface area between mathematical statistics and statistical physics relevant to the study of Boltzmann machines. The Boltzmann machine learning algorithm is based on a variational principle (Gibbs' lemma for relative entropy). This fact suggests the possibility of a scheme of successive approximations: here we consider successive approximations parametrized by the order of many-body interactions among individual units. We prove bounds on the gain in relative entropy in the crucial step of adding, and estimating by Hebb's rule, a new parameter. We address the problem of providing, on the basis of local observations, upper and lower bounds on the entropy. While upper bounds are easily obtained by subadditivity, lower bounds involve localization of Hirschman bounds on a dual quantum system.