An effect of initial distribution covariance for annealing Gaussian restricted Boltzmann machines

Abstract

In this paper, we investigate an effect that the covariance of an initial distribution for annealed importance sampling (AIS) exertson the estimation accuracy for the partition functions of Gaussian restricted Boltzmann machines (RBMs). A common choicefor an AIS initial distribution is a Gaussian RBM (GRBM) with zero weight connections. Such an initial distribution does notshow any covariance between variables. However, target distributions generally allow a finite covariance between variables. Wepropose a method to design the covariance matrix of an initial distribution for GRBMs. We empirically analyze the effect ofthe initial distribution covariance on the estimation accuracy of AIS. The proposed method for designing initial distributionsoutperforms conventional methods under various conditions.