Improving mixing rate with tempered transition for learning restricted Boltzmann machines

Abstract

Recently, as the building block of deep generative models such as Deep Belief Networks (DBNs), Restricted Boltzmann Machines (RBMs) have attracted much attention. RBM is a Markov Random Field (MRF) associated with a bipartite undirected graph which is famous for powerful expression and tractable inference. While training an RBM, we need to sample from the model. The larger the mixing rate is, the smaller the bias of the samples is. However, neither Gibbs sampling based training methods such as Contrastive Divergence (CD) nor Parallel Tempering based training methods can achieve satisfying mixing rate, which causes poor rendering of the diversity of the modes captured by these trained models. This property may hinder the existing methods to approximate the likelihood gradient. In order to alleviate this problem, we attempt to introduce Tempered Transition, an advanced tempered Markov Chain Monte Carlo method, into training RBMs to replace Gibbs sampling or Parallel Tempering for sampling from RBMs. Experimental results show that our proposed method outperforms the existing methods to achieve better mixing rate and to help approximate the likelihood gradient.