Analysis on Noisy Boltzmann Machines and Noisy Restricted Boltzmann Machines

Abstract

The Boltzmann machine (BM) and restricted Boltzmann machine (RBM) models are representative stochastic neural networks, in which neuron states are determined by stochastic activation functions. They are widely used in many applications. However, when analog circuits are used to realize a neural network, noise are not avoidable. The noise could come from external environments, such as power supplies and thermal noise, and they will affect the neurons’ stochastic behaviour in the BM and RBM models. Hence it is important to theoretically study how the noise affect the operation of the BM and RBM models. To best of our knowledge, there are little works related to the analysis on noisy BMs and noisy RBMs. This paper considers that there are additive noise in the inputs of the neurons, and theoretically studies the behaviors of the two models under this imperfect condition. It is found that the effect of additive noise is similar to increasing the temperature factors of the two models. Since the input noise may make the networks to have wrong stochastic behaviour, there is Kullback Leibler (KL) divergence loss in noisy BMs and noisy RBMs. Based on the Gaussian-distributed noise assumption, a noise compensation method is proposed to suppress the effect of additive noise. Experiments show that the proposed noise compensation method can greatly suppress the KL divergence loss. In addition, from the experimental results, our method is also effective for handling non-Gaussian noise.