Dynamical analysis of the Wake–Sleep algorithm

Abstract

AbstractWe employ statistical dynamics to study the convergence of the Wake–Sleep (W‐S) algorithm, which is a learning algorithm for neural network models having hidden units. Although there have been several reports on the effectiveness of the W‐S algorithm based on experimental methods, the theoretical side is not clear even for a simple network. In this paper, we investigate the dynamic characteristics of the W‐S algorithm expressed by a single factor analysis problem, which is the simplest state setting. The advantage of our approach is the ability to quantitatively evaluate the effect that the learning coefficients have on the convergence, which is difficult when using other methods. The result was that the settings of the learning coefficients, particularly in the Sleep step, had a substantial effect on the convergence of the algorithm. © 2001 Scripta Technica, Electron Comm Jpn Pt 3, 85(1): 41–49, 2002