Abstract
Restricted Boltzmann machines (RBMs) are commonly used as pre-training methods for deep learning models. Contrastive divergence (CD) and parallel tempering (PT) are traditional training algorithms of RBMs. However, these two algorithms have shortcomings in processing high-dimensional and complex data. In particular, the number of temperature chains in PT has a significant impact on the training effect, and the PT algorithm cannot fully utilize parallel sampling from multiple temperature chains for the divergence of the algorithm. The training can quickly converge with fewer temperature chains, but this impacts the accuracy. More temperature chains can help PT achieve higher accuracy in theory, but severe divergence at the beginning of the training may ruin the training result. To exploit fully the advantages of PT and improve the ability of RBMs to process high-dimensional and complex models, this article proposes dynamic tempering chains (DTC). By dynamically changing the number of temperature chains during the training process, DTC starts training with fewer temperature chains and gradually increase the number of temperature chains with training going on, and finally get an accurate RBM. And one-step reconstruction error is proposed to measure the convergence, which can decrease the influence of the dynamic training strategy on reconstruction error. Experiments on MNIST, MNORB, Cifar 10, and Cifar 100 indicate that, compared with PT, the classification accuracy of DTC algorithm improved by up to 8%. DTC quickly converges in the early stage of training because of few exchanges among temperature chains and produces higher accuracy at the end for the global optimum model learned by more temperature chains, especially when learning high-dimensional and complex data. This proves that the DTC algorithm effectively utilizes parallel sampling of multiple temperature chains, overcomes divergence challenges, and further improves the training effect of the RBM.
Highlights
The prototype of deep learning was proposed in 1967 [1]; the training problem was not solved until 2006 [2]
Different stacking strategies can contribute to a deep belief network (DBN) [7] and a deep Boltzmann machine (DBM) [8] as bipartite graphs
This paper continues with a brief introduction to the theoretical background of Restricted Boltzmann machines (RBMs) (Section II) and an analysis of the conventional Contrastive divergence (CD) and parallel tempering (PT) training algorithms (Section III)
Summary
The prototype of deep learning was proposed in 1967 [1]; the training problem was not solved until 2006 [2]. Dynamic Gibbs sampling (DGS) [18] and gradient fixing parallel tempering (GFPT) [19] have been proposed as improvements to the RBM training algorithm. To overcome the drawback of PT, this paper proposes the dynamic tempering chains (DTC) algorithm. The proposed algorithm dynamically changes the number of tempering chains in the training process to fit the energy of the RBM. This paper continues with a brief introduction to the theoretical background of RBMs (Section II) and an analysis of the conventional CD and PT training algorithms (Section III) Based on this analysis, the DTC algorithm is proposed (Section IV) and compared with the latest conventional algorithms, utilizing several indicators to analyze the performance (Section V). The contributions of this paper have been summarized as follows. 1) A novel algorithm is proposed to deal with highdimensional and complex data in RBM training. 2) Dynamically changing the number of temperature chains during the training process result in rapid convergence in the early stage of training and produce higher accuracy at the end. 3) A new indicator is proposed to measure the convergence, which improves upon the classical reconstruction error
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
