Combining normal sparse into discriminative deep belief networks

Abstract

Training deep models are time consuming and face many local minima. For dealing with this problem, we can use DBN (Deep Belief Network) with Contrastive Divergence (CD). Sparse representations are more efficient. This paper aims to find the best structure of incorporating sparsity into discriminative DBN. We use a DBN architecture with 784 units as input, two layers each with 500 hidden units, one layer with 2000 hidden units, and 10 units as the final output. We argue that combining sparsity and discriminative DBN may increase the accuracy, but no previous studies suggest the best structure or configuration of that combination that can give the best accuracy. We took three stages of experiments to find the best configuration, namely preliminary, intermediate, and final stages. Each analysis of each stage serves as a background for consideration of the next experiment We use normal sparse for generative DBN and discriminative DBN. Experimental studies on MNIST dataset show that the best structure or scenario to combine normal sparse into deep belief networks is as follows: input — generative (CD) — generative (CD) — normal sparse discriminative (CD).