Parallel design of sparse deep belief network with multi-objective optimization

Abstract

Deep belief network (DBN) is an import deep learning model and restricted Boltzmann machine (RBM) is one of its basic models. The traditional DBN and RBM have numerous redundant features. Hence an improved strategy is required to perform sparse operations on them. Previously, we have proposed our own sparse DBN (SDBN): using a multi-objective optimization (MOP) algorithm to learn sparse features, which solves the contradiction between the reconstruction error and network sparsity of RBM. Due to the optimization algorithm and millions of parameters of the network itself, the training process is difficult. Therefore, in this paper, we propose an efficient parallel strategy to speed up the training of SDBN networks. Self-adaptive Quantum Multi-objectives Evolutionary algorithm based on Decomposition (SA-QMOEA/D) that we have proposed as the multi-objective optimization algorithm has the hidden parallelism of populations. Based on this, we not only parallelize the DBN network but also realize the parallelism of the multi-objective optimization algorithm. In order to further verify the advantages of our approach, we apply it to the problem of facial expression recognition (FER). The obtained experimental results demonstrate that our parallel algorithm achieves a significant speedup performance and a higher accuracy rate over previous CPU implementations and other conventional methods.