Deep compression of probabilistic graphical networks

Abstract

Probabilistic Graphical Models (PGMs) are important and active research areas in machine learning and artificial intelligence. The well-known representatives of PGMs include Restricted Boltzmann Machines (RBMs), Deep Belief Networks (DBNs), Deep Boltzmann Machines (DBMs), and their variants. These PGMs open a new dimension of machine learning and directly lead to the revival of deep learning since 2006. Compared with deep deterministic models, such as deep convolutional neural networks (CNNs), PGMs are more flexible, robust and powerful for both unsupervised and supervised learning problems. As the generative models on the one hand, PGMs have the good capability of generating samples after extracting informative features of input data. On the other hand, they also can provide a good initialization of hierarchical structures for supervised learning. The PGMs with deep structures are called deep Probabilistic Graphical Networks (PGNs) due to their hierarchical structures and close relationships with deep neural networks. There are a large number of CNN-based deep deterministic networks proposed for computer vision tasks. However, these deep deterministic models are shown to possess extremely large number of model parameters, and have serious redundancy problems. Therefore, deep compression of these deterministic models have been proposed in the last few years to reduce the number of connections and nodes, while preserving the classification accuracy. This paper is the first attempt to combine deep PGNs and deep compression techniques together to derive sparse versions of the deep probabilistic models. We firstly examine that whether deep PGNs have serious problem of parameter redundancy, and compress them if necessary. From the experimental results on MNIST, Fashion-MNIST and CIFAR-10 dataset, deep PGNs can be compressed by at least 50% of their parameters through a deep pruning and retraining approach, while keeping their capabilities in both generative and discriminative tasks.