Sparse Restricted Boltzmann Machines Research Articles

Pruning a restricted Boltzmann machine for quantum state reconstruction

Restricted Boltzmann machines (RBMs) have proven to be a powerful tool for learning quantum wavefunction representations from qubit projective measurement data. Since the number of classical parameters needed to encode a quantum wavefunction scales rapidly with the number of qubits, the ability to learn efficient representations is of critical importance. In this paper we study magnitude-based pruning as a way to compress the wavefunction representation in an RBM, focusing on RBMs trained on data from the transverse-field Ising model in one dimension. We find that pruning can reduce the total number of RBM weights, but the threshold at which the reconstruction accuracy starts to degrade varies significantly depending on the phase of the model. In a gapped region of the phase diagram, the RBM admits pruning over half of the weights while still accurately reproducing relevant physical observables. At the quantum critical point however, even a small amount of pruning can lead to significant loss of accuracy in the physical properties of the reconstructed quantum state. Our results highlight the importance of tracking all relevant observables as their sensitivity varies strongly with pruning. Finally, we find that sparse RBMs are trainable and discuss how a successful sparsity pattern can be created without pruning.

Simulating disordered quantum Ising chains via dense and sparse restricted Boltzmann machines.

In recent years, generative artificial neural networks based on restricted Boltzmann machines (RBMs) have been successfully employed as accurate and flexible variational wave functions for clean quantum many-body systems. In this article, we explore their use in simulations of disordered quantum Ising chains. The standard dense RBM with all-to-all interlayer connectivity is not particularly appropriate for large disordered systems, since in such systems one cannot exploit translational invariance to reduce the amount of parameters to be optimized. To circumvent this problem, we implement sparse RBMs, whereby the visible spins are connected only to a subset of local hidden neurons, thus reducing the amount of parameters. We assess the performance of sparse RBMs as a function of the range of the allowed connections, and we compare it with that of dense RBMs. Benchmark results are provided for two sign-problem-free Hamiltonians, namely pure and random quantum Ising chains. The RBM Ansätzes are trained using the unsupervised learning scheme based on projective quantum Monte Carlo (PQMC) algorithms. We find that the sparse connectivity facilitates the training process and allows sparse RBMs to outperform their dense counterparts. Furthermore, the use of sparse RBMs as guiding functions for PQMC simulations allows us to perform PQMC simulations at a reduced computational cost, avoiding possible biases due to finite random-walker populations. We obtain unbiased predictions for the ground-state energies and the magnetization profiles with fixed boundary conditions, at the ferromagnetic quantum critical point. The magnetization profiles agree with the Fisher-de Gennes scaling relation for conformally invariant systems, including the scaling dimension predicted by the renormalization-group analysis.

Approximating power of machine-learning ansatz for quantum many-body states

An artificial neural network (ANN) with the restricted Boltzmann machine (RBM) architecture was recently proposed as a versatile variational quantum many-body wave function. In this work we provide physical insights into the performance of this ansatz. We uncover the connection between the structure of the RBM and perturbation series, which explains the excellent precision achieved by the RBM ansatz in certain simple models, demonstrated in G. Carleo and M. Troyer [Science 355, 602 (2017)]. Based on this relation, we improve the numerical algorithm to achieve a better performance of the RBM in cases where local minima complicate the convergence to the global one. Furthermore, we study the performance of a sparse RBM for approximating ground states of random, translationally invariant models in one dimension, as well as random matrix-product states. We find that the error in approximating such states exhibits a broad distribution and shows a positive correlation with the entanglement properties of the targeted state.

An Adaptive Deep Belief Network With Sparse Restricted Boltzmann Machines

Deep belief network (DBN) is an efficient learning model for unknown data representation, especially nonlinear systems. However, it is extremely hard to design a satisfactory DBN with a robust structure because of traditional dense representation. In addition, backpropagation algorithm-based fine-tuning tends to yield poor performance since its ease of being trapped into local optima. In this article, we propose a novel DBN model based on adaptive sparse restricted Boltzmann machines (AS-RBM) and partial least square (PLS) regression fine-tuning, abbreviated as ARP-DBN, to obtain a more robust and accurate model than the existing ones. First, the adaptive learning step size is designed to accelerate an RBM training process, and two regularization terms are introduced into such a process to realize sparse representation. Second, initial weight derived from AS-RBM is further optimized via layer-by-layer PLS modeling starting from the output layer to input one. Third, we present the convergence and stability analysis of the proposed method. Finally, our approach is tested on Mackey-Glass time-series prediction, 2-D function approximation, and unknown system identification. Simulation results demonstrate that it has higher learning accuracy and faster learning speed. It can be used to build a more robust model than the existing ones.

A New Sparse Restricted Boltzmann Machine

Although existing sparse restricted Boltzmann machine (SRBM) can make some hidden units activated, the major disadvantage is that the sparseness of data distribution is usually overlooked and the reconstruction error becomes very large after the hidden unit variables become sparse. Different from the SRBMs which only incorporate a sparse constraint term in the energy function formula from the original restricted Boltzmann machine (RBM), an energy function constraint SRBM (ESRBM) is proposed in this paper. The proposed ESRBM takes into account the sparseness of the data distribution so that the learned features can better reflect the intrinsic features of data. Simulations show that compared with SRBM, ESRBM has smaller reconstruction error and lower computational complexity, and that for supervised learning classification, ESRBM obtains higher accuracy rates than SRBM, classification RBM, and Softmax classifier.

Photo Aesthetics Analysis via DCNN Feature Encoding

We propose an automatic framework for quality assessment of a photograph as well as analysis of its aesthetic attributes. In contrast to the previous methods that rely on manually designed features to account for photo aesthetics, our method automatically extracts such features using a pretrained deep convolutional neural network (DCNN). To make the DCNN-extracted features more suited to our target tasks of photo quality assessment and aesthetic attribute analysis, we propose a novel feature encoding scheme, which supports vector machines-driven sparse restricted Boltzmann machines, which enhances sparseness of features and discrimination between target classes. Experimental results show that our method outperforms the current state-of-the-art methods in automatic photo quality assessment, and gives aesthetic attribute ratings that can be used for photo editing. We demonstrate that our feature encoding scheme can also be applied to general object classification task to achieve performance gains.

Improving Design Preference Prediction Accuracy Using Feature Learning

Quantitative preference models are used to predict customer choices among design alternatives by collecting prior purchase data or survey answers. This paper examines how to improve the prediction accuracy of such models without collecting more data or changing the model. We propose to use features as an intermediary between the original customer-linked design variables and the preference model, transforming the original variables into a feature representation that captures the underlying design preference task more effectively. We apply this idea to automobile purchase decisions using three feature learning methods (principal component analysis (PCA), low rank and sparse matrix decomposition (LSD), and exponential sparse restricted Boltzmann machine (RBM)) and show that the use of features offers improvement in prediction accuracy using over 1 million real passenger vehicle purchase data. We then show that the interpretation and visualization of these feature representations may be used to help augment data-driven design decisions.

Object recognition in remote sensing images using sparse deep belief networks

Object recognition has been one of the hottest issues in the field of remote sensing image analysis. In this letter, a new pixel-wise learning method based on deep belief networks (DBNs) for object recognition is proposed. The method is divided into two stages, the unsupervised pre-training stage and the supervised fine-tuning stage. Given a training set of images, a pixel-wise unsupervised feature learning algorithm is utilized to train a mixed structural sparse restricted Boltzmann machine (RBM). After that, the outputs of this RBM are put into the next RBM as inputs. By stacking several layers of RBM, the deep generative model of DBNs is built. At the fine-tuning stage, a supervised layer is attached to the top of the DBN and labels of the data are put into this layer. The whole network is then trained using the back-propagation (BP) algorithm with sparse penalty. Finally, the deep model generates good joint distribution of images and their labels. Comparative experiments are conducted on our dataset acquired by QuickBird with 60 cm resolution and the recognition results demonstrate the accuracy and efficiency of our proposed method.

Learning representative and discriminative image representation by deep appearance and spatial coding

How to build a suitable image representation remains a critical problem in computer vision. Traditional Bag-of-Feature (BoF) based models build image representation by the pipeline of local feature extraction, feature coding and spatial pooling. However, three major shortcomings hinder the performance, i.e., the limitation of hand-designed features, the discrimination loss in local appearance coding and the lack of spatial information. To overcome the above limitations, in this paper, we propose a generalized BoF-based framework, which is hierarchically learned by exploring recently developed deep learning methods. First, with raw images as input, we densely extract local patches and learn local features by stacked Independent Subspace Analysis network. The learned features are then transformed to appearance codes by sparse Restricted Boltzmann Machines. Second, we perform spatial max-pooling on a set of over-complete spatial regions, which is generated by covering various spatial distributions, to incorporate more flexible spatial information. Third, a structured sparse Auto-encoder is proposed to explore the region representations into the image-level signature. To learn the proposed hierarchy, we layerwise pre-train the network in unsupervised manner, followed by supervised fine-tuning with image labels. Extensive experiments on different benchmarks, i.e., UIUC-Sports, Caltech-101, Caltech-256, Scene-15 and MIT Indoor-67, demonstrate the effectiveness of our proposed model.

Speech emotion recognition with unsupervised feature learning

Emotion-based features are critical for achieving high performance in a speech emotion recognition (SER) system. In general, it is difficult to develop these features due to the ambiguity of the ground-truth. In this paper, we apply several unsupervised feature learning algorithms (including K-means clustering, the sparse auto-encoder, and sparse restricted Boltzmann machines), which have promise for learning task-related features by using unlabeled data, to speech emotion recognition. We then evaluate the performance of the proposed approach and present a detailed analysis of the effect of two important factors in the model setup, the content window size and the number of hidden layer nodes. Experimental results show that larger content windows and more hidden nodes contribute to higher performance. We also show that the two-layer network cannot explicitly improve performance compared to a single-layer network.

Enhancing performance of restricted Boltzmann machines via log-sum regularization

Restricted Boltzmann machines (RBMs) are often used as building blocks to construct a deep belief network. By optimizing several RBMs, the deep networks can be trained quickly to achieve good performance on the tasks of interest. To further improve the performance of data representation, many researches focus on incorporating sparsity into RBMs. In this paper, we propose a novel sparse RBM model, referred to as LogSumRBM. Instead of constraining the expected activation of every hidden unit to the same low level of sparsity as done in [27], we explicitly encourage the hidden units to be sparse through adding a log-sum norm constraint on the totality of the hidden units’ activation probabilities. In this approach, we do not need to keep the “firing rate” of each hidden unit at a certain level that is set beforehand, and therefore the level of sparsity corresponding to each hidden unit can be automatically learnt based on the task at hand. Some experiments conducted on several image data sets of different scales show that LogSumRBM learns sparser and more discriminative representations compared with the related state-of-the-art models, and stacking two LogSumRBMs learns more significant features which mimic computations in the cortical hierarchy. Meanwhile, LogSumRBM can also be used to pre-train deep networks, and achieve better classification performance.

Multitasking Associative Networks

We introduce a bipartite, diluted and frustrated, network as a sparse restricted Boltzmann machine and we show its thermodynamical equivalence to an associative working memory able to retrieve several patterns in parallel without falling into spurious states typical of classical neural networks. We focus on systems processing in parallel a finite (up to logarithmic growth in the volume) amount of patterns, mirroring the low-level storage of standard Amit-Gutfreund-Sompolinsky theory. Results obtained through statistical mechanics, the signal-to-noise technique, and MonteCarlo simulations are overall in perfect agreement and carry interesting biological insights. Indeed, these associative networks pave new perspectives in the understanding of multitasking features expressed by complex systems, e.g., neural and immune networks.