Training Restricted Boltzmann Machines Research Articles

Role of education and awareness programs in fostering energy conservation behavior in cities: Empowering urban sustainability using deep learning approach

This paper introduces a pioneering framework that emphasizes the critical role of education in promoting energy conservation behavior within urban environments. By synergizing deep learning methodologies with education-awareness formulations and algorithm, the framework aims to bolster energy conservation security in cities. A central aspect of this framework is the development of an advanced Intrusion Detection System (IDS) model, utilizing Restricted Boltzmann Machines (RBMs) tailored to enhance energy consumption security. Through six strategic integrations of educational components, the IDS not only identifies but also mitigates potential threats to energy systems with unprecedented efficiency. Additionally, the research introduces a novel evolutionary feature selection method based on Memetic Algorithms (MAs), which strategically identifies crucial features for RBM training, ensuring optimal model performance. The incorporation of Memetic Algorithms facilitates individual learning and problem-specific knowledge transmission, further enhancing the adaptive capabilities of the system. This multifaceted approach underscores the pivotal role of education in fostering urban sustainability, pushing the boundaries of energy conservation security. By proactively responding to emerging threats, the proposed framework contributes significantly to the overarching goal of sustainable and secure urban energy management.

Computational design of novel Cas9 PAM-interacting domains using evolution-based modelling and structural quality assessment.

We present here an approach to protein design that combines (i) scarce functional information such as experimental data (ii) evolutionary information learned from a natural sequence variants and (iii) physics-grounded modeling. Using a Restricted Boltzmann Machine (RBM), we learn a sequence model of a protein family. We use semi-supervision to leverage available functional information during the RBM training. We then propose a strategy to explore the protein representation space that can be informed by external models such as an empirical force-field method (FoldX). Our approach is applied to a domain of the Cas9 protein responsible for recognition of a short DNA motif. We experimentally assess the functionality of 71 variants generated to explore a range of RBM and FoldX energies. Sequences with as many as 50 differences (20% of the protein domain) to the wild-type retained functionality. Overall, 21/71 sequences designed with our method were functional. Interestingly, 6/71 sequences showed an improved activity in comparison with the original wild-type protein sequence. These results demonstrate the interest in further exploring the synergies between machine-learning of protein sequence representations and physics grounded modeling strategies informed by structural information.

Quantum image denoising: a framework via Boltzmann machines, QUBO, and quantum annealing

We investigate a framework for binary image denoising via restricted Boltzmann machines (RBMs) that introduces a denoising objective in quadratic unconstrained binary optimization (QUBO) form well-suited for quantum annealing. The denoising objective is attained by balancing the distribution learned by a trained RBM with a penalty term for derivations from the noisy image. We derive the statistically optimal choice of the penalty parameter assuming the target distribution has been well-approximated, and further suggest an empirically supported modification to make the method robust to that idealistic assumption. We also show under additional assumptions that the denoised images attained by our method are, in expectation, strictly closer to the noise-free images than the noisy images are. While we frame the model as an image denoising model, it can be applied to any binary data. As the QUBO formulation is well-suited for implementation on quantum annealers, we test the model on a D-Wave Advantage machine, and also test on data too large for current quantum annealers by approximating QUBO solutions through classical heuristics.

Software techniques for training restricted Boltzmann machines on size-constrained quantum annealing hardware

Restricted Boltzmann machines are common machine learning models that can utilize quantum annealing devices in their training processes as quantum samplers. While this approach has shown promise as an alternative to classical sampling methods, the limitations of quantum annealing hardware, such as the number of qubits and the lack of connectivity between the qubits, still pose a barrier to wide-scale adoption. We propose the use of multiple software techniques such as dropout method, passive labeling, and parallelization techniques for addressing these hardware limitations. The study found that using these techniques along with quantum sampling showed comparable results to its classical counterparts in certain contexts, while in others the increased complexity of the sampling process hindered the performance of the trained models. This means that further research into the behavior of quantum sampling needs to be done to apply quantum annealing to training tasks of more complicated RBM models.

Effective learning algorithm for restricted Boltzmann machines via spatial Monte Carlo integration

Restricted Boltzmann machines (RBMs) are a type of statistical machine learning model used in various applications. However, training RBM models is computationally difficult owing to the requirement of calculating expectations that have a combinatorial explosion. We provide a new and effective learning algorithm based on spatial Monte Carlo integration, which is an extension of the standard Monte Carlo integration and can approximate such intractable expectations with high accuracy. The proposed method exhibited superior performance compared to the standard learning method, i.e., contrastive divergence, in terms of accuracy and learning speed. However, there were cases in which the proposed learning method exhibited reduced performances. Thus, we further demonstrate that a heuristic initialization of the learning parameters can suppress this degradation.

Equilibrium and non-equilibrium regimes in the learning of restricted Boltzmann machines* *This article is an updated version of: Decelle A, Furtlehner C and Seoane B 2021 Equilibrium and non-equilibrium regimes in the learning of restricted Boltzmann machines Advances in Neural Information Processing Systems vol 34 ed M Ranzato, A Beygelzimer, Y Dauphin, P S Liang and J Wortman Vaughan (New

Training restricted Boltzmann machines (RBMs) have been challenging for a long time due to the difficulty of precisely computing the log-likelihood gradient. Over the past few decades, many works have proposed more or less successful training recipes but without studying the crucial quantity of the problem: the mixing time, i.e. the number of Monte Carlo iterations needed to sample new configurations from a model. In this work, we show that this mixing time plays a crucial role in the dynamics and stability of the trained model, and that RBMs operate in two well-defined regimes, namely equilibrium and out-of-equilibrium, depending on the interplay between this mixing time of the model and the number of steps, k, used to approximate the gradient. We further show empirically that this mixing time increases with the learning, which often implies a transition from one regime to another as soon as k becomes smaller than this time. In particular, we show that using the popular k (persistent) contrastive divergence approaches, with k small, the dynamics of the learned model are extremely slow and often dominated by strong out-of-equilibrium effects. On the contrary, RBMs trained in equilibrium display faster dynamics, and a smooth convergence to dataset-like configurations during the sampling. Finally, we discuss how to exploit in practice both regimes depending on the task one aims to fulfill: (i) short k can be used to generate convincing samples in short learning times, (ii) large k (or increasingly large) is needed to learn the correct equilibrium distribution of the RBM. Finally, the existence of these two operational regimes seems to be a general property of energy based models trained via likelihood maximization.

Clinical Characteristics and Mathematical Analysis of Curative Effect of Hemodialysis in Curing Poisoning Caused by Snakebite.

In order to explore the clinical characteristics of hemodialysis in curing poisoning from snakebites, a two-classification model of nuclear logistic neural network based on restricted Boltzmann machine is proposed. The model combines kernel logistic regression with artificial neural networks, enabling the model to both learn autonomously and handle linearly inseparable problems. The network first performs feature learning through unsupervised training of restricted Boltzmann machines and obtains the initial values of the parameters to be identified, which reduces the influence of the randomness of the initial parameters. The variable universe learning rate with scaling factor is used to learn the parameters to be identified, and the model convergence speed is improved by dynamic adjustment of the learning rate. Experimental results show the following: Compared with before treatment, patient's activated partial thromboplastin time (APTT) after treatment and the prothrombin time (PT) level decrease, fibrinogen (FIB) levels are elevated, aspartate transferase (AST) and creatine kinase isoenzyme (CK-MB) level decreased, and the differences were statistically significant (p < 0.05). It is proved that continuous hemodiafiltration combined with plasma exchange treatment can effectively improve the blood coagulation index and myocardial index of severe snakebite poisoning patients.

Exact Training of Restricted Boltzmann Machines on Intrinsically Low Dimensional Data.

The restricted Boltzmann machine is a basic machine learning tool able, in principle, to model the distribution of some arbitrary dataset. Its standard training procedure appears, however, delicate and obscure in many respects. We bring some new insights to it by considering the situation where the data have low intrinsic dimension, offering the possibility of an exact treatment and revealing a fundamental failure of the standard training procedure. The reasons for this failure-like the occurrence of first-order phase transitions during training-are clarified thanks to a Coulomb interactions reformulation of the model. In addition, a convex relaxation of the original optimization problem is formulated, thereby resulting in a unique solution, obtained in precise numerical form on d=1, 2 study cases, while a constrained linear regression solution can be conjectured on the basis of an information theory argument.

Mode-assisted joint training of deep Boltzmann machines

The deep extension of the restricted Boltzmann machine (RBM), known as the deep Boltzmann machine (DBM), is an expressive family of machine learning models which can serve as compact representations of complex probability distributions. However, jointly training DBMs in the unsupervised setting has proven to be a formidable task. A recent technique we have proposed, called mode-assisted training, has shown great success in improving the unsupervised training of RBMs. Here, we show that the performance gains of the mode-assisted training are even more dramatic for DBMs. In fact, DBMs jointly trained with the mode-assisted algorithm can represent the same data set with orders of magnitude lower number of total parameters compared to state-of-the-art training procedures and even with respect to RBMs, provided a fan-in network topology is also introduced. This substantial saving in number of parameters makes this training method very appealing also for hardware implementations.

Training Restricted Boltzmann Machines With a D-Wave Quantum Annealer

Restricted Boltzmann Machine (RBM) is an energy-based, undirected graphical model. It is commonly used for unsupervised and supervised machine learning. Typically, RBM is trained using contrastive divergence (CD). However, training with CD is slow and does not estimate the exact gradient of the log-likelihood cost function. In this work, the model expectation of gradient learning for RBM has been calculated using a quantum annealer (D-Wave 2000Q), where obtaining samples is faster than Markov chain Monte Carlo (MCMC) used in CD. Training and classification results of RBM trained using quantum annealing are compared with the CD-based method. The performance of the two approaches is compared with respect to the classification accuracies, image reconstruction, and log-likelihood results. The classification accuracy results indicate comparable performances of the two methods. Image reconstruction and log-likelihood results show improved performance of the CD-based method. It is shown that the samples obtained from quantum annealer can be used to train an RBM on a 64-bit “bars and stripes” dataset with classification performance similar to an RBM trained with CD. Though training based on CD showed improved learning performance, training using a quantum annealer could be useful as it eliminates computationally expensive MCMC steps of CD.

A cautionary tale for machine learning generated configurations in presence of a conserved quantity

We investigate the performance of machine learning algorithms trained exclusively with configurations obtained from importance sampling Monte Carlo simulations of the two-dimensional Ising model with conserved magnetization. For supervised machine learning, we use convolutional neural networks and find that the corresponding output not only allows to locate the phase transition point with high precision, it also displays a finite-size scaling characterized by an Ising critical exponent. For unsupervised learning, restricted Boltzmann machines (RBM) are trained to generate new configurations that are then used to compute various quantities. We find that RBM generates configurations with magnetizations and energies forbidden in the original physical system. The RBM generated configurations result in energy density probability distributions with incorrect weights as well as in wrong spatial correlations. We show that shortcomings are also encountered when training RBM with configurations obtained from the non-conserved Ising model.

Comparison of Use of a 2000 Qubit D-Wave Quantum Annealer and MCMC for Sampling, Image Reconstruction, and Classification

The next generation D-Wave quantum computer (QC) having more than 2000 qubits was demonstrated to be adequate for embedding graphs of restricted Boltzmann machines (RBMs) having the connectivity between RBM units (though still limited) already sufficient for successful training by classical algorithms. As a verification of the validity and quality of the employed embedding approach, the RBM was trained by classical contrastive divergence (CD), while the D-Wave was initially used only in the recognition step for reconstruction of incomplete test images (8 × 8 bars-and-stripes), and for their classification. The label classification errors by the D-Wave compared favorably to those obtained by classical Gibbs sampling, indicating that the QC was successfully finding the most energetically favorable combinations of the unknown visible RBM units/labels for the given fixed incomplete input image. The valid RBM embedding was further applied to investigate opportunities for using QCs in the RBM training, and specifically for replacing the classical Gibbs technique in generating a representative sample from the RBM model distribution. Statistical comparison of samples obtained by the Gibbs technique versus 10000 sample states generated from the D-Wave revealed significant differences in the observed outcomes. The D-Wave samples were found insufficiently representative of the model distribution, specifically by missing many local valley found by the Gibbs sampling. On the other hand, the D-Wave possessed an ability to find local valleys of the configuration space that were consistently missed by the classical sampling algorithms, which could potentially be very interesting for sampling applications.

Adaptive hyperparameter updating for training restricted Boltzmann machines on quantum annealers

Restricted Boltzmann Machines (RBMs) have been proposed for developing neural networks for a variety of unsupervised machine learning applications such as image recognition, drug discovery, and materials design. The Boltzmann probability distribution is used as a model to identify network parameters by optimizing the likelihood of predicting an output given hidden states trained on available data. Training such networks often requires sampling over a large probability space that must be approximated during gradient based optimization. Quantum annealing has been proposed as a means to search this space more efficiently which has been experimentally investigated on D-Wave hardware. D-Wave implementation requires selection of an effective inverse temperature or hyperparameter (beta ) within the Boltzmann distribution which can strongly influence optimization. Here, we show how this parameter can be estimated as a hyperparameter applied to D-Wave hardware during neural network training by maximizing the likelihood or minimizing the Shannon entropy. We find both methods improve training RBMs based upon D-Wave hardware experimental validation on an image recognition problem. Neural network image reconstruction errors are evaluated using Bayesian uncertainty analysis which illustrate more than an order magnitude lower image reconstruction error using the maximum likelihood over manually optimizing the hyperparameter. The maximum likelihood method is also shown to out-perform minimizing the Shannon entropy for image reconstruction.

PM2.5 concentration modeling and prediction by using temperature-based deep belief network

Air quality prediction is a global hot issue, and PM2.5 is an important factor affecting air quality. Due to complicated causes of formation, PM2.5 prediction is a thorny and challenging task. In this paper, a novel deep learning model named temperature-based deep belief networks (TDBN) is proposed to predict the daily concentrations of PM2.5 for the next day. Firstly, the location of PM2.5 concentration prediction is Chaoyang Park in Beijing of China from January 1, 2018 to October 27, 2018. The auxiliary variables are selected as input variables of TDBN by Partial Least Square (PLS), and the corresponding data is divided into three independent sections: training samples, validating samples and testing samples. Secondly, the TDBN is composed of temperature-based restricted Boltzmann machine (RBM), where temperature is considered as an effective physical parameter in energy balance of training RBM. The structural parameters of TDBN are determined by minimizing the error in the training process, including hidden layers number, hidden neurons and value of temperature. Finally, the testing samples are used to test the performance of the proposed TDBN on PM2.5 prediction, and the other similar models are tested by the same testing samples for convenience of comparison with TDBN. The experimental results demonstrate that TDBN performs better than its peers in root mean square error (RMSE), mean absolute error (MAE) and coefficient of determination (R2).

Toward sampling from undirected probabilistic graphical models using a D-Wave quantum annealer

A D-Wave quantum annealer (QA) having a 2048 qubit lattice, with no missing qubits and couplings, allowed embedding of a complete graph of a restricted Boltzmann machine (RBM). A handwritten digit OptDigits dataset having 8 × 7 pixels of visible units was used to train the RBM using classical contrastive divergence. Embedding of the classically trained RBM into the D-Wave lattice was used to demonstrate that the QA offers a high-efficiency alternative to the classical Markov chain Monte Carlo (MCMC) for reconstructing missing labels of the test images as well as a generative model on a classically trained RBM. At any training iteration, the D-Wave-based classification had classification error less than half of that in MCMC. The main goal of this study was to investigate the quality of the QA sample from the RBM model probability distribution and compare it to a classical MCMC during the Gibbs sampling traditionally used in RBM training. For the OptDigits dataset, the states in the D-Wave sample belonged to about twice as many local valleys (LVs) as in the MCMC sample. All the lowest energy (the highest joint probability), local minima in the MCMC sample were also found by the D-Wave. The D-Wave missed many of the higher-energy LVs, while also finding many “new” LVs consistently missed by the MCMC. It was established that the “new” LVs that the D-Wave finds are important for the model distribution in terms of the energy of the corresponding local minima, the width of the LVs and the height of the escape barrier.

Variational mean-field theory for training restricted Boltzmann machines with binary synapses.

Unsupervised learning requiring only raw data is not only a fundamental function of the cerebral cortex, but also a foundation for a next generation of artificial neural networks. However, a unified theoretical framework to treat sensory inputs, synapses, and neural activity together is still lacking. The computational obstacle originates from the discrete nature of synapses, and complex interactions among these three essential elements of learning. Here, we propose a variational mean-field theory in which the distribution of synaptic weights is considered. The unsupervised learning can then be decomposed into two intertwined steps: A maximization step is carried out as a gradient ascent of the lower bound on the data log-likelihood, in which the synaptic weight distribution is determined by updating variational parameters, and an expectation step is carried out as a message passing procedure on an equivalent or dual neural network whose parameter is specified by the variational parameters of the weight distribution. Therefore, our framework provides insights on how data (or sensory inputs), synapses, and neural activities interact with each other to achieve the goal of extracting statistical regularities in sensory inputs. This variational framework is verified in restricted Boltzmann machines with planted synaptic weights and handwritten-digits learning.

A Novel Restricted Boltzmann Machine Training Algorithm with Fast Gibbs Sampling Policy

The restricted Boltzmann machine (RBM) is one of the widely used basic models in the field of deep learning. Although many indexes are available for evaluating the advantages of RBM training algorithms, the classification accuracy is the most convincing index that can most effectively reflect its advantages. RBM training algorithms are sampling algorithms essentially based on Gibbs sampling. Studies focused on algorithmic improvements have mainly faced challenges in improving the classification accuracy of the RBM training algorithms. To address the above problem, in this paper, we propose a fast Gibbs sampling (FGS) algorithm to learn the RBM by adding accelerated weights and adjustment coefficient. An important link based on Gibbs sampling theory was established between the update of the network weights and mixing rate of Gibbs sampling chain. The proposed FGS method was used to accelerate the mixing rate of Gibbs sampling chain by adding accelerated weights and adjustment coefficients. To further validate the FGS method, numerous experiments were performed to facilitate comparisons with the classical RBM algorithm. The experiments involved learning the RBM based on standard data. The results showed that the proposed FGS method outperformed the CD, PCD, PT5, PT10, and DGS algorithms, particularly with respect to the handwriting database. The findings of our study suggest the potential applications of FGS to real-world problems and demonstrate that the proposed method can build an improved RBM for classification.

Restricted Boltzmann machine as an aggregation technique for binary descriptors

The article presents a novel approach to the challenge of real-time image classification with deep neural networks. The proposed architecture of the neural network exploits computationally efficient local binary descriptors and uses a restricted Boltzmann machine (RBM) as a feature space projection step so that the resulting depth of the deep neural network can be reduced. A contrastive divergence procedure is used both for RBM training and for feature projection. The resulting neural networks exhibit performance close to the current state-of-the-art but are characterized by a small model memory footprint (i.e., number of parameters) and extremely efficient computational complexity (i.e., response time). The low number of parameters makes these architectures applicable in embedded systems with limited memory or reduced computational capabilities.

DBM Optimization with Additional Category Information

For unconstrained training of restricted Boltzmann machine (RBM), it is easy to appear that feature homogenization leads to poor generalization ability. This paper introduces category conditions into DBM training, proposes label conditional RBM which is used in the construction of DBM. According to unsupervised training characteristics of RBM, this paper adds the category information as the model hidden unit training condition to the implicit unit posterior activation probability calculation. This paper applies the model as the underlying structure of the deep Boltzmann machine (DBM) to the deep network construction. Through handwritten digit recognition set test, compared with the shallow model, the new model after adding the category condition has a great improvement in the model training speed and feature extraction effectiveness, and can effectively enhance the feature learning ability of the deep model.

Accelerate Training of Restricted Boltzmann Machines via Iterative Conditional Maximum Likelihood Estimation.

Restricted Boltzmann machines (RBMs) have become a popular tool of feature coding or extraction for unsupervised learning in recent years. However, there still lacks an efficient algorithm for training the RBM due to that its likelihood function contains an intractable normalizing constant. The existing algorithms, such as contrastive divergence and its variants, approximate the gradient of the likelihood function using Markov chain Monte Carlo. However, the approximation is time consuming and, moreover, the approximation error often impedes the convergence of the training algorithm. This paper proposes a fast algorithm for training RBMs by treating the hidden states as missing data and then estimating the parameters of the RBM via an iterative conditional maximum likelihood estimation approach, which avoids the issue of intractable normalizing constants. The numerical results indicate that the proposed algorithm can provide a drastic improvement over the contrastive divergence algorithm in RBM training. This paper also presents an extension of the proposed algorithm for how to cope with missing data in RBM training and illustrates its application using an example about drug-target interaction prediction.