Restricted Boltzmann Machine Model Research Articles

A Deep Learning-based Model for Evaluating the Sustainability Performance of Accounting Firms

The harmonious and stable development of society is strongly related to the sustainable development of enterprises. In order to better face the challenges of environmental resources, sustainable development must be included in the development focus of accounting enterprises. The research proposes a performance evaluation model based on deep learning, improves RBMs model on the basis of deep belief network (DBN), improves the accuracy of the model through reverse fine-tuning technology, and effectively combines multiple restricted Boltzmann machines (RBMs) and Softmax classifiers to build a modular multi classification model to complete the sustainable development performance evaluation of accounting enterprises. The performance of RBM fine tuning classifier is higher than that of RBM expression and PCA (Principal Component Analysis) expression, which mainly shows the effectiveness and stability of feature extraction. The network output results of test samples are converted into prediction performance evaluation. The model is evaluated by average precision (AP), average recall (AR), and prediction accuracy. The AP, AR, and prediction accuracy of the proposed method are 86.95%, 89.74%, and 88.29% respectively, which are higher than Softmax classifiers, Back Propagation (BP) neural networks, and DBN based Softmax methods, It shows that this method is superior to other algorithms in the application of performance evaluation model for sustainable development of accounting enterprises, and it is feasible and effective, which is of great significance to the establishment of performance evaluation model for the accounting industry.

PQPS: Prior-Art Query-Based Patent Summarizer Using RBM and Bi-LSTM

A prior-art search on patents ascertains the patentability constraints of the invention through an organized review of prior-art document sources. This search technique poses challenges because of the inherent vocabulary mismatch problem. Manual processing of every retrieved relevant patent in its entirety is a tedious and time-consuming job that demands automated patent summarization for ease of access. This paper employs deep learning models for summarization as they take advantage of the massive dataset present in the patents to improve the summary coherence. This work presents a novel approach of patent summarization named PQPS: prior-art query-based patent summarizer using restricted Boltzmann machine (RBM) and bidirectional long short-term memory (Bi-LSTM) models. The PQPS also addresses the vocabulary mismatch problem through query expansion with knowledge bases such as domain ontology and WordNet. It further enhances the retrieval rate through topic modeling and bibliographic coupling of citations. The experiments analyze various interlinked smart device patent sample sets. The proposed PQPS demonstrates that retrievability increases both in extractive and abstractive summaries.

Multimodal Tucker Decomposition for Gated RBM Inference

Gated networks are networks that contain gating connections in which the output of at least two neurons are multiplied. The basic idea of a gated restricted Boltzmann machine (RBM) model is to use the binary hidden units to learn the conditional distribution of one image (the output) given another image (the input). This allows the hidden units of a gated RBM to model the transformations between two successive images. Inference in the model consists in extracting the transformations given a pair of images. However, a fully connected multiplicative network creates cubically many parameters, forming a three-dimensional interaction tensor that requires a lot of memory and computations for inference and training. In this paper, we parameterize the bilinear interactions in the gated RBM through a multimodal tensor-based Tucker decomposition. Tucker decomposition decomposes a tensor into a set of matrices and one (usually smaller) core tensor. The parameterization through Tucker decomposition helps reduce the number of model parameters, reduces the computational costs of the learning process and effectively strengthens the structured feature learning. When trained on affine transformations of still images, we show how a completely unsupervised network learns explicit encodings of image transformations.

Personalized news recommendation based on an improved conditional restricted Boltzmann machine

PurposeBecause of the extensive user coverage of news sites and apps, greater social and commercial value can be realized if users can access their favourite news as easily as possible. However, news has a timeliness factor; there are serious cold start and data sparsity in news recommendation, and news users are more susceptible to recent topical news. Therefore, this study aims to propose a personalized news recommendation approach based on topic model and restricted Boltzmann machine (RBM).Design/methodology/approachFirstly, the model extracts the news topic information based on the LDA2vec topic model. Then, the implicit behaviour data are analysed and converted into explicit rating data according to the rules. The highest weight is assigned to recent hot news stories. Finally, the topic information and the rating data are regarded as the conditional layer and visual layer of the conditional RBM (CRBM) model, respectively, to implement news recommendations.FindingsThe experimental results show that using LDA2vec-based news topic as a conditional layer in the CRBM model provides a higher prediction rating and improves the effectiveness of news recommendations.Originality/valueThis study proposes a personalized news recommendation approach based on an improved CRBM. Topic model is applied to news topic extraction and used as the conditional layer of the CRBM. It not only alleviates the sparseness of rating data to improve the efficient in CRBM but also considers that readers are more susceptible to popular or trending news.

Automated arrhythmia detection from electrocardiogram signal using stacked restricted Boltzmann machine model

Significant advances in deep learning techniques have made it possible to offer technologically advanced methods to detect cardiac abnormalities. In this study, we have proposed a new deep learning based Restricted Boltzmann machine (RBM) model for the classification of arrhythmias from Electrocardiogram (ECG) signal. The work is divided into three phases where, in the first phase, signal processing is performed, including the normalization of the heartbeats as well as the segmentation of the heartbeats. In the second phase, the stacked RBM model is implemented which extracts the essential features from the ECG signal. Finally, a SoftMax activation function is used that classifies the ECG signal into four types of heartbeat classes according to ANSI/AAMA standards. This stacked RBM model is offered as three types of experiments, patient independent data classification for multi-class, patient independent data for binary classification, and patient specific classification. The best result was obtained using patient independent binary classification with an overall accuracy of 99.61%. For Patient Independent Multi Class classification, accuracy obtained was 98.61% and for patient specific data, the accuracy was 95.13%. The experimental results shows that the developed RBM model has better performance in terms of accuracy, sensitivity and specificity as compared to work mentioned in the other research papers.Article highlightsThe proposed RBM model is skilled to automatically classify ECG heartbeat according to the ANSI- AAMI standards with accuracy, Recall, specificity.The performance of the RBM model to correctly classify heartbeat classes was found to be improved.The model is fully automatic, hence there is no requirement of additional system like feature extraction, feature selection, and classification.

Research on e-Commerce User Behavior Analysis Based on Big Data Collaborative Recommendation Algorithm

With the rapid development of Internet, users tend to purchase their favorite products through Internet transactions and online payment. The general trend of e-commerce development in China is that physical trading places are gradually replaced by online trading platforms on the Internet. In this paper, the restricted Boltzmann machine based on category conditions is used to describe the user's own interest preference by using the objective label of the project itself. In this process, only the project information that the user has scored is used, which strengthens the user's personalized needs. The method fully mines user behavior information, replaces commodity content big data with user behavior information as a recommended data set, and can actively push commodity content that users may be interested in to users. Experimental results show that the accuracy of RBM ( Restricted Boltzmann machine) model with nearest neighbor is higher than that of the original model, and the anti-over-fitting ability of the model is also improved.

Comparison of Use of a 2000 Qubit D-Wave Quantum Annealer and MCMC for Sampling, Image Reconstruction, and Classification

The next generation D-Wave quantum computer (QC) having more than 2000 qubits was demonstrated to be adequate for embedding graphs of restricted Boltzmann machines (RBMs) having the connectivity between RBM units (though still limited) already sufficient for successful training by classical algorithms. As a verification of the validity and quality of the employed embedding approach, the RBM was trained by classical contrastive divergence (CD), while the D-Wave was initially used only in the recognition step for reconstruction of incomplete test images (8 × 8 bars-and-stripes), and for their classification. The label classification errors by the D-Wave compared favorably to those obtained by classical Gibbs sampling, indicating that the QC was successfully finding the most energetically favorable combinations of the unknown visible RBM units/labels for the given fixed incomplete input image. The valid RBM embedding was further applied to investigate opportunities for using QCs in the RBM training, and specifically for replacing the classical Gibbs technique in generating a representative sample from the RBM model distribution. Statistical comparison of samples obtained by the Gibbs technique versus 10000 sample states generated from the D-Wave revealed significant differences in the observed outcomes. The D-Wave samples were found insufficiently representative of the model distribution, specifically by missing many local valley found by the Gibbs sampling. On the other hand, the D-Wave possessed an ability to find local valleys of the configuration space that were consistently missed by the classical sampling algorithms, which could potentially be very interesting for sampling applications.

Malicious Code Detection under 5G HetNets Based on a Multi-Objective RBM Model

The fifth generation (5G) mobile communication technology brings people a higher perceived rate experience, the high-quality service of high-density user connection, and other commercial applications. As an important means of data processing in 5G heterogeneous networks (HetNets), data fusion technology is faced with a large number of malicious code attacks. Thus, it is particularly important to find an efficient malicious code detection method. However, in the traditional research, due to dataset imbalance, the complexity of the deep learning network model, the use of a single-objective algorithm, and other factors, it brings greater loss and lower detection accuracy. Therefore, how to choose a suitable network model and improve the data classification accuracy in HetNets is a big challenge. To enhance the model's robustness, a multi-objective restricted Boltzmann machine (RBM) model is designed for training. In this article, evaluation indices are used to comprehensively measure the effect of data classification, introducing a strategy pool to improve the effect of data fusion and using non-dominated sorting genetic algorithms (NSGA-II) to deal with the imbalanced malware family. Experimental results demonstrate that the proposed multi-objective RBM model combined with NSGA-II can effectively enhance the data classification accuracy of HetNets and reduce the loss in the process of data fusion.

Analysis on Noisy Boltzmann Machines and Noisy Restricted Boltzmann Machines

The Boltzmann machine (BM) and restricted Boltzmann machine (RBM) models are representative stochastic neural networks, in which neuron states are determined by stochastic activation functions. They are widely used in many applications. However, when analog circuits are used to realize a neural network, noise are not avoidable. The noise could come from external environments, such as power supplies and thermal noise, and they will affect the neurons’ stochastic behaviour in the BM and RBM models. Hence it is important to theoretically study how the noise affect the operation of the BM and RBM models. To best of our knowledge, there are little works related to the analysis on noisy BMs and noisy RBMs. This paper considers that there are additive noise in the inputs of the neurons, and theoretically studies the behaviors of the two models under this imperfect condition. It is found that the effect of additive noise is similar to increasing the temperature factors of the two models. Since the input noise may make the networks to have wrong stochastic behaviour, there is Kullback Leibler (KL) divergence loss in noisy BMs and noisy RBMs. Based on the Gaussian-distributed noise assumption, a noise compensation method is proposed to suppress the effect of additive noise. Experiments show that the proposed noise compensation method can greatly suppress the KL divergence loss. In addition, from the experimental results, our method is also effective for handling non-Gaussian noise.

Identifying the degree of genetic interactions using Restricted Boltzmann Machine-A study on colorectal cancer.

The phenomenon of two or more genes affecting the expression of each other in various ways in the development of a single character of an organism is known as gene interaction. Gene interaction not only applies to normal human traits but to the diseased samples as well. Thus, an analysis of gene interaction could help us to differentiate between the normal and the diseased samples or between the two/more phases any diseased samples. At the first stage of this work we have used restricted Boltzmann machine model to find such significant interactions present in normal and/or cancer samples of every gene pairs of 20 genes of colorectal cancer data set (GDS4382) along with the weight/degree of those interactions. Later on, we are looking for those interactions present in adenoma and/or carcinoma samples of the same 20 genes of colorectal cancer data set (GDS1777). The weight/degree of those interactions represents how strong/weak an interaction is. At the end we will create a gene regulatory network with the help of those interactions, where the regulatory genes are identified by using Naïve Bayes Classifier. Experimental results are validated biologically by comparing the interactions with NCBI databases.

Learning Individualized Treatment Rules for Multiple-Domain Latent Outcomes

For many mental disorders, latent mental status from multiple-domain psychological or clinical symptoms may perform as a better characterization of the underlying disorder status than a simple summary score of the symptoms, and they may also serve as more reliable and representative features to differentiate treatment responses. Therefore, to address the complexity and heterogeneity of treatment responses for mental disorders, we provide a new paradigm for learning optimal individualized treatment rules (ITRs) by modeling patients’ latent mental status. We first learn the multi-domain latent states at baseline from the observed symptoms under a restricted Boltzmann machine (RBM) model, which encodes patients’ heterogeneous symptoms using an economical number of latent variables and yet remains flexible. We then optimize a value function defined by the latent states after treatment by exploiting a transformation of the observed symptoms based on the RBM without modeling the relationship between the latent mental states before and after treatment. The optimal treatment rules are derived using a weighted large margin classifier. We derive the convergence rate of the proposed estimator under the latent models. Simulation studies are conducted to test the performance of the proposed method. Finally, we apply the developed method to real world studies and we demonstrate the utility and advantage of our method in tailoring treatments for patients with major depression, and identify patient subgroups informative for treatment recommendations. Supplementary materials for this article are available online.

Toward sampling from undirected probabilistic graphical models using a D-Wave quantum annealer

A D-Wave quantum annealer (QA) having a 2048 qubit lattice, with no missing qubits and couplings, allowed embedding of a complete graph of a restricted Boltzmann machine (RBM). A handwritten digit OptDigits dataset having 8 × 7 pixels of visible units was used to train the RBM using classical contrastive divergence. Embedding of the classically trained RBM into the D-Wave lattice was used to demonstrate that the QA offers a high-efficiency alternative to the classical Markov chain Monte Carlo (MCMC) for reconstructing missing labels of the test images as well as a generative model on a classically trained RBM. At any training iteration, the D-Wave-based classification had classification error less than half of that in MCMC. The main goal of this study was to investigate the quality of the QA sample from the RBM model probability distribution and compare it to a classical MCMC during the Gibbs sampling traditionally used in RBM training. For the OptDigits dataset, the states in the D-Wave sample belonged to about twice as many local valleys (LVs) as in the MCMC sample. All the lowest energy (the highest joint probability), local minima in the MCMC sample were also found by the D-Wave. The D-Wave missed many of the higher-energy LVs, while also finding many “new” LVs consistently missed by the MCMC. It was established that the “new” LVs that the D-Wave finds are important for the model distribution in terms of the energy of the corresponding local minima, the width of the LVs and the height of the escape barrier.

Image recognition based on improved convolutional deep belief network model

Aiming at the homogeneity of convolution kernels in Convolutional Deep Belief Network (CDBN), a cross-entropy-based sparse penalty mechanism suitable for Convolutional Restricted Boltzmann Machine (CRBM) model is introduced which makes the hidden layer units of the whole network in a lower activation state. On this basis, a parameter learning algorithm is applied to compensate the gradient by introducing the prior information of the samples, which alternates the supervised learning and unsupervised learning. The experimental results show that the proposed model can weaken the homogeneity of convolution kernels and improve the supervising and predicting ability of the network. The recognition rate on simulated dataset achieves 97.45%, which is increased 5.12% and 1.29% than Convolutional Neural Network (CNN) and the traditional CDBN model, respectively. At the same time, test error rate on common dataset MNIST also shows that the proposed model is more effective than some other state-of-the-art deep learning models.

The accuracy of restricted Boltzmann machine models of Ising systems

Restricted Boltzmann machines (RBM) provide a general framework for modeling physical systems, but their behavior is dependent on hyperparameters such as the learning rate, the number of hidden nodes and the form of the threshold function. This article accordingly examines in detail the influence of these parameters on Ising spin system calculations. A tradeoff is identified between the accuracy of statistical quantities such as the specific heat and that of the joint distribution of energy and magnetization. The optimal structure of the RBM therefore depends intrinsically on the physical problem to which it is applied.

Restricted Boltzmann Machine-driven Interactive Estimation of Distribution Algorithm for personalized search

Effective and efficient personalized search is one of the most pursued objectives in the era of big data. The challenge of this problem lies in its complex quantifying evaluations and dynamic user preferences. A user-involved interactive evolutionary algorithm is a good choice if it has reliable preference surrogate and powerful evolutionary strategies. A Restricted Boltzmann Machine (RBM) assisted Interactive Estimation of Distribution Algorithm (IEDA) is presented to enhance the IEDA in solving the personalized search. Specifically, a dual-RBM module is developed to simultaneously provide a preference surrogate and a probability model for conducting the individual selection and generation of the IEDA. Firstly, the positive and negative preferences of the currently involved user in IEDA are distinguished and combined to achieve a dual-RBM, and then the weighted energy functions of the RBM model together with social group information from users with similar preferences are designed as the preference surrogate. The probability of the trained positive RBM on the visible units is fetched as the reproduction model of EDA since it reflects the attribute distributions of more preferred items. Some benchmarks from the Movielens and Amazon datasets are applied to experimentally demonstrate the superiority of the proposed algorithm in improving the efficiency and effectiveness of the interactive evolutionary computations served personalized search.

Parallelized Training of Restricted Boltzmann Machines Using Markov-Chain Monte Carlo Methods

Restricted Boltzmann machine (RBM) is a generative stochastic neural network that can be applied to collaborative filtering technique used by recommendation systems. Prediction accuracy of the RBM model is usually better than that of other models for recommendation systems. However, training the RBM model involves Markov-Chain Monte Carlo method, which is computationally expensive. In this paper, we have successfully applied distributed parallel training using Horovod framework to improve the training time of the RBM model. Our tests show that the distributed training approach of the RBM model has a good scaling efficiency. We also show that this approach effectively reduces the training time to little over 12 min on 64 CPU nodes compared to 5 h on a single CPU node. This will make RBM models more practically applicable in recommendation systems.

Unsupervised Rotation Factorization in Restricted Boltzmann Machines.

Finding suitable image representations for the task at hand is critical in computer vision. Different approaches extending the original Restricted Boltzmann Machine (RBM) model have recently been proposed to offer rotation-invariant feature learning. In this paper, we present an extended novel RBM that learns rotation invariant features by explicitly factorizing for rotation nuisance in 2D image inputs within an unsupervised framework. While the goal is to learn invariant features, our model infers an orientation per input image during training, using information related to the reconstruction error. The training process is regularised by a Kullback-Leibler divergence, offering stability and consistency. We used the γ -score, a measure that calculates the amount of invariance, to mathematically and experimentally demonstrate that our approach indeed learns rotation invariant features. We show that our method outperforms the current state-of-the-art RBM approaches for rotation invariant feature learning on three different benchmark datasets, by measuring the performance with the test accuracy of an SVM classifier. Our implementation is available at https://bitbucket.org/tuttoweb/rotinvrbm.

Development and Application of Matrix Variate Restricted Boltzmann Machine

Matrix Variate Restricted Boltzmann Machine (MVRBM) model could maintain valuable spatial information using much fewer model parameters than the classic RBM, thus it has been used in image analysis, pattern recognition and machine learning for data analysis. This paper introduces two extensions of MVRBM: Matrix Variate Deep Belief Network (MVDBN) and Multimodal MVRBM (MMVRBM), both of them are composed of two or more MVRBMs whose input and latent variables are in matrix form. The MVDBN and MMVRBM have much fewer model parameters and deeper layer with better features to avoid overfitting and more accurate model easier to be learned. We test their efficiency on two real-world applications: image super-resolution and handwritten digit recognition. The experiments show the feasibility and superiority of the proposed models.

Conditional restricted Boltzmann machine for item recommendation

Recommender systems provide an excellent solution to the issue of information overload by generating item recommendation from a huge collection of items based on users’ preferences. In terms of modeling users’ rating data, existing methods are mainly neighborhood- and factorization-based methods, most of which are rating oriented. Among network-based methods, the restricted Boltzmann machine (RBM) model is also applied to rating prediction tasks. However, item recommendation tasks play a more important role in the real world, due to the large item space as well as users’ limited attention. In this paper, we treat users’ rating behaviors from a new perspective and study the effectiveness of conditional RBM (CRBM) in modeling users’ rating preferences for top-k recommendation. We conduct extensive empirical studies on four real-world datasets and find that our proposed CRBM-IR is very competitive in exploiting users’ explicit rating feedback in comparison with the closely related works.

Multi-view RBM with posterior consistency and domain adaptation

The restricted Boltzmann machine (RBM) and extensions are rarely used in the field of multi-view learning. In this paper, we first present a multi-view RBM model for classification, which is named RBM with posterior consistency (PCRBM). PCRBM computes multiple representations by regularizing the marginal likelihood function with the consistency among representations from multiple views. However, PCRBM ignores the specific information of each view, where the learned representations just contain the consistency information among multiple views. To address this problem, we propose a novel multi-view RBM model and name it as RBM with posterior consistency and domain adaptation (PDRBM). PDRBM divides the hidden units of each view into two groups: the one is the view-consistency group that contains the consistency information among multiple views, and the other is the view-specific group that contains the information unique to this separate view. In addition, PDRBM balances the relationship among view-consistency hidden representations for multi-view classification. Contrasting with existing multi-view classification methods, PDRBM has achieved satisfactory results on two-class and multi-class classification multi-view datasets.