Restricted Boltzmann Machine Model Research Articles

A novel collaborative filtering algorithm of machine learning by integrating restricted Boltzmann machine and trust information

With rapidly increasing information on the Internet, it can be more difficult and time consuming to find what one really wants, especially in e-commerce. Systems and methods based on machine learning are emerging to generate recommendations based on various factors. Existing methods face issues such as data sparsity and cold starts. To alleviate their effects, this paper proposes a novel social recommendation method combined with a restricted Boltzmann machine model and trust information to improve the performance of recommendations. Specifically, users’ preferences and ratings of items are used as data inputs in a restricted Boltzmann machine model to learn the probability distribution. In addition, user similarities are calculated by weighting user similarity and user trust values derived from trust information (i.e., trust statements explicitly given by users). Predictions are made by integrating user-history ratings and ratings of trusted users from a well-trained restricted Boltzmann machine model. Experimental results show that the proposed method has better prediction accuracy than other common collaborative filtering algorithms of machine learning.

A Hybrid Imputation Method Based on Denoising Restricted Boltzmann Machine

Data imputation is an important issue in data processing and analysis which has serious impact on the results of data mining and learning. Most of the existing algorithms are either utilizing whole data sets for imputation or only considering the correlation among records. Aiming at these problems, the article proposes a hybrid method to fill incomplete data. In order to reduce interference and computation, denoising restricted Boltzmann machine model is developed for robust feature extraction from incomplete data and clustering. Then, the article proposes partial-distance and co-occurrence matrix strategies to measure correlation between records and attributes, respectively. Finally, quantifiable correlation is converted to weights for imputation. Compared with different algorithms, the experimental results confirm the effectiveness and efficiency of the proposed method in data imputation.

Isolation Forest as an Alternative Data-Driven Mineral Prospectivity Mapping Method with a Higher Data-Processing Efficiency

Mineral exploration targets can be delineated through multivariate analysis. These targets are usually recognized as anomalies in the procedure of data mining using a detection algorithm. In this paper, isolation forest, which is a data-mining algorithm for anomaly detection, was adapted to map gold prospectivity in the Laotudingzi–Xiaosiping district in Jilin Province, northeastern China. The performance of the isolation forest model was compared with those of restricted Boltzmann machine and logistic regression models for mapping gold prospectivity. The isolation forest model outperforms the restricted Boltzmann machine model but not the logistic regression model based on the receiver operating characteristic curve and area under the curve; however, the isolation forest model is much more efficient than the other two models in terms of data-processing efficiency. It takes 15.48, 254.74, and 57.33 s, respectively, to integrate the predictor map layers using the three models. Gold mineral exploration targets were optimally delineated by using the Youden index to maximize spatial association between delineated gold mineral exploration targets and the discovered gold deposits. Gold mineral exploration targets predicted by the three models occupy 10.97, 10.36, and 8.40% of the study area and contain 88, 81, and 81% of the discovered gold deposits. Therefore, the isolation forest algorithm is a potentially useful mineral prospectivity mapping method.

Mapping mineral prospectivity by using one-class support vector machine to identify multivariate geological anomalies from digital geological survey data

ABSTRACTMineral targets are local geological anomalies. In a study area of a number of unit cells, mapping mineral prospectivity can be implemented by identifying anomaly cells from the unit cell population. One-class support vector machine (OCSVM) models can yield useful results in anomaly detection in high-dimensional data or without any assumptions on the distribution of the inlying data. The OCSVM model was applied to mapping gold prospectivity of the Laotudingzi-Xiaosiping district, an area with a complex geological background, in Jilin Province, China. The decision function value of each unit cell belonging to an anomaly was computed on the basis of the trained OCSVM model and used to express gold prospectivity of the cell. The receiver operating characteristic (ROC) curve, area under curve (AUC) and data-processing efficiency were used to compare the performance of the OCSVM model and a restricted Boltzmann machine (RBM) model in mapping gold prospectivity. The results show that the OCSVM model outperforms the RBM model in terms of ROC, AUC and data-processing efficiency. Gold targets were optimally delineated by using the Youden index to maximise the spatial association between the delineated gold targets and known gold deposits. The gold targets delineated by the OCSVM model occupy 11% of the study area and contain 88% of the known gold deposits; and the gold targets delineated by the RBM model occupy 10% of the study area and contain 81% of the known gold deposits. Therefore, the OCSVM model is a feasible mineral prospectivity mapping approach.

Elastic restricted Boltzmann machines for cancer data analysis

Restricted Boltzmann machines (RBMs) are endowed with the universal power of modeling (binary) joint distributions. Meanwhile, as a result of their confining network structure, training RBMs confronts less difficulties when dealing with approximation and inference issues. But little work has been developed to fully exploit the capacity of these models to analyze cancer data, e.g., cancer genomic, transcriptomic, proteomic and epigenomic data. On the other hand, in the cancer data analysis task, the number of features/predictors is usually much larger than the sample size, which is known as the “p ≫ N” problem and is also ubiquitous in other bioinformatics and computational biology fields. The “p ≫ N” problem puts the bias-variance trade-off in a more crucial place when designing statistical learning methods. However, to date, few RBM models have been particularly designed to address this issue. We propose a novel RBMs model, called elastic restricted Boltzmann machines (eRBMs), which incorporates the elastic regularization term into the likelihood function, to balance the model complexity and sensitivity. Facilitated by the classic contrastive divergence (CD) algorithm, we develop the elastic contrastive divergence (eCD) algorithm which can train eRBMs efficiently. We obtain several theoretical results on the rationality and properties of our model.We further evaluate the power of our model based on a challenging task — predicting dichotomized survival time using the molecular profiling of tumors. The test results show that the prediction performance of eRBMs is much superior to that of the state-of-the-art methods. The proposed eRBMs are capable of dealing with the “p ≫ N” problems and have superior modeling performance over traditional methods. Our novel model is a promising method for future cancer data analysis.

Deep Multimodal Fusion: A Hybrid Approach

We propose a novel hybrid model that exploits the strength of discriminative classifiers along with the representation power of generative models. Our focus is on detecting multimodal events in time varying sequences as well as generating missing data in any of the modalities. Discriminative classifiers have been shown to achieve higher performances than the corresponding generative likelihood-based classifiers. On the other hand, generative models learn a rich informative space which allows for data generation and joint feature representation that discriminative models lack. We propose a new model that jointly optimizes the representation space using a hybrid energy function. We employ a Restricted Boltzmann Machines (RBMs) based model to learn a shared representation across multiple modalities with time varying data. The Conditional RBMs (CRBMs) is an extension of the RBM model that takes into account short term temporal phenomena. The hybrid model involves augmenting CRBMs with a discriminative component for classification. For these purposes we propose a novel Multimodal Discriminative CRBMs (MMDCRBMs) model. First, we train the MMDCRBMs model using labeled data by training each modality, followed by training a fusion layer. Second, we exploit the generative capability of MMDCRBMs to activate the trained model so as to generate the lower-level data corresponding to the specific label that closely matches the actual input data. We evaluate our approach on ChaLearn dataset, audio-mocap, as well as the Tower Game dataset, mocap-mocap as well as three multimodal toy datasets. We report classification accuracy, generation accuracy, and localization accuracy and demonstrate its superiority compared to the state-of-the-art methods.

A Joint Deep Boltzmann Machine (jDBM) Model for Person Identification Using Mobile Phone Data

We propose an audio–visual person identification approach based on a joint deep Boltzmann machine (jDBM) model. The proposed jDBM model is trained in three steps: 1) learning the unimodal DBM models corresponding to the speech and facial image modalities, 2) learning the shared layer parameters using a joint restricted Boltzmann machine (jRBM) model, and 3) the fine-tuning of the jDBM model after the initialization with the parameters of the unimodal DBMs and the shared layer. The activation probabilities of the units of the shared layer are used as the joint features and a logistic regression classifier is used for the combined speech and facial image recognition. We show that by learning the shared layer parameters using a jRBM, a higher accuracy can be achieved compared to the greedy layer-wise initialization. The performance of our proposed model is also compared with a state-of-the art support vector machine (SVM), deep belief network (DBN), and the deep auto-encoder (DAE) models. In addition, our experimental results show that the joint representations obtained from the proposed jDBM model are robust to noise and missing information. Experiments were carried out on the challenging MOBIO database, which includes audio–visual data captured using mobile phones.

Object-oriented ensemble classification for polarimetric SAR Imagery using restricted Boltzmann machines

ABSTRACTA series of deep learning algorithms have recently shown excellent performances in many different fields. Deep models are usually generated by stacking similar modules, e.g., restricted Boltzmann machines (RBMs), and they are especially suitable for discriminating complex objects through the use of ‘big data’. However, object-oriented classification (OOC) for polarimetric synthetic aperture radar (PolSAR) imagery is based on homogeneous regions instead of pixels, which results in a degraded performance for the deep models, as the data volume is inadequate. To solve this problem, we adopt an RBM as the module, and use it to construct an adaptive boosting (AdaBoost) model instead of a stacked deep model, to carry out OOC for PolSAR imagery. The experimental results demonstrate that the proposed model is superior to the stacked RBM model and the other common methods for OOC.

Weight Uncertainty in Boltzmann Machine

Based on restricted Boltzmann machine (RBM), the deep learning models can be roughly divided into deep belief networks (DBNs) and deep Boltzmann machine (DBM). However, the overfitting problems commonly exist in neural networks and RBM models. In order to alleviate the overfitting problem, lots of research has been done. This paper alleviated the overfitting problem in RBM and proposed the weight uncertainty semi-restricted Boltzmann machine (WSRBM) to improve the ability of image recognition and image reconstruction. First, this paper built weight uncertainty RBM model based on maximum likelihood estimation. And in the experimental section, this paper verified the effectiveness of the weight uncertainty deep belief network and the weight uncertainty deep Boltzmann machine. Second, in order to obtain better reconstructed images, this paper used the semi-restricted Boltzmann machine (SRBM) as the feature extractor and built the WSRBM. Lastly, this paper used hybrid Monte Carlo sampling and cRBM to improve the classification ability of WSDBM. The experiments showed that the weight uncertainty RBM, weight uncertainty DBN and weight uncertainty DBM were effective compared with the dropout method. And the WSDBM model performed well in image recognition and image reconstruction as well. This paper introduced the weight uncertainty method to RBM, and proposed a WSDBM model, which was effective in image recognition and image reconstruction.

PATTERN: Pain Assessment for paTients who can't TEll using Restricted Boltzmann machiNe.

BackgroundAccurately assessing pain for those who cannot make self-report of pain, such as minimally responsive or severely brain-injured patients, is challenging. In this paper, we attempted to address this challenge by answering the following questions: (1) if the pain has dependency structures in electronic signals and if so, (2) how to apply this pattern in predicting the state of pain. To this end, we have been investigating and comparing the performance of several machine learning techniques.MethodsWe first adopted different strategies, in which the collected original n-dimensional numerical data were converted into binary data. Pain states are represented in binary format and bound with above binary features to construct (n + 1) -dimensional data. We then modeled the joint distribution over all variables in this data using the Restricted Boltzmann Machine (RBM).ResultsSeventy-eight pain data items were collected. Four individuals with the number of recorded labels larger than 1000 were used in the experiment. Number of avaliable data items for the four patients varied from 22 to 28. Discriminant RBM achieved better accuracy in all four experiments.ConclusionThe experimental results show that RBM models the distribution of our binary pain data well. We showed that discriminant RBM can be used in a classification task, and the initial result is advantageous over other classifiers such as support vector machine (SVM) using PCA representation and the LDA discriminant method.

MapReduce based distributed learning algorithm for Restricted Boltzmann Machine

Deep learning is recently regarded as the closest artificial intelligence model to human brain. It is about learning multiple levels of representation and abstraction that help to make sense of data such as images, sound, and text. One deep model often consists of a hierarchical architecture that has the capability to model super non-linear and stochastic problems. Restricted Boltzmann Machine (RBM) is the main constructing block of current deep networks, as most of deep architectures are built with it. Based on MapReduce framework and Hadoop distributed file system, this paper proposes a distributed algorithm for training the RBM model. Its implementation and performance are evaluated on Big Data platform-Hadoop. The main contribution of the new learning algorithm is that it solves the scalability problem that limits the development of deep learning. The intelligence growing process of human brain requires learning from Big Data. The distributed learning mechanism for RBM makes it possible to abstract sophisticated and informative features from Big Data to achieve high-level intelligence. The evaluations of the proposed learning algorithm are carried out on image inpainting and classification problems based on the BAS dataset and MNIST hand-written digits dataset.

Multiple Emotion Tagging for Multimedia Data by Exploiting High-Order Dependencies Among Emotions

In this paper, a novel approach of multiple emotional multimedia tagging is proposed, which explicitly models the higher-order relations among emotions. First, multimedia features are extracted from the multimedia data. Second, a traditional multi-label classifier is used to obtain the measurements of the multi-emotion labels. Then, we propose a three-layer restricted Boltzmann machine (TRBM) model to capture the higher-order relations among emotion labels, as well as the relations between labels and measurements . Finally , the TRBM model is used to infer the samples' multi- emotion labels by combining the emotion measurements with the dependencies among multi- emotions . Experimental results on four databases demonstrate that our method is more effective than both feature -driven methods and current model-based methods, which capture the pairwise relations among labels by the Bayesian network (BN). Furthermore , the comparison of BN models and the proposed TRBM model verifies that the patterns captured by the latent units of TRBM contain not only all the dependencies captured by the BN but also many other dependencies that the BN cannot capture.

Infrared ultraspectral signature classification based on a restricted Boltzmann machine with sparse and prior constraints

The state-of-the-art ultraspectral technology brings a new hope for the high precision applications due to its high spectral resolution. However, it comes with new challenges brought by the improvement of spectral resolution such as the Hughes phenomenon and over-fitting issue, and our work is aimed at addressing these problems. As new Markov random field (MRF) models, the restricted Boltzmann machines (RBMs) have been used as generative models for many different pattern recognition and artificial intelligence applications showing promising and outstanding performance. In this article, we propose a new method for infrared ultraspectral signature classification based on the RBMs, which adopt the regularization-based techniques to improve the classification accuracy and robustness to noise compared to traditional RBMs. First, we add an arctan-like term to the objective function as a sparse constraint to improve the classification accuracy. Second, we utilize a Gaussian prior to avoid the over-fitting problem. Third, to further improve the classification performance, a multi-layer RBM model, a deep belief network (DBN), is adopted for infrared ultraspectral signature classification. Experiments using different spectral libraries provided by the Advanced Spaceborne Thermal Emission and Reflection Radiometer (ASTER) and the Environmental Protection Agency (EPA) were performed to evaluate the performance of the proposed method by comparing it with other traditional methods, including spectral coding-based classifiers (binary coding (BC), spectral feature-based binary coding (SFBC), and spectral derivative feature coding (SDFC) matching methods), a novel feature extraction method termed crosscut feature extraction matching (CF), and three machine learning methods (artificial deoxyribonucleic acid (DNA)-based spectral matching (ADSM), DBN, and sparse deep belief network (SparseDBN)). Experimental results demonstrate that the proposed method is superior to the other methods with which it was compared and can simultaneously improve the accuracy and robustness of classification.

Training Restricted Boltzmann Machines

Restricted Boltzmann Machines (RBMs), two-layered probabilistic graphical models that can also be interpreted as feed forward neural networks, enjoy much popularity for pattern analysis and generation. Training RBMs however is challenging. It is based on likelihood maximization, but the likelihood and its gradient are computationally intractable. Therefore, training algorithms such as Contrastive Divergence (CD) and learning based on Parallel Tempering (PT) rely on Markov chain Monte Carlo methods to approximate the gradient. The presented thesis contributes to understanding RBM training methods by presenting an empirical and theoretical analysis of the bias of the CD approximation and a bound on the mixing rate of PT. Furthermore, the thesis improves RBM training by proposing a new transition operator leading to faster mixing Markov chains, by investigating a different parameterization of the RBM model class referred to as centered RBMs, and by exploring estimation techniques from statistical physics to approximate the likelihood. Finally, an analysis of the representational power of deep belief networks with real-valued visible variables is given.

A generative restricted Boltzmann machine based method for high-dimensional motion data modeling

Many computer vision applications involve modeling complex spatio-temporal patterns in high-dimensional motion data. Recently, restricted Boltzmann machines (RBMs) have been widely used to capture and represent spatial patterns in a single image or temporal patterns in several time slices. To model global dynamics and local spatial interactions, we propose to theoretically extend the conventional RBMs by introducing another term in the energy function to explicitly model the local spatial interactions in the input data. A learning method is then proposed to perform efficient learning for the proposed model. We further introduce a new method for multi-class classification that can effectively estimate the infeasible partition functions of different RBMs such that RBM is treated as a generative model for classification purpose. The improved RBM model is evaluated on two computer vision applications: facial expression recognition and human action recognition. Experimental results on benchmark databases demonstrate the effectiveness of the proposed algorithm.

A Neural Autoregressive Approach to Attention-based Recognition

Tasks that require the synchronization of perception and action are incredibly hard and pose a fundamental challenge to the fields of machine learning and computer vision. One important example of such a task is the problem of performing visual recognition through a sequence of controllable fixations; this requires jointly deciding what inference to perform from fixations and where to perform these fixations. While these two problems are challenging when addressed separately, they become even more formidable if solved jointly. Recently, a restricted Boltzmann machine (RBM) model was proposed that could learn meaningful fixation policies and achieve good recognition performance. In this paper, we propose an alternative approach based on a feed-forward, auto-regressive architecture, which permits exact calculation of training gradients (given the fixation sequence), unlike for the RBM model. On a problem of facial expression recognition, we demonstrate the improvement gained by this alternative approach. Additionally, we investigate several variations of the model in order to shed some light on successful strategies for fixation-based recognition.

Modeling Higher-Order Correlations within Cortical Microcolumns

We statistically characterize the population spiking activity obtained from simultaneous recordings of neurons across all layers of a cortical microcolumn. Three types of models are compared: an Ising model which captures pairwise correlations between units, a Restricted Boltzmann Machine (RBM) which allows for modeling of higher-order correlations, and a semi-Restricted Boltzmann Machine which is a combination of Ising and RBM models. Model parameters were estimated in a fast and efficient manner using minimum probability flow, and log likelihoods were compared using annealed importance sampling. The higher-order models reveal localized activity patterns which reflect the laminar organization of neurons within a cortical column. The higher-order models also outperformed the Ising model in log-likelihood: On populations of 20 cells, the RBM had 10% higher log-likelihood (relative to an independent model) than a pairwise model, increasing to 45% gain in a larger network with 100 spatiotemporal elements, consisting of 10 neurons over 10 time steps. We further removed the need to model stimulus-induced correlations by incorporating a peri-stimulus time histogram term, in which case the higher order models continued to perform best. These results demonstrate the importance of higher-order interactions to describe the structure of correlated activity in cortical networks. Boltzmann Machines with hidden units provide a succinct and effective way to capture these dependencies without increasing the difficulty of model estimation and evaluation.

Enhancing performance of restricted Boltzmann machines via log-sum regularization

Restricted Boltzmann machines (RBMs) are often used as building blocks to construct a deep belief network. By optimizing several RBMs, the deep networks can be trained quickly to achieve good performance on the tasks of interest. To further improve the performance of data representation, many researches focus on incorporating sparsity into RBMs. In this paper, we propose a novel sparse RBM model, referred to as LogSumRBM. Instead of constraining the expected activation of every hidden unit to the same low level of sparsity as done in [27], we explicitly encourage the hidden units to be sparse through adding a log-sum norm constraint on the totality of the hidden units’ activation probabilities. In this approach, we do not need to keep the “firing rate” of each hidden unit at a certain level that is set beforehand, and therefore the level of sparsity corresponding to each hidden unit can be automatically learnt based on the task at hand. Some experiments conducted on several image data sets of different scales show that LogSumRBM learns sparser and more discriminative representations compared with the related state-of-the-art models, and stacking two LogSumRBMs learns more significant features which mimic computations in the cortical hierarchy. Meanwhile, LogSumRBM can also be used to pre-train deep networks, and achieve better classification performance.

Privacy-preserving restricted boltzmann machine.

With the arrival of the big data era, it is predicted that distributed data mining will lead to an information technology revolution. To motivate different institutes to collaborate with each other, the crucial issue is to eliminate their concerns regarding data privacy. In this paper, we propose a privacy-preserving method for training a restricted boltzmann machine (RBM). The RBM can be got without revealing their private data to each other when using our privacy-preserving method. We provide a correctness and efficiency analysis of our algorithms. The comparative experiment shows that the accuracy is very close to the original RBM model.

Semantic Topics Modeling Approach for Community Detection

ABSTRACT Social networks play an increasingly important role in online world as it enables individuals to easily share opinions, experiences and expertise. The capability to extract latent communities based on user interest is becoming vital for a wide variety of applications. However, existing literature on community extraction has largely focused on methods based on the link structure of a given social network. Such link-based methods ignore the content of social interactions, which may be crucial for accurate and meaningful community extraction. In this paper, we present a novel approach for community extraction which naturally incorporates the content published within the social network with its semantic features. Two layer generative Restricted Boltzmann Machines model is applied for community discovery. The model assumes that users within a community communicate based on topics of mutual interest. The proposed model naturally allows users to belong to multiple communities. Through extensive experiments on the Twitter data for scientific papers, we demonstrate that the model is able to extract well-connected and topically meaningful communities.