Gaussian Restricted Boltzmann Machine Research Articles

Self-supervised Gaussian Restricted Boltzmann Machine via joint contrastive representation and contrastive divergence

In this paper, we propose a novel self-supervised Gaussian Restricted Boltzmann Machine with contrastive learning (CL-GRBM), which fuses contrastive representation learning and contrastive divergence to optimize and enhance the representation of GRBM. Built upon the concept of contrastive representation learning, CL-GRBM aims to enhance the representation capacity of the GRBM. Firstly, a pair of positive samples are constructed by one times Gibbs sampling with the original data. Then, the contrastive loss is used to pull the positive samples closer and push other samples further apart, making similar representations closer and different representations farther apart. In summary, during the training process of CL-GRBM using the CD algorithm, the objective is to align the sampling probability distribution of the visible layer in CL-GRBM as closely as possible with the empirical distribution of the original data. In the feature space of the sampling probabilities in the hidden layer, the distance between positive samples is minimized to capture the intrinsic structure of the data. According to the experimental verification, the proposed CL-GRBM shows better performance than contrastive models. As a shallow self-supervised model, it has even better performance than some excellent deep self-supervised models.

An Efficient Water Quality Prediction and Assessment Method Based on the Improved Deep Belief Network—Long Short-Term Memory Model

The accuracy of water quality prediction and assessment has always been the focus of environmental departments. However, due to the high complexity of water systems, existing methods struggle to capture the future internal dynamic changes in water quality based on current data. In view of this, this paper proposes a data-driven approach to combine an improved deep belief network (DBN) and long short-term memory (LSTM) network model for water quality prediction and assessment, avoiding the complexity of constructing a model of the internal mechanism of water quality. Firstly, using Gaussian Restricted Boltzmann Machines (GRBMs) to construct a DBN, the model has a better ability to extract continuous data features compared to classical DBN. Secondly, the extracted time-series data features are input into the LSTM network to improve predicting accuracy. Finally, due to prediction errors, noise that randomly follows the Gaussian distribution is added to the assessment results based on the predicted values, and the probability of being at the current water quality level in the future is calculated through multiple evolutionary computations to complete the water quality assessment. Numerical experiments have shown that our proposed algorithm has a greater accuracy compared to classical algorithms in challenging scenarios.

Lapping Quality Prediction of Ceramic Fiber Brush Based on Gaussian-Restricted Boltzmann Machine.

Although ceramic fiber brushes have been widely used for deburring and surface finishing, the associated relationship between process parameters and lapping quality is still unclear. In order to optimize the lapping process of ceramic fiber brushes, this paper proposes a multi-layer neural network based on the Gaussian-restricted Boltzmann machine (GRBM), and verified its prediction effectiveness. Compared with a traditional back-propagation neural network, its prediction error was reduced from 7.6% to 4.5%, and the determination coefficient was increased from 0.96 to 0.98, respectively. The comparison results showed that the proposed model can better grasp the relationship between process parameters and machining quality, which can be used as a decision-making foundation for lapping-process optimization.

Hybrid recommendation algorithm based on real-valued RBM and CNN.

With the unprecedented development of big data, it is becoming hard to get the valuable information hence, the recommendation system is becoming more and more popular. When the limited Boltzmann machine is used for collaborative filtering, only the scoring matrix is considered, and the influence of the item content, the user characteristics and the user evaluation content on the predicted score is not considered. To solve this problem, the modified hybrid recommendation algorithm based on Gaussian restricted Boltzmann machine is proposed in the paper. The user text information and the item text information are input to the embedding layer to change the text information into numerical vector. The convolutional neural network is used to get the latent feature vector of the text information. The latent vector is connected to rating vector to get the item and the user vector. The user vector and the item vector are fused together to get the user-item matrix which is input to the visual layer of Gaussian restricted Boltzmann Machine to predict the ratings. Some simulation experiments have been performed on the algorithm, and the results of the experiments proved that the algorithm is feasible.

Sound quality evaluation of electronic expansion valve using Gaussian restricted Boltzmann machines based DBN

The operational noise of electronic expansion valve (EEV) is the main component of noise source for the indoor unit of the air conditioner. The sound quality of EEV noise directly influences the consumer’s perceptions. Based on deep belief network (DBN) technique, an objective model has been conducted to evaluate the sound quality of EEV. Gaussian Restricted Boltzmann Machines (GRBM) provides the capability of modeling continuous data and has been used to develop a DBN model in this paper. The interior noise of 46 EEVs under working conditions were measured and corresponding subjective evaluation were implemented. A linear regression-based DBN (LR-DBN) is proposed with 6 psychoacoustic metrics and 26 Mel-frequency cepstral coefficients (MFCC) as input features. The performance of LR-DBN was validated against an ordinary DBN, a multiple linear regression (MLR) and a back-propagation neural network (BPNN). The results show that the LR-DBN has higher correlation coefficient and lower prediction error with human perception compared to the other considered methods. In addition, LR-DBN shows better stability than the other models. This present method may be a reliable approach for evaluating EEV sound.

A Global Method for Non-Rigid Registration of Cell Nuclei in Live Cell Time-Lapse Images.

Non-rigid registration of cell nuclei in time-lapse microscopy images can be achieved through estimating the deformation fields using optical flow methods. In contrast to local optical flow models employed in the existing non-rigid registration methods, we introduce approaches based on a global optical flow model. Our registration model consists of a data fidelity term and a regularization term. We compared different regularizers for the deformation fields and found that a convex quadratic function is more suitable than non-convex ones. To improve the robustness, we propose an adaptive weighting scheme based on the statistics of the noise in fluorescence microscopy images as well as a combined local-global scheme. Moreover, we extend the global method by exploiting high-order image features. The best suitable high-order features are determined through learning two generative image models, namely, fields of experts and convolutional Gaussian restricted Boltzmann machine, whose model formulations are both consistent with the assumption of high-order feature constancy in the registration model. Using multiple data sets of real 2D and 3D live cell microscopy image sequences as well as synthetic image data, we demonstrate that our proposed approach outperforms the previous methods in terms of both registration accuracy and computational efficiency.

A Gaussian-Gaussian-Restricted-Boltzmann-Machine-based Deep Neural Network Technique for Photovoltaic System Generation Forecasting

Abstract This paper proposes a new Gaussian-Gaussian-Restricted-Boltzmann-Machine-based method for forecasting photovoltaic (PV) system generation forecasting. Although renewable energy such as PV system and wind power generation has been used to suppress greenhouse gases in the world, it has a drawback that weather conditions influence the generation output significantly. Thus, it is not easy to perform Economic Load Dispatch (ELD) and Unit Commitment in power systems smoothly. From a standpoint of power system operation, more accurate predication models are required to deal with predicted values of PV system generation. In this paper, an efficient Deep Neural Network (DNN) model with Gaussian Gaussian Restricted Boltzmann Machine is presented to predict one-step-ahead PV system generation output. The model is based on Restricted Boltzmann Machine as a feature extractor and MultiLayer Perceptron (MLP) as ANN. The effectiveness of the proposed method is demonstrated for real data of a PV system.

Robust spike-and-slab deep Boltzmann machines for face denoising

The robust Gaussian restricted Boltzmann machine can effectively learn the structure of noise to achieve better results in the face denoising task. The robust Gaussian restricted Boltzmann machine model contains two types of the restricted Boltzmann machine (RBM) model, where a general RBM is used to model the structure of the noise and a Gaussian RBM is used to model the clean data. The spike-and-slab RBM shows better learning abilities than the Gaussian RBM in real images modeling. In addition, the deep Boltzmann machine (DBM) shows powerful image reconstruction ability. To model the real images better, we first stack the spike-and-slab RBM and the RBM to create the spike-and-slab DBM. And then, we utilize the spike-and-slab DBM instead of the Gaussian RBM to model the density of the clean data in the Robust Gaussian RBM, and the proposed method is named as the robust spike-and-slab DBM which can obtain clearer denoising images. Finally, in order to obtain better denoising results, we make use of the learned spike-and-slab DBM model and the mean field method to multi-inference the denoising data learned from the robust spike-and-slab DBM. Experimental results show that the robust spike-and-slab DBM is an effective neural network denoising method.

A topological insight into restricted Boltzmann machines

Restricted Boltzmann Machines (RBMs) and models derived from them have been successfully used as basic building blocks in deep artificial neural networks for automatic features extraction, unsupervised weights initialization, but also as density estimators. Thus, their generative and discriminative capabilities, but also their computational time are instrumental to a wide range of applications. Our main contribution is to look at RBMs from a topological perspective, bringing insights from network science. Firstly, here we show that RBMs and Gaussian RBMs (GRBMs) are bipartite graphs which naturally have a small-world topology. Secondly, we demonstrate both on synthetic and real-world datasets that by constraining RBMs and GRBMs to a scale-free topology (while still considering local neighborhoods and data distribution), we reduce the number of weights that need to be computed by a few orders of magnitude, at virtually no loss in generative performance. Thirdly, we show that, for a fixed number of weights, our proposed sparse models (which by design have a higher number of hidden neurons) achieve better generative capabilities than standard fully connected RBMs and GRBMs (which by design have a smaller number of hidden neurons), at no additional computational costs.

Dynamical analysis of contrastive divergence learning: Restricted Boltzmann machines with Gaussian visible units

The restricted Boltzmann machine (RBM) is an essential constituent of deep learning, but it is hard to train by using maximum likelihood (ML) learning, which minimizes the Kullback–Leibler (KL) divergence. Instead, contrastive divergence (CD) learning has been developed as an approximation of ML learning and widely used in practice. To clarify the performance of CD learning, in this paper, we analytically derive the fixed points where ML and CDn learning rules converge in two types of RBMs: one with Gaussian visible and Gaussian hidden units and the other with Gaussian visible and Bernoulli hidden units. In addition, we analyze the stability of the fixed points. As a result, we find that the stable points of CDn learning rule coincide with those of ML learning rule in a Gaussian–Gaussian RBM. We also reveal that larger principal components of the input data are extracted at the stable points. Moreover, in a Gaussian–Bernoulli RBM, we find that both ML and CDn learning can extract independent components at one of stable points. Our analysis demonstrates that the same feature components as those extracted by ML learning are extracted simply by performing CD1 learning. Expanding this study should elucidate the specific solutions obtained by CD learning in other types of RBMs or in deep networks.

Mean-Field Inference in Gaussian Restricted Boltzmann Machine

A Gaussian restricted Boltzmann machine (GRBM) is a Boltzmann machine defined on a bipartite graph and is an extension of usual restricted Boltzmann machines. A GRBM consists of two different layers: a visible layer composed of continuous visible variables and a hidden layer composed of discrete hidden variables. In this paper, we derive two different inference algorithms for GRBMs based on the naive mean-field approximation (NMFA). One is an inference algorithm for whole variables in a GRBM, and the other is an inference algorithm for partial variables in a GBRBM. We compare the two methods analytically and numerically and show that the latter method is better.

A Novel Bearing Fault Diagnosis Method Based on Gaussian Restricted Boltzmann Machine

To realize the fault diagnosis of bearing effectively, this paper presents a novel bearing fault diagnosis method based on Gaussian restricted Boltzmann machine (Gaussian RBM). Vibration signals are firstly resampled to the same equivalent speed. Subsequently, the envelope spectrums of the resampled data are used directly as the feature vectors to represent the fault types of bearing. Finally, in order to deal with the high-dimensional feature vectors based on envelope spectrum, a classifier model based on Gaussian RBM is applied. Gaussian RBM has the ability to provide a closed-form representation of the distribution underlying the training data, and it is very convenient for modeling high-dimensional real-valued data. Experiments on 10 different data sets verify the performance of the proposed method. The superiority of Gaussian RBM classifier is also confirmed by comparing with other classifiers, such as extreme learning machine, support vector machine, and deep belief network. The robustness of the proposed method is also studied in this paper. It can be concluded that the proposed method can realize the bearing fault diagnosis accurately and effectively.

Deep Reconstruction Models for Image Set Classification

Image set classification finds its applications in a number of real-life scenarios such as classification from surveillance videos, multi-view camera networks and personal albums. Compared with single image based classification, it offers more promises and has therefore attracted significant research attention in recent years. Unlike many existing methods which assume images of a set to lie on a certain geometric surface, this paper introduces a deep learning framework which makes no such prior assumptions and can automatically discover the underlying geometric structure. Specifically, a Template Deep Reconstruction Model (TDRM) is defined whose parameters are initialized by performing unsupervised pre-training in a layer-wise fashion using Gaussian Restricted Boltzmann Machines (GRBMs). The initialized TDRM is then separately trained for images of each class and class-specific DRMs are learnt. Based on the minimum reconstruction errors from the learnt class-specific models, three different voting strategies are devised for classification. Extensive experiments are performed to demonstrate the efficacy of the proposed framework for the tasks of face and object recognition from image sets. Experimental results show that the proposed method consistently outperforms the existing state of the art methods.

An effect of initial distribution covariance for annealing Gaussian restricted Boltzmann machines

In this paper, we investigate an effect that the covariance of an initial distribution for annealed importance sampling (AIS) exertson the estimation accuracy for the partition functions of Gaussian restricted Boltzmann machines (RBMs). A common choicefor an AIS initial distribution is a Gaussian RBM (GRBM) with zero weight connections. Such an initial distribution does notshow any covariance between variables. However, target distributions generally allow a finite covariance between variables. Wepropose a method to design the covariance matrix of an initial distribution for GRBMs. We empirically analyze the effect ofthe initial distribution covariance on the estimation accuracy of AIS. The proposed method for designing initial distributionsoutperforms conventional methods under various conditions.

Deep Adaptive Networks for Visual Data Classification

This paper proposes a classifier called deep adaptive networks (DAN) based on deep belief networks (DBN) for visual data classification. First, we construct a directed deep belief nets by using a set of Restricted Boltzmann Machines (RBM) and a Gaussian RBM via greedy and layerwise unsupervised learning. Then, we refine the parameter space of the deep architecture to adapt the classification requirement by using global gradient-descent based supervised learning. An exponential loss function is utilized to maximize the separability of different classes. Moreover, we apply DAN to visual data classification task and observe an important fact that the learning ability of deep architecture is seriously underrated in real-world applications, especially when there are not enough labeled data. Experiments conducted on standard datasets of different types and different scales demonstrate that the proposed classifier outperforms the representative classification techniques and deep learning methods.

Continuous blood pressure assessment from a photoplethysmographic signal with Deep Belief Networks (LB674)

Continuous blood pressure assessment from a photoplethysmographic signal with Deep Belief Networks Vicent J. Ribas Ripoll PhD; Anna Wojdel PhD; Anna Sáez de Tejada Cuenca; Juan C. Ruiz‐Rodríguez, MD; Adolf Ruiz‐Sanmartín, MD; Miriam de Nadal MD, PhD; Enrique Romero PhD; Alfredo Vellido PhD A prospective study was conducted at a critical care department and post‐anesthesia care unit of a university teaching hospital in Barcelona, Spain. The study recruited 707 patients with invasive BP and finger PPG waves over a period of 26 months. Exclusion criteria were presence of major arrhythmia, immediate death condition and disturbances in the arterial or PPG curve morphology. For each patient we automatically recorded the systolic blood pressure (SBP), mean arterial pressure (MAP), diastolic blood pressure (DBP) and PPG curve for 30 minutes. The PPG signal was further processed to obtain a set of features that were used to construct a Deep Belief Network with Gaussian Restricted Boltzmann Machine (DBN‐RBM). The available dataset was split into three subsets (Training, Validation and Testing). The training and validation datasets included 85% of data and the testing dataset included 15% of the available data. The regression error was assessed through a Bland‐Altman analysis and the AAMI standard. The mean prediction error were ‐2.98+‐19.35 mmHg for SBP, ‐3.38+‐10.35 mmHg for MAP and 3.65+‐8.69 mmHg for DBP.The results obtained are promising for the assessment of MAP and DBP with DBN‐RBM. Further research and clinical validation are needed to bring this technology to standard medical practice.Grant Funding Source: Supported by Shockomics (FP7 framework)