Gaussian-Bernoulli Restricted Boltzmann Machine Research Articles

Effective sampling on Gaussian-Bernoulli restricted Boltzmann machines

Effective sampling on Gaussian-Bernoulli restricted Boltzmann machines

Quasi-free energy evaluation of Gaussian-Bernoulli restricted Boltzmann machine for anomaly detection

Quasi-free energy evaluation of Gaussian-Bernoulli restricted Boltzmann machine for anomaly detection

Attention in a Family of Boltzmann Machines Emerging From Modern Hopfield Networks.

Hopfield networks and Boltzmann machines (BMs) are fundamental energy-based neural network models. Recent studies on modern Hopfield networks have broadened the class of energy functions and led to a unified perspective on general Hopfield networks, including an attention module. In this letter, we consider the BM counterparts of modern Hopfield networks using the associated energy functions and study their salient properties from a trainability perspective. In particular, the energy function corresponding to the attention module naturally introduces a novel BM, which we refer to as the attentional BM (AttnBM). We verify that AttnBM has a tractable likelihood function and gradient for certain special cases and is easy to train. Moreover, we reveal the hidden connections between AttnBM and some single-layer models, namely the gaussian-Bernoulli restricted BM and the denoising autoencoder with softmax units coming from denoising score matching. We also investigate BMs introduced by other energy functions and show that the energy function of dense associative memory models gives BMs belonging to exponential family harmoniums.

Learning State Transition Rules from High-Dimensional Time Series Data with Recurrent Temporal Gaussian-Bernoulli Restricted Boltzmann Machines

Understanding the dynamics of a system is crucial in various scientific and engineering domains. Machine learning techniques have been employed to learn state transition rules from observed time-series data. However, these data often contain sequences of noisy and ambiguous continuous variables, while we typically seek simplified dynamics rules that capture essential variables. In this work, we propose a method to extract a small number of essential hidden variables from high-dimensional time-series data and learn state transition rules between hidden variables. Our approach is based on the Restricted Boltzmann Machine (RBM), which models observable data in the visible layer and latent features in the hidden layer. However, real-world data, such as video and audio, consist of both discrete and continuous variables with temporal relationships. To address this, we introduce the Recurrent Temporal Gaussian-Bernoulli Restricted Boltzmann Machine (RTGB-RBM), which combines the Gaussian-Bernoulli Restricted Boltzmann Machine (GB-RBM) to handle continuous visible variables and the Recurrent Temporal Restricted Boltzmann Machine (RT-RBM) to capture time dependencies among discrete hidden variables. Additionally, we propose a rule-based method to extract essential information as hidden variables and represent state transition rules in an interpretable form. We evaluate our proposed method on the Bouncing Ball, Moving MNIST, and dSprite datasets. Experimental results demonstrate that our approach effectively learns the dynamics of these physical systems by extracting state transition rules between hidden variables. Moreover, our method can predict unobserved future states based on observed state transitions.

Monitoring and diagnosis of complex production process based on free energy of Gaussian–Bernoulli restricted Boltzmann machine

Monitoring and diagnosis of complex production process based on free energy of Gaussian–Bernoulli restricted Boltzmann machine

Channel Estimation for UAV Communication Systems Using Deep Neural Networks

Channel modeling of unmanned aerial vehicles (UAVs) from wireless communications has gained great interest for rapid deployment in wireless communication. The UAV channel has its own distinctive characteristics compared to satellite and cellular networks. Many proposed techniques consider and formulate the channel modeling of UAVs as a classification problem, where the key is to extract the discriminative features of the UAV wireless signal. For this issue, we propose a framework of multiple Gaussian–Bernoulli restricted Boltzmann machines (GBRBM) for dimension reduction and pre-training utilization incorporated with an autoencoder-based deep neural network. The developed system used UAV measurements of a town’s already existing commercial cellular network for training and validation. To evaluate the proposed approach, we run ray-tracing simulations in the program Remcom Wireless InSite at a distinct frequency of 28 GHz and used them for training and validation. The results demonstrate that the proposed method is accurate in channel acquisition for various UAV flying scenarios and outperforms the conventional DNNs.

Learning Gaussian-Bernoulli RBMs Using Difference of Convex Functions Optimization.

The Gaussian-Bernoulli restricted Boltzmann machine (GB-RBM) is a useful generative model that captures meaningful features from the given n -dimensional continuous data. The difficulties associated with learning GB-RBM are reported extensively in earlier studies. They indicate that the training of the GB-RBM using the current standard algorithms, namely contrastive divergence (CD) and persistent contrastive divergence (PCD), needs a carefully chosen small learning rate to avoid divergence which, in turn, results in slow learning. In this work, we alleviate such difficulties by showing that the negative log-likelihood for a GB-RBM can be expressed as a difference of convex functions if we keep the variance of the conditional distribution of visible units (given hidden unit states) and the biases of the visible units, constant. Using this, we propose a stochastic difference of convex (DC) functions programming (S-DCP) algorithm for learning the GB-RBM. We present extensive empirical studies on several benchmark data sets to validate the performance of this S-DCP algorithm. It is seen that S-DCP is better than the CD and PCD algorithms in terms of speed of learning and the quality of the generative model learned.

Deep learning of complex process data for fault classification based on sparse probabilistic dynamic network

BackgroundThe dynamic and nonlinear characteristics of process data have become the major issue in data-driven process monitoring. Traditional data-driven methods are often only able to extract a single feature in process data. Therefore, how to effectively extract multi-dimensional features has become the focus of current research. MethodsSparse probabilistic dynamic network (SPDN) is a deep learning model proposed in this paper for the purpose of fault classification. The method mainly takes the advantages of the sparse Gaussian-Bernoulli Restricted Boltzmann Machine (GRBM) and the recurrent neural network (RNN) with long-short term memory (LSTM) units. First, the sparse GRBM is used for nonlinear feature extraction in an unsupervised way. Then, LSTM is introduced to realize the modeling of sequence data which can effectively handle the dynamic feature of the data. FindingsIn the Tennessee-Eastman benchmark process, the classification accuracies of the proposed method are proved to be far superior to MLP, RNN and PDN. Meanwhile, in order to prove the influence of the data dynamics and the internal parameters of the structure on the fault classification results, two additional experiments were carried out.

DDoS Attacks Detection in the IoT Using Deep Gaussian-Bernoulli Restricted Boltzmann Machine

Distributed denial of service (DDoS) attack is generally known as one of the most significant threats to the internet of things (IoT). Current detection technologies of DDoS attacks are not adequate for IoT systems because of the peculiar features of IoT such as resource constraint nodes, specific network architecture, and specific network protocols. Providing adequate DDoS attacks detection systems to IoT, however, becomes a necessity since IoT is ubiquitous. This study hence developed a deep learning-based model for detecting DDoS in IoT, while considering its peculiarities. The proposed deep learning-based model was formulated using a deep Gaussian-Bernoulli restricted Boltzmann machine (DBM) because of its capability to learn high-level features from input following the unsupervised approach and its ability to manage real-time data that is common in the IoT network. Furthermore, the SoftMax regression was used for classification. The accuracy of the proposed model on the network socket layer-knowledge discovery in databases was obtained as 93.52%. The outcome of the study shows that the proposed DBM can efficiently detect DDoS attacks in IoT.

Bluetooth Indoor Localization With Gaussian–Bernoulli Restricted Boltzmann Machine Plus Liquid State Machine

The location information of sensors and devices plays an important role in Internet of Things (IoT). Low cost and energy efficient Bluetooth Low Energy (BLE)-based localization solutions are ideal for extensive use in IoT applications. The main challenge is to combat the signal fluctuation in non-line-of-sight (NLOS) propagation. Most Machine Learning (ML) algorithms available in the literature to address this issue belong to classification and batch learning. This article proposes a novel online learning algorithm for BLE localization based on Gaussian–Bernoulli Restricted Boltzmann Machine (GBRBM) plus Liquid State Machine (LSM), which learns the training data one-by-one. Unsupervised GBRBM is able to extract sound patterns of fluctuated RSS inputs, and LSM manages to map the patterns to real-time position estimation. Extensive experimental results demonstrate the superiority of the proposed method over other state-of-the-art batch learning methods in terms of localization accuracy and complexity.

An improved deep belief network based hybrid forecasting method for wind power

The stochastic nature of wind speed hinders the forecasting of wind power generation. To improve the accuracy of wind power forecasting and effectively utilize the capability of principal component analysis (PCA) to process high-dimensional data, and take the advantages of deep belief network (DBN) to process complex data and spatial correlation (SC) in considering geographical position and terrain, a hybrid forecasting method using numerical weather prediction (NWP) is presented in this paper. First, an improved DBN is proposed by introducing Gaussian-Bernoulli restricted Boltzmann machine, and an adaptive learning step technique is applied to improve the convergence speed. Furthermore, the principal components are extracted from high-dimension original data by PCA, which are further input to the improved DBN. Then, a wind speed correction model is established to address the inaccuracy of NWP. Moreover, the output of the target site is forecasted using the data of its neighboring observation sites. Finally, the advantages of the above methods are combined, and the sliding window strategy is utilized to adaptively update the training data. The simulation results verify the effectiveness of the proposed improved DBN and the hybrid method compared to the traditional DBN, the corresponding average increase of forecasting accuracy is 15.8975% and 29.3725%, respectively.

Deep Learning-Based Trajectory Tracking Control forUnmanned Surface Vehicle

Trajectory tracking control based on waypoint behavior is a promising way for unmanned surface vehicle (USV) to achieve autonomous navigation. This study is aimed at the guidance progress in the kinematics; the artificial intelligence method of deep learning is adopted to improve the trajectory tracking level of USV. First, two deep neural network (DNN) models are constructed to evaluate navigation effects and to estimate guidance law parameters in real time, respectively. We then pretrain the DNN using a Gaussian–Bernoulli restricted Boltzmann machine to further improve the accuracy of predicting navigation effect. Finally, two DNNs are connected in parallel with the control loop of USV to provide predictive supervision and auxiliary decision making for traditional control methods. This kind of parallel way conforms to the ship manipulation of habit. Furthermore, we develop a new application on the basis of Mission Oriented Operating Suite Interval Programming named “pDeepLearning.” It can predict the navigation effect online by DNN and adjust the guidance law parameters according to the effect level. The experimental results show that, compared with the original waypoint behavior of USV, the prediction model proposed in this study reduces the trajectory tracking error by 19.0% and increases the waypoint behavior effect level.

Gamma Boltzmann Machine for Audio Modeling

This paper presents an energy-based probabilistic model that handles nonnegative data in consideration of both linear and logarithmic scales. In audio applications, magnitude of time-frequency representation, including spectrogram, is regarded as one of the most important features. Such magnitude-based features have been extensively utilized in learning-based audio processing. Since a logarithmic scale is important in terms of auditory perception, the features are usually computed with a logarithmic function. That is, a logarithmic function is applied within the computation of features so that a learning machine does not have to explicitly model the logarithmic scale. We think in a different way and propose a restricted Boltzmann machine (RBM) that simultaneously models linear- and log-magnitude spectra. RBM is a stochastic neural network that can discover data representations without supervision. To manage both linear and logarithmic scales, we define an energy function based on both scales. This energy function results in a conditional distribution (of the observable data, given hidden units) that is written as the gamma distribution, and hence the proposed RBM is termed gamma-Bernoulli RBM. The proposed gamma-Bernoulli RBM was compared to the ordinary Gaussian-Bernoulli RBM by speech representation experiments. Both objective and subjective evaluations illustrated the advantage of the proposed model.

A deep fusion model based on restricted Boltzmann machines for traffic accident duration prediction

Traffic accidents causing nonrecurrent congestion can decrease the capacity of highways and increase car emissions. Some models in previous studies have been built based on artificial intelligence or statistical theory because estimating the duration of an accident can aid traffic operation and management. However, only characteristics of traffic accidents were considered in most models; the spatial–temporal correlations of traffic flow were always ignored. In this study, a deep fusion model, which can simultaneously handle categorical and continuous variables, is proposed. The model considers not only the characteristics of traffic accidents but also the spatial–temporal correlations in traffic flow. In this model, a stacked restricted Boltzmann machine (RBM) is used to handle the categorical variables, a stacked Gaussian-Bernoulli RBM is used to handle the continuous variables, and a joint layer is used to fuse the extracted features. With extracted I-80 data, the performance of the proposed model was evaluated and compared to some benchmark models. Furthermore, the target variable (duration) was divided into ten groups, and then the evaluation criteria of the models of each group were calculated. The results show that the novel model outperforms some previous models and that the fusion of different types of variables can improve prediction accuracy. In conclusion, the proposed model can fully mine nonlinear and complex patterns in traffic accident data and traffic flow data. The fusion of features is important to predict traffic accident durations.

Coupled application of deep learning model and quantile regression for travel time and its interval estimation using data in different dimensions

The rapid development of sensing and computing methods and their application to transportation engineering in recent years provide us data support to traffic flow prediction. However, the travel time prediction is still a complex and difficult task in the intelligent transportation system because of its nonlinear and nonstationary characteristics. In this study, a hybrid model coupling the deep learning model and the quantile regression (QR) has been proposed to achieve the deterministic and probabilistic travel time prediction. To consider multiple correlations of the traffic flow, a spatial–temporal state-space matrix has been developed. Then, a novel deep belief network stacked by several Gaussian Bernoulli Restricted Boltzmann Machine (GBRBM) to extract important features and a regression layer to finish the prediction were developed. Moreover, to strengthen the reliability of results, the QR was applied to generate a prediction interval. Using real-world data sets, the proposed hybrid model was evaluated and contrasted with several benchmark models. The results show the deep learning model outperform the shallow learning model. The prediction interval providing by QR is better than that provided by the traditional method. It indicates that our proposed hybrid model can obtain a more perfect and reliable prediction for travel time which is meaningful to the advanced traveler information system.

Learning time-frequency mask for noisy speech enhancement using gaussian-bernoulli pre-trained deep neural networks

Speech enhancement is a very important problem in various speech processing applications. Recently, supervised speech enhancement using deep learning approaches to estimate a time-frequency mask have proved remarkable performance gain. In this paper, we have proposed time-frequency masking-based supervised speech enhancement method for improving intelligibility and quality of the noisy speech. We believe that a large performance gain can be achieved if deep neural networks (DNNs) are layer-wise pre-trained by stacking Gaussian-Bernoulli Restricted Boltzmann Machine (GB-RBM). The proposed DNN is called as Gaussian-Bernoulli Deep Belief Network (GB-DBN) and are optimized by minimizing errors between the estimated and pre-defined masks. Non-linear Mel-Scale weighted mean square error (LMW-MSE) loss function is used as training criterion. We have examined the performance of the proposed pre-training scheme using different DNNs which are established on three time-frequency masks comprised of the ideal amplitude mask (IAM), ideal ratio mask (IRM), and phase sensitive mask (PSM). The results in different noisy conditions demonstrated that when DNNs are pre-trained by the proposed scheme provided a persistent performance gain in terms of the perceived speech intelligibility and quality. Also, the proposed pre-training scheme is effective and robust in noisy training data.

A Drift-Compensating Novel Deep Belief Classification Network to Improve Gas Recognition of Electronic Noses

Electronic nose (E-nose) systems have a good effect on the identification of distinct odours. However, the properties of chemical gas sensors indicate that ageing, poisoning, fluctuation of environmental conditions (moisture, temperature, etc.) and a lack of fabrication repeatability, etc. have a large impact on the sensitivity and accuracy of sensors, which leads to sensor data drift. Although previous studies have indicated the feasibility and validity of deep learning in drift compensation of gas sensor data, the actual performances of these deep learning models are less impressive compared with some existing methods. Thus, we intend to further explore a novel deep learning model for drift compensation for E-noses. In this paper, we investigate the drift compensation effect of E-nose data based on a deep belief network (DBN) and constructed a Gaussian deep belief classification network (GDBCN) model by cascading a Gaussian-Bernoulli restricted Boltzmann machines based DBN with a softmax classifier layer to compensate for sensor drift at the decision level. The merits of our method are as follows: 1) it is a unified classification model for drift auto-compensation at the decision level rather than a feature extractor; 2) it couples unsupervised and supervised techniques by modelling the intrinsic distribution of the data from different domains in an unsupervised manner and fine-tunes the model parameters by leveraging the label information of the source domain; 3) the supervised fine-tuning process for the coupled GDBCN model fits well with the nature of the supervised task and guarantees that the parameters of the DBN will be useful for classification; 4) the GDBCN model is a classification model and thus automatically compensates for drift without manually setting specific model rules for domain alignment before classification. Experimental results on real sensor datasets demonstrate the effectiveness and superiority compared with several existing control methods.

Bus travel time prediction based on deep belief network with back-propagation

In an intelligent transportation system, accurate bus information is vital for passengers to schedule their departure time and make reasonable route choice. In this paper, an improved deep belief network (DBN) is proposed to predict the bus travel time. By using Gaussian–Bernoulli restricted Boltzmann machines to construct a DBN, we update the classical DBN to model continuous data. In addition, a back-propagation (BP) neural network is further applied to improve the performance. Based on the real traffic data collected in Shenyang, China, several experiments are conducted to validate the technique. Comparison with typical forecasting methods such as k-nearest neighbor algorithm (k-NN), artificial neural network (ANN), support vector machine (SVM) and random forests (RFs) shows that the proposed method is applicable to the prediction of bus travel time and works better than traditional methods.

Real-Time Fault Detection for IIoT Facilities Using GBRBM-Based DNN

Fault detection is a fundamental requirement for Industrial Internet of Things (IIoT), such as the process industry. This article first reviews the recent studies focusing on applying the fault detection techniques to the IIoT networks. However, we find that numerous studies focus on the resource utilization and workload allocation. The fault detection toward IIoT facilities is still in its immature stage because the existing approaches are not accurate enough for the stringent fault detection in IIoT networks. To this end, we present a novel algorithm, named Gaussian Bernoulli restricted Boltzmann machines (GBRBMs)-based deep neural network (DNN), to transform the fault detection into a classification problem. The real trace-driven experiments show that the proposed scheme outperforms other baseline machine learning methods. We anticipate that this article can inspire blooming studies on the related topics of smart IIoT networks.

Improved Gaussian–Bernoulli restricted Boltzmann machine for learning discriminative representations

Restricted Boltzmann machines (RBMs) have received considerable research interest in recent years because of their capability to discover latent representations in an unsupervised manner. The standard RBM is only suitable for processing binary-valued data. To address this limitation, the Gaussian–Bernoulli RBM (GRBM) has been designed to model real-valued data, particularly images. A GRBM that seeks to map real-valued data nonlinearly into a latent representation space is typically trained by contrastive divergence learning. However, most existing GRBM-based models neglect the inherent interpoint affinity information from the original data that can be used to enhance the expression ability of the model. In this study, a novel interpoint-affinity-based GRBM (abGRBM) is proposed to learn discriminative representations in the hidden layer. By incorporating the interpoint affinity information into the training process of the GRBM, the proposed model can not only utilize the GRBM’s powerful latent representation learning capabilities for real-valued data, it can also transform the original data into another space with improved separability. We prove the availability of our model using several image datasets of Microsoft Research Asia Multimedia for unsupervised clustering and supervised classification tasks. The experimental results show the superior performance of the GRBM in discovering discriminative representations and demonstrate the effectiveness of the affinity information.