Gaussian-Bernoulli Restricted Boltzmann Research Articles

Quasi-free energy evaluation of Gaussian-Bernoulli restricted Boltzmann machine for anomaly detection

Quasi-free energy evaluation of Gaussian-Bernoulli restricted Boltzmann machine for anomaly detection

Effective sampling on Gaussian-Bernoulli restricted Boltzmann machines

Effective sampling on Gaussian-Bernoulli restricted Boltzmann machines

Learning State Transition Rules from High-Dimensional Time Series Data with Recurrent Temporal Gaussian-Bernoulli Restricted Boltzmann Machines

Understanding the dynamics of a system is crucial in various scientific and engineering domains. Machine learning techniques have been employed to learn state transition rules from observed time-series data. However, these data often contain sequences of noisy and ambiguous continuous variables, while we typically seek simplified dynamics rules that capture essential variables. In this work, we propose a method to extract a small number of essential hidden variables from high-dimensional time-series data and learn state transition rules between hidden variables. Our approach is based on the Restricted Boltzmann Machine (RBM), which models observable data in the visible layer and latent features in the hidden layer. However, real-world data, such as video and audio, consist of both discrete and continuous variables with temporal relationships. To address this, we introduce the Recurrent Temporal Gaussian-Bernoulli Restricted Boltzmann Machine (RTGB-RBM), which combines the Gaussian-Bernoulli Restricted Boltzmann Machine (GB-RBM) to handle continuous visible variables and the Recurrent Temporal Restricted Boltzmann Machine (RT-RBM) to capture time dependencies among discrete hidden variables. Additionally, we propose a rule-based method to extract essential information as hidden variables and represent state transition rules in an interpretable form. We evaluate our proposed method on the Bouncing Ball, Moving MNIST, and dSprite datasets. Experimental results demonstrate that our approach effectively learns the dynamics of these physical systems by extracting state transition rules between hidden variables. Moreover, our method can predict unobserved future states based on observed state transitions.

Monitoring and diagnosis of complex production process based on free energy of Gaussian–Bernoulli restricted Boltzmann machine

Monitoring and diagnosis of complex production process based on free energy of Gaussian–Bernoulli restricted Boltzmann machine

Channel Estimation for UAV Communication Systems Using Deep Neural Networks

Channel modeling of unmanned aerial vehicles (UAVs) from wireless communications has gained great interest for rapid deployment in wireless communication. The UAV channel has its own distinctive characteristics compared to satellite and cellular networks. Many proposed techniques consider and formulate the channel modeling of UAVs as a classification problem, where the key is to extract the discriminative features of the UAV wireless signal. For this issue, we propose a framework of multiple Gaussian–Bernoulli restricted Boltzmann machines (GBRBM) for dimension reduction and pre-training utilization incorporated with an autoencoder-based deep neural network. The developed system used UAV measurements of a town’s already existing commercial cellular network for training and validation. To evaluate the proposed approach, we run ray-tracing simulations in the program Remcom Wireless InSite at a distinct frequency of 28 GHz and used them for training and validation. The results demonstrate that the proposed method is accurate in channel acquisition for various UAV flying scenarios and outperforms the conventional DNNs.

Learning Gaussian-Bernoulli RBMs Using Difference of Convex Functions Optimization.

The Gaussian-Bernoulli restricted Boltzmann machine (GB-RBM) is a useful generative model that captures meaningful features from the given n -dimensional continuous data. The difficulties associated with learning GB-RBM are reported extensively in earlier studies. They indicate that the training of the GB-RBM using the current standard algorithms, namely contrastive divergence (CD) and persistent contrastive divergence (PCD), needs a carefully chosen small learning rate to avoid divergence which, in turn, results in slow learning. In this work, we alleviate such difficulties by showing that the negative log-likelihood for a GB-RBM can be expressed as a difference of convex functions if we keep the variance of the conditional distribution of visible units (given hidden unit states) and the biases of the visible units, constant. Using this, we propose a stochastic difference of convex (DC) functions programming (S-DCP) algorithm for learning the GB-RBM. We present extensive empirical studies on several benchmark data sets to validate the performance of this S-DCP algorithm. It is seen that S-DCP is better than the CD and PCD algorithms in terms of speed of learning and the quality of the generative model learned.

Deep learning of complex process data for fault classification based on sparse probabilistic dynamic network

BackgroundThe dynamic and nonlinear characteristics of process data have become the major issue in data-driven process monitoring. Traditional data-driven methods are often only able to extract a single feature in process data. Therefore, how to effectively extract multi-dimensional features has become the focus of current research. MethodsSparse probabilistic dynamic network (SPDN) is a deep learning model proposed in this paper for the purpose of fault classification. The method mainly takes the advantages of the sparse Gaussian-Bernoulli Restricted Boltzmann Machine (GRBM) and the recurrent neural network (RNN) with long-short term memory (LSTM) units. First, the sparse GRBM is used for nonlinear feature extraction in an unsupervised way. Then, LSTM is introduced to realize the modeling of sequence data which can effectively handle the dynamic feature of the data. FindingsIn the Tennessee-Eastman benchmark process, the classification accuracies of the proposed method are proved to be far superior to MLP, RNN and PDN. Meanwhile, in order to prove the influence of the data dynamics and the internal parameters of the structure on the fault classification results, two additional experiments were carried out.

DDoS Attacks Detection in the IoT Using Deep Gaussian-Bernoulli Restricted Boltzmann Machine

Distributed denial of service (DDoS) attack is generally known as one of the most significant threats to the internet of things (IoT). Current detection technologies of DDoS attacks are not adequate for IoT systems because of the peculiar features of IoT such as resource constraint nodes, specific network architecture, and specific network protocols. Providing adequate DDoS attacks detection systems to IoT, however, becomes a necessity since IoT is ubiquitous. This study hence developed a deep learning-based model for detecting DDoS in IoT, while considering its peculiarities. The proposed deep learning-based model was formulated using a deep Gaussian-Bernoulli restricted Boltzmann machine (DBM) because of its capability to learn high-level features from input following the unsupervised approach and its ability to manage real-time data that is common in the IoT network. Furthermore, the SoftMax regression was used for classification. The accuracy of the proposed model on the network socket layer-knowledge discovery in databases was obtained as 93.52%. The outcome of the study shows that the proposed DBM can efficiently detect DDoS attacks in IoT.

An improved deep belief network based hybrid forecasting method for wind power

The stochastic nature of wind speed hinders the forecasting of wind power generation. To improve the accuracy of wind power forecasting and effectively utilize the capability of principal component analysis (PCA) to process high-dimensional data, and take the advantages of deep belief network (DBN) to process complex data and spatial correlation (SC) in considering geographical position and terrain, a hybrid forecasting method using numerical weather prediction (NWP) is presented in this paper. First, an improved DBN is proposed by introducing Gaussian-Bernoulli restricted Boltzmann machine, and an adaptive learning step technique is applied to improve the convergence speed. Furthermore, the principal components are extracted from high-dimension original data by PCA, which are further input to the improved DBN. Then, a wind speed correction model is established to address the inaccuracy of NWP. Moreover, the output of the target site is forecasted using the data of its neighboring observation sites. Finally, the advantages of the above methods are combined, and the sliding window strategy is utilized to adaptively update the training data. The simulation results verify the effectiveness of the proposed improved DBN and the hybrid method compared to the traditional DBN, the corresponding average increase of forecasting accuracy is 15.8975% and 29.3725%, respectively.

Deep Learning-Based Trajectory Tracking Control forUnmanned Surface Vehicle

Trajectory tracking control based on waypoint behavior is a promising way for unmanned surface vehicle (USV) to achieve autonomous navigation. This study is aimed at the guidance progress in the kinematics; the artificial intelligence method of deep learning is adopted to improve the trajectory tracking level of USV. First, two deep neural network (DNN) models are constructed to evaluate navigation effects and to estimate guidance law parameters in real time, respectively. We then pretrain the DNN using a Gaussian–Bernoulli restricted Boltzmann machine to further improve the accuracy of predicting navigation effect. Finally, two DNNs are connected in parallel with the control loop of USV to provide predictive supervision and auxiliary decision making for traditional control methods. This kind of parallel way conforms to the ship manipulation of habit. Furthermore, we develop a new application on the basis of Mission Oriented Operating Suite Interval Programming named “pDeepLearning.” It can predict the navigation effect online by DNN and adjust the guidance law parameters according to the effect level. The experimental results show that, compared with the original waypoint behavior of USV, the prediction model proposed in this study reduces the trajectory tracking error by 19.0% and increases the waypoint behavior effect level.

A Drift-Compensating Novel Deep Belief Classification Network to Improve Gas Recognition of Electronic Noses

Electronic nose (E-nose) systems have a good effect on the identification of distinct odours. However, the properties of chemical gas sensors indicate that ageing, poisoning, fluctuation of environmental conditions (moisture, temperature, etc.) and a lack of fabrication repeatability, etc. have a large impact on the sensitivity and accuracy of sensors, which leads to sensor data drift. Although previous studies have indicated the feasibility and validity of deep learning in drift compensation of gas sensor data, the actual performances of these deep learning models are less impressive compared with some existing methods. Thus, we intend to further explore a novel deep learning model for drift compensation for E-noses. In this paper, we investigate the drift compensation effect of E-nose data based on a deep belief network (DBN) and constructed a Gaussian deep belief classification network (GDBCN) model by cascading a Gaussian-Bernoulli restricted Boltzmann machines based DBN with a softmax classifier layer to compensate for sensor drift at the decision level. The merits of our method are as follows: 1) it is a unified classification model for drift auto-compensation at the decision level rather than a feature extractor; 2) it couples unsupervised and supervised techniques by modelling the intrinsic distribution of the data from different domains in an unsupervised manner and fine-tunes the model parameters by leveraging the label information of the source domain; 3) the supervised fine-tuning process for the coupled GDBCN model fits well with the nature of the supervised task and guarantees that the parameters of the DBN will be useful for classification; 4) the GDBCN model is a classification model and thus automatically compensates for drift without manually setting specific model rules for domain alignment before classification. Experimental results on real sensor datasets demonstrate the effectiveness and superiority compared with several existing control methods.

Bus travel time prediction based on deep belief network with back-propagation

In an intelligent transportation system, accurate bus information is vital for passengers to schedule their departure time and make reasonable route choice. In this paper, an improved deep belief network (DBN) is proposed to predict the bus travel time. By using Gaussian–Bernoulli restricted Boltzmann machines to construct a DBN, we update the classical DBN to model continuous data. In addition, a back-propagation (BP) neural network is further applied to improve the performance. Based on the real traffic data collected in Shenyang, China, several experiments are conducted to validate the technique. Comparison with typical forecasting methods such as k-nearest neighbor algorithm (k-NN), artificial neural network (ANN), support vector machine (SVM) and random forests (RFs) shows that the proposed method is applicable to the prediction of bus travel time and works better than traditional methods.

Real-Time Fault Detection for IIoT Facilities Using GBRBM-Based DNN

Fault detection is a fundamental requirement for Industrial Internet of Things (IIoT), such as the process industry. This article first reviews the recent studies focusing on applying the fault detection techniques to the IIoT networks. However, we find that numerous studies focus on the resource utilization and workload allocation. The fault detection toward IIoT facilities is still in its immature stage because the existing approaches are not accurate enough for the stringent fault detection in IIoT networks. To this end, we present a novel algorithm, named Gaussian Bernoulli restricted Boltzmann machines (GBRBMs)-based deep neural network (DNN), to transform the fault detection into a classification problem. The real trace-driven experiments show that the proposed scheme outperforms other baseline machine learning methods. We anticipate that this article can inspire blooming studies on the related topics of smart IIoT networks.

Improved Gaussian–Bernoulli restricted Boltzmann machine for learning discriminative representations

Restricted Boltzmann machines (RBMs) have received considerable research interest in recent years because of their capability to discover latent representations in an unsupervised manner. The standard RBM is only suitable for processing binary-valued data. To address this limitation, the Gaussian–Bernoulli RBM (GRBM) has been designed to model real-valued data, particularly images. A GRBM that seeks to map real-valued data nonlinearly into a latent representation space is typically trained by contrastive divergence learning. However, most existing GRBM-based models neglect the inherent interpoint affinity information from the original data that can be used to enhance the expression ability of the model. In this study, a novel interpoint-affinity-based GRBM (abGRBM) is proposed to learn discriminative representations in the hidden layer. By incorporating the interpoint affinity information into the training process of the GRBM, the proposed model can not only utilize the GRBM’s powerful latent representation learning capabilities for real-valued data, it can also transform the original data into another space with improved separability. We prove the availability of our model using several image datasets of Microsoft Research Asia Multimedia for unsupervised clustering and supervised classification tasks. The experimental results show the superior performance of the GRBM in discovering discriminative representations and demonstrate the effectiveness of the affinity information.

Development and evaluation of a deep learning model for real-time ground vehicle semantic segmentation from UAV-based thermal infrared imagery

Real-time unmanned aerial vehicles (UAVs)-based thermal infrared images processing, due to high spatial resolution and knowledge of the various infrared radiant energy level distribution of solid bodies, has important applications such as monitoring and control of the various phenomena in different natural situations. One of these applications is monitoring the ground vehicles in cities by using detection or semantic segmentation of them in the thermal images. In this research, our purpose is to improve the performance of deep learning combined model by using Gaussian-Bernoulli Restricted Boltzmann Machine (GB-RBM) specifications for the segmentation of the ground vehicles from UAV-based thermal infrared imagery. The proposed model is studied in three steps. First, designing the proposed model by using an encoder-decoder structure and addition of extracted features from convolutional layers and restricted Boltzmann machine in the network. Second, the implementation of the research goals on four sets of UAV-based thermal infrared imagery named NPU_CS_UAV_IR_DATA that was collected from some streets of China by using FLIR TAU2 thermal infrared sensor in 2017. Finally, analyzing the performance of the proposed model by using five state-of-the-art models in semantic segmentation. The results evaluated the performance of the proposed model as a robust model with the average precision and average processing time of approximately 0.97, and 19.73 s for all datasets, respectively.

Probabilistic Sequential Network for Deep Learning of Complex Process Data and Soft Sensor Application

Soft sensing of quality/key variables is critical to the control and optimization of industrial processes. One of the main drawbacks of data-driven soft sensors is to deal with the dynamic and nonlinear characteristics of process data. This paper proposes a deep learning structure and corresponding training algorithm for the purpose of soft sensor, which is called probabilistic sequential network. The proposed model merges unsupervised feature extraction and supervised dynamic modeling approaches to improve the prediction performance. It is mainly based on the Gaussian-Bernoulli restricted Boltzmann machine and the recurrent neural network structure. To avoid the overfitting problem in the training procedure of deep learning algorithms, the L2 regularization and dropout technique are adopted. The new method can not only deeply extract the nonlinear feature but also widely capture dynamic characteristic of process data. Effectiveness and superiority of the new method are validated through an actual CO2 absorption column, compared to traditional methods.

A Parallel Gaussian–Bernoulli Restricted Boltzmann Machine for Mining Area Classification With Hyperspectral Imagery

In this paper, a novel feature extraction method is proposed for hyperspectral image classification using a Gaussian–Bernoulli restricted Boltzmann machine (GBRBM) in parallel. The proposed approach employs several GBRBMs with different hidden layers to extract deep features from hyperspectral images, which are nonlinear and local invariant. Based on the learned deep features, a logistic regression layer is trained for classification. The proposed approaches are carried out on two public hyperspectral datasets: Pavia University dataset and Salinas dataset, and a new dataset obtained by HySpex imaging spectrometer in the mining area in Xuzhou. The obtained results reveal that the proposed approach offers superior performance compared to traditional classifiers. The advantage of the proposed GBRBM is that it can extract deep features in an unsupervised way and reduce the prediction time by using GPU. In particular, the classification results of the mining area provide valuable suggestions to improve environmental protection.

Unsupervised Neural Network for Modulation Format Discrimination and Identification

We propose a new method to discriminate and identify the modulation format of signals based on an unsupervised neural network named convolutional Gaussian-Bernoulli restricted Boltzmann machine (CGBRBM). Tests are performed to demonstrate how the proposed method works and to evaluate the discrimination/identification accuracy for different input combinations. Signals of five modulation formats are used to test the CGBRBM-based algorithm including QPSK, 8QAM, 16QAM, 32QAM, and 64QAM. The results indicate the performance of the proposed method when dealing with various application scenarios and reveal the relation between discrimination accuracy and identification accuracy.

Background subtraction using Gaussian–Bernoulli restricted Boltzmann machine

The background subtraction is an important technique in computer vision which segments moving objects into video sequences by comparing each new frame with a learned background model. In this work, the authors propose a novel background subtraction method based on Gaussian-Bernoulli restricted Boltzmann machines (GRBMs). The GRBM is different from the ordinary restricted Boltzmann machine (RBM) by using real numbers as inputs, resulting in a constrained mixture of Gaussians, which is one of the most widely used techniques to solve the background subtraction problem. The GRBM makes it easy to learn the variance of pixel values and takes the advantage of the generative model paradigm of the RBM. They present a simple technique to reconstruct the learned background model from a given input frame and to extract the foreground from the background using the variance learned for each pixel. Furthermore, they demonstrate the effectiveness of the proposed technique with extensive experimentation and quantitative evaluation on several commonly used public data sets for background subtraction.

An improved advertising CTR prediction approach based on the fuzzy deep neural network.

Combining a deep neural network with fuzzy theory, this paper proposes an advertising click-through rate (CTR) prediction approach based on a fuzzy deep neural network (FDNN). In this approach, fuzzy Gaussian-Bernoulli restricted Boltzmann machine (FGBRBM) is first applied to input raw data from advertising datasets. Next, fuzzy restricted Boltzmann machine (FRBM) is used to construct the fuzzy deep belief network (FDBN) with the unsupervised method layer by layer. Finally, fuzzy logistic regression (FLR) is utilized for modeling the CTR. The experimental results show that the proposed FDNN model outperforms several baseline models in terms of both data representation capability and robustness in advertising click log datasets with noise.