Deep Restricted Boltzmann Machine Research Articles

An efficient attention-based hybridized deep learning network with deep RBM features for customer behavior prediction in digital marketing

PurposeAn efficient customer behavior prediction model is designed using deep learning techniques. The necessary data used for the implementation are taken from standard datasets and presented to perform subsequent tasks. Here, deep restricted Boltzmann machines (RBM) features are retrieved from the input images. Further, the extracted deep RBM features are presented to the customer behavior prediction phase. Here, the attention-based hybrid deep learning (A-HDL) technique is designed based on the incorporation of a dilated deep temporal convolutional network (dilated-DTCN) and a weighted recurrent neural network (weighted RNN). Moreover, the weights in RNN are tuned using a modernized random parameter-based cheetah optimizer (MRPCO). Further, various experiments were performed on the implemented framework, and it secured an enhanced customer behavior prediction rate than the conventional models.Design/methodology/approachA novel hybrid deep network-based customer behavior prediction model was developed to predict the behavior of the customer so the companies yield more income by advertising their products based on the predicted results.FindingsWhen considering the first dataset, the designed customer behavior prediction mechanism produced 94% accuracy, which is higher than the conventional techniques such as long short-term memory (LSTM), DTCN, RNN and A-HDL with 88%, 87%, 89% and 93%.Originality/valueThe precision and the accuracy of the developed MRPCO-A-HDL-based customer behavior prediction model progressed than the conventional techniques and algorithms.

DRBM-ClustNet: A Deep Restricted Boltzmann-Kohonen Architecture for Data Clustering.

A Bayesian deep restricted Boltzmann-Kohonen architecture for data clustering termed deep restricted Boltzmann machine (DRBM)-ClustNet is proposed. This core-clustering engine consists of a DRBM for processing unlabeled data by creating new features that are uncorrelated and have large variance with each other. Next, the number of clusters is predicted using the Bayesian information criterion (BIC), followed by a Kohonen network (KN)-based clustering layer. The processing of unlabeled data is done in three stages for efficient clustering of the nonlinearly separable datasets. In the first stage, DRBM performs nonlinear feature extraction by capturing the highly complex data representation by projecting the feature vectors of d dimensions into n dimensions. Most clustering algorithms require the number of clusters to be decided a priori; hence, here, to automate the number of clusters in the second stage, we use BIC. In the third stage, the number of clusters derived from BIC forms the input for the KN, which performs clustering of the feature-extracted data obtained from the DRBM. This method overcomes the general disadvantages of clustering algorithms, such as the prior specification of the number of clusters, convergence to local optima, and poor clustering accuracy on nonlinear datasets. In this research, we use two synthetic datasets, 15 benchmark datasets from the UCI Machine Learning repository, and four image datasets to analyze the DRBM-ClustNet. The proposed framework is evaluated based on clustering accuracy and ranked against other state-of-the-art clustering methods. The obtained results demonstrate that the DRBM-ClustNet outperforms state-of-the-art clustering algorithms.

Deep Learning to Hash with Application to Cross-View Nearest Neighbor Search

Learning hash functions for approximate nearest neighbor search of high-dimensional data has received a surge of interests in recent years. Most existing methods are often concerned with learning hash functions for nearest neighbor search on high-dimensional data from a single source. In many real-world applications, data can be collected from diverse sources or represented using different feature descriptors. This raises an open challenge, i.e., the Cross-View Nearest Neighbor Search (CVNNS), where the representation of a query instance can be different from that of target instances to be retrieved in database. The key challenge of cross-view search is to learn an effective shared representation which can effectively connect the query instance and the target instances to be retrieved. In this paper, we present a new cross-view nearest neighbor search scheme by applying the emerging deep learning to hash techniques. In particular, we investigate two different architectures of deep Restricted Boltzmann Machines (RBMs) for learning to hash toward cross-view nearest neighbor search, and conduct extensive experiments to examine their empirical performance on diverse settings of cross-view image retrieval tasks. The encouraging results show that our technique outperforms the state-of-the-art approaches.

Malware Attack Detection in Large Scale Networks using the Ensemble Deep Restricted Boltzmann Machine

Today, cyber attackers use Artificial Intelligence (AI) to boost the sophistication and scope of their attacks. On the defense side, AI is used to improve defense plans, robustness, flexibility, and efficiency of defense systems by adapting to environmental changes. With the developments in information and communication technologies, various exploits that are changing rapidly constitute a danger sign for cyber security. Cybercriminals use new and sophisticated tactics to boost their attack speed and size. Consequently, there is a need for more flexible, adaptable, and strong cyber defense systems that can identify a wide range of threats in real time. In recent years, the adoption of AI approaches has increased and maintained a vital role in the detection and prevention of cyber threats. This paper presents an Ensemble Deep Restricted Boltzmann Machine (EDRBM) to classify cybersecurity threats in large-scale network environments. EDRBM acts as a classification model that enables the classification of malicious flowsets in a large-scale network. Simulations were carried out to evaluate the efficacy of the proposed EDRBM model under various malware attacks. The results showed that the proposed method achieved a promising malware classification rate in malicious flowsets.

Minimax formula for the replica symmetric free energy of deep restricted Boltzmann machines

Minimax formula for the replica symmetric free energy of deep restricted Boltzmann machines

Gaussian Mixture with Max Expectation Guide for Stacked Architecture of Denoising Autoencoder and DRBM for Medical Chest Scans and Disease Identification

Artificial intelligence (AI), in particular deep learning, has proven to be efficient in medical diagnosis. This paper introduces a new hybrid deep learning model for pneumonia diagnosis based on chest CT scans. At the core of the model, a Gaussian mixture is combined with the expectation-maximization algorithm (EMGMM) to extract the regions of interest (ROI), while a convolutional denoising autoencoder (DAE) and deep restricted Boltzmann machine (DRBM) are combined for the classification. In order to prevent the model from learning trivial solutions, stochastic noises were added as an input to the unsupervised learning phase. The dataset used in this work is a publicly available dataset of chest X-rays for pneumonia on the Kaggle website; it contains 5856 images with 1583 normal cases and 4273 pneumonia cases, with an imbalance ratio (IR) of 0.46. Several operations including zooming, flipping, shifting and rotation were used in the augmentation phase to balance the data distribution across the different classes, which led to enhancing the IR value to 0.028. The computational analysis of the results show that the proposed model is promising as it provides an average accuracy value of 98.63%, sensitivity value of 96.5%, and specificity value of 94.8%.

Air-CSL: Chinese Sign Language Recognition Based on the Commercial WiFi Devices

Artificial intelligence and Internet of Things (IoT) devices are experiencing explosive growth. Currently, the commonly used gesture recognition methods are difficult to deploy and expensive, so this paper uses the Channel State Information (CSI) for Chinese sign language recognition. Aiming at the problems of current gesture recognition methods, such as strong personnel dependence, high computational resource consumption, and low robustness, we proposed a Chinese sign language gesture recognition method named Air-CSL. In this method, the Local Outlier Factor (LOF) removal algorithm and the Discrete Wavelet Transform (DWT) are used to reduce the noise in the data, and the subcarriers that best represent the gesture data are selected by principal component analysis. After denoising, mathematical statistics were extracted from the gesture waveform as the eigenvalues, and the features were fused by the Deep Restricted Boltzmann Machine (DBM). Finally, the result of gesture classification and recognition is obtained by the Gated Recurrent Unit (GRU). In this way, the prediction model realizes as well as the classification of sign language gestures. The results show that the proposed method can effectively recognize Chinese sign language gestures of different people in different environments and has good robustness.

Information flow in Deep Restricted Boltzmann Machines: An analysis of mutual information between inputs and outputs

Empirical evidence suggests the existence of an entangled relationship between the information flow from inputs features to hidden representations of a deep neural network and its ability to generalize from training samples to unobserved data. For instance, regularization techniques often used to control statistical generalization, are expected to impact this information flow. In this work, we study MI (mutual information) between inputs and representation outputs, and its relationship with various regularization methods commonly used in Restricted Boltzmann Machines (RBM) and their generalizations: Deep Belief Networks and Deep Boltzmann Machines. Our theoretical findings show the existence of fundamental connections between the hyperparameters associated with the regularization and the MI, including relevant practical ingredients such as: network dimension, matrix norms and dropout probability, which are well-known to influence the generalization ability of the network. These results are experimentally corroborated on various visual datasets. Code is avaliable at https://codeocean.com/capsule/3175474/tree.

Emotion Analysis and Happiness Evaluation for Graduates During Employment.

Happiness can be regarded as an evaluation of life satisfaction. A high level of wellbeing can promote self-fulfillment and build a rational, peaceful, self-esteem, self-confidence, and positive social mentality. Therefore, the analysis of the factors of happiness is of great significance for the continuous improvement of the individual’s sense of security and gain and the realization of the maximization of self-worth. Emotion is not only an important internal factor that affects happiness, but it can also accurately reflect the individual’s happiness. However, most of current happiness evaluation methods based on the emotional analysis belong to shallow learning paradigm, making the deep learning method unexploited for automatically happiness decoding. In this article, we analyzed the emotions of graduates during their employment and studied its influence on personal happiness at work. We proposed deep restricted Boltzmann machine (DRBM) for graduates’ happiness evaluation during employment. Furthermore, to mitigate the information loss when passing through many network layers, we introduced the skip connections to DRBM and proposed a deep residual RBM (DRRBM) for enhancing the valuable information. We further introduced an attention mechanism to DRRBM to focus on the important factors. To verify the effectiveness of the proposed method on the happiness evaluation tasks, we conducted extensive experiments on the statistical data of the China Comprehensive Social Survey (CGSS). Compared with the state-of-the-art methods, our method shows better performance, which proves the practicability and feasibility of our method for happiness evaluation.

Sustainable Investment Forecasting of Power Grids Based on the Deep Restricted Boltzmann Machine Optimized by the Lion Algorithm

This paper proposes a new power grid investment prediction model based on the deep restricted Boltzmann machine (DRBM) optimized by the Lion algorithm (LA). Firstly, two factors including transmission and distribution price reform (TDPR) and 5G station construction were comprehensively incorporated into the consideration of influencing factors, and the fuzzy threshold method was used to screen out critical influencing factors. Then, the LA was used to optimize the parameters of the DRBM model to improve the model's prediction accuracy, and the model was trained with the selected influencing factors and investment. Finally, the LA-DRBM model was used to predict the investment of a power grid enterprise, and the final prediction result was obtained by modifying the initial result with the modifying factors. The LA-DRBM model compensates for the deficiency of the single model, and greatly improves the investment prediction accuracy of the power grid. In this study, a power grid enterprise was taken as an example to carry out an empirical analysis to prove the validity of the model, and a comparison with the RBM, support vector machine (SVM), back propagation neural network (BPNN), and regression model was conducted to verify the superiority of the model. The conclusion indicates that the proposed model has a strong generalization ability and good robustness, is able to abstract the combination of low-level features into high-level features, and can improve the efficiency of the model's calculations for investment prediction of power grid enterprises.

A transport equation approach for deep neural networks with quenched random weights

We consider a multi-layer Sherrington–Kirkpatrick spin-glass as a model for deep restricted Boltzmann machines with quenched random weights and solve for its free energy in the thermodynamic limit by means of Guerra’s interpolating techniques under the RS and 1RSB ansatz. In particular, we recover the expression already known for the replica-symmetric case. Further, we drop the restriction constraint by introducing intra-layer connections among spins and we show that the resulting system can be mapped into a modular Hopfield network, which is also addressed via the same techniques up to the first step of replica symmetry breaking.

DeepO: An ontology-based deep learning system for disease prediction

Recently, there is a lot of research about building supporting diagnosis and treatment in medical care using deep learning. In this paper, we propose a method for building a deep learning model base on ontology structure for disease prediction (DeepO). The ontology structure will be automatically built based on the improvements of formal concepts analysis and pattern structure. We use deep restricted Boltzmann machine (DRBM) for the disease prediction task in DeepO. By using OCA in DeepO, our model learns more knowledge than RBM model. In experiments, we build a digestive disease diagnosis model with Vietnamese patient dataset.

Recognition of EEG Signal Motor Imagery Intention Based on Deep Multi-View Feature Learning.

Recognition of motor imagery intention is one of the hot current research focuses of brain-computer interface (BCI) studies. It can help patients with physical dyskinesia to convey their movement intentions. In recent years, breakthroughs have been made in the research on recognition of motor imagery task using deep learning, but if the important features related to motor imagery are ignored, it may lead to a decline in the recognition performance of the algorithm. This paper proposes a new deep multi-view feature learning method for the classification task of motor imagery electroencephalogram (EEG) signals. In order to obtain more representative motor imagery features in EEG signals, we introduced a multi-view feature representation based on the characteristics of EEG signals and the differences between different features. Different feature extraction methods were used to respectively extract the time domain, frequency domain, time-frequency domain and spatial features of EEG signals, so as to made them cooperate and complement. Then, the deep restricted Boltzmann machine (RBM) network improved by t-distributed stochastic neighbor embedding(t-SNE) was adopted to learn the multi-view features of EEG signals, so that the algorithm removed the feature redundancy while took into account the global characteristics in the multi-view feature sequence, reduced the dimension of the multi-visual features and enhanced the recognizability of the features. Finally, support vector machine (SVM) was chosen to classify deep multi-view features. Applying our proposed method to the BCI competition IV 2a dataset we obtained excellent classification results. The results show that the deep multi-view feature learning method further improved the classification accuracy of motor imagery tasks.

Targeted Sentiments and Hierarchical Events based Learning Model for Stock Market Forecasting from Multi-Source Data

Stock market price movement forecast from multi-source data has gained massive interest in recent years. Studies were focussed on extracting the events and sentiments from different source data and employ them in learning the stock price movement patterns. This approach provided accurate and highly reliable forecasting as it involves multiple stock price indicators. However, some aspects of sentiment analysis and event extraction increase the training time and computation complexity in big data stock analysis. To overcome these issues, the hierarchical event extraction and the target dependent sentiment analysis are performed in this paper to improve the learning rate stock price movement patterns. In this paper, the events are hierarchically extracted from news articles using Deep Restricted Boltzmann Machine (DRBM). The target based sentiments from the tweets are detected using Improved Extreme Learning machine (IELM) whose parameters are optimally selected using Spotted Hyena Optimizer (SHO). The stock indicators obtained from these two processes are used in the learning process performed using Tolerant Flexible Multi-Agent Deep Reinforcement Learning (TFMA-DRL) model for analysing the stock patterns and forecasting the future stock trends. The forecasting results obtained by using the TFMA-DRL model by combining the stock indicators of targeted sentiments and hierarchical events are trustworthy and reliable. Evaluations are performed using three datasets collected for 12 months period from three sources of Twitter, Market News and Stock exchange. Results highlighted that the proposed stock forecasting model achieved 90% accuracy with minimum training time.

Knowledge extraction and insertion to deep belief network for gearbox fault diagnosis

Deep neural network (DNN) with a complex structure and multiple nonlinear processing units has achieved great success for feature learning in machinery fault diagnosis. Due to the “black box” problem in DNNs, there are still many obstacles to the application of DNNs in fault diagnosis. This paper proposes a new DNN model, knowledge-based deep belief network (KBDBN), which inserts confidence and classification rules into the deep network structure. This not only enables the model to have good pattern recognition performance but also to adaptively determine the network structure and obtain a good understanding of the features learned by the deep network. The knowledge extraction algorithm is proposed to offer a good representation of layerwise networks (i.e., restricted Boltzmann machines (RBMs)). The layerwise extraction can produce an improvement in feature learning of RBMs. Moreover, the extracted confidence rules that characterize the deep network offers a novel method for insertion of prior knowledge in the deep RBM. The classification knowledge extracted from the data is further inserted into the classification layer of DBN. KBDBN is used to generate the discriminant features from the data and then construct a complex mapping between vibration signals and gearbox defects. The testing results of KBDBN on a gearbox test rig not only effectively extracts knowledge from the deep network, but also shows better classification performance than the typical classifiers and DBNs. Moreover, the interpretable network model helps us understand what DBN has learned from vibration signals and then makes it be applied easily in real-world cases.

Discriminant analysis of mine quake type and intensity based on a deep neural network

ABSTRACT The occurrence of mine quake is subject to coupling control of multiple factors. To study the influence of mine quake distribution on rock burst under different incentive conditions, based on massive microseismic monitoring data, this paper established time-varying signal analysis model for deep recurrent neural network and restricted Boltzmann machine deep process neural network, which supports multi-perspective and high-dimensional analysis of complex mine quake. For feature extraction of multi-source acoustic emission real-time monitoring signals and the discrimination of microquake and mine quake intensity, a deep neural network model was proposed for the identification of mine quake type and intensity. Practice has shown that the discrimination effect is fine.

Velocity forecasts using a combined deep learning model in hybrid electric vehicles with V2V and V2I communication

Vehicle velocity forecast is an important clue in improving the performance of energy management in hybrid electric vehicles (HEV). This paper presents a new combined model for predicting vehicle’s velocity time series. The main features of the model are to combine the feature extraction capability of deep restricted Boltzmann machines (DBM) and sequence pattern predicting capability of bidirectional long short-term memory (BLSTM). Hence, the model is named as DBMBLSTM. In addition, the DRMBLSTM model utilizes the vehicle driving information and roadside infrastructure information provided respectively through vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communication channels to predict vehicle velocity at various length of prediction horizon. Furthermore, the predictions results of this study are compared with the state of the art of vehicle velocity forecasts. The root mean square error (RMSE) is used as an evaluation criteria of predictions accuracy. Finally, these compared prediction model are applied in model predictive control (MPC) energy management strategy for the verifications of fuel economy improvement of a HEV. Simulation results confirm that the proposed combined deep learning model performs better than other five prediction methods. Therefore, it is a means of arriving at a reliable forecast model for HEV.

Learning from Longitudinal Face Demonstration—Where Tractable Deep Modeling Meets Inverse Reinforcement Learning

Modeling the face aging process is a challenging task due to large and non-linear variations present in different stages of face development. This paper presents a deep model approach for face age progression that can efficiently capture the non-linear aging process and automatically synthesize a series of age-progressed faces in various age ranges. In this approach, we first decompose the long-term age progress into a sequence of short-term changes and model it as a face sequence. The Temporal Deep Restricted Boltzmann Machines based age progression model together with the prototype faces are then constructed to learn the aging transformation between faces in the sequence. In addition, to enhance the wrinkles of faces in the later age ranges, the wrinkle models are further constructed using Restricted Boltzmann Machines to capture their variations in different facial regions. The geometry constraints are also taken into account in the last step for more consistent age-progressed results. The proposed approach is evaluated using various face aging databases, i.e. FG-NET, Cross-Age Celebrity Dataset (CACD) and MORPH, and our collected large-scale aging database named AginG Faces in the Wild (AGFW). In addition, when ground-truth age is not available for input image, our proposed system is able to automatically estimate the age of the input face before aging process is employed.

Revealing causality between heterogeneous data sources with deep restricted Boltzmann machines

In a number of real life applications, scientists do not have access to temporal data, since budget for data acquisition is always limited. Here we challenge the problem of causal inference between groups of heterogeneous non-temporal observations obtained from multiple sources. We consider a family of probabilistic algorithms for causal inference based on an assumption that in case where X causes Y, P(X) and P(Y|X) are statistically independent. For a number of real world applications, deep learning methods were reported to achieve the most accurate empirical performance, what motivates us to use deep Boltzmann machines to approximate the marginal and conditional probabilities of heterogeneous observations as accurate as possible.We introduce a novel algorithm to infer causal relationships between blocks of variables. The proposed method was tested on a benchmark of multivariate cause-effect pairs. We show by our experiments that our method achieves the state-of-the-art empirical accuracy, and sometimes outperforms the state-of-the-art methods. An important part of our contribution is an application of the proposed algorithm to an original medical data set, where we explore relations between alimentary patters, human gut microbiome composition, and health status.

Quantum deep learning

In recent years, deep learning has had a profound impact on machine learning and artificial intelligence. At the same time, algorithms for quantum computers have been shown to efficiently solve some problems that are intractable on conventional, classical computers. We show that quantum computing not only reduces the time required to train a deep restricted Boltzmann machine, but also provides a richer and more comprehensive framework for deep learning than classical computing and leads to significant improvements in the optimization of the underlying objective function. Our quantum methods also permit efficient training of multilayer and fully connected models.