Layers Of Restricted Boltzmann Machine Research Articles

Improved RBM‐based feature extraction for credit risk assessment with high dimensionality

AbstractTo address the high‐dimensional issues in credit risk assessment, an improved multilayer restricted Boltzmann machine (RBM) based feature extraction method is proposed. In the improved multilayer RBM methodology, the reconstruction error method is first applied to ensure the number of RBM layers to construct an optimal model and then the weighted pruning approach is used to remove redundant and irrelevant traits. For verification purposes, two real‐world credit datasets are employed to demonstrate the effectiveness of the proposed multilayer RBM methodology. The experimental results reveal that a significant improvement in credit classification performance can be obtained by the improved multilayer RBM methodology. This indicates the improved multilayer RBM model proposed in this paper can be used as a promising tool to solve the high‐dimensionality issues in credit risk evaluation.

Fine-tuning Road Classification Models: Optimization Strategies for Deep Belief Networks in Transportation Big Data

Road traffic accidents are very essential for common people, consequential an estimated 1.2 million deaths and 50 million injuries all over the world every year. In this emerging world, the road accidents are among the principal reason of fatality and injury. The concern of traffic safety has heaved immense alarms across the manageable enhancement of contemporary traffic and transportation. The analysis on road traffic accident grounds can detect the major aspects quickly, professionally and afford instructional techniques to the prevention of traffic accidents and reduction of road traffic accident, which might significantly decrease personal victim by means of road traffic accidents. Data Mining techniques are used in the process of knowledge discovery for many domains’ problems. Feature Selection plays a vital role for a large number of datasets. In this paper, the classification of road accident in transportation domain was analyzed with the assistance of the proposed Intelligent classification technique. In this proposed technique, the DBN hidden layers weights are optimized by using evolutionary Genetic algorithm. This GA is utilized to enhance the classification accuracy by applying the hidden layers of Restricted Boltzmann Machine (RBM). The comparative results show that the proposed intelligent classifier gives the improved accuracy, specificity, precision, Sensitivity, F-Measure, and reduced false positive rate.

Hybrid intelligent intrusion detection system for multiple Wi‐Fi attacks in wireless networks using stacked restricted Boltzmann machine and deep belief networks

SummaryThe recent cutting edge mobile and internet advancements have increased Internet of Things based gadgets more pervasive and cohesive into our day to day lives. There emerges a need to beat the security challenges, prevalently came about because of the uncovered idea of a remote medium, for example a Wi‐Fi network setup. The linked gadgets are ubiquitous, creating high‐dimensional information on an expansive scale. It tends to be seen those illegitimate actions like illegal information access, information stealing, information alteration different other interruption exercises are quickly developing during a decade ago. Wireless network data transfers will be anticipated to surge strongly in the coming years. Feature learning, still, can evade the probable complications, possibly formed by means of large wireless network dimensions. Therefore, the proposed system focuses on developing a Hybrid Intelligent Intrusion Detection System (IDS) for detecting multiple assaults such as Impersonation, Injection, flooding along with other unknown malicious Wi‐Fi attacks using Deep Learning and Machine Learning algorithms. The proposed system is investigated on well referenced Aegean Wi‐Fi Intrusion Dataset (AWID) and the experimental results prove that the utility of deep learning algorithms in feature representation and classification has achieved higher detection accuracy along with lower false alarm rate. To get an optimum feature set and perform dimensionality reduction on Wi‐Fi network data, the proposed framework uses Deep Belief Network (DBN) with stacked layers of Restricted Boltzmann Machine (RBM). In order to identify and categories various Wi‐Fi threats, Deep Neural Networks, and algorithms like SVM, MLKNN, and RAkEL are employed. The anticipated DBN model was given an optimization strategy to enhance classification performance. The optimization strategy involved selecting the best possible feature subset using a feature selection technique, optimizing the DBN model using the best possible hyperparameter values, less reconstruction error values, and utilizing the regularization method to account for weight decay. The experimental findings demonstrate that the use of deep learning algorithms in feature representation and classification has produced improved detection accuracy coupled with a decreased false alarm rate when applied to the proposed hybrid scheme based Intelligent IDS using the well‐referenced AWID dataset.

Image Classification and Recognition of Rice Diseases: A Hybrid DBN and Particle Swarm Optimization Algorithm.

Rice blast, rice sheath blight, and rice brown spot have become the most popular diseases in the cold areas of northern China. In order to further improve the accuracy and efficiency of rice disease diagnosis, a framework for automatic classification and recognition of rice diseases is proposed in this study. First, we constructed a training and testing data set including 1,500 images of rice blast, 1,500 images of rice sheath blight, and 1,500 images of rice brown spot, and 1,100 healthy images were collected from the rice experimental field. Second, the deep belief network (DBN) model is designed to include 15 hidden restricted Boltzmann machine layers and a support vector machine (SVM) optimized with switching particle swarm (SPSO). It is noted that the developed DBN and SPSO-SVM can simultaneously learn three proposed features including color, texture, and shape to recognize the disease type from the region of interest obtained by preprocessing the disease images. The proposed model leads to a hit rate of 91.37%, accuracy of 94.03%, and a false measurement rate of 8.63%, with the 10-fold cross-validation strategy. The value of the area under the receiver operating characteristic curve (AUC) is 0.97, whose accuracy is much higher than that of the conventional machine learning model. The simulation results show that the DBN and SPSO-SVM models can effectively extract the image features of rice diseases during recognition, and have good anti-interference and robustness.

Eye State Identification Utilizing EEG Signals: A Combined Method Using Self-Organizing Map and Deep Belief Network

Measuring brain activity through Electroencephalogram (EEG) analysis for eye state prediction has attracted attention from machine learning researchers. There have been many methods for EEG analysis using supervised and unsupervised machine learning techniques. The tradeoff between the accuracy and computation time of these methods in performing the analysis is an important issue that is rarely investigated in the previous research. This paper accordingly proposes a new method for EEG signal analysis through Self-Organizing Map (SOM) clustering and Deep Belief Network (DBN) approaches to efficiently improve the computation and accuracy of the previous methods. The method is developed using SOM clustering and DBN, which is a deep layer neural network with multiple layers of Restricted Boltzmann Machines (RBMs). The results on a dataset with 14980 instances and 15 attributes representing the values of the electrodes demonstrated that the method is efficient for EEG analysis. In addition, compared with the other supervised methods, the proposed method was able to significantly improve the accuracy of the EEG prediction.

Insider Attack Detection Using Deep Belief Neural Network in Cloud Computing

Cloud computing is a high network infrastructure where users, owners, third users, authorized users, and customers can access and store their information quickly. The use of cloud computing has realized the rapid increase of information in every field and the need for a centralized location for processing efficiently. This cloud is nowadays highly affected by internal threats of the user. Sensitive applications such as banking, hospital, and business are more likely affected by real user threats. An intruder is presented as a user and set as a member of the network. After becoming an insider in the network, they will try to attack or steal sensitive data during information sharing or conversation. The major issue in today's technological development is identifying the insider threat in the cloud network. When data are lost, compromising cloud users is difficult. Privacy and security are not ensured, and then, the usage of the cloud is not trusted. Several solutions are available for the external security of the cloud network. However, insider or internal threats need to be addressed. In this research work, we focus on a solution for identifying an insider attack using the artificial intelligence technique. An insider attack is possible by using nodes of weak users’ systems. They will log in using a weak user id, connect to a network, and pretend to be a trusted node. Then, they can easily attack and hack information as an insider, and identifying them is very difficult. These types of attacks need intelligent solutions. A machine learning approach is widely used for security issues. To date, the existing lags can classify the attackers accurately. This information hijacking process is very absurd, which motivates young researchers to provide a solution for internal threats. In our proposed work, we track the attackers using a user interaction behavior pattern and deep learning technique. The usage of mouse movements and clicks and keystrokes of the real user is stored in a database. The deep belief neural network is designed using a restricted Boltzmann machine (RBM) so that the layer of RBM communicates with the previous and subsequent layers. The result is evaluated using a Cooja simulator based on the cloud environment. The accuracy and F-measure are highly improved compared with when using the existing long short-term memory and support vector machine.

Intelligent Recognition of Insulator Contamination Grade Based on the Deep Learning of Ultraviolet Discharge Image Information

In order to achieve the noncontact detection of the contamination grade of insulators and to provide guidance for preventing the contamination flashover of insulators based on the pollution state, we propose a contamination grade recognition method based on the deep learning of ultraviolet discharge images using a sparse autoencoder (SAE) and a deep belief network (DBN). Under different humidity conditions, we filmed and preprocessed the ultraviolet discharge images of insulators at different contamination grades and we obtained the ultraviolet spot area sequence as original data for contamination grade recognition. A double-layer sparse autoencoder was used to extract sparse features that could characterize different contamination grades from the ultraviolet spot area sequence. Using the extracted features, a DBN composed of three layers of restricted Boltzmann machine was trained to provide contamination grade recognition. To verify the effectiveness of the method proposed in this paper, high-voltage experiments were performed on contaminated insulators at relative humidity levels of 80%, 85%, and 90%, and ultraviolet images were recorded. The proposed SAE–DBN method was used to identify the ultraviolet images of the insulators with different contamination grades. The recognition accuracy rates at the three humidity levels were 91.25%, 93.125%, and 92.5%. The experimental results showed that this method could accurately recognize the contamination grade of the insulator and provide guidance for the prevention of contamination flashover based on the pollution severity.

Microgrid Fault Detection and Classification: Machine Learning Based Approach, Comparison, and Reviews

Accurate fault classification and detection for the microgrid (MG) becomes a concern among the researchers from the state-of-art of fault diagnosis as it increases the chance to increase the transient response. The MG frequently experiences a number of shunt faults during the distribution of power from the generation end to user premises, which affects the system reliability, damages the load, and increases the fault line restoration cost. Therefore, a noise-immune and precise fault diagnosis model is required to perform the fast recovery of the unhealthy phases. This paper presents a review on the MG fault diagnosis techniques with their limitations and proposes a novel discrete-wavelet transform (DWT) based probabilistic generative model to explore the precise solution for fault diagnosis of MG. The proposed model is made of multiple layers with a restricted Boltzmann machine (RBM), which enables the model to make the probability reconstruction over its inputs. The individual RBM layer is trained with an unsupervised learning approach where an artificial neural network (ANN) algorithm tunes the model for minimizing the error between the true and predicted class. The effectiveness of the proposed model is studied by varying the input signal and sampling frequencies. A level of considered noise is added with the sample data to test the robustness of the studied model. Results prove that the proposed fault detection and classification model has the ability to perform the precise diagnosis of MG faults. A comparative study among the proposed, kernel extreme learning machine (KELM), multi KELM, and support vector machine (SVM) approaches is studied to confirm the robust superior performance of the proposed model.

Mapping the susceptibility to landslides based on the deep belief network: a case study in Sichuan Province, China

A dataset of landslides from Sichuan Province in China, containing 1551 historical individual landslides, is a result of two teams’ effort in the past few years to map the susceptibility to landslides. Considering complex internal relations among the triggering factors, logistic regression (LR) and shallow neural networks, such as back-propagation neural network (BPNN), are often limited. In this paper, we make a straightforward development that the deep belief network (DBN) based on deep learning technology is introduced to map the regional susceptibility to landslides. Seven factors with respect to geomorphology, geology and hydrology are considered and verified through the collinearity test. A DBN model containing three pre-trained layers of restricted Boltzmann machines by stochastic gradient descent method is configured to obtain the susceptibility to landslides. Susceptibility results evaluated by DBN model are compared with those by LR and BPNN in the receive operator characteristic (ROC) analysis, showing that DBN has a better prediction precision, with a lower rate of false alarms and fake alarms. The case study also indicates different sensitivities of the triggering factors to the landslide susceptibility, that the factors of altitude, distance to drainage network and average annual rainfall have significant impact in mapping the susceptibility to landslides in the region. This research will contribute to a better-performance model for regional-scale mapping for the susceptibility to landslides, in particular, at the area where triggering factors show complex relations and relative independence.

Adaptive Threshold Estimation of Open Set Voiceprint Recognition Based on OTSU and Deep Learning

Aiming at the problem of open set voiceprint recognition, this paper proposes an adaptive threshold algorithm based on OTSU and deep learning. The bottleneck technology of open set voiceprint recognition lies in the calculation of similarity values and thresholds of speakers inside and outside the set. This paper combines deep learning and machine learning methods, and uses a Deep Belief Network stacked with three layers of Restricted Boltzmann Machines to extract deep voice features from basic acoustic features. And by training the Gaussian Mixture Model, this paper calculates the similarity value of the feature, and further determines the threshold of the similarity value of the feature through OTSU. After experimental testing, the algorithm in this paper has a false rejection rate of 3.00% for specific speakers, a false acceptance rate of 0.35% for internal speakers, and a false acceptance rate of 0 for external speakers. This improves the accuracy of traditional methods in open set voiceprint recognition. This proves that the method is feasible and good recognition effect.

Self-Organizing Traffic Flow Prediction with an Optimized Deep Belief Network for Internet of Vehicles.

To assist in the broadcasting of time-critical traffic information in an Internet of Vehicles (IoV) and vehicular sensor networks (VSN), fast network connectivity is needed. Accurate traffic information prediction can improve traffic congestion and operation efficiency, which helps to reduce commute times, noise and carbon emissions. In this study, we present a novel approach for predicting the traffic flow volume by using traffic data in self-organizing vehicular networks. The proposed method is based on using a probabilistic generative neural network techniques called deep belief network (DBN) that includes multiple layers of restricted Boltzmann machine (RBM) auto-encoders. Time series data generated from the roadside units (RSUs) for five highway links are used by a three layer DBN to extract and learn key input features for constructing a model to predict traffic flow. Back-propagation is utilized as a general learning algorithm for fine-tuning the weight parameters among the visible and hidden layers of RBMs. During the training process the firefly algorithm (FFA) is applied for optimizing the DBN topology and learning rate parameter. Monte Carlo simulations are used to assess the accuracy of the prediction model. The results show that the proposed model achieves superior performance accuracy for predicting traffic flow in comparison with other approaches applied in the literature. The proposed approach can help to solve the problem of traffic congestion, and provide guidance and advice for road users and traffic regulators.

Voiceprint Identification for Limited Dataset Using the Deep Migration Hybrid Model Based on Transfer Learning.

The convolutional neural network (CNN) has made great strides in the area of voiceprint recognition; but it needs a huge number of data samples to train a deep neural network. In practice, it is too difficult to get a large number of training samples, and it cannot achieve a better convergence state due to the limited dataset. In order to solve this question, a new method using a deep migration hybrid model is put forward, which makes it easier to realize voiceprint recognition for small samples. Firstly, it uses Transfer Learning to transfer the trained network from the big sample voiceprint dataset to our limited voiceprint dataset for the further training. Fully-connected layers of a pre-training model are replaced by restricted Boltzmann machine layers. Secondly, the approach of Data Augmentation is adopted to increase the number of voiceprint datasets. Finally, we introduce fast batch normalization algorithms to improve the speed of the network convergence and shorten the training time. Our new voiceprint recognition approach uses the TLCNN-RBM (convolutional neural network mixed restricted Boltzmann machine based on transfer learning) model, which is the deep migration hybrid model that is used to achieve an average accuracy of over 97%, which is higher than that when using either CNN or the TL-CNN network (convolutional neural network based on transfer learning). Thus, an effective method for a small sample of voiceprint recognition has been provided.

Deep learning-based fault diagnosis of variable refrigerant flow air-conditioning system for building energy saving

The fault diagnosis of air-conditioning systems is of great significance to the energy saving of buildings. This study proposes a novel fault diagnosis approach for building energy saving based on the deep learning method which is deep belief network, and its application potential in the air conditioning fault diagnosis field is investigated. Then, a parameter optimization selection strategy is developed for model optimization. Four kinds of faults of the variable flow refrigerant system under heating mode are used to evaluate the performance of the models. The fault diagnosis results show that the deep belief network model with initial parameters can be used to diagnose the faults of the variable flow refrigerant system. Through the parameter optimization selection strategy, the fault diagnosis correct rate of the optimized model is 97.7%, which is improved by 5.05% compared with the model with initial parameters. The number of hidden layers of the deep belief network model is selected to be 2 layers. This result indicates that the fault diagnosis for variable flow refrigerant systems may not require a very deep model. Additionally, the performance of the optimized deep belief network model is compared with that of the traditional back propagation neural network, and the former is better. This finding also shows that the unsupervised restricted Boltzmann machine layer for data feature reconstruction can improve the fault diagnosis performance.

Modelling of a post-combustion CO2 capture process using deep belief network

This paper presents a study on using deep learning for the modelling of a post-combustion CO2 capture process. Deep learning has emerged as a very powerful tool in machine learning. Deep learning technique includes two phases: an unsupervised pre-training phase and a supervised back-propagation phase. In the unsupervised pre-training phase, a deep belief network (DBN) is pre-trained to obtain initial weights of the subsequent supervised phase. In the supervised back-propagation phase, the network weights are fine-tuned in a supervised manner. DBN with many layers of Restricted Boltzmann Machine (RBM) can extract a deep hierarchical representation of training data. In terms of the CO2 capture process, the DBN model predicts CO2 production rate and CO2 capture level using the following variables as model inputs: inlet flue gas flow rate, CO2 concentration in inlet flue gas, pressure of flue gas, temperature of flue gas, lean solvent flow rate, MEA concentration and temperature of lean solvent. A greedy layer-wise unsupervised learning algorithm is introduced to optimize DBN, which can bring better generalization than a single hidden layer neural network. The developed deep architecture network models can then be used in the optimisation of the CO2 capture process.

Single and combined fault diagnosis of reciprocating compressor valves using a hybrid deep belief network

In this paper, a hybrid deep belief network is proposed to diagnose single and combined faults of suction and discharge valves in a reciprocating compressor. This hybrid integrates the deep belief network structured by multiple stacked restricted Boltzmann machines for pre-training and simplified fuzzy ARTMAP (SFAM) for fault classification. In the pre-training procedure, an algorithm for selecting local receptive fields is used to group the most similar features into the receptive fields of which top values are the units of each layer, and then restricted Boltzmann machine is applied to these units to construct a network. Unsupervised learning is also carried out for each restricted Boltzmann machine layer in this procedure to compute the network weights and biases. Finally, the network output is fed into SFAM to perform fault classification. In order to diagnose the valve faults, three signal types of vibration, pressure, and current are acquired from a two-stage reciprocating air compressor under different valve conditions such as suction leakages, discharge leakages, spring deterioration, and their combination. These signals are subsequently processed so that the useful fault information from the signals can be revealed; next, statistical features in the time and frequency domains are extracted from the signals and used as the inputs for hybrid deep belief network. Performance of hybrid deep belief network in fault classification is compared with that of the original deep belief network and the deep belief network combined with generalized discriminant analysis, where softmax regression is used as a classifier for the latter two models. The results indicate that hybrid deep belief network is more capable of improving the diagnosis accuracy and is feasible in industrial applications.

Temperature-Based Deep Boltzmann Machines

Deep learning techniques have been paramount in the last years, mainly due to their outstanding results in a number of applications, that range from speech recognition to face-based user identification. Despite other techniques employed for such purposes, Deep Boltzmann Machines (DBMs) are among the most used ones, which are composed of layers of Restricted Boltzmann Machines stacked on top of each other. In this work, we evaluate the concept of temperature in DBMs, which play a key role in Boltzmann-related distributions, but it has never been considered in this context up to date. Therefore, the main contribution of this paper is to take into account this information, as well as the impact of replacing a standard Sigmoid function by another one and to evaluate their influence in DBMs considering the task of binary image reconstruction. We expect this work can foster future research considering the usage of different temperatures during learning in DBMs.

Identification framework for cracks on a steel structure surface by a restricted Boltzmann machines algorithm based on consumer-grade camera images

This paper proposes an identification framework based on a restricted Boltzmann machine (RBM) for crack identification and extraction from images containing cracks and complicated background inside steel box girders of bridges. The original images that include fatigue crack and other background information are obtained by a consumer-grade camera inside the steel box girder. The original images are cut into a number of elements with small size as the input dataset, and a state representation vector is artificially labeled to every image element used for the crack identification. A deep learning model or network consisting of multiple processing RBM layers to learn the abstract features is constructed to match the input image elements with corresponding state representation vectors. Next, a three-layer RBM with 500; 500; and 2,000 hidden units is trained as the hidden layers in the deep learning network. A contrastive divergence learning algorithm is employed for training the deep network to update and obtain the optimal parameters (i.e., the biases and weights). The new input image elements labeled as crack are sorted out and assembled to form an output image. A deep network is modeled through the consumer-grade camera images containing cracks and complicated background information using the proposed approach. The accuracy and ability to identify cracks from new images with different resolutions using the trained deep network are validated. Furthermore, effects of element size on reconstruction error and identification accuracy are investigated. The results show that there exists optimal element size; that is, too small and too large element sizes both increase the reconstruction error and decrease the identification accuracy.

Event Recognition Based on Deep Learning in Chinese Texts

Event recognition is the most fundamental and critical task in event-based natural language processing systems. Existing event recognition methods based on rules and shallow neural networks have certain limitations. For example, extracting features using methods based on rules is difficult; methods based on shallow neural networks converge too quickly to a local minimum, resulting in low recognition precision. To address these problems, we propose the Chinese emergency event recognition model based on deep learning (CEERM). Firstly, we use a word segmentation system to segment sentences. According to event elements labeled in the CEC 2.0 corpus, we classify words into five categories: trigger words, participants, objects, time and location. Each word is vectorized according to the following six feature layers: part of speech, dependency grammar, length, location, distance between trigger word and core word and trigger word frequency. We obtain deep semantic features of words by training a feature vector set using a deep belief network (DBN), then analyze those features in order to identify trigger words by means of a back propagation neural network. Extensive testing shows that the CEERM achieves excellent recognition performance, with a maximum F-measure value of 85.17%. Moreover, we propose the dynamic-supervised DBN, which adds supervised fine-tuning to a restricted Boltzmann machine layer by monitoring its training performance. Test analysis reveals that the new DBN improves recognition performance and effectively controls the training time. Although the F-measure increases to 88.11%, the training time increases by only 25.35%.

Deep belief network based electricity load forecasting: An analysis of Macedonian case

A number of recent studies use deep belief networks (DBN) with a great success in various applications such as image classification and speech recognition. In this paper, a DBN made up from multiple layers of restricted Boltzmann machines is used for electricity load forecasting. The layer-by-layer unsupervised training procedure is followed by fine-tuning of the parameters by using a supervised back-propagation training method. Our DBN model was applied to short-term electricity load forecasting based on the Macedonian hourly electricity consumption data in the period 2008–2014. The obtained results are not only compared with the latest actual data, but furthermore, they are compared with the predicted data obtained from a typical feed-forward multi-layer perceptron neural network and with the forecasted data provided by the Macedonian system operator (MEPSO). The comparisons show that the applied model is not only suitable for hourly electricity load forecasting of the Macedonian electric power system, it actually provides superior results than the ones obtained using traditional methods. The mean absolute percentage error (MAPE) is reduced by up to 8.6% when using DBN, compared to the MEPSO data for the 24-h ahead forecasting, and the MAPE for daily peak forecasting is reduced by up to 21%.

Sentiment analysis for Chinese microblog based on deep neural networks with convolutional extension features

Related research for sentiment analysis on Chinese microblog is aiming at the analysis procedure of posts. The length of short microblog text limits feature extraction of microblog. Tweeting is the process of communication with friends, so that microblog comments are important reference information for related post. A contents extension framework is proposed in this paper combining posts and related comments into a microblog conversation for features extraction. A novel convolutional auto encoder is adopted which can extract contextual information from microblog conversation as features for the post. A customized DNN (Deep Neural Network) model, which is stacked with several layers of RBM (Restricted Boltzmann Machine), is implemented to initialize the structure of neural network. The RBM layers can take probability distribution samples of input data to learn hidden structures for better high level features representation. A ClassRBM (Classification RBM) layer, which is stacked on top of RBM layers, is adopted to achieve the final sentiment classification label for the post. Experimental results show that, with proper structure and parameters, the performance of proposed DNN on sentiment classification is better than state-of-the-art surface learning models such as SVM or NB, which proves that the proposed DNN model is suitable for short-length document classification with the proposed feature dimensionality extension method.