High-dimensional and large-scale anomaly detection using a linear one-class SVM with deep learning

Recognizing multivariate geochemical anomalies for mineral exploration by combining deep learning and one-class support vector machine

Kernelized support vector machine with deep learning: An efficient approach for extreme multiclass dataset

Deep learning has been popular for a few years, and it shows great capability on unsupervised leaning of representation. Deep belief network consists of multi layers of restricted Boltzmann machine(RBM) and a deep auto-encoder, which uses a stack architecture learning feature layer by layer. The learning rule is that one deeper layer learns more complex representations, which are the high level features of the input data, from the representations learnt by the layer before. Fuzzy C-Means(FCM) is one of the most popular clustering algorithms, which allows one piece of data belong to several clusters. In this paper the authors propose a novel clustering model, and introduce a novel clustering technique(DBNOC) which combines deep belief network and fuzzy c-means. The main idea is that: first, it clusters with the high level representations learnt by stacked RBM to produce the initial cluster center, then it uses the fine-tune step including one center holding clustering algorithm and deep auto-encoder to optimize the cluster center and membership between input data and every cluster by cross iteration. The authors use FCM clustering algorithm to fulfill the model and do experiment on both low dimensional datasets and high dimensional datasets. The experiment results suggest that the proposed deep belief network oriented clustering method is better than the standard K-Means and FCM algorithm on the test datasets. Even on high dimensional datasets, the DBNOC clustering method show more generalization. What's more, the proposed model is suitable both in theoretical and practical.

Human Activity Recognition (HAR) has become one of the most prominent research topics in the field of ubiquitous computing and pattern recognition over the last decade. In this paper, a comparative analysis of 17 different algorithms is done using a 4-core 940mx machine and a 16-core G4dn.4xlarge Elastic Compute (EC2) instance on a public domain HAR dataset using accelerometer and gyroscope data from the inertial sensors in smartphones. The results are evaluated using the metrics accuracy, F1-score, precision, recall, training time, and testing time. The Machine Learning (ML) models implemented include Logistic Regression (LR), Support Vector Classifier (SVC), Random Forest (RF), Decision Trees (DT), Gradient Boosted Decision Trees (GBDT), linear and Radial Basis Function (RBF) kernel Support Vector Machines (SVM), K- Nearest Neighbors (KNN) and Naive Bayes (NB). The Deep Learning (DL) models implemented include Convolutional Neural Networks (CNN), Long Short Term Memory (LSTM), a combination of CNN-LSTM and Bidirectional LSTM. Neural Structure Learning was also implemented over a CNN-LSTM model along with Deep Belief Networks (DBN). It is identified that the Deep Learning models CNN, LSTM, CNN-LSTM & CNN-BLSTM consistently confuse between dynamic activities and that the machine learning models confuse between static activities. A Divide and Conquer approach was implemented on the dataset and CNN achieved an accuracy of 99.92% on the dynamic activities, whereas the CNN-LSTM model achieved an accuracy of 96.73% eliminating confusion between the static and dynamic activities. Maximum classification accuracy of 99.02% was achieved by DBN on the full dataset after Gaussian standardization. The proposed DBN model is much more efficient, lightweight, accurate, and faster in its classification than the existing models.KeywordsMachine learningDeep learningUCI HARInertial sensorsDivide and ConquerDeep Belief Networks

One class Support Vector Machine (OCSVM) is remarkably an efficient semi-supervised learning method for classifying one class anomaly in the applications such as fault detection in hardware, document classification, novelty detection, etc. However, many studies showed that due to the presence of anomalies in the training data the boundary measured by OCSVM is biased towards anomalies and thereby results in to low classification accuracy. Classifying ozone measurements obtained from the environment is one such application where the dataset composes huge number of anomalies due to irregularities in the deployed sensors. To this end, this paper presents a technique to improve the anomaly classification accuracy of OCSVM using Deep Belief Networks (DBN). First, the data are pre-processed and then DBN is used for extracting linearly separable data. This outcome is then given to the OCSVM for classification of anomalous ozone measurements in the next step. It is observed from the simulation results that the proposed method shows better classification and achieved high accuracy of 92.71%.

This paper addresses the problem of anomaly detection for high-dimensional sensing data. The one-class support vector machine (OCSVM) is one of the most popular unsupervised methods for anomaly detection. When data are high dimensional and large scale, however, the efficiency of OCSVM-based methods in anomaly detection suffers. Although dimensionality-reduction tools, such as deep belief networks, can be applied to compress the high-dimensional data to alleviate the problem, the accuracy and timely detection are still hard to improve due to the inherent features of OCSVM. In this paper, we propose a new form of OCSVM model based on the structure of the compressed data and the characteristics of OCSVM. Based on the new model, we design both optimal and approximate methods for model training and testing. We evaluate the performance of our methods with extensive experiments on four real-world datasets. The experimental results demonstrate that our new methods, both optimal and approximate ones, not only significantly outperform the state-of-the-art in accuracy and efficiency, but also achieve the good performance without the need of manual parameter tuning. In addition, our approximate training and testing mechanism can reduce the computing time by three orders of magnitude with a negligible loss in accuracy.

Ultra-short term wind prediction with wavelet transform, deep belief network and ensemble learning

Air quality management and monitoring are vital to maintaining clean air, which is necessary for the health of human, vegetation, and ecosystems. Ozone pollution is one of the main pollutants that negatively affect human health and ecosystems. This paper reports the development of an unsupervised and efficient scheme to detecting anomalies in unlabeled ozone measurements. This scheme combines a deep belief networks (DBNs) model and a one-class support vector machine (OCSVM). The DBN model accounts for nonlinear variations in the ground-level ozone concentrations, while OCSVM detects the abnormal ozone measurements. The performance of this approach is evaluated using real data from Isère in France. We also compare the detection quality of DBN-based detection schemes to that of deep stacked auto-encoders, restricted Boltzmann machines-based OCSVM, and DBN-based clustering procedures (i.e., K-means, Birch, and expectation-maximization). The results show that the developed strategy is able to identify anomalies in ozone measurements.

This paper presents a fault detection method based on an unsupervised deep learning to monitor operating conditions of wastewater treatment plants (WWTPs). This method uses Deep Belief Networks (DBNs) model and one-class support vector machine (OCSVM). Here, DBN model is introduced to account for nonlinear aspects of WWTPs, while OCSVM is employes to reliably detect a fault in WWTP. The developed DBN-OCSVM approach has been tested through practical application on data from a decentralized wastewater treatment plant in Golden, CO, USA. Results show the effectiveness of the developed approach to monitor the WWTP.

Statistical monitoring of a wastewater treatment plant: A case study

A new semi-supervised approach based on deep learning and active learning for classification of electrocardiogram signals (ECG) is proposed. The objective of the proposed work is to model a scientific method for classification of cardiac irregularities using electrocardiogram beats. The model follows the Association for the Advancement of medical instrumentation (AAMI) standards and consists of three phases. In phase I, feature representation of ECG is learnt using Gaussian-Bernoulli deep belief network followed by a linear support vector machine (SVM) training in the consecutive phase. It yields three deep models which are based on AAMI-defined classes, namely N, V, S, and F. In the last phase, a query generator is introduced to interact with the expert to label few beats to improve accuracy and sensitivity. The proposed approach depicts significant improvement in accuracy with minimal queries posed to the expert and fast online training as tested on the MIT-BIH Arrhythmia Database and the MIT-BIH Supra-ventricular Arrhythmia Database (SVDB). With 100 queries labeled by the expert in phase III, the method achieves an accuracy of 99.5% in "S" versus all classifications (SVEB) and 99.4% accuracy in "V " versus all classifications (VEB) on MIT-BIH Arrhythmia Database. In a similar manner, it is attributed that an accuracy of 97.5% for SVEB and 98.6% for VEB on SVDB database is achieved respectively. Graphical Abstract Reply- Deep belief network augmented by active learning for efficient prediction of arrhythmia.

Novel deep generative simultaneous recurrent model for efficient representation learning

In this invited paper, my overview material on the same topic as presented in the plenary overview session of APSIPA-2011 and the tutorial material presented in the same conference [1] are expanded and updated to include more recent developments in deep learning. The previous and the updated materials cover both theory and applications, and analyze its future directions. The goal of this tutorial survey is to introduce the emerging area of deep learning or hierarchical learning to the APSIPA community. Deep learning refers to a class of machine learning techniques, developed largely since 2006, where many stages of non-linear information processing in hierarchical architectures are exploited for pattern classification and for feature learning. In the more recent literature, it is also connected to representation learning, which involves a hierarchy of features or concepts where higher-level concepts are defined from lower-level ones and where the same lower-level concepts help to define higher-level ones. In this tutorial survey, a brief history of deep learning research is discussed first. Then, a classificatory scheme is developed to analyze and summarize major work reported in the recent deep learning literature. Using this scheme, I provide a taxonomy-oriented survey on the existing deep architectures and algorithms in the literature, and categorize them into three classes: generative, discriminative, and hybrid. Three representative deep architectures – deep autoencoders, deep stacking networks with their generalization to the temporal domain (recurrent networks), and deep neural networks (pretrained with deep belief networks) – one in each of the three classes, are presented in more detail. Next, selected applications of deep learning are reviewed in broad areas of signal and information processing including audio/speech, image/vision, multimodality, language modeling, natural language processing, and information retrieval. Finally, future directions of deep learning are discussed and analyzed.

The Deep Learning models have drawn ever-increasing research interest owing to their intrinsic capability of overcoming the drawback of traditional algorithm. Hence, we have adopted the representative Deep Learning methods which are Deep Belief Network (DBN) and Stacked Auto-Encoder (SAE), to initialize deep supervised Neural Networks (NN), besides of Back Propagation Neural Networks (BPNN) applied to face classification task. Moreover, our contribution is to extract hierarchical representations of face image based on the Deep Learning models which are: DBN, SAE and BPNN. Then, the extracted feature vectors of each model are used as input of NN classifier. Next, to test our approach and evaluate its performance, a simulation series of experiments were performed on two facial databases: BOSS and MIT. Our proposed approach which is (DBN,NN) has a significant improvement on the classification error rate compared to (SAE,NN) and BPNN which we get 1.14% and 1.96% in terms of error rate with BOSS and MIT respectively.

Solar energy predictive models designed to emulate the long-term (e.g., monthly) global solar radiation (GSR) trained with satellite-derived predictors can be employed as decision tenets in the exploration, installation and management of solar energy production systems in remote and inaccessible solar-powered sites. In spite of a plethora of models designed for GSR prediction, deep learning, representing a state-of-the-art intelligent tool, remains an attractive approach for renewable energy exploration, monitoring and forecasting. In this paper, algorithms based on deep belief networks and deep neural networks are designed to predict long-term GSR. Deep learning algorithms trained with publicly-accessible Moderate Resolution Imaging Spectroradiometer (MODIS) satellite data are tested in Australia’s solar cities to predict the monthly GSR: single hidden layer and ensemble models. The monthly-scale MODIS-derived predictors (2003–2018) are adopted, with 15 diverse feature selection approaches including a Gaussian Emulation Machine for sensitivity analysis used to select optimal MODIS-predictor variables to simulate GSR against ground-truth values. Several statistical score metrics are adopted to comprehensively verify surface GSR simulations to ascertain the practicality of deep belief and deep neural networks. In the testing phase, deep learning models generate significantly lower absolute percentage bias (≤3%) and high Kling–Gupta efficiency (≥97.5%) values compared to the single hidden layer and ensemble model. This study ascertains that the optimal MODIS input variables employed in GSR prediction for solar energy applications can be relatively different for diverse sites, advocating a need for feature selection prior to the modelling of GSR. The proposed deep learning approach can be adopted to identify solar energy potential proactively in locations where it is impossible to install an environmental monitoring data acquisition instrument. Hence, MODIS and other related satellite-derived predictors can be incorporated for solar energy prediction as a strategy for long-term renewable energy exploration.