Semi-supervised Anomaly Detection Research Articles

Semi-Supervised Anomaly Detection Algorithm Using Probabilistic Labeling (SAD-PL)

To detect abnormal data via semi-supervised learning, unlabeled data are generally assumed to be normal data. This assumption, however, causes inevitable performance degradation when a small fraction of abnormal data is included in the unlabeled dataset. To overcome the degradation and to maintain stable detection performance, we propose a semi-supervised anomaly detection algorithm using probabilistic labeling (SAD-PL) for unlabeled data. The proposed SAD-PL is composed of two steps: (1) estimating local outlier factor (LOF) scores of latent vectors from both labeled and unlabeled data and (2) estimating labeling probability on the unlabeled data by using the prior missing probability of the labeled data via the Neyman-Pearson (NP) criterion. The SAD-PL runs iteratively by using the proposed complementary learning functions until the rate of label changes is lower than the predefined threshold. Experimental results reveal that the SAD-PL shows superior detection probability over the existing algorithms and stable performance regardless of the normal to abnormal data ratio in unlabeled data and the ratio of change variation of unlabeled data statistics to labeled data statistics.

MADICS: A Methodology for Anomaly Detection in Industrial Control Systems

Industrial Control Systems (ICSs) are widely used in critical infrastructures to support the essential services of society. Therefore, their protection against terrorist activities, natural disasters, and cyber threats is critical. Diverse cyber attack detection systems have been proposed over the years, in which each proposal has applied different steps and methods. However, there is a significant gap in the literature regarding methodologies to detect cyber attacks in ICS scenarios. The lack of such methodologies prevents researchers from being able to accurately compare proposals and results. In this work, we present a Methodology for Anomaly Detection in Industrial Control Systems (MADICS) to detect cyber attacks in ICS scenarios, which is intended to provide a guideline for future works in the field. MADICS is based on a semi-supervised anomaly detection paradigm and makes use of deep learning algorithms to model ICS behaviors. It consists of five main steps, focused on pre-processing the dataset to be used with the machine learning and deep learning algorithms; performing feature filtering to remove those features that do not meet the requirements; feature extraction processes to obtain higher order features; selecting, fine-tuning, and training the most appropriate model; and validating the model performance. In order to validate MADICS, we used the popular Secure Water Treatment (SWaT) dataset, which was collected from a fully operational water treatment plant. The experiments demonstrate that, using MADICS, we can achieve a state-of-the-art precision of 0.984 (as well as a recall of 0.750 and F1-score of 0.851), which is above the average of other works, proving that the proposed methodology is suitable for use in real ICS scenarios.

Semi-supervised anomaly detection with dual prototypes autoencoder for industrial surface inspection

Anomaly detection in the automated optical quality inspection is of great important for guaranteeing the surface quality of industrial products. Most related methods are based on supervised learning techniques, which require a large number of normal and anomalous samples to obtain a robust classifier. However, the diversity of potential defects and low availability of defective samples during manufacturing bring more challenges to anomaly detection. Based on the encoder-decoder-encoder paradigm, a semi-supervised anomaly detection method Dual Prototype Auto-Encoder (DPAE) is proposed in this paper. At the training stage, the dual prototype loss and reconstruction loss are introduced to encourage the latent vectors generated by the encoders to keep closer to their own prototype. Therefore, two latent vectors of the normal image tend to be closer, and large distance between the latent vectors indicates an anomaly. And we also construct the Aluminum Profile Surface Defect (APSD) dataset for the anomaly detection task. Finally, extensive experiments on four datasets show that DPAE is effective and outperforms state-of-the-art methods.

Statistical approaches for semi-supervised anomaly detection in machining

Numerous methods have been developed to detect process anomalies during machining. Statistical approaches for semi-supervised anomaly detection compute decision boundaries using information of normal running processes for process evaluation. In this paper, two statistical approaches for semi-supervised anomaly detection in machining based on envelopes are presented and compared. The proposed parametric approach assumes normal distributed envelopes to compute decision boundaries. However, experiments show that deviations from a normal distribution can reduce the monitoring quality. The new approach is non-parametric and employs kernel density estimation (KDE) to estimate the probability density function of the envelopes. Both approaches were evaluated for several machining processes. It is found that the parametric approach is robust against high scattering processes and yields low false alarm rates. By means of the selected safety factor, the number of detected anomalies can be increased using the non-parametric approach.

Incremental collusive fraud detection in large-scale online auction networks

An online auction network (OAN) is a community of users who buy or sell items through an auction site. Along with the growing popularity of auction sites, concerns about auction frauds and criminal activities have increased. As a result, fraud detection in OANs has attracted renewed interest from researchers. Since most real OANs are large-scale networks, detecting fraudulent users is usually difficult, especially when multiple users collude with each other and new online auctions are continuously added. Although collusive auction frauds are not as popular as other types of auction frauds, they are more horrible and catastrophic because they often bring huge financial losses. To tackle this issue, some techniques have been proposed to detect collusive frauds in OANs. While all of the techniques have demonstrated promising results, they often suffer from low detection performance or slow convergence, especially in large-scale OANs. In this paper, we overcome these deficiencies by presenting ICAFD, a novel technique that recasts the problem of detecting collusive frauds in large-scale OANs as an incremental semi-supervised anomaly detection problem. In this technique, we propagate reputations from a small set of labeled benign users to unlabeled users along the auction relationships between them and then incrementally update reputations when a new auction gets added to the OAN. This increases the convergence of ICAFD and allows it to avoid wasteful recalculation of reputations from scratch. Our experimental results show that ICAFD can successfully detect different types of collusive auction frauds in a reasonable detection time.

Autoencoder based Semi-Supervised Anomaly Detection in Turbofan Engines

This paper proposes a semi-supervised autoencoder based approach for the detection of anomalies in turbofan engines. Data used in this research is generated through simulation of turbofan engines created using a tool known as Commercial Modular Aero-Propulsion System Simulation (CMAPSS). C-MAPSS allows users to simulate various operational settings, environmental conditions, and control settings by varying various input parameters. Optimal architecture of autoencoder is discovered using Bayesian hyperparameter tuning approach. Autoencoder model with optimal architecture is trained on data representing normal behavior of turbofan engines included in training set. Performance of trained model is then tested on data of engines included in test set. To study the effect of redundant features removal on performance, two approaches are implemented and tested: with and without redundant features removal. Performance of proposed models is evaluated using various performance evaluation metrics like F1-score, Precision and Recall. Results have shown that best performance is achieved when autoencoder model is used without redundant feature removal.

Segmentation Improved Label Propagation for Semi-Supervised Anomaly Detection in Fused Magnesia Furnace Process

This article introduces a semi-supervised solution to the practical task of identifying the semi-molten working condition for fused magnesium furnace. The classifier is trained under the semi-supervised learning framework using video data which are partially labeled, and it works on-line to classify the semi-molten working condition from the monitoring video of fused magnesium furnace. Firstly, a deep auto-encoder is used to extract features from the video. Next, the feature time series is segmented into subsequence through a bottom-up algorithm which maximizes the interior data correlation. Then, a graph based label propagation algorithm is employed to iteratively train a LSTM classifier using the subsequences as training samples and improve the accuracy of the LSTM classifier. The main contribution of this work is the novel segmentation algorithm for sequential data which can remarkably improve the classification accuracy. The advantages of the proposed method are demonstrated by experiments and comparative analysis based on industrial data.

An Integrated Model Based on O-GAN and Density Estimation for Anomaly Detection

Anomaly detection is a classic and crucial problem in the field of artificial intelligence, which aims to find instances that deviate so much from the main distribution of data or differ from known instances. This paper explores how to combine advanced deep learning techniques with traditional probabilistic statistical methods for anomaly detection. We propose a very effective and concise semi-supervised anomaly detection method named “ORGAN-KDE” based on the orthogonal generative adversarial network (O-GAN) and kernel density estimation. In the training phase, we use the encoder of O-GAN to learn the latent representation of normal data, namely the code of normal data, and then use the kernel density estimation to solve the probability density function of code. The code of normal sample obtained through the trained encoder can get a larger probability value when passing through the trained kernel density estimator, while the code of anomalous sample can get a smaller probability value, so as to achieve the purpose of anomaly detection. Compared with other anomaly detection methods based on GAN, our method has a very simple network structure, and experiments have proved that it performs well on both structured datasets and image datasets.

Detecting Gas Turbine Combustor Anomalies Using Semi-supervised Anomaly Detection with Deep Representation Learning

Deep learning (DL), regarded as a breakthrough machine learning technique, has proven to be effective for a variety of real-world applications. However, DL has not been actively applied to condition monitoring of industrial assets, such as gas turbine combustors. We propose a deep semi-supervised anomaly detection (deepSSAD) that has two key components: (1) using DL to learn representations or features from multivariate, time-series sensor measurements; and (2) using one-class classification to model normality in the learned feature space, thus performing anomaly detection. Both steps use normal data only; thus our anomaly detection falls into the semi-supervised anomaly detection category, which is advantageous for industrial asset condition monitoring where abnormal or faulty data is rare. Using the data collected from a real-world gas turbine combustion system, we demonstrate that our proposed approach achieved a good detection performance (AUC) of 0.9706 ± 0.0029. Furthermore, we compare the detection performance of the proposed approach against that of other different designs, including different features (i.e., the deep learned, handcrafted and PCA features) and different detection models (i.e., one-class ELM, one-class SVM, isolation forest, and Gaussian mixture model). The proposed approach significantly outperforms others. The proposed combustor anomaly detection approach is effective in detecting combustor anomalies or faults.

Gryphon: a semi-supervised anomaly detection system based on one-class evolving spiking neural network

The backbone of the economy, security and sustainability of a state is inseparably linked to the security of its critical infrastructure. Critical infrastructures define goods, systems or subsystems that are essential to maintain the vital functions of society, health, physical protection, security plus economic and social well-being of citizens. The digital security of critical infrastructures is a very important priority for the well-being of every country, especially nowadays, because of the direct threats dictated by the current international conjuncture and due to the emerging interactions or interconnections developed between the National Critical Infrastructures, internationally. The aim of this research is the development and testing of an Anomaly Detection intelligent algorithm that has the advantage to run very fast with a small portion of the available data and to perform equally well with the existing approaches. Such a system must be characterized by high efficiency and very fast execution. Thus, we present the Gryphon advanced intelligence system. Gryphon is a Semi-Supervised Unary Anomaly Detection System for big industrial data which is employing an evolving Spiking Neural Network (eSNN) One-Class Classifier (eSNN-OCC). This machine learning algorithm corresponds to a model capable of detecting very fast and efficiently, divergent behaviors and abnormalities associated with cyberattacks, which are known as Advanced Persistent Threat (APT). The training process is performed on data related to the normal function of a critical infrastructure.

A Novel and Efficient CVAE-GAN-Based Approach With Informative Manifold for Semi-Supervised Anomaly Detection

Semi-supervised anomaly detection identifies abnormal (testing) observations which are different from normal (training) observations. In many practical situations, anomalies are poorly insufficient and not well defined, while the normal data are easily sampled, have a wide variety, and may not be classified. For this paradigm, we propose a novel end-to-end deep network as an anomaly detector only trained on normal samples. Our architecture consists of a conditional variational auto-encoder (CVAE), a feature discriminator (FD), and an adversarially trained WGAN-GP discriminator. The CVAE is designed as a generator to reconstruct images. It leverages underlying category information and multivariate Gaussian distributions to regularize the latent space, enabling a smooth and informative manifold. For anomalies which have a certain similarity to normal data, we perform active negative training by generating potential outliers from the latent space to limit network generative capability. In order to capture data characteristics, we maximize the mutual information between the inputs and the latent codes by the FD. It enhances the relationship between the high-dimensional image space and corresponding encoded vectors. To promote reconstruction, a structural similarity loss is applied to robustly recover local texture details and the WGAN-GP discriminator is employed to aid in generating photo-realistic images. We distinguish anomalies by computing a reconstruction-based anomaly score. Different from recent encoder-decoder or GAN-based architectures, our approach considers input categories, constructs, and exploits a useful manifold in an unsupervised manner and has a stronger reconstruction capability. The experimental results demonstrate that the proposed approach outperforms state-of-the-art methods over several benchmark datasets.

Comparison analysis of electricity theft detection methods for advanced metering infrastructure in smart grid

While smart grid technologies are deployed to help achieve improved grid reliability and efficiency, energy companies are vulnerable to cyber-attacks leading to billions of dollars loss. The selection of an appropriate classification method for the electricity theft detection relies on operational requirements and resource constraints in real scenarios. Since unsupervised methods have high error rates, we employ a semi-supervised anomaly detection method [the principal component analysis (PCA)] technique for the electricity theft detection. PCA is compared with the peer-to-peer (P2P) method based on linear equations. The P2P method assumes known particular electricity theft patterns and in the absence of which, the P2P method detection system results in 100% false alarm. While PCA does not require any prior assumptions about the pattern of the electricity theft, 4% false alarm rate is observed. Our analysis shows an average of 45% improvement in the detection accuracy rate in comparison with the P2P method.

Comparison analysis of electricity theft detection methods for advanced metering infrastructure in smart grid

While smart grid technologies are deployed to help achieve improved grid reliability and efficiency, energy companies are vulnerable to cyber-attacks leading to billions of dollars loss. The selection of an appropriate classification method for the electricity theft detection relies on operational requirements and resource constraints in real scenarios. Since unsupervised methods have high error rates, we employ a semi-supervised anomaly detection method [the principal component analysis (PCA)] technique for the electricity theft detection. PCA is compared with the peer-to-peer (P2P) method based on linear equations. The P2P method assumes known particular electricity theft patterns and in the absence of which, the P2P method detection system results in 100% false alarm. While PCA does not require any prior assumptions about the pattern of the electricity theft, 4% false alarm rate is observed. Our analysis shows an average of 45% improvement in the detection accuracy rate in comparison with the P2P method.

An Overview of Deep Learning Based Methods for Unsupervised and Semi-Supervised Anomaly Detection in Videos

Videos represent the primary source of information for surveillance applications. Video material is often available in large quantities but in most cases it contains little or no annotation for supervised learning. This article reviews the state-of-the-art deep learning based methods for video anomaly detection and categorizes them based on the type of model and criteria of detection. We also perform simple studies to understand the different approaches and provide the criteria of evaluation for spatio-temporal anomaly detection.

Statistical Anomaly Detection in Human Dynamics Monitoring Using a Hierarchical Dirichlet Process Hidden Markov Model

Understanding of human dynamics has drawn attention to various areas. The wide spread of positioning technologies, such as GPS facilitates location information to be obtained with high spatio-temporal resolution as well as at low costs. By collecting individual location information in real time, monitoring of human dynamics has recently become possible and is expected to the area of dynamic traffic control. In this monitoring, detecting anomalous states of human dynamics become important. This research aims to define an anomaly detection problem of the human dynamics monitoring with time-series gridded population data and develop an anomaly detection method for this problem. According to the result of a review we have conducted, we discussed the characteristics of the anomaly detection in human dynamics monitoring and categorized our problem to a semi-supervised anomaly detection problem that detects contextual anomalies behind time-series data. We developed an anomaly detection method based on a sticky hierarchical Dirichlet process hidden Markov model, which is able to estimate the number of latent states according to the input data. Results of the experiment with synthetic data showed that our proposed method has good fundamental performance with respect to the detection rate. Through the experiments with real gridded population data, anomalies were detected when and where an actual social event had occurred.

A Hybrid Semi-Supervised Anomaly Detection Model for High-Dimensional Data.

Anomaly detection, which aims to identify observations that deviate from a nominal sample, is a challenging task for high-dimensional data. Traditional distance-based anomaly detection methods compute the neighborhood distance between each observation and suffer from the curse of dimensionality in high-dimensional space; for example, the distances between any pair of samples are similar and each sample may perform like an outlier. In this paper, we propose a hybrid semi-supervised anomaly detection model for high-dimensional data that consists of two parts: a deep autoencoder (DAE) and an ensemble k-nearest neighbor graphs- (K-NNG-) based anomaly detector. Benefiting from the ability of nonlinear mapping, the DAE is first trained to learn the intrinsic features of a high-dimensional dataset to represent the high-dimensional data in a more compact subspace. Several nonparametric KNN-based anomaly detectors are then built from different subsets that are randomly sampled from the whole dataset. The final prediction is made by all the anomaly detectors. The performance of the proposed method is evaluated on several real-life datasets, and the results confirm that the proposed hybrid model improves the detection accuracy and reduces the computational complexity.

STATISTICAL ANOMALY DETECTION FOR MONITORING OF HUMAN DYNAMICS

Abstract. Understanding of human dynamics has drawn attention to various areas. Due to the wide spread of positioning technologies that use GPS or public Wi-Fi, location information can be obtained with high spatial-temporal resolution as well as at low cost. By collecting set of individual location information in real time, monitoring of human dynamics is recently considered possible and is expected to lead to dynamic traffic control in the future. Although this monitoring focuses on detecting anomalous states of human dynamics, anomaly detection methods are developed ad hoc and not fully systematized. This research aims to define an anomaly detection problem of the human dynamics monitoring with gridded population data and develop an anomaly detection method based on the definition. According to the result of a review we have comprehensively conducted, we discussed the characteristics of the anomaly detection of human dynamics monitoring and categorized our problem to a semi-supervised anomaly detection problem that detects contextual anomalies behind time-series data. We developed an anomaly detection method based on a sticky HDP-HMM, which is able to estimate the number of hidden states according to input data. Results of the experiment with synthetic data showed that our proposed method has good fundamental performance with respect to the detection rate. Through the experiment with real gridded population data, an anomaly was detected when and where an actual social event had occurred.

FineSplice, enhanced splice junction detection and quantification: a novel pipeline based on the assessment of diverse RNA-Seq alignment solutions

Alternative splicing is the main mechanism governing protein diversity. The recent developments in RNA-Seq technology have enabled the study of the global impact and regulation of this biological process. However, the lack of standardized protocols constitutes a major bottleneck in the analysis of alternative splicing. This is particularly important for the identification of exon–exon junctions, which is a critical step in any analysis workflow. Here we performed a systematic benchmarking of alignment tools to dissect the impact of design and method on the mapping, detection and quantification of splice junctions from multi-exon reads. Accordingly, we devised a novel pipeline based on TopHat2 combined with a splice junction detection algorithm, which we have named FineSplice. FineSplice allows effective elimination of spurious junction hits arising from artefactual alignments, achieving up to 99% precision in both real and simulated data sets and yielding superior F1 scores under most tested conditions. The proposed strategy conjugates an efficient mapping solution with a semi-supervised anomaly detection scheme to filter out false positives and allows reliable estimation of expressed junctions from the alignment output. Ultimately this provides more accurate information to identify meaningful splicing patterns. FineSplice is freely available at https://sourceforge.net/p/finesplice/.

FRaC: a feature-modeling approach for semi-supervised and unsupervised anomaly detection

Anomaly detection involves identifying rare data instances (anomalies) that come from a different class or distribution than the majority (which are simply called "normal" instances). Given a training set of only normal data, the semi-supervised anomaly detection task is to identify anomalies in the future. Good solutions to this task have applications in fraud and intrusion detection. The unsupervised anomaly detection task is different: Given unlabeled, mostly-normal data, identify the anomalies among them. Many real-world machine learning tasks, including many fraud and intrusion detection tasks, are unsupervised because it is impractical (or impossible) to verify all of the training data. We recently presented FRaC, a new approach for semi-supervised anomaly detection. FRaC is based on using normal instances to build an ensemble of feature models, and then identifying instances that disagree with those models as anomalous. In this paper, we investigate the behavior of FRaC experimentally and explain why FRaC is so successful. We also show that FRaC is a superior approach for the unsupervised as well as the semi-supervised anomaly detection task, compared to well-known state-of-the-art anomaly detection methods, LOF and one-class support vector machines, and to an existing feature-modeling approach.