Collective Anomaly Detection Research Articles

ASAP: Automatic Synthesis of Attack Prototypes, an online-learning, end-to-end approach

Zero-day attack detection and categorization is an open-research field where four main context factors need to be taken into account: novel or zero-day attacks (i) are unlabeled by definition, (ii) may correspond to out-of-distribution data, (iii) can arise concurrently, and (iv) distribution shifts in the feature space need online-learning. Given such constraints, the online detection and categorization of new cyber threats can be modeled as a heterogeneous collective anomaly detection problem, for which no online-learning solutions exist purely based on back-propagation. To this respect, this paper presents an online-learning, end-to-end back-propagation strategy for Automatically Synthesizing the potential signatures or Attack Prototypes of novel cyber threats (asap). The presented framework incorporates automatic feature engineering, operating over raw data from the OpenFlow monitoring API and raw bytes of traffic captures. In asap, specialized inductive biases enhance the training data efficiency and accommodate the inference machinery to resource-constrained scenarios such as the Internet of Things. Finally, the validity of this framework is demonstrated in a live training experiment comprising IoT traffic emulation 33To foster research in the field, the source-code of asap alongside instructions to reproduce the experiments are made publicly available among the set of SmartVille NID models in [1]..

Erratum to “Clustering-Based Granular Representation of Time Series With Application to Collective Anomaly Detection”

Erratum to “Clustering-Based Granular Representation of Time Series With Application to Collective Anomaly Detection”

A Process Mining-based unsupervised Anomaly Detection technique for the Industrial Internet of Things

Industrial Internet of Things (IIoT) applications in Industry 4.0 collect and process Time Series (TS) originating from heterogeneous sources. Many data-driven techniques have been proposed over the years for unsupervised collective Anomaly Detection (AD) to detect anomalous TS and improve quality of service. These techniques build statistical and/or behavioral models through data-driven algorithms often based on machine and deep learning. However, these algorithms may have black-box behavior, may require too much computational effort, and may not be trustworthy. In order to address these challenges, this paper proposes and evaluates an unsupervised AD technique that pre-processes noisy and complex TS and detects anomalous patterns. Pre-processing handles TS complexity through Feature Extraction and Dimensionality Reduction based on Autoencoders (AEs), whereas anomalous patterns are classified through Process Mining (PM), which unsupervisedly captures TS patterns and compares unknown behavior to such patterns for collective AD. We apply the technique to a scale replica of an assembly plant adopted in a smart automotive factory to evaluate our technique with respect to several configurations to analyze their impact on both detection and timing performance. Our contribution improves and extends the existing state-of-the-art work in the literature regarding the application of PM to IIoT for collective AD in TS. Moreover, it compares the obtained results with a baseline approach based on AEs.

A Survey of Collective Anomaly Detection on Sequence Dataset

Anomaly detection on sequence dataset typically focuses on the detection of collective anomalies, aiming to find anomalous patterns consisting of sequences of data with specific relationships rather than individual observations. In this survey, existing studies are summarized to align with temporal sequence dataset and spatial sequence dataset. For the first category, the detection can be subdivided into symbolic dataset based and time series dataset based, which include similarity, probabilistic, and trend approaches. For the second category, it can be subdivided into homogeneous datasets based heterogeneous datasets based, which include multi-dataset fusion and joint approaches. Compared to the state-of-the-art survey papers, the contribution of this paper lies in providing a deep analysis of various representations of collective anomaly in different application field and their corresponding detection methods, representative techniques. As a result, practitioners can receive some guidance for selecting the most suitable methods for their particular case.

A new collective anomaly detection approach using pitch frequency and dissimilarity: Pitchy anomaly detection (PAD)

Anomaly detection in time series is an important process that can aid in both preprocessing and postprocessing, particularly in biomedical data modalities where anomalies often signify the presence of disorders that require identification. However, quasi-periodic data behavior frequently poses a challenge in detecting collective anomalies, especially when they fall within the normal data range. This difficulty makes detecting collective anomalies particularly challenging. This paper proposes a new method for detecting collective anomalies using pitch frequency information from the frequency domain and dissimilarity metric scores in a sliding window approach. The proposed method called pitchy anomaly detection (PAD) is expected to be effective in the analysis of quasi-periodic time series data. To evaluate the effectiveness of the PAD approach, it is compared to three other benchmark time series anomaly detection methods. Due to the particular interest in quasi-periodic data modalities, the comparative analysis is conducted under five different anomalous conditions on anomaly-added benchmark electrocardiography (ECG) datasets. The results show that the PAD approach yields promising classification performance results for detecting collective anomalies.

Clustering-Based Granular Representation of Time Series With Application to Collective Anomaly Detection

Clustering-Based Granular Representation of Time Series With Application to Collective Anomaly Detection

A Collective Anomaly Detection Technique to Detect Crypto Wallet Frauds on Bitcoin Network

The popularity and remarkable attractiveness of cryptocurrencies, especially Bitcoin, absorb countless enthusiasts every day. Although Blockchain technology prevents fraudulent behavior, it cannot detect fraud on its own. There are always unimaginable ways to commit fraud, and the need to use anomaly detection methods to identify abnormal and fraudulent behaviors has become a necessity. The main purpose of this study is to use the Blockchain technology of symmetry and asymmetry in computer and engineering science to present a new method for detecting anomalies in Bitcoin with more appropriate efficiency. In this study, a collective anomaly approach was used. Instead of detecting the anomaly of individual addresses and wallets, the anomaly of users was examined. In addition to using the collective anomaly detection method, the trimmed_Kmeans algorithm was used for clustering. The results of this study show the anomalies are more visible among users who had multiple wallets. The proposed method revealed 14 users who had committed fraud, including 26 addresses in 9 cases, whereas previous works detected a maximum of 7 addresses in 5 cases of fraud. The suggested approach, in addition to reducing the processing overhead for extracting features, detect more abnormal users and anomaly behavior.

Network traffic analysis over clustering-based collective anomaly detection

Due to the ever-growing presence of network traffic, there has been a considerable amount of research on anomaly detection in network traffic by clustering. Most of them have not considered the problem that collective anomaly detection in network traffic. Collective anomaly might scatter among multiple clusters when applying the clustering-based algorithms in the anomaly detection. In this paper, we propose a progressive exploration framework for collective anomaly detection on network traffic based on a clustering method, called CCAD. CCAD enables analysts to effectively explore collective anomaly in network traffic. This framework is different from the other anomaly detection methods. It is based on the analysis of the influence of collective anomaly on the clustering results in the network traffic stream data. CCAD framework efficiently supports the collective anomaly exploration. As demonstrated by our extensive experiments with real-world data, CCAD has high detection rate in comparison with other existing methods.

CSCAD: Correlation Structure-based Collective Anomaly Detection in Complex System

Detecting anomalies in large complex systems is a critical and challenging task. The difficulties arise from several aspects. First, collecting ground truth labels or prior knowledge for anomalies is hard in real-world systems, which often lead to limited or no anomaly labels in the dataset. Second, anomalies in large systems usually occur in a collective manner due to the underlying dependency structure among devices or sensors. Lastly, real-time anomaly detection for high-dimensional data requires efficient algorithms that are capable of handling different types of data (i.e. continuous and discrete). We propose a correlation structure-based collective anomaly detection (CSCAD) model for high-dimensional anomaly detection problem in large systems, which is also generalizable to semi-supervised or supervised settings. Our framework utilize graph convolutional network combining a variational autoencoder to jointly exploit the feature space correlation and reconstruction deficiency of samples to perform anomaly detection. We propose an extended mutual information (EMI) metric to mine the internal correlation structure among different data features, which enhances the data reconstruction capability of CSCAD. The reconstruction loss and latent standard deviation vector of a sample obtained from reconstruction network can be perceived as two natural anomalous degree measures. An anomaly discriminating network can then be trained using low anomalous degree samples as positive samples, and high anomalous degree samples as negative samples. Experimental results on five public datasets demonstrate that our approach consistently outperforms all the competing baselines.

Subset Multivariate Collective and Point Anomaly Detection

In the recent years, there has been a growing interest in identifying anomalous structure within multivariate data sequences. We consider the problem of detecting collective anomalies, corresponding to intervals where one, or more, of the data sequences behaves anomalously. We first develop a test for a single collective anomaly that has power to simultaneously detect anomalies that are either rare, that is affecting few data sequences, or common. We then show how to detect multiple anomalies in a way that is computationally efficient but avoids the approximations inherent in binary segmentation-like approaches. This approach is shown to consistently estimate the number and location of the collective anomalies—a property that has not previously been shown for competing methods. Our approach can be made robust to point anomalies and can allow for the anomalies to be imperfectly aligned. We show the practical usefulness of allowing for imperfect alignments through a resulting increase in power to detect regions of copy number variation. Supplemental files for this article are available online.

A Novel Vehicle Gearbox Fault Diagnosis Approach Based on Collective Anomaly Detection

Targeting the problem of gearbox fault diagnosis, we proposed a novel semi-supervised approach based on collective anomaly detection. Based on the limited sample data, the principle of the approach is to detect whether a test dataset contains abnormal patterns by using data distribution as the metric. The sequence obeying unexpected distribution will be identified as collective anomaly, which may be generated by fault patterns. The approach consists of three steps. First, the mixture of multivariate Gaussian distribution is used to fit the structure of sample dataset and test dataset. Then, based on maximum likelihood estimate algorithm, we hope to search the optimal parameters which can fit the data distribution with the highest degree. Finally, the fixed point iteration algorithm is used to solve likelihood estimate functions. Experimental results demonstrate that the proposed approach can be used to find fault patterns of gearbox without the prior knowledge of their generated mechanisms.

Random Partitioning Forest for Point-Wise and Collective Anomaly Detection—Application to Network Intrusion Detection

In this paper, we propose DiFF-RF, an ensemble approach composed of random partitioning binary trees to detect point-wise and collective (as well as contextual) anomalies. Thanks to a distance-based paradigm used at the leaves of the trees, this semi-supervised approach solves a drawback that has been identified in the isolation forest (IF) algorithm. Moreover, taking into account the frequencies of visits in the leaves of the random trees allows to significantly improve the performance of DiFF-RF when considering the presence of collective anomalies. DiFF-RF is fairly easy to train, and excellent performance can be obtained by using a simple semi-supervised procedure to setup the extra hyper-parameter that is introduced. We first evaluate DiFF-RF on a synthetic data set to i) verify that the limitation of the IF algorithm is overcome, ii) demonstrate how collective anomalies are actually detected and iii) to analyze the effect of the meta-parameters it involves. We assess the DiFF-RF algorithm on a large set of datasets from the UCI repository, as well as two benchmarks related to intrusion detection applications. Our experiments show that DiFF-RF almost systematically outperforms the IF algorithm, but also challenges the one-class SVM baseline and a deep learning variational auto-encoder architecture. Furthermore, our experience shows that DiFF-RF can work well in the presence of small-scale learning data, which is conversely difficult for deep neural architectures. Finally, DiFF-RF is computationally efficient and can be easily parallelized on multi-core architectures.

Deep learning for collective anomaly detection

Deep learning has been performing well in a number of application domains. Inspired by its popularity in domains such as image processing, speech recognition, etc., in this paper we explore the effectiveness of deep learning and other supervised learning algorithms for collective anomaly detection. Recently, collective anomaly has become popular for denial of service (DoS) attack detection, however, all these approaches are unsupervised in nature and often have high false alarm rate due to being unsupervised. Therefore, to reduce the false alarm rates, we have experimented using the deep learning method which is supervised in nature. Our experimental results on UNSW-NB15 and KDD Cup 1999 datasets show that the deep learning implemented using H2O achieves ≈97% recall for collective anomaly detection. Deep learning outperforms a wide range of unsupervised techniques for collective anomaly detection. The key insight of this paper is to report the efficiency of deep learning for collective anomaly detection. To the best of our knowledge, this paper is the first one to address the collective anomaly detection problem using deep learning.

A novel approach to identify regional fault of urban power grid based on collective anomaly detection

As a classical data form, the collective anomaly is used to describe the abnormality which cannot be identified by individual data. According to the data characteristics of current signals in the urban power grid, this paper proposes a novel detection approach, which transforms the diagnosis of regional fault into the detection of collective anomaly from the data of current fluctuation signal. Besides, in the proposed approach, an improved multi-layered clustering algorithm based on fixed point iteration (FPIML-clustering algorithm) is designed to enhance the detection efficiency. The experiment is tested on the power grid operation data of a Chinese city. The results demonstrate that the proposed approach can be used to detect regional faults before they reveal obvious fault characteristics.

A novel approach to identify regional fault of urban power grid based on collective anomaly detection

As a classical data form, the collective anomaly is used to describe the abnormality which cannot be identified by individual data. According to the data characteristics of current signals in the urban power grid, this paper proposes a novel detection approach, which transforms the diagnosis of regional fault into the detection of collective anomaly from the data of current fluctuation signal. Besides, in the proposed approach, an improved multi-layered clustering algorithm based on fixed point iteration (FPIML-clustering algorithm) is designed to enhance the detection efficiency. The experiment is tested on the power grid operation data of a Chinese city. The results demonstrate that the proposed approach can be used to detect regional faults before they reveal obvious fault characteristics.

Deep learning for collective anomaly detection

Deep learning for collective anomaly detection

A Collective Anomaly Detection Approach for Multidimensional Streams in Mobile Service Security

Anomaly detection in many applications is becoming more and more important, especially for security and privacy in mobile service computing domains with the development of mobile internet and mobile cloud computing, in which data are typical multidimensional time series data. However, the collective anomaly detection for multidimensional streams exists lots of problems, owing to the differences between the anomaly detection in multidimensional time series and univariate time series data. For example, the temporal continuity of multidimensional time series is much weaker than univariate time series and it is unreasonable to judge the entire multidimensional data as an anomaly if a certain dimension is abnormal. In this paper, we consider the statistical features of the subsequence of streams, proposing a novel collective anomaly detection algorithm for multidimensional streams based on iForest in a cloud environment, namely iForestFS. When using different features about mobile cloud services' metrics suggested by domain knowledge, iForestFS could detect different kinds of anomalies for mobile service security. Furthermore, we implement a distributed iForestFS using spark framework in order to improve time performance and scalability. The experimental results performed on three datasets (mainly about network security) derived from the UCI repository demonstrate that the proposed algorithm can effectively detect a collective anomaly of multidimensional streams in the security domain.

Novel Approach for Network Traffic Pattern Analysis using Clustering-based Collective Anomaly Detection

There is increasing interest in the data mining and network management communities in improving existing techniques for the prompt analysis of underlying traffic patterns. Anomaly detection is one such technique for detecting abnormalities in many different domains, such as computer network intrusion, gene expression analysis, financial fraud detection and many more. Clustering is a useful unsupervised method for both identifying underlying patterns in data and anomaly detection. However, existing clustering-based techniques have high false alarm rates and consider only individual data instances for anomaly detection. Interestingly, there are traffic flows which seem legitimate but are targeted at disrupting a normal computing environment, such as the Denial of Service (DoS) attack. The presence of such anomalous data instances explains the poor performances of existing clustering-based anomaly detection techniques. In this paper, we formulate the problem of detecting DoS attacks as a collective anomaly which is a pattern in the data when a group of similar data instances behave anomalously with respect to the entire dataset. We propose a framework for collective anomaly detection using a partitional clustering technique to detect anomalies based on an empirical analysis of an attack’s characteristics. We validate our approach by comparing its results with those from existing techniques using benchmark datasets.