Scalable Anomaly Detection Research Articles

ASTREAM: Data-Stream-Driven Scalable Anomaly Detection With Accuracy Guarantee in IIoT Environment

Intrusion detection exerts a crucial influence on securing the IIoT driven by anomaly detection approaches. Dissimilar with the static data, the intrusion detection data is in the form of a dynamic data stream possessing the properties of infiniteness, correlations, and data distribution change.However, these properties cause some issues for current anomaly detection approaches.Firstly, it is impractical to save the whole dataset due to the infiniteness.Secondly, the correlations are hardly considered.Thirdly, the data distribution change cant be appropriately handled due to a lack of model update and change detection strategy. Thus, we propose ASTREAM(anomaly detection in data streams), a novel anomaly detection approach that merges sliding window, model update, and change detection strategies into LSHiForest to achieve accurate and efficient anomaly detection with better scalability. ASTREAM has the following characteristics:(a)the sliding window can be utilized to handle the infiniteness of data streams;(b)the introduced PCA can consider the correlations between different attributes;(c)the change detection and model update can detect data distribution change in time and train the new model. Comprehensive experiments are implemented on the KDDCUP99 dataset to validate ASTREAM performance.Experiment results reveal that ASTREAM outperforms baselines in aspects of accuracy and efficiency and has better scalability.

Warping resilient scalable anomaly detection in time series

Time series data is ubiquitous in the real-world problems across various domains including healthcare, social media, and crime surveillance. Detecting anomalies, or irregular and rare events, in time series data, can enable us to find abnormal events in any natural phenomena, which may require special treatment. Moreover, labeled instances of anomaly are hard to get in time series data. On the other hand, time series data, due to its nature, often exhibits localized expansions and compressions in the time dimension which is called warping. These two challenges make it hard to detect anomalies in time series as often such warpings could get detected as anomalies erroneously. Our objective is to build an anomaly detection model that is robust to such warping variations. In this paper, we propose a novel unsupervised time series anomaly detection method, WaRTEm-AD, that operates in two stages. Within the key stage of representation learning, we employ data augmentation through bespoke time series operators which are passed through a twin autoencoder architecture to learn warping-robust representations for time series data. Second, adaptations of state-of-the-art anomaly detection methods are employed on the learnt representations to identify anomalies. We will illustrate that WaRTEm-AD is designed to detect two types of time series anomalies: point and sequence anomalies. We compare WaRTEm-AD with the state-of-the-art baselines and establish the effectiveness of our method both in terms of anomaly detection performance and computational efficiency.

Scalable anomaly detection in manufacturing systems using an interpretable deep learning approach

Anomaly detection in manufacturing systems has great potential for the prevention of critical quality faults. In recent years, unsupervised deep learning has shown to frequently outperform conventional methods for anomaly detection. However, tuning, deploying and debugging deep learning models is a time-consuming task, limiting their practical applicability in manufacturing systems. We approach this problem by developing a deep learning model that learns interpretable shapes that can be used for anomaly detection in temporal process data. Application of the model to assembly tightening processes in the automotive industry shows a significant improvement in model interpretability and scalability.

Detection o f Cyber Attack i n Broad Scale Smart Grids u sing Deep a nd Scalable Unsupervised Machine Learning System

The increase in the reliability, efficiency and security of the electrical grids was credited to the innovation of the smart grid. It is also a fact that the smart grids a very dependable on the digital communication technology that in turn gives rise to undiscovered weaknesses which have to be reconsidered for dependable and coherent power distribution. In this paper, we propose an unsupervised anomaly detection which is mainly focused the statistical correlation among the data. The main aim is to create a scalable anomaly detection system suitable for huge-scale smart grids, which are capable to denote a difference between a real fault from a disruption and an intelligent cyber-attack. We have presented a methodology that applies the concept of attribute extraction by the use of Symbolic Dynamic Filtering (SDF) to decrease compilation drift whilst uncovering usual interactions among subsystems. Results of simulation obtained on IEEE 39, 118 and 2848 bus systems confirm the execution of the method, proposed in this paper, under various working conditions. The results depict a precision of almost 99 percent, along with 98 percent of true positive rate and less than 2 percent of false positive rate.

Value-added tax fraud detection with scalable anomaly detection techniques

The tax fraud detection domain is characterized by very few labelled data (known fraud/legal cases) that are not representative for the population due to sample selection bias. We use unsupervised anomaly detection (AD) techniques, which are uncommon in tax fraud detection research, to deal with these domain issues. We analyse a unique dataset containing the VAT declarations and client listings of all Belgian VAT numbers pertaining to ten sectors. Our methodology consists in applying AD methods to firms belonging to the same sector and enables an efficient auditing strategy that can be adopted by tax authorities worldwide. The high lifts and hit rates observed in most sectors demonstrate the success of this approach. Sectoral differences exist due to varying market conditions and legal requirements across sectors and we show that the optimal AD method is sector dependent. We focus on three methodological problems that show issues in the related literature. (1) Can we design suitable input features? We develop new fraud indicators from specific fields of the VAT form and client listings and show the predictive value of the combination of these features. (2) Can we design fast algorithms to deal with the large data sizes that can occur in the tax domain? New methods are developed and we demonstrate their scalability both theoretically as well as empirically. (3) How should fraud detection performance be assessed? A new evaluation methodology is proposed that provides reliable performance indications and guarantees that fraud cases are effectively detected by the proposed methods.

Scalable Anomaly Detection for Large-Scale Heterogeneous Data in Cloud Using Optimal Elliptic Curve Cryptography and Gaussian Kernel Fuzzy C-Means Clustering

In most systems, a smart functionality is enabled through an essential vital service such as detecting anomalies from complex, large-scale and dynamic data. However, ensuring the privacy and security for the cloud data is the most crucial and challenging task in the present world. Moreover, it is important to safeguard the security of sensitive data and its privacy from unauthorized parties who are trying to access the data. Therefore, to accomplish this task, several encryption, decryption and key generation mechanisms were introduced in the existing works for privacy preserving in cloud platform. But, there still remain open issues such as increased communication overhead, reduced security and increased time consumption. Also, these existing works followed the symmetric key cryptographic mechanism for privacy preservation of data; hence, a single secret key is shared by several users for accessing the original data. Due to this fact, a high security risk arises and it allows unauthorized parties to access the data. Thus, this work introduces a cloud-based privacy preserving model for offering a scalable and reliable anomaly detection service for sensor data through holding the benefits of cloud resources. Also, this paper aims to impose a newly developed Elliptic Curve Cryptography-based Collective Decision Optimization (ECDO) approach over the proposed framework for improving the privacy and security of the data. Furthermore, to perform the data clustering computation we used the Gaussian kernel fuzzy [Formula: see text]-means clustering (GKFCM) algorithm within the cloud platform, especially for data partitioning and to classify the anomalies. Thus, the computational difficulties are limited by adopting this suitable privacy preserving model which collaborates a private server and a set of public servers through a cloud data center. Moreover, on encrypted data the granular anomaly detection operations are performed by the virtual nodes executed over public servers. Experimental validation was performed on four datasets resulting from Intel Labs publicly available sensor data. The experimental outcomes demonstrate the ability of the proposed framework in providing high anomaly detection accuracy without any degradation in data privacy.

A Deep and Scalable Unsupervised Machine Learning System for Cyber-Attack Detection in Large-Scale Smart Grids

Smart grid technology increases reliability, security, and efficiency of the electrical grids. However, its strong dependencies on digital communication technology bring up new vulnerabilities that need to be considered for efficient and reliable power distribution. In this paper, an unsupervised anomaly detection based on statistical correlation between measurements is proposed. The goal is to design a scalable anomaly detection engine suitable for large-scale smart grids, which can differentiate an actual fault from a disturbance and an intelligent cyber-attack. The proposed method applies feature extraction utilizing symbolic dynamic filtering (SDF) to reduce computational burden while discovering causal interactions between the subsystems. The simulation results on IEEE 39, 118, and 2848 bus systems verify the performance of the proposed method under different operation conditions. The results show an accuracy of 99%, true positive rate of 98%, and false positive rate of less than 2%

Streaming anomaly detection using randomized matrix sketching

Data is continuously being generated from sources such as machines, network traffic, application logs, etc. Timely and accurate detection of anomalies in massive data streams has important applications such as in preventing machine failures, intrusion detection, and dynamic load balancing. In this paper, we introduce a novel (unsupervised) anomaly detection framework which can be used to detect anomalies in a streaming fashion by making only one pass over the data while utilizing limited storage. We adapt ideas from matrix sketching to maintain, in a streaming model, a set of few orthogonal vectors that form a good approximate basis for all the observed data. Using this constructed orthogonal basis, anomalies in new incoming data are detected based on a simple reconstruction error test. We theoretically prove that our algorithm compares favorably with an offline approach based on expensive global singular value decomposition (SVD) updates. Additionally, we apply ideas from randomized low-rank matrix approximations to further speedup the algorithm. The experimental results show the effectiveness and efficiency of our approach over other popular scalable anomaly detection approaches.

A scalable anomaly detection and mitigation architecture for legacy networks via an OpenFlow middlebox

AbstractIn this paper, we investigate the applicability of inserting an OpenFlow middlebox to enhance the remotely triggered black hole routing mechanism, to mitigate distributed denial of service (DDoS) attacks in legacy networks. Specifically, we propose a modular architecture that exploits the network programmability of software‐defined networking within the context of network functions virtualization, deploying on‐demand virtualized network functions (VNFs) capable to manipulate and filter malicious traffic. Leveraging on the OpenFlow control functionality, we match and handle traffic on a per‐flow level, preserving connectivity to/from the victim while pushing the mitigation process upstream, towards the edge of the affected network. To that end, a multilevel anomaly detection and identification mechanism was developed, pinpointing the victim in case an attack is detected. Subsequently, a virtualized network function instructs the edge router to forward all traffic destined to the victim to an OpenFlow switch, acting as a middlebox capable to filter malicious traffic identified by an OpenFlow controller, while preserving benign flows. The proposed architecture was implemented and evaluated based on the combination of datasets containing traces of real DDoS attacks and normal background traffic from our university campus network. Our analysis illustrated a clear clustering of Internet protocol prefixes used by malicious sources; thus, we implemented a longest common prefix aggregation algorithm to enable scaling of the proposed mitigation process, overcoming constraints due to hardware limitations of OpenFlow devices. Our analysis verifies that the proposed modular and scalable schema can efficiently identify DDoS attack victims and filter malicious traffic, without exhausting system and network resources. Copyright © 2015 John Wiley & Sons, Ltd.

Scalable anomaly detection in graphs

Scalable anomaly detection in graphs

Scalable anomaly detection in large homogeneous populations

Anomaly detection in large populations is a challenging but highly relevant problem. It is essentially a multi-hypothesis problem, with a hypothesis for every division of the systems into normal and anomalous systems. The number of hypothesis grows rapidly with the number of systems and approximate solutions become a necessity for any problem of practical interest. In this paper we take an optimization approach to this multi-hypothesis problem. It is first shown to be equivalent to a non-convex combinatorial optimization problem and then is relaxed to a convex optimization problem that can be solved distributively on the systems and that stays computationally tractable as the number of systems increase. An interesting property of the proposed method is that it can under certain conditions be shown to give exactly the same result as the combinatorial multi-hypothesis problem and the relaxation is hence tight.