Intelligent Big Data Summarization for Rare Anomaly Detection

Abstract

Identifying interesting patterns from a huge amount of data is a challenging task across a wide range of application domain. Especially, for cyber security being able to identify rare types of network activities or anomalies from network traffic data (a.k.a. Big Data!) is an important but time-consuming data analysis task having moderate computing resources. Existing research has shown that it is possible to detect rare anomalies from the summarized version of big data. Therefore, summarization is an effective preprocessing function before applying anomaly detection techniques. This aim of this paper is to improve and quantify the scalability and accuracy of the anomaly detection techniques by using summarization. Hence, we propose a sampling-based summarization technique (SUCh: Summarization Using Chernoff-Bound) which is computationally effective than the existing techniques and also performs better in identifying rare anomalies from twelve benchmark network traffic datasets. The experimental results show that, instead of using original dataset, a summary of the data yields better performance in terms of true positive and false positive rates, when used for anomaly detection with less time required.