Big data analytics for network anomaly detection from netflow data

Abstract

Cyber-attacks was organized in a simple and random way in the past. However attacks are carried out systematically and long term nowadays. In addition, the high calculation volume and continuous changes in network data distribution have made it more difficult to analyze data and detect abnormal behaviors within. For this reason, big data solutions have become essential. In this paper, firstly network anomaly and attack detection studies on big data has been reviewed. Then, a public big network data was analyzed with a new unsupervised anomaly detection approach on Apache Spark cluster in Azure HD Insight. Finally, the results obtained from a case study were evaluated, %96 accuracy was achieved. The results were visualized after dimension reduction using Principal Component Analysis (PCA). The identified anomalies may provide usable outputs to understand the behavior of the network, distinguishing the attacks, providing better cyber security, and protecting critical infrastructures.