Evolving anomaly detection for network streaming data

Abstract

Network security has always been a concern because it remains to be an unresolved problem. Unlike signature-based methods, anomaly-based methods can detect novel attacks and thus have gained increasing attention over the past decades. However, as the huge and unbounded network data samples continuously arrive at an unprecedented rate and always evolve and change, building a precise network normal pattern has become extremely difficult. In this study, an evolving anomaly detection method for network streaming data is proposed. Clusters are incrementally updated as the new network samples arrive at the incremental updating phase. The outliers, which include not only the global outliers but also the local outliers, are detected using the local density and global density thresholds at the anomaly detection phase. Meanwhile, a buffer is used to store temporary outliers, which may subsequently become normal samples, to avoid normal network samples being deleted as outliers.Three prominent streaming data (packet-based KDDCUP’99, NSL_KDD, and flow-based CIDDS-001) are used to validate the proposed algorithm. The detection rate of the proposed algorithm can achieve the best result. The result is nearly 100% on KDDCUP’99 and CIDDS-001. The false positive rate and accuracy are 0.0125 and 0.9886 on CIDDS-001, respectively. Experimental results indicate that the proposed algorithm can process real-time network anomaly detection with a much lower time and memory computational cost, and it outperforms other unsupervised anomaly detection methods and most supervised anomaly detection methods reported in the literature in terms of detection rate, false-positive rate, and detection accuracy.