Autonomous anomaly detection for streaming data

Abstract

Anomaly detection from data streams is a hotly studied topic in the machine learning domain. It is widely considered a challenging task because the underlying patterns exhibited by the streaming data may dynamically change at any time. In this paper, a new algorithm is proposed to detect anomalies autonomously for streaming data. The proposed algorithm is nonparametric and does not require any threshold to be preset by users. The algorithmic procedure of the proposed algorithm is composed of the following three complementary stages. Firstly, the potentially anomalous samples that represent highly different patterns from others are identified from data streams based on data density. Then, these potentially anomalous samples are clustered online using the evolving autonomous data partitioning algorithm. Finally, true anomalies are identified from these minor clusters with the least amounts of samples associated with them. Numerical examples based on three benchmark datasets demonstrated the potential of the proposed algorithm as a highly effective approach for anomaly detection from data streams.