Anomaly detection for high‐dimensional dynamic data stream using stacked habituation autoencoder and union kernel density estimator

Abstract

AbstractWith the rapid development of communication technology, various complex heterogeneous sensor network applications produce a large number of high‐dimensional dynamic data streams, which results in more difficult to anomaly detection than ever before. So, anomaly detection for high‐dimensional dynamic data streams is of a more and more challenging problem. This paper proposes a novel method for detecting anomalies in high‐dimensional dynamic data streams by utilizing several components. Firstly, it uses a stacked habituation autoencoder with habituation physiological mechanism to detect similarity anomalies more easily and capture feature relationships. Secondly, a union kernel density estimator with micro‐cluster is designed to improve online anomaly detection accuracy by estimating the data density. Lastly, candidate anomaly sets and a delayed processing approach are utilized to cope with conceptual drift and evolution in the data stream, allowing the system to adapt to changes in the data over time. Extensive experiments on four high‐dimensional dynamic data streams of the Internet of Things show that the proposed method is very effective.