AREP: an adaptive, machine learning-based algorithm for real-time anomaly detection on network telemetry data

Abstract

Abnormal behaviour detection is an essential task of real-time monitoring to secure the reliable operation of ICT infrastructures. This paper presents AREP, an adaptive, long short-term memory-based machine learning algorithm for real-time anomaly detection on network telemetry data. AREP is an improved version of Alter-Re^2, the direct predecessor algorithm developed by our research team. AREP introduces automatic tuning of its two key parameters and includes an offset compensation component to increase accuracy. Unfortunately, AREP and its predecessors perform well only on time series showing specific patterns. Thus, we propose also a data type classification method to identify patterns on which AREP performs best. Moreover, we use an extended range of metrics in our performance evaluations, including area under the curve (AUC). AUC computation is based on receiver operating characteristic (ROC) curves. However, generating ROC curves is not straightforward due to the inherent adaptive threshold technique used by AREP and its predecessors, so we had to develop a novel ROC curve generation approach for these algorithms. We show through rigorous experiments that on network time series following specific data patterns AREP overperforms its predecessors and produces similar or even better performance than other state-of-the-art algorithms.