SALAD: A split active learning based unsupervised network data stream anomaly detection method using autoencoders

Abstract

Machine learning based intrusion detection systems monitor network data streams for cyber attacks. Challenges in this space include detecting unknown attacks, adapting to changes in the data stream such as changes in underlying behavior, the human cost of labeling data to retrain the machine learning model and the processing and memory constraints of a real-time data stream. Failure to manage the aforementioned factors could result in missed attacks, degraded detection performance, unnecessary expense or delayed detection times. This research proposes a new semi-supervised network data stream anomaly detection method, Split Active Learning Anomaly Detector (SALAD), which combines our novel Adaptive Anomaly Threshold and Stochastic Anomaly Threshold with Fading Factor methods. A novel Reconstruction Error based Distance from Threshold strategy is proposed and evaluated as part of an active stream framework to demonstrate reduction in labeling costs. The proposed methods are evaluated with the KDD Cup 1999, and UNSW-NB15 data sets, using the scikit-multiflow framework. Results demonstrated that the proposed SALAD method offered equivalent performance to full labeled and alternative Naïve Bayes (NB) and Hoeffding Adaptive Tree (HAT) methods, with a labeling budget of just 20%, significantly reducing the required human expertise to annotate the network data. Processing times of the SALAD method were demonstrated to be significantly lower than NB and HAT methods, allowing for greatly improved responsiveness to attacks occurring in real time.