AE‐Integrated: Real‐time network intrusion detection with Apache Kafka and autoencoder

Abstract

SummaryUnknown cyber‐attack detection in network traffic streams is challenging but crucial to ensure network security. It is observed that new security threats occur on a daily basis and make cyberspace vulnerable. In the literature, machine learning and deep learning‐based network intrusion detection systems have gained a lot of success but still face many challenges in detecting new security threats and unknown cyber‐attacks in real‐time. Additionally, high false alarm rates and real‐time detection in constantly evolving high‐dimensional network data streams are open issues for the research community. To address this issue, a DL‐based solution is developed to detect real‐time network anomalies in streaming data with high detection accuracy, precision, recall and low false negative and positive scenarios. The proposed novel algorithm, AE‐Integrated, is developed and evaluated on the latest CICIDS‐2017 dataset. The AE‐Integrated is updated with the newest network traffic data stream by the human administrator after a certain period to maintain its prediction accuracy for future inference. The simulation study is conducted with the Apache Kafka and Slack API to get real‐time anomaly alerts. Finally, we compared the result with recent state‐of‐the‐art research to evaluate the significance of the proposed algorithm. It is concluded that combining multiple lightweight autoencoders into a single large architecture provides optimal results. The accuracy, recall, and AUC of AE‐Integrated obtained are 99.54%, 99.53%, and 0.998, respectively.