Real-Time Intrusion Detection in Network Traffic Using Adaptive and Auto-Scaling Stream Processor

Abstract

Advanced intrusion detection systems are beginning to utilize the power and flexibility offered by Complex Event Processing (CEP) engines. Adapting to new attacks and optimizing CEP rules are two challenges in this domain. Optimizing CEP rules requires a complete framework which can be ported to stream processors because a CEP rule cannot run without a stream processor. External dependencies of stream processors make CEP rule a black box which is hard to optimize. In this paper, we present a novel adaptive and functionally auto-scaling stream processor: “Wisdom” with a built-in hybrid optimizer developed using Particle Swarm Optimization, and Bisection algorithms to optimize CEP rule parameters. We show that an adaptive “Wisdom” rule tuned by the proposed optimization algorithm is able to detect selected attacks in CICIDS 2017 dataset with an average precision of 99.98% and an average recall of 93.42% while processing over 2.5 million events per second. The proposed distributed functionally auto-scaling deployment mode consumes significantly fewer system resources than the monolithic deployment of CEP rules.