Real-time behaviour profiling for network monitoring

Abstract

This paper presents the design and implementation of a real-time behaviour profiling system for internet links. The system uses flow-level information, and applies data mining and information-theoretic techniques to automatically discover significant events based on communication patterns. We demonstrate the operational feasibility of the system by implementing it and performing benchmarking of CPU and memory costs using packet traces from backbone links. To improve the robustness of this system against sudden traffic surges, we propose a novel filtering algorithm. The proposed algorithm successfully reduces the CPU and memory cost while maintaining high profiling accuracy. Finally, we devise and evaluate simple yet effective blocking strategies to reduce prevalent exploit traffic, and build a simple event analysis engine to generate ACL rules for filtering unwanted traffic.