A streaming-based network monitoring and threat detection system

Abstract

The unyielding trend of increasing cyber threats has made cyber security paramount in protecting personal and private intellectual property. In order to provide the most highly secured network environment, network traffic monitoring and threat detection systems must handle real-time data from varied and branching places in enterprise networks. Though numerous investigations have yielded real-time threat detection systems, in this paper we addressed the issue of handling the large volumes of network traffic data of enterprise systems, while simultaneously providing real-time monitoring and detection remain unsolved. Particularly, we introduced and evaluated a streaming-based threat detection system that can rapidly analyze highly intensive network traffic data in real-time, utilizing the streaming-based clustering algorithms to detect abnormal network activities. The developed system integrates the streaming and high-performance data analysis capabilities of Flume, Sharp, and Hadoop into a cloud-computing environment to provide network monitoring and intrusion detection. Our performance evaluation and experimental results demonstrate that the developed system can cope with a significant volume streaming data with high detection accuracy and good system performance.