Network Traffic Monitoring System Based on Big Data Technology

Abstract

With the rapid growth of network traffic and the increasing rich methods of network attacks, traditional network traffic monitoring system cannot meet the requirements of data storage and query in real time. Therefore, how to monitor the large scale network traffic effectively has become an important challenge for network security management. Aiming at it, we propose a new network monitoring system where Netflow as the monitoring object based on big data technology, which has four main functions: it can use Filebeat to collect Netflow in real time; it transfers the data reliably based on Logstash; it stores the data in ElasticSearch, it analyzes and displays the data in real time through Kabana. The experimental results show that our system is capable of meeting millisecond responses to 100 million of Netflows. It can meet the requirements of real-time monitoring for large-scale network traffic, and provide the basis for network security control.