Network Traffic Monitoring Algorithm Based on Big Data Analysis

Abstract

Due to the ever-increasing variety of network threats and the need to store and query data in real time, conventional network traffic monitoring systems are becoming inadequate. As a result, the task of figuring out how to properly monitor massive amounts of network traffic has risen to the forefront of the field of network security management. In order to achieve this goal, we have proposed a new network monitoring system that uses big data technology and uses Netflow as the monitoring object. This system has four primary functions: it can utilize Filebeat to collect Netflow in real time; it transmits the data safely based on Logstash; it stores the data in ElasticSearch; and it analyzes and presents the data in real time using Kibana. The results of our experiments demonstrate that our system can provide millisecond-level replies to 100 million Netflows. It’s able to give the foundation for network security management and fulfill the need for real-time monitoring of massive amounts of network traffic.