Internet Traffic Analysis Using Community Detection and Apache Spark

Abstract

With the rapid development of Internet, Internet traffic and end hosts continue to grow in size. Traffic behavior analysis for a large-scale network is becoming more and more difficult. To address these challenges, this paper proposes an Internet traffic analysis approach based on community detection to discover community consisted of end hosts with similar traffic behavior in a large campus network. First, we use only the IP-to-IP information without packet payloads to model the similarity of end hosts in campus network. Then the similarity graph which represent the social behavior similarity of all end hosts is constructed. Finally, we leverage Label Propagation algorithm to discover end hosts community on the similarity graph. To satisfy demands for the scalable analysis of ever-growing Internet traffic data, a Spark-based Internet traffic analysis system is developed, including implementing the above algorithm. The experimental results based on real campus network traffic show the benefits of the proposed approach in analyzing traffic behavior of a large-scale network on host community level and detecting potential anomalous traffic behavior. The proposed approach reduces the complexity of analyzing the traffic behavior of a large network compare with analyzing individual host. In addition, the experimental results also demonstrate the Spark-based Internet traffic analysis system can analyze Internet traffic efficiently.