Automatic Clustering of Malicious IP Flow Records Using Unsupervised Learning

Abstract

Anomaly based intrusion detection systems classify network traffic into normal and malicious categories. The intrusion detection system raises an alert when maliciousness is detected in the traffic. A security administrator inspects these alerts and takes corrective action to protect the network from intrusions and unauthorized access. Manual inspection of the alerts is also necessary because anomaly based intrusion detection systems have a high false positive rate. The alerts can be in very large number and their manual inspection is a challenging task. We propose an extension for anomaly based intrusion detection system which automatically groups malicious IP flows into different attack clusters. Our technique creates attack clusters from a training set of unlabeled IP flows using unsupervised learning. Every attack cluster consists of malicious IP flows which are similar to each other. We analyze IP flows in every cluster and assign an attack label to them. After the clusters are created, an incoming malicious IP flow is compared with all clusters and the label of the closest cluster is assigned to the IP flow. The intrusion detection system uses labeled flows to raise consolidated anomaly alert for a set of similar IP flows. This approach significantly reduces the overall number of alerts and also generates a high-level map of attack population. We use unsupervised learning techniques for automatic clustering of IP flows. Unsupervised learning is advantageous over supervised learning because the availability of a labeled training set for supervised learning is not always guaranteed. Three unsupervised learning techniques, k-means, self-organizing maps (SOM) and DBSCAN are considered for clustering of malicious IP flows. We evaluated our technique on a flow-based data-set containing different types of malicious flows. Experimental results show that our scheme gives good performance and places majority of the IP flows in correct attack clusters.