Discovering anomaly on the basis of flow estimation of alert feature distribution

Abstract

ABSTRACTA challenge faced by many system administrators in utilizing the intrusion detection system (IDS) is to sift out genuine alerts buried with overwhelming alerts of benign activities generated by the IDS, especially for IDS deployed in large networks. Existing methods propose to identify the real alerts to aid the administrators. In our paper, we extend the idea of filtering irrelevant alerts based on alert volumes. And we formulate the flow estimation of abrupt changes in feature distribution caused by anomalies, by computing Kullback–Leibler distance of alert feature values under observation in comparison with a reference distribution, which is the mixture of a distribution drawing a tread from historical alerts, and a distribution derived from expertise provided by administrators. Experimental studies on the Defense Advanced Research Projects Agency dataset as well as real‐life data gathered from the IDS of a large network show that our method is able to distinguish and highlight genuine anomalies arising from the tremendous number of intrusion alerts, including different kinds of attacks and network failures. Application of this technique to alerts greatly helps the administrators in identifying real alerts and then reduces the alert load in the future. Copyright © 2013 John Wiley & Sons, Ltd.