IFS: Intelligent flow sampling for network security–an adaptive approach

Abstract

SummaryIn order to cope with an increasing volume of network traffic, flow sampling methods are deployed to reduce the volume of log data stored for monitoring, attack detection and forensic purposes. Sampling frequently changes the statistical properties of the data and can reduce the effectiveness of subsequent analysis or processing. We propose two concepts that mitigate the negative impact of sampling on the data. Late sampling is based on a simple idea that the features used by the analytic algorithms can be extracted before the sampling and attached to the surviving flows. The surviving flows thus carry the representation of the original statistical distribution in these attached features. The second concept we introduce is that of adaptive sampling. Adaptive sampling deliberatively skews the distribution of the surviving data to over‐represent the rare flows or flows with rare feature values. This preserves the variability of the data and is critical for the analysis of malicious traffic, such as the detection of stealthy, hidden threats. Our approach has been extensively validated on standard NetFlow data, as well as on HTTP proxy logs that approximate the use‐case of enriched IPFIX for the network forensics. Copyright © 2015 John Wiley & Sons, Ltd.