A BasisEvolution framework for network traffic anomaly detection

Abstract

Traffic anomalies arise from network problems, and so detection and diagnosis are useful tools for network managers. A great deal of progress has been made on this problem so far, but most approaches can be thought of as forcing the data to fit a single mould. Existing anomaly detection methods largely work by separating traffic signals into “normal” and “anomalous” types using historical data, but do so inflexibly, either requiring a long description for “normal” traffic, or a short, but inaccurate description. In essence, preconceived “basis” functions limit the ability to fit data, and the static nature of many algorithms prevents true adaptivity despite the fact that real Internet traffic evolves over time. In our approach we allow a very general class of functions to represent traffic data, limiting them only by invariant properties of network traffic such as diurnal and weekly cycles. This representation is designed to evolve so as to adapt to changing traffic over time. Our anomaly detection uses thresholding approximation residual error, combined with a generic clustering technique to report a group of anomalous points as a single anomaly event. We evaluate our method with orthogonal matching pursuit, principal component analysis, robust principal component analysis and back propagation neural network, using both synthetic and real world data, and obtaining very low false-alarm probabilities in comparison.