An Anomaly Detection Algorithm Based on Lossless Compression

Abstract

Anomaly detection is essential in network security. It has been researched for decades. Many anomaly detection methods have been proposed. Because of the simplicity of principles, statistical and Markovian methods dominate these approaches. However, their effectiveness is constrained by specific preconditions, which make them work for only appropriate data sets which satisfy their premises. Other than statistical and Markovian model, information theory provides a different perspective about anomaly detection. However, the computation of information theoretic measures is still based on statistics. In this paper, we present a novel, information theoretic anomaly detection framework. Instead of statistics, it employs lossless compression for measuring the information quantity, and detects outliers according to compression result. We also discuss the selection of underlying compression algorithm, and choose a grammar compression for utilizing the structure of data. With grammar compression, our method overcomes the shortcomings of statistical and Markovian methods. In addition, the implementation and operation of our method is even simpler than traditional approaches. We test our method on four data sets about text analyzing, host intrusion detection and bug detection. Experimental results show that, even traditional methods fail in some situations, our simple method works well in all cases.