Unsupervised field segmentation of unknown protocol messages

Abstract

In network security systems working on intrusion detection, deep packet inspection, and protocol fuzzing, protocol specifications analyzed by Protocol Reverse Engineering(PRE) play an important role as fundamental input. For binary protocols having fixed-length fields, the location of those field boundaries has great impact on the subsequent analysis as well as the final performance. In this paper, we discuss the field segmentation problem formally, and develop a reasonable method ProSeg by introducing and optimize statistics(self-information and mutual information) from Information Theory. By analyzing the format structure of messages from unknown protocol vertically, the boundaries of fixed-length fields could be located by an expert voting strategy successfully. In experiments and analysis on several common protocols, our method turns out to be effective relatively and the results of ProSeg are consistent with standard segmentations to a great extent.