MimicHunter: A General Passive Network Protocol Mimicry Detection Framework

Abstract

Network based intrusions and information theft events are becoming more and more popular today. To bypass the network security devices such as firewall, intrusion detection/prevention system (IDS/IPS) and web application firewall, attackers use evasive techniques to circumvent them, of which protocol mimicry is a very useful approach. The technique camouflages malicious communications as common protocols or generally innocent applications to avoid network security audit, which has been widely used in advanced Trojans, botnets, as well as anonymous communication systems, bringing a great challenge to current network management and security. To this end, we propose a general network protocol mimicry behavior discovery framework named MimicHunter to detect such evasive masquerade behaviors, which exploits protocol structure and state transition verifications, as well as primary protocol behavior elements. Experiment results on several datasets demonstrate the effectiveness of our method in practice. Besides, MimicHunter is flexible in deployment and can be easily implemented in passive detection systems with only a little cost compared with the active methods.