MalHunter: Automatic generation of multiple behavioral signatures for polymorphic malware detection

Abstract

Malicious software, also called malware, is one of the major threats on the Internet today. Despite various antivirus programs, thousands of Internet hosts are daily infected with malware, such as viruses, worms, and Trojan horses. Due to using a variety of obfuscation techniques, polymorphic malware can easily evade signature-based detection techniques by continually changing their appearance or patterns. However, all polymorphic malware samples in the same malware family often follow the same behavioral pattern that can be used to generate a behavioral signature. In this paper, we propose MalHunter, a novel method based on sequence clustering and sequence alignment to automatic generation of behavioral signatures for polymorphic malware detection. We first generate a set of behavioral sequences for different samples of a polymorphic malware, each of which represents a thread's behavior. We then group similar behavioral sequences into the same cluster and generate an alignment pattern for each cluster. We finally build a multiple behavioral signature for the polymorphic malware. MalHunter stores fewer signatures in the signature database due to the generation of a multiple behavioral signature for different samples of each polymorphic malware. The experimental results on a malware collection suggest that MalHunter is both precise and succinct for effective matching and detection of polymorphic malware.