Targeted Malicious Email Detection Using Hypervisor-Based Dynamic Analysis and Ensemble Learning

Abstract

At present, email is still one of the most frequently used communication tools for organizations and individuals. With the leakage of personal privacy information, targeted malicious email (TME) is becoming a prominent targeted cyber attack vector in today's Internet. This type of attack often uses personal information, about an individual, group of individuals, or an organization, to make a TME more believable and personalized. TME is effective to penetrate email defense system because it is fundamentally difficult for traditional email security method to distinguish legitimate emails from malicious emails. And TMEs often contain malicious URLs or malicious attachments, which are extremely aggressive and destructive. In order to effectively deal with this new type of malicious email attack, this paper proposes a dynamic detection method for malicious email. We simulate the recipient opening the email in the virtual machine (VM), accessing the URL and activating the attachment. And we use the virtual machine introspection (VMI) and memory forensics analysis (MFA) technology to obtain the dynamic features of the email by the out-of-VM. Then we use AdaBoostM1 ensemble learning method and Voting combination strategy to combine three base classifiers such as BayesNet, SMO and J48 to build a powerful classification model for detecting TME attacks. The AdaBoostM1 classifier achieved the high detection rates, with an AUC of 0.997, true positive rate (TPR) of 0.997, and false positive rate (FPR) of 0.015. In addition, our proposed detection method is superior to the 56 anti-virus engines on VirusTotal and most of the existing research works.