Efficient and Robust Malware Detection Based on Control Flow Traces Using Deep Neural Networks

Abstract

Nowadays, the rapid growth of the number and variety of malware brings great security challenges. Machine learning has become a mainstream tool for effective malware detection, which can mainly be classified into static and dynamic analysis methods.The purpose of malware detection is to have a good and stable detection performance for different software forms. However, many static analysis methods are easily affected by packing and other code obfuscation techniques, and dynamic analysis methods are commonly believed more robust, while the impact of packing on them has received little attention. In addition, adversarial sample attacks against dynamic analysis methods have also been conducted. This indicates the need to investigate more accurate and robust malware classification methods.In this paper, we propose a new robust dynamic analysis method for malware detection by using specific fine-grained behavioral features, i.e., control flow traces. Further, a malware classifier is constructed by converting control flow traces into byte sequences and applying a combination of convolutional neural networks and long short term memory.The proposed classifier can effectively detect malware with an accuracy of up to 95.7%, as well as detect unseen malware with an accuracy of 94.6% (indicating a good performance in handling the evolution of malware). Meanwhile, it is first found experimentally that packing has specific interference with existing behavior-based malware classifiers, resulting in even worse performance than static classifiers in some cases. However, the proposed classifier performs well in terms of robustness, showing stable performance under the interference of an uneven packing distribution in the dataset, with a 26.3% higher true positive rate for unpacked samples compared to the API call-based classifier. In addition, the classifier is also more robust against adversarial samples, with a detection rate of at least 83%.