Leveraging Reinforcement Learning and Generative Adversarial Networks to Craft Mutants of Windows Malware against Black-box Malware Detectors

Abstract

To build an effective malware detector, it is required to collect a diversity of malware samples and their evolution, since malware authors always try to evade detectors through strategies of malware mutation. So, this paper explores the ability to craft mutants of malware for gathering numerous mutated samples in training a machine learning (ML)-based malware detector. Specifically, we leverage Reinforcement Learning (RL) and Generative Adversarial Networks (GAN) to generate adversarial malware samples against ML-based detectors. The more we use this approach with different targeted antivirus and malware samples in training the RL agent as a malware mutator, the more it learns how to avoid black box malware detectors. The experimental results in real-world dataset indicate that RL can help GAN in crafting variants of malware with executability preservation to evade ML-based detectors and VirusTotal. Finally, this approach can be used as an automated tool for benchmarking the robustness of malware detectors against the metamorphic malwares.