M]allotROPism: a metamorphic engine for malicious software variation development

Abstract

For decades, code transformations have been a vital open problem in the field of system security, especially for cases like malware mutation engines that generate semantically equivalent forms of given malicious payloads. While there are abundant works on malware and on malware phylogenies classification and detection in general, the fundamental principles about malicious transformations to evade detection have been neglected. In the present work, we introduce a mutation engine, named [m]allotROPism, to generate malicious code deviations with equivalent semantics from a static-analysis point of view. To achieve this, we reduce the problem of generating semantically equivalent solutions of given assembly code into a decision problem, and we solve it with the aid of satisfiability modulo theories. Moreover, we leverage return-oriented programming techniques to alter the traditional execution control flow from text to stack memory segment. We have implemented our proposed mutation engine and evaluated its detection evasion capabilities. Results show that so far, our approach is undetectable against popular free and commercial anti-malware products. We release the implementation of [m]allotROPism as open source. Our intention is to provide a method to generate malware families for experimental purposes and inspire further state-of-the-art research in the field of malware analysis.