VMAttack

Abstract

We present VMAttack, a deobfuscation tool for virtualization-packed binaries based on automated static and dynamic analysis, which offers a simplified view of the disassembly. VMAttack is implemented as a plug-in for IDA Pro and as such, integrates seamlessly with manual reverse engineering. The complexity of the disassembly view is notably reduced by analyzing the inner working principles of the VM layer of protected binaries. Using static analysis, complex bytecode sequences of the VM are mapped to easy-to-read pseudo-code instructions, based on an intermediate representation specifically designed for stack-based virtual machines. Using dynamic analysis, we identify structural components like the interpreter loop and compress instruction sequences by filtering out semantically redundant instructions of the execution trace. The integrated result, which rates both static and dynamic analysis's results, provides the reverse engineer with a deobfuscated disassembly that tolerates weaknesses of a single analysis technique. VMAttack is currently limited to stack-based virtual machines like VMProtect. We evaluated VMAttack using binaries obfuscated with VMProtect and achieved an average execution trace reduction of 89.86% for the dynamic and 96.67% for the combined static and dynamic analysis.