Resurrecting anti-virtualization and anti-debugging: Unhooking your hooks

Abstract

Dynamic malware analysis involves the debugging of the associated binary files and the monitoring of changes in sandboxed environments. This allows the investigator to manipulate the code execution path and environment to develop an understanding of the malware’s internal workings, aims and modus operandi. However, the malware may incorporate anti-virtual environment (VM) and anti-debugging countermeasures (e.g. determining whether the malware is being executed in a VM, or using a debugger prior to payload execution). In essence, the malware needs to adopt a “defence in depth” paradigm. Beyond the malicious uses, software vendors seeking to preserve the intellectual property rights of their products often resort to similar methods to deter competitors from gaining intelligence from the binaries or prevent customers from using their products without unauthorization. In this work, we illustrate how the Windows architecture impedes the work of debuggers in the analysis of armoured binaries. The debugger and the malware have the same privileges, so the attacker may manipulate the address space that the debugger operates to bypass detection. We showcase this by presenting a new framework (ANTI), which automates the procedure of integrating anti-debugging and anti-VM in the binary. Specifically, ANTI introduces an anti-hooking method targeting Windows binaries, where hooks applied by state of the art debuggers are removed and injects its code in other processes. This significantly compounds the challenge of binary analysis. Our extensive evaluation also demonstrates that ANTI successfully circumvents detection from state-of-the-art detection methods. In other words, we show using ANTI that implementation gaps in current tools for dynamic analysis can be exploited to allow binaries to bypass them. More concerningly, ANTI shows how one can use well-known methods to “resurrect” old attacks.