Memory behavior-based automatic malware unpacking in stealth debugging environment

Abstract

Malware analysts have to first extract hidden original code from a packed executable to analyze malware because most recent malware is obfuscated by a packer in order to disrupt analysis by debuggers and dis-assemblers. There are several studies on automatic extraction of hidden original code, which executes malware in an isolated environment, monitors write memory accesses and instruction fetches at runtime, determines if the code under execution is newly generated, then dumps specific memory areas into a file as candidates for the original code. However, the conventional techniques output many dump files as candidates for the original code when experiments are conducted on malware in the wild. Thus, manual identification of the true original code is needed. In this paper, we present “memory behavior-based unpacking,” an algorithm that automatically identifies the true original code from among many candidates depending on the change in the trend of accessed memory addresses before and after the dumping points. To achieve this algorithm, we have implemented Stealth Debugger, a virtual machine monitor for debugging and monitoring all memory accesses of a process without interruption by any anti-debug functions of the malware. We have evaluated our proposed system by using malware obfuscated by various common packers. The results show that our proposed system successfully finds the original entry points and obtains the original code of the malware.