Secure and advanced unpacking using computer emulation

Abstract

The purpose of this article is firstly to present a secure unpacker which is specifically designed for a security analyst when studying viruses but also any anti-virus scanner. Such a tool is in fact required when assessing security requirements of an anti-virus scanner through a black box approach. During testing of anti-virus software, a security analyst needs to build virus populations required for several penetration tests. Virus unpacking is a first mandatory step before gaining the ability to apply obfuscation transformation or any information extraction algorithm on a viral set. A secure unpacker is also useful when checking security robustness against reverse engineering of any packed or protected security product. Several static and dynamic analysis tools already implement unpacking algorithms, but these often require human intervention and are not well designed to automatically unpack such a dangerous program as a virus. A new algorithm for automatically unpacking encrypted viruses is presented in this paper. Forensics techniques to reconstruct an unpacked executable and advanced heuristics are also presented in order to decrypt more sophisticated self-protected Malwares. We present several detection techniques which are specifically designed to deceive virtual machine monitors and discuss the security of our tool against these low-level viral attacks. Our secure unpacker figures among a set of several tools. We then present in this paper a proof-of-concept human analysis framework which implements most standard components of an anti-virus scanner (real-time scanner, emulator engine) and in addition proposes a reliable system for automatically gaining information about a virus and its interaction with the OS executive (stealth native API hooking), but focuses on human decision as a detection process without the same resource limitation constraint as product oriented anti-virus scanners. This framework is used as a basis/reference for the comparative analysis of security aspects of anti-virus scanners and deals with the robustness of their driver stack and the efficiency of their de-obfuscation and unpacking algorithms.