Holography: a behavior-based profiler for malware analysis

Abstract

Behavior-based detection and signature-based detection are two popular approaches to malware (malicious software) analysis. The security industry, such as the sector selling antivirus tools, has been using signature and heuristic-based technologies for years. However, this approach has been proven to be inefficient in identifying unknown malware strains. On the other hand, the behavior-based malware detection approach has a greater potential in identifying previously unknown instances of malicious software. The accuracy of this approach relies on techniques to profile and recognize accurate behavior models. Unfortunately, with the increasing complexity of malicious software and limitations of existing automatic tools, the current behavior-based approach cannot discover many newer forms of malware either. In this paper, we implement ‘holography platform’, a behavior-based profiler on top of a virtual machine emulator that intercepts the system processes and analyzes the CPU instructions, CPU registers, and memory. The captured information is stored in a relational database, and data mining techniques are used to extract information. We demonstrate the breadth of the ‘holography platform’ by conducting two experiments: a packed binary behavior analysis and a malvertising (malicious advertising) incident tracing. Both tasks are known to be very difficult to do efficiently using existing methods and tools. We demonstrate how the precise behavior information can be easily obtained using the ‘holography platform’ tool. With these two experiments, we show that the ‘holography platform’ can provide security researchers and automatic malware detection systems with an efficient malicious software behavior analysis solution. Copyright © 2011 John Wiley & Sons, Ltd.