A Similarity based Technique for Detecting Malicious Executable files for Computer Forensics

Abstract

With the rapidly increasing complexity of computer systems and the sophistication of hacking tools and techniques, there is a crucial need for computer forensic analysis techniques. Very few techniques exist to support forensic analysis of unknown executable files. The existing techniques primarily inspect executable files to detect known signatures or are based on metadata information. A key goal of such forensic investigation is to identify malicious executable files that hackers might have installed in a targeted system. Finding such malware in a compromised system is difficult because it is hard to identify the purpose of the fragments of executable files. In this paper, we present a similarity-based technique that analyzes targeted executable files to identify a malware present in a compromised system. The technique involves assigning a similarity value to the fragments of executable files present in a compromised hard disk against a set of source files. We present some results based on the comparison of assembly instruction sequences of well-known hacking tools with those of various executable files, and suggest various ways to reduce the false positives.