Control Flow Graph Matching for Detecting Obfuscated Programs

Abstract

Malicious programs like the viruses, worms, Trojan horses, and backdoors infect host computers by taking advantage of flaws of the software and thereby introducing some kind of secret functionalities. The authors of these malicious programs attempt to find new methods to get avoided from detection engines. They use different obfuscation techniques such as dead code insertion, instruction substitution to make the malicious programs more complex. Initially, obfuscation techniques those are used by software developers to protect their software from piracy are now misused by these malware authors. This paper intends to detect such obfuscated programs or malware using control flow graph (CFG) matching technique, using VF2 algorithm. If the original CFG of the executable is found to be isomorphic to subgraph of obfuscated CFG (under examination), then it can be classified as an obfuscated one.