Detecting Malicious Behavior Using Critical API-Calling Graph Matching

Abstract

The proliferation of new malware in recent years has presented a serious security threat to our society. Research shows that variants of some known ones take a large amount of new malware, so one of the challenges in malware detection is how to find the similarities between known malware and its variants. Since API(Application Programming Interface)functions is used extensively to achieve the function of one program and It is difficult for different malware versions to conceal similarity on the functional flow level, making use of the similarity of their API-calling sequences is an essential detection method.In this paper, we present a new approach of malware detection based on critical API-calling Graph (CAG) matching rather than considering all API calls. More attention is paid on the illustration of how to extract a CAG from a control flow graph (CFG) for each malware to define a malicious behavior and how to detect the behavior in a suspicious executable using the CAG matching. This approach can overcome the drawbacks of two most common techniques adopted by current antivirus in detection of variants and unknown malware, which is demonstrated by the favorable experimental results.