Deriving common malware behavior through graph clustering

Abstract

Detection of malicious software (malware) continues to be a problem as hackers devise new ways to evade available methods. The proliferation of malware and malware variants requires new advanced methods to detect them. This paper proposes a method to construct a common behavioral graph representing the execution behavior of a family of malware instances. The method generates one common behavioral graph by clustering a set of individual behavioral graphs, which represent kernel objects and their attributes based on system call traces. The resulting common behavioral graph has a common path, called HotPath, which is observed in all the malware instances in the same family. The proposed method shows high detection rates and false positive rates close to 0%. The derived common behavioral graph is highly scalable regardless of new instances added. It is also robust against system call attacks.