Trusted system-calls analysis methodology aimed at detection of compromised virtual machines using sequential mining

Abstract

Most organizations today employ cloud-computing environments and virtualization technology; Due to their prevalence and importance in providing services to the entire organization, virtual-servers are constantly targeted by cyber-attacks, and specifically by malware. Existing solutions, consisting of the widely-used antivirus (AV) software, fail to detect newly created and unknown-malware; moreover, by the time the AV is updated, the organization has already been attacked. In this paper, we present a during run-time analysis methodology for a trusted detection of unknown malware on virtual machines (VMs). We conducted trusted analysis of volatile memory dumps taken from a VM and focused on analyzing their system-calls using a sequential-mining-method. We leveraged the most informative system-calls by machine-learning algorithms for the efficient detection of malware in widely used VMs within organizations (i.e. IIS and Email server). We evaluated our methodology in a comprehensive set of experiments over a collections of real-world, advanced, and notorious malware (both ransomware and RAT), and legitimate programs. The results show that our suggested methodology is able to detect the presence of unknown malware, in an average of 97.9% TPR and 0% FPR. Such results and capabilities can form the ground for the development of practical detection-tools for both corporates and companies.