Designing in-VM-assisted lightweight agent-based malware detection framework for securing virtual machines in cloud computing

Abstract

The security of cloud services and underlying resources is a major concern due to vulnerabilities existing in current implementation of the virtualization. Thus, there is a need of detecting system-level attacks like viruses, worms, malware, etc. In this paper, we extend our previous work on vulnerability assessment and patching by integrating in-VM-assisted agent-based malware detection (AMD) framework for securing high-risk virtual machines (VMs) in cloud. The proposed framework has two components, viz. agent at VM and anomaly detection at hypervisor. An agent continuously looks for the new deployment of the executable in-VM and applies the signature-based detection to detect known malware. For detecting unknown attacks, it generates the profile with optimal static features for new executable. The optimal features are derived using an extended binary bat algorithm with two new fitness functions. The profile is transferred to hypervisor where anomaly detection using random forest classifier is applied. It classifies the executable to either normal or malware and generates an alert to VM user. The functionality of the proposed AMD framework is validated over cloud testbed at NIT Goa, as well as with the latest malware datasets. In addition, we analyze the VM security requirements fulfilled by the proposed framework.