VMGuard: A VMI-based Security Architecture for Intrusion Detection in Cloud Environment

Abstract

Cloud security is of paramount importance in the new era of computing. Advanced malware can hide their behavior on detection of the presence of a security tool at a tenant virtual machine (TVM). Hence, TVM-layer security solutions are not reliable. In this paper, we propose a Virtual Machine Introspection (VMI) based security architecture design for fine granular monitoring of the virtual machines to detect known attacks and their variants. We have developed techniques for monitoring the TVMs at the process level and system call level to detect attacks such as those based on malicious hidden processes, attacks that disable security tools in the virtual machines and attacks that alter the behavior of legitimate applications to access sensitive data. Our architecture, VMGuard , utilizes the introspection feature at the VMM-layer to analyze system call traces of programs running on TVM. VMGuard applies the software breakpoint injection technique which is OS agnostic and can be used to trap the execution of programs. Motivated by text mining approaches, VMGuard provides ‘Bag of n-grams (BonG)’ approach integrated with Term Frequency-Inverse Document Frequency (TF-IDF) method, to extract and select features of normal and attack traces. It then applies the Random Forest classifier to produce a generic behavior for different categories of intrusions of the monitored TVM. We have implemented a prototype and conducted a detailed analysis using University of New Mexico (UNM) datasets and a Windows malware dataset obtained from the University of California. The results obtained are promising and demonstrate the applicability of the VMGuard. We compare VMGuard with existing techniques and discuss its advantages.