Design of a Hypervisor-based Rootkit Detection Method for Virtualized Systems in Cloud Computing Environments

Abstract

Cloud computing is becoming increasingly popular. Many companies utilize cloud computing services to minimize IT infrastructure costs. The popularity of cloud computing has attracted the interest of cyber criminals. As the result, virtualized environments are a valid and attractive target for APT attacks. Since the key components in APT attacks are rootkit malware that provides stealth, detecting rootkits is an effective measure for protecting against APT attacks. Traditional rootkit detection algorithms are based on non- virtualized environments, where a detection agent tries to identify incoherency in OS system calls to detect rootkits. However, applying these algorithms to cloud computing environments entails installing a copy of the detection agent in every virtual machine, resulting in inefficient storage use and performance degradation. We propose a hypervisor-based, out-of-the-box rootkit detection system that takes cloud computing environments into consideration. The method utilizes vIPS platform to gain many beneficial traits including hypervisor-independency, agentless virtual security appliance structure, and usability. Therefore the method provides effective protection against rootkits in cloud computing environments.