Checking virtual machine kernel control-flow integrity using a page-level dynamic tracing approach

Abstract

Kernel control-flow integrity (CFI) of virtual machines is very important to cloud security. VMI-based dynamic tracing and analyzing methods are promising options for checking kernel CFI in cloud. However, the CFI monitors based on tracing always work at instruction or branch level and result in serious virtual machine performance degradation. To meet the performance requirements in the cloud, we present a page-level dynamic VMI-based kernel CFI checking solution. We trace VM kernel execution at page level, which means that the in-page instruction execution cannot trigger our monitor. As a result, the tracing overhead can be greatly reduced. Based on page-level execution information, we propose two policies to describe the kernel control-flow so as to build the secure kernel control-flow database in the learning stage. In the monitoring stage, we compare runtime execution information with the secure database to check kernel CFI. To further reduce the monitoring overhead, we propose two performance optimization strategies. We implement the prototype on Xen and leverage hardware events to trace VM memory page execution. Then, we evaluate the effectiveness and performance of the prototype. The experimental results prove that our system has enough detection capability and the overhead is acceptable.