Detecting Kernel Control-Flow Modifying Rootkits

Abstract

Kernel Control-flow Modifying Rootkits are the most common kernel rootkits and pose the most threat to system security. Existing host-based and Virtual Machine Monitor (VMM) based techniques have limitations in security and suffer from system performance overhead. We propose a VMM-based framework to detect control-flow modifying kernel rootkits in a guest Virtual Machine (VM) by checking the number of certain hardware events that occur during the execution of a system call. Our technique leverages the Hardware Performance Counters (HPCs) to securely and efficiently count the monitored hardware events. By using HPCs, the checking cost is significantly reduced and the temper-resistance is enhanced.