VMInsight: Hardware Virtualization-Based Process Security Monitoring System

Abstract

Malicious software is one of the primary threats to information system on Internet, while the traditional host-based and network-based monitoring systems are vulnerable to prevent the malicious behavior of software because most current malicious software is capable of resisting security monitoring. Virtualization technology gives an impactful approach to monitoring the behavior of malicious software since it can provide an abstraction layer between the operating system and the hardware. In this paper, we propose a hardware-virtualization-based security monitor system named VMInsight, which can provide load-time and run-time monitoring for processes. VMInsight intercepts system calls and process behaviors by monitoring changes in the virtual machine CPU register, and it is implemented in the hyper visor, thus is completely transparent to the software and operating system running in the virtual machine. The experimental results show that the performance overhead of VMInsight is less than 10%, and it can be easily applied to the third-party security monitoring system.