A protective mechanism for the access control system in the virtual domain

Abstract

In traditional framework, mandatory access control (MAC) system and malicious software are run in kernel mode. Malicious software can stop MAC systems to be started and make it do invalid. This problem cannot be solved under the traditional framework if the operating system (OS) is comprised since malwares are running in ring0 level. In this paper, we propose a novel way to use hypervisors to protect kernel integrity and the access control system in commodity operating systems. We separate the access control system into three parts: policy management (PM), security server (SS) and policy enforcement (PE). Policy management and the security server reside in the security domain to protect them against malware and the isolation feather of the hypervisor can protect them from attacks. We add an access vector cache (AVC) between SS and PE in the guest OS, in order to speed up communication between the guest OS and the security domain. The policy enforcement module is retained in the guest OS for performance. The security of AVC and PE can be ensured by using a memory protection mechanism. The goal of protecting the OS kernel is to ensure the security of the execution path. We implement the system by a modified Xen hypervisor. The result shows that we can secure the security of the access control system in the guest OS with no overhead compared with modules in the latter. Our system offers a centralized security policy for virtual domains in virtual machine environments.