Control of system calls from outside of virtual machines

Abstract

A virtual machine monitor (VMM) can isolate virtual machines (VMs) for trusted programs from VMs for untrusted ones. The security of VMs for untrusted programs can be enhanced by monitoring and controlling the behavior of the VMs with security systems running in a VM for trusted programs. However, programs running outside of a monitored VM usually obtain only low-level events and states such as interrupts and register values. Therefore, it is not straight-forward for the programs to understand the high-level behavior of an operating system in a monitored VM and to control resources managed by the operating system. In this paper, we propose a security system that controls the execution of processes from the outside of VMs. It consists of a modified VMM and a program running in a trusted VM. The system intercepts system calls invoked in a monitored VM and controls the execution according to a security policy. To fill the semantic gap between low-level events and high-level behavior, the system uses knowledge of the structure of a given operating system kernel. The user creates the knowledge with a tool when building an operating system. We implemented the system using Xen, and measured the overhead through experiments using microbenchmarks and a benchmark for the Apache web server.