Transparent Process Monitoring in a Virtual Environment

Abstract

PsycoTrace is a system that integrates static and dynamic tools to protect a process from attacks that alter the process self as specified by the program source code. The static tools build a context-free grammar that describes the sequences of system calls the process may issue and a set of assertions on the process state, one for each invocation. The dynamic tools parse the call trace of the process to check that it belongs to the grammar language and evaluate the assertions. This paper describes the architecture of PsycoTrace, which exploits virtualization to introduce two virtual machines, the monitored and the monitoring virtual machines, to increase both the robustness and the transparency of the monitoring because the machine that implements all the checks is strongly separated from the monitored one. We discuss the modification to the kernel of the monitored machine to trace system call invocations, the definition of the legal traces and the checks to prove the trace is valid. We describe how PsycoTrace applies introspection to evaluate the assertions and analyze the state of the monitored machine and of its data structures. Finally, we present the security and performance results of the dynamic tools, and the implementation of the static tools.