Improving monolithic kernel security and robustness through intra-kernel sandboxing

Abstract

The structure of commodity operating systems kernels remains largely unchanged despite radical changes in underlying hardware and security risks. Existing research has managed to increase overall monolithic kernel security using various defense mechanisms, such as kernel control-flow integrity, and through the use of active vulnerability discovery techniques such as system call fuzzing. However, these mitigation mechanisms often focus on a class of vulnerabilities while failing to address the broader, underlying architectural issues which amplify the impact of these issues.This paper presents a novel architectural approach that aims to increase the robustness and security of monolithic operating system kernels. We propose an operating system model which focuses on strict decomposition and runtime separation between individual monolithic kernel subsystems through separate execution contexts. We propose a novel, SMP-capable nested kernel architecture that enforces separation policies in an effective, efficient and mechanism-agnostic manner, complemented by a special compiler pass and a domain-specific language that provides a handy and intuitive way of specifying separation policies and automating their integration.We implement a prototype system based on the FreeBSD operating system and the Clang/LLVM compiler. We run a series of intense benchmarks to evaluate our model and separation mechanisms.