Hardware-software collaboration for secure coexistence with kernel extensions

Abstract

Our society is dependent upon computer systems that are the target of a never-ending siege against their resources. One powerful avenue for exploitation is the operating system kernel, which has complete control of a computer system's resources. The current methodology for kernel design, which involves loadable extensions from third parties, facilitates compromises. Most of these extensions are benign, but in general they pose a threat to system trustworthiness: they run as part of the kernel and some of them can be vulnerable or malicious. This situation is paradoxical from a security point of view: modern OSes depend, and must co-exist, with untrustworthy but needed extensions. Similarly, the immune system is continuously at war against various types of invaders and, through evolution, has developed highly successful defense mechanisms. Collaboration is one of these mechanisms, where many players throughout the body effectively communicate to share attack intelligence. Another mechanism is foreign body co-existence with its microbiota. Remarkably, these properties are not leveraged in kernel defense approaches. Security approaches at the OS and virtual machine layers do not cooperate with each other or with the hardware. This paper advocates a new paradigm for OS defense based on close collaboration between an OS and the hardware infrastructure, and describes a hardware-software architecture realizing this vision. It also discusses the architecture design at the OS and hardware levels, including experimental results from an emulator-based prototype, and aspects of an ongoing hardware implementation. The emulator-based proof-of-concept prototype, Ianus, uses Linux as the OS and the Bochs x86 emulator as the architecture layer. It successfully minimized kernel extensions interactions with the original kernel. Its security was evaluated with real rootkits and benign extensions. Ianus' performance was analyzed with system and CPU benchmarks and it caused a small overhead to the system (approximately 12%).