Sabotaging the system boundary: A study of the inter-boundary vulnerability

Abstract

The hierarchy theory is the foundation of the modern computer system design. However, the interaction part between different system layers is usually the weak point of the system, which tends to have security flaws. When communicating across the system boundary, failure to enforce the required synchronization in the shared memory can cause data inconsistency of the communication partners. Especially when there is a privilege gap between different boundary sides, such data inconsistency can lead to security vulnerability and sabotage the trust boundary. In this paper, we propose the concept of inter-boundary vulnerability and give the first in-depth study of them. We investigate three typical boundaries in the system that inter-boundary vulnerabilities are prone to occur, including the kernel-user boundary, the hardware-OS boundary, and the VMM-guest OS boundary. Then, based on the investigation of 115 real-world vulnerability cases, we extract four vulnerability types and provide analysis for each type to illustrate the principle. Finally, we discuss the state-of-the-art techniques that are relevant to the detection, prevention, and exploitation of such vulnerabilities, aiming to light the future research on this topic.