Automatic Invariant Generation for Monitoring OS Kernel Integrity

Abstract

System administrators have used integrity checkers to prevent the system from malicious infections. Especially, checking the integrity of the kernel is important, since the infections of the kernel affect the entire system. Most of the previous works to prevent such infections rely on the developers or administrators to write specifications to detect them. Those works require high engineering cost and may incur vulnerabilities. The other previous works use virtualization techniques to trace the memory usage of the target system. However, they require hardware supports for the virtualization to avoid significant overhead. Most of embedded systems do not have such hardware supports. In addition, the overhead of the integrity checking affects all of the guest OSes, because they check integrity of the target OS in the virtualization layer. Therefore, they are difficult to be applied to multi-core environment.In this paper, we propose a method to generate the integrity checker automatically. The integrity checker runs on a virtualization layer and checks the integrity of kernel data structures of the target OS kernel from the outside of it. The virtualization layer does not require a special hardware support for the virtualization, because the integrity checker only reads memory area used by the target OS. Moreover, the integrity checker is executed as a guest OS, and thereforeit does not affect the entire system performance when it runs on multicore environment. The integrity checker checks the kernel data structures using invariants of them. In order to generate the invariants automatically, our system analyzes obtained kernel data structures. However, checking all of the kernel data structures is not feasible, since there are a lot of kernel data structures and an analyzer uses relationships with them to generate invariants. Therefore, our challenge is to reduce the target kernel data structures while avoiding false positives and false negatives as much as possible.