Nonvolatile kernel rootkit detection using cross‐view clean boot in cloud computing

Abstract

SummaryMalware attacks on kernel rootkits have become increasingly sophisticated and extremely difficult to detect; hence, they have a reign of power over the functionalities of the kernel. These kernel rootkits adopt stealth techniques to conceal the system processes, kernel modules, and other control structures, making it quite a challenge to detect their presence in the victim system. Many current efforts to detect the rootkits are based on known sources and are primarily system specific and hence are ineffective for newly mutating, hidden, and unknown rootkits. Therefore, in this paper, a kernel rootkit hidden file detection view (KRHFDV) system is proposed to detect such rootkits by identifying hidden files. This detection process uses a cross‐view clean‐boot‐based approach and defines a process monitoring framework that continuously maintains a list of active files and can detect both known and unknown rootkits with minimal performance overhead. KRHFDV overcomes the semantic gap by intercepting system call events of the tainted operating system in a nonintrusive manner and monitors the kernel to reconstruct a semantic‐level process information structure. The results from the extensive performance evaluation carried out with 64 rootkit samples in a cloud environment for both Linux and Windows kernels show that KRHFDV is able to identify file hiding behaviours of all samples in the least detection time.