Machine Learning and Deep Learning Based Model for the Detection of Rootkits Using Memory Analysis

Abstract

Rootkits are malicious programs designed to conceal their activities on compromised systems, making them challenging to detect using conventional methods. As the threat landscape continually evolves, rootkits pose a serious threat by stealthily concealing malicious activities, making their early detection crucial to prevent data breaches and system compromise. A promising strategy for monitoring system activities involves analyzing volatile memory. This study proposes a rootkit detection model that combines memory analysis with Machine Learning (ML) and Deep Learning (DL) techniques. The model aims to identify suspicious patterns and behaviors associated with rootkits by analyzing the contents of a system’s volatile memory. To train the model, a diverse dataset of known rootkit samples is employed, and ML and deep learning algorithms are utilized. Through extensive experimentation and evaluation using SVM, RF, DT, k-NN, and LSTM algorithms, it is determined that SVM achieves the highest accuracy rate of 96.2%, whereas Execution Time (ET) shows that k-NN depicts the best performance, and LSTM (a DL model) shows the worst performance among the tested algorithms. This research contributes to the development of advanced defense mechanisms and enhances system security against the constantly evolving threat of rootkit attacks.