Deep-Hook: A trusted deep learning-based framework for unknown malware detection and classification in Linux cloud environments

Abstract

Since the beginning of the 21st century, the use of cloud computing has increased rapidly, and it currently plays a significant role among most organizations’ information technology (IT) infrastructure. Virtualization technologies, particularly virtual machines (VMs), are widely used and lie at the core of cloud computing. While different operating systems can run on top of VM instances, in public cloud environments the Linux operating system is used 90% of the time. Because of their prevalence, organizational Linux-based virtual servers have become an attractive target for cyber-attacks, mainly launched by sophisticated malware designed at causing harm, sabotaging operations, obtaining data, or gaining financial profit. This has resulted in the need for an advanced and reliable unknown malware detection mechanism for Linux cloud-based environments. Antivirus software and today’s even more advanced malware detection solutions have limitations in detecting new, unseen, and evasive malware. Moreover, many existing solutions are considered untrusted, as they operate on the inspected machine and can be interfered with, and can even be detected by the malware itself, allowing malware to evade detection and cause damage. In this paper, we propose Deep-Hook, a trusted framework for unknown malware detection in Linux-based cloud environments. Deep-Hook hooks the VM’s volatile memory in a trusted manner and acquires the memory dump to discover malware footprints while the VM operates. The memory dumps are transformed into visual images which are analyzed using a convolutional neural network (CNN) based classifier. The proposed framework has some key advantages, such as its agility, its ability to eliminate the need for features defined by a cyber domain expert, and most importantly, its ability to analyze the entire memory dump and thus to better utilize the existing indication it conceals, thus allowing the induction of a more accurate detection model. Deep-Hook was evaluated on widely used Linux virtual servers; four state-of-the-art CNN architectures; eight image resolutions; and a total of 22,400 volatile memory dumps representing the execution of a broad set of benign and malicious Linux applications. Our experimental evaluation results demonstrate Deep-Hook’s ability to effectively, efficiently, and accurately detect and classify unknown malware (even evasive malware like rootkits), with an AUC and accuracy of up to 99.9%.