Behavioral Malware Detection Research Articles

Detection Strategies for COM, WMI, and ALPC-Based Multi-Process Malware.

Behavioral malware detection is based on attributing malicious actions to processes. Malicious processes may try to hide by changing the behavior of other benign processes to achieve their goals. We showcase how Component Object Model (COM) and Windows Management Instrumentation (WMI) can be used to create such spoofing attacks. We discuss the internals of COM and WMI and Asynchronous Local Procedure Call (ALPC). We present multiple functional monitoring techniques to identify the spoofing and discuss the strong and weak points of each technique. We create a robust process monitoring system that can correctly identify the source of malicious actions spoofed via COM, WMI and ALPC with a low performance impact. Finally, we discuss how malicious actors use COM, WMI and ALPC by examining real-world malware detected by our monitoring system.

Explainability in AI-based behavioral malware detection systems

Nowadays, our security and privacy are strongly threatened by malware programs which aim to steal our confidential data and make our systems out of service, among other things. While traditional signature-based malware detection methods or statistical analysis have proven to be ineffective and time-consuming, recently data-driven Artificial Intelligence (AI) techniques, i.e. Machine Learning (ML) and Deep Learning (DL) approaches, have been successfully applied leveraging the behavior of malware in terms of API calls, and achieving promising performances. However, their black-box behavior leads to a lack of explainability thus preventing their application in real world scenarios. In light of this, eXplainable Artificial Intelligence (XAI) methodologies and tools can be effectively embedded within an AI-based malware detection process in order to make more understandable the produced results. In this paper, we propose a XAI framework for behavioral malware detection problems and evaluate the usefulness of four XAI methods (SHAP, LIME, LRP and Attention mechanism) on three datasets with different size, sequence length and number of classes, by which we could evaluate the strengths and weaknesses – from effectiveness and efficiency point of views – of recurrent deep architectures (i.e. Long-Short Term Memory (LSTM) and Gated Recurrent Unit (GRU) models), and their applicability in the modern Cyber Security (CS) scenarios.

Evading behavioral classifiers: a comprehensive analysis on evading ransomware detection techniques

Recent progress in machine learning has led to promising results in behavioral malware detection. Behavioral modeling identifies malicious processes via features derived by their runtime behavior. Behavioral features hold great promise as they are intrinsically related to the functioning of each malware, and are therefore considered difficult to evade. Indeed, while a significant amount of results exists on evasion of static malware features, evasion of dynamic features has seen limited work. This paper examines the robustness of behavioral ransomware detectors to evasion and proposes multiple novel techniques to evade them. Ransomware behavior differs significantly from that of benign processes, making it an ideal best case for behavioral detectors, and a difficult candidate for evasion. We identify and propose a set of novel attacks that distribute the overall malware workload across a small set of independent, cooperating processes in order to avoid the generation of significant behavioral features. Our most effective attack decreases the accuracy of a state-of-the-art classifier from 98.6 to 0% using only 18 cooperating processes. Furthermore, we show our attacks to be effective against commercial ransomware detectors in a black-box setting. Finally, we evaluate a detector designed to identify our most effective attack, as well as discuss potential directions to mitigate our most advanced attack.

A Novel Method for Detecting Future Generations of Targeted and Metamorphic Malware Based on Genetic Algorithm

This paper presents a novel solution for detecting rare and mutating malware programs and provides a strategy to address the scarcity of datasets for modeling these types of malware. To provide sufficient training data for malware behavioral modeling, genetic algorithms are used together with an optimization strategy that selectively creates generations of mutated elite malware samples. In our unique method, a sequence of system API calls is extracted using tracker filter drivers in a sandbox environment. The most obfuscated and metamorphic malware are chosen by an elite selection method. The behavioral chromosomes are formed by mapping extracted APIs to genes using linear regression. Our analysis system includes an Internet simulator and a human emulator to deceive intelligent classes of malware to successfully execute themselves and prevent system halting. The evolution process is performed through crossover and permutation of genes, which are encoded based on the addresses of the kernel-level system functions. An objective function has been defined to optimize the vital indicators of malignancy and tracking rate with a linear time complexity. This guarantees that new generations of malware are more destructive and stealthy than their parents. J48 and deep neural networks were employed in our experiments as they are two popular modeling and classification strategies in the area of behavioral malware detection. Real-world malware samples from valid references were used for the performance evaluation of our approach. Comprehensive scenarios were involved in the experiments to evaluate the performance of our proposed strategy. The results demonstrate significant improvement in detection accuracy - up to 5% considering rare and metamorphic malware. The results also demonstrated a considerable enhancement in true positive rate for the proposed deep-learning algorithm.

Combined dynamic multi-feature and rule-based behavior for accurate malware detection

Malware have become the scourge of the century, as they are continuously evolving and becoming more complex with increasing damages. Therefore, an adequate protection against such threats is vital. Behavior-based malware detection techniques have shown to be effective at overcoming the weaknesses of the signature-based ones. However, they are known for their high false alarms, which is still a very challenging problem. In this article, we address this shortcoming by proposing a rule-based behavioral malware detection system, which inherits the advantages of both signature and behavior-based approaches. We apply the proposed detection system on a combined set of three types of dynamic features, namely, (1) list of application programming interface calls; (2) application programming interface sequences; and (3) network traffic, which represents the IP addresses and domain names used by malware to connect to remote command-and-control servers. Feature selection and construction techniques, that is, term frequency–inverse document frequency and longest common subsequence, are performed on the three extracted features to generate new set of features, which are used to build behavioral Yet Another Recursive Acronym rules. The proposed malware detection approach is able to achieve an accuracy of 97.22% and a false positive rate of 4.69%.

HLMD: a signature-based approach to hardware-level behavioral malware detection and classification

Malicious programs, or malware, often use code obfuscation techniques to make static analysis difficult. To deal with this problem, various behavioral detection techniques have been proposed that focus on runtime behavior to distinguish between benign and malicious programs. The majority of them are based on the analysis and modeling of system call traces, which are a common type of audit data often used to describe the interaction between programs and the operating system. However, the techniques are not widely used in practice because of high performance overheads. An alternative approach is to perform behavioral detection at the hardware level. The basic idea is to use information that is accessible through hardware performance counters, which are a set of special purpose registers built into modern processors providing detailed information about hardware and software events. In this paper, we pursue this line of research by presenting HLMD, a novel approach that uses behavioral signatures generated from hardware performance counter traces to instantly detect and disable malicious programs at the beginning of their execution. HLMD is especially suitable for independent malicious programs that can be run standalone without having to be attached to a host program. Each behavioral signature is composed of some number of singular values and singular vectors, obtained by applying the singular value decomposition to the hardware performance counter traces of a known malware family. HLMD follows a two-stage heuristic matching strategy to increase the detection performance to an acceptable level while reducing the detection complexity to linear time. The results of our experiments performed on a dataset of benign and malicious programs show that HLMD can achieve an average precision, recall, and F-measure of 95.19%, 89.96%, and 92.50%, respectively.

SmartMal: a service-oriented behavioral malware detection framework for mobile devices.

This paper presents SmartMal—a novel service-oriented behavioral malware detection framework for vehicular and mobile devices. The highlight of SmartMal is to introduce service-oriented architecture (SOA) concepts and behavior analysis into the malware detection paradigms. The proposed framework relies on client-server architecture, the client continuously extracts various features and transfers them to the server, and the server's main task is to detect anomalies using state-of-art detection algorithms. Multiple distributed servers simultaneously analyze the feature vector using various detectors and information fusion is used to concatenate the results of detectors. We also propose a cycle-based statistical approach for mobile device anomaly detection. We accomplish this by analyzing the users' regular usage patterns. Empirical results suggest that the proposed framework and novel anomaly detection algorithm are highly effective in detecting malware on Android devices.