Machine Learning Based Malware Detection Research Articles

A Novel Feature Encoding Scheme for Machine Learning Based Malware Detection Systems

A Novel Feature Encoding Scheme for Machine Learning Based Malware Detection Systems

Employing combined spatial and frequency domain image features for machine learning-based malware detection

<p>The ubiquitous adoption of Android devices has unfortunately brought a surge in malware threats, compromising user data, privacy concerns, and financial and device integrity, to name a few. To combat this, numerous efforts have explored automated botnet detection mechanisms, with anomaly-based approaches leveraging machine learning (ML) gaining attraction due to their signature-agnostic nature. However, the problem lies in devising accurate ML models which capture the ever evolving landscape of malwares by effectively leveraging all the possible features from Android application packages (APKs).This paper delved into this domain by proposing, implementing, and evaluating an image-based Android malware detection (AMD) framework that harnessed the power of feature hybridization. The core idea of this framework was the conversion of text-based data extracted from Android APKs into grayscale images. The novelty aspect of this work lied in the unique image feature extraction strategies and their subsequent hybridization to achieve accurate malware classification using ML models. More specifically, four distinct feature extraction methodologies, namely, Texture and histogram of oriented gradients (HOG) from spatial domain, and discrete wavelet transform (DWT) and Gabor from the frequency domain were employed to hybridize the features for improved malware identification. To this end, three image-based datasets, namely, Dex, Manifest, and Composite, derived from the information security centre of excellence (ISCX) Android Malware dataset, were leveraged to evaluate the optimal data source for botnet classification. Popular ML classifiers, including naive Bayes (NB), multilayer perceptron (MLP), support vector machine (SVM), and random forest (RF), were employed for the classification task. The experimental results demonstrated the efficacy of the proposed framework, achieving a peak classification accuracy of 93.03% and recall of 97.1% for the RF classifier using the Manifest dataset and a combination of Texture and HOG features. These findings validate the proof-of-concept and provide valuable insights for researchers exploring ML/deep learning (DL) approaches in the domain of AMD.</p>

Malware Detection in Files and URL’s using Machine Learning

The use of Support Vector Machine (SVM) to machine learning-based malware detection is the main goal of this study. Since malware threats are always changing, more advanced detection techniques are required. With the help of SVM, a potent classification algorithm, our project is able to precisely identify malware based on a range of attributes. By developing a strong and effective solution that efficiently identifies and reduces malware threats, the aim is to improve cybersecurity and support continuous efforts to protect computer systems and networks. Malware has been posing a serious threat to embedded systems in recent times, and traditional software solutions like antivirus and patching haven't been able to keep up with the sophisticated and constantly changing bad programmes. In this work, we present guardol, a hardware-enhanced architecture designed to identify online malware. Guardol is a hybrid technique that combines FPGA and CPU. Our method seeks to capture malware's malevolent behaviour, or high-level semantics. In order to do this, we first suggest the frequency-centric model for building features out of benign samples and known malware's system call patterns. We next create a machine learning strategy in FPGA to train a classifier with these features, utilising a multilayer perceptron. The trained classifier is applied at runtime to categorise the unknown data as benign or malicious, with In this day and age of advanced technology, the internet has been embraced by most people. And with it has come an increased risk of malevolent cyberattacks by cybercriminals. The attacks are completed.

Leveraging Dual-Generative Adversarial Networks for Adversarial Malware Detection via Ensemble Learning


 
 
 
 In the expanding realm of cybersecurity, machine learning-based malware detection has emerged as a vital line of defense. However, the growing sophistication of malware attacks poses formidable challenges to conventional detection systems. To address this, this paper uses a Generative Adversarial Network that utilizes dual generators for adversarial learning on malware, designed to enhance the detection of harmful Portable Executable (PE) files. Our model employs a two-tiered generator system within the GAN architecture, where the secondary generator intervenes when the primary generator yields a malware PE executable dismissed by the detector.
 The detection unit leverages ensemble learning techniques to analyze the PE software feature vector, capitalizing on the synergy of multiple learning models for improved performance and generalization. This setup empowers the system to generate a broader range of adversarial examples and respond to them effectively, enhancing the robustness of the detector against previously unseen or variable malware types.
 
 
 

A Review on Machine Learning-based Malware Detection Techniques for Internet of Things (IoT) Environments

A Review on Machine Learning-based Malware Detection Techniques for Internet of Things (IoT) Environments

Optimasi Algoritma Random Forest menggunakan Principal Component Analysis untuk Deteksi Malware

Malware is a type of software designed to harm various devices. As malware evolves and diversifies, traditional signature-based detection methods have become less effective against advanced types such as polymorphic, metamorphic, and oligomorphic malware. To address this challenge, machine learning-based malware detection has emerged as a promising solution. In this study, we evaluated the performance of several machine learning algorithms in detecting malware and applied Principal Component Analysis (PCA) to the best-performing algorithm to reduce the number of features and improve performance. Our results showed that the Random Forest algorithm outperformed Adaboost, Neural Network, Support Vector Machine, and k-Nearest Neighbor algorithms with an accuracy and recall rate of 98.3%. By applying PCA, we were able to further improve the performance of Random Forest to 98.7% for both accuracy and recall while reducing the number of features from 1084 to 32.

MalwD&C: A Quick and Accurate Machine Learning-Based Approach for Malware Detection and Categorization

Malware, short for malicious software, is any software program designed to cause harm to a computer or computer network. Malware can take many forms, such as viruses, worms, Trojan horses, and ransomware. Because malware can cause significant damage to a computer or network, it is important to avoid its installation to prevent any potential harm. This paper proposes a machine learning-based malware detection method called MalwD&C to allow the secure installation of Programmable Executable (PE) files. The proposed method uses machine learning classifiers to analyze the PE files and classify them as benign or malware. The proposed MalwD&C scheme was evaluated on a publicly available dataset by applying several machine learning classifiers in two settings: two-class classification (malware detection) and multi-class classification (malware categorization). The results showed that the Random Forest (RF) classifier outperformed all other chosen classifiers, achieving as high as 99.56% and 97.69% accuracies in the two-class and multi-class settings, respectively. We believe that MalwD&C will be widely accepted in academia and industry due to its speed in decision making and higher accuracy.

Leveraging the first line of defense: a study on the evolution and usage of android security permissions for enhanced android malware detection

Android security permissions are built-in security features that constrain what an app can do and access on the system, that is, its privileges. Permissions have been widely used for Android malware detection, mostly in combination with other relevant app attributes. The available set of permissions is dynamic, refined in every new Android OS version release. The refinement process adds new permissions and deprecates others. These changes directly impact the type and prevalence of permissions requested by malware and legitimate applications over time. Furthermore, malware trends and benign apps’ inherent evolution influence their requested permissions. Therefore, the usage of these features in machine learning-based malware detection systems is prone to concept drift issues. Despite that, no previous study related to permissions has taken into account concept drift. In this study, we demonstrate that when concept drift is addressed, permissions can generate long-lasting and effective malware detection systems. Furthermore, the discriminatory capabilities of distinct set of features are tested. We found that the initial set of permissions, defined in Android 1.0 (API level 1), are sufficient to build an effective detection model, providing an average 0.93 F1 score in data that spans seven years. In addition, we explored and characterized permissions evolution using local and global interpretation methods. In this regard, the varying importance of individual permissions for malware and benign software recognition tasks over time are analyzed.

MACHINE LEARNING TECHNIQUES TO PERCEIVE INTRUSIVE SOFTWARE

Machine learning based malware detection algorithms are usually integrated into antivirus software and therefore they are black-box systems to malware authors. It is hard for malware authors to know which classifier a malware detection system uses and the parameters of the classifier. However, it is possible to figure out what features a malware detection algorithm uses by feeding some carefully designed test cases to the black-box algorithm. Malware authors can modify a benign program’s DLL or API names to malware’s DLL or API names, and vice versa. If the detection results change after most of the modifications, they can judge that the malware detection algorithm uses DLL or API features. Therefore, in this paper we assume that malware authors can know what features a malware detection algorithm uses but know nothing about the machine learning model. The proposed model in this paper uses GANs, which generates adversarial examples to attack black-box malware detection algorithms. A substitute detector is trained to fit the black-box malware detection algorithm, and a generative network is used to transform malware samples into adversarial examples. Experimental results show that almost all the adversarial examples generated by GANs successfully bypass the detection algorithms.

Cross-device behavioral consistency: Benchmarking and implications for effective android malware detection

Most of the proposed solutions using dynamic features for Android malware detection collect and test their systems using a single and particular data collection device, either a real device or an emulator. The results obtained using these particular devices are then generalized to any Android platform. This extensive generalization is based on the assumption of consistent behavior of apps across devices. This study performs an extensive benchmarking of this assumption for system calls, executing Android malware and benign samples under the same conditions in 9 different collection devices, including real and virtual devices. The results indicate the existence of significant differences between real devices and emulators in system calls usage and, consequently, in the collected behavioral profiles obtained from running the same set of applications on different devices. Furthermore, the impact of these differences on machine learning-based malware detection models is evaluated. In this regard, a significant degenerative effect on the detection performance of the model is produced when data collected on different devices are used in the training and testing sets. Therefore, the empirical findings do not support the assumption of cross-device consistent behavior of Android apps when system calls are used as descriptive features.

Compact feature hashing for machine learning based malware detection

Machine learning can detect variant malware files that can evade signature-based detection. Feature hashing is used to convert features into a fixed-length vector. In this paper, we study the appropriate vector size for feature hashing for a large dataset of malware files. Through exhaustive experiments on more than 280,000 real malware and benign files, we find for the first time that the default vector size of current feature hashing practices is unnecessarily large. We experimentally explore the appropriate vector size, which not only reduces memory space by 70% but also increases the detection accuracy, compared with the state-of-the-art scheme.

On-device Resilient Android Malware Detection using Incremental Learning

In most cases, the Android runs in a battery-driven, low-power connected, embedded real-time environment. New malware attacks are reported every day targeting Android embedded devices. This is because the existing Android malware detection engines are not resilient to zero day attacks. The machine learning based malware detection engines must be re-trained with the newly reported malware and benign samples to manage the zero-day attacks. The re-training is very expensive in terms of CPU and memory. Re-training at embedded devices is very challenging because of its hardware, memory, and energy constraints. In these scenarios, a lightweight, resilient and incremental learning based dynamic Android malware detection mechanism is becoming mandatory. Very little evidence is observed in the related literature; most of the existing incremental learning approaches have not demonstrated their experiments in a real device environment and have not worked in bench-marked real device dynamic analysis datasets. In this work, we select the latest KronoDroid Android dynamic hybrid dataset and propose efficient incremental learning based Android malware detection mechanism using the linear incremental algorithm. We have incrementally trained our mechanism in the real device Android environment and proved the consistent performance keeping the resilient aspect. We have evaluated our mechanism and found that we could achieve the highest incremental accuracy of 97.94%, the lowest training time of 0.5 seconds, and lowest detection time of 0.04 seconds with the smallest model size of 456 bytes when experimented within an on-device Android environment. To the best of our knowledge, this is the first work that proves the incremental learning and resilience scenario using a bench-marked Android real-device dynamic dataset keeping the lowest computational complexity and consistent performance.

Machine learning-based malware detection on Android devices using behavioral features

The most recent focus of security vulnerabilities has been on Android devices. Several permissions were developed to detect malware, but they can affect the device through malware applications that deal with dangerous permissions. To detect malware, behavioral features and machine learning techniques are used in this work. The purpose of this work is to evaluate the impact of malware occurrence using a behavioral features dataset. For malware detection, the features are incorporated into the prediction model using LBGM, ENB0, and EEXR. There is a Collector Phase, a Cleaner Phase, a Selector Phase, and a Detector Phase in this proposed system. This proposed system does the relevant process for detecting malware through these four phases. According to the performance results, the proposed prediction model EEXR gives high accuracy of 96.75% and less error rate of 3.28% while predicting more than other methods.

Malware Detection Using Ensemble N-gram Opcode Sequences

Conventional approaches to tackling malware attacks have proven to be futile at detecting never-before-seen (zero-day) malware. Research however has shown that zero-day malicious files are mostly semantic-preserving variants of already existing malware, which are generated via obfuscation methods. In this paper we propose and evaluate a machine learning based malware detection model using ensemble approach. We employ a strategy of ensemble where multiple feature sets generated from different n-gram sizes of opcode sequences are trained using a single classifier. Model predictions on the trained multi feature sets are weighted and combined on average to make a final verdict on whether a binary file is malicious or benign. To obtain optimal weight combination for the ensemble feature sets, we applied a grid search on a set of pre-defined weights in the range 0 to 1. With a balanced dataset of 2000 samples, an ensemble of n-gram opcode sequences of n sizes 1 and 2 with respective weight pair 0.3 and 0.7 yielded the best detection accuracy of 98.1% using random forest (RF) classifier. Ensemble n-gram sizes 2 and 3 obtained 99.7% as best precision using weight 0.5 for both models.

Android Malware Detection Using Machine Learning with Feature Selection Based on the Genetic Algorithm

Since the discovery that machine learning can be used to effectively detect Android malware, many studies on machine learning-based malware detection techniques have been conducted. Several methods based on feature selection, particularly genetic algorithms, have been proposed to increase the performance and reduce costs. However, because they have yet to be compared with other methods and their many features have not been sufficiently verified, such methods have certain limitations. This study investigates whether genetic algorithm-based feature selection helps Android malware detection. We applied nine machine learning algorithms with genetic algorithm-based feature selection for 1104 static features through 5000 benign applications and 2500 malwares included in the Andro-AutoPsy dataset. Comparative experimental results show that the genetic algorithm performed better than the information gain-based method, which is generally used as a feature selection method. Moreover, machine learning using the proposed genetic algorithm-based feature selection has an absolute advantage in terms of time compared to machine learning without feature selection. The results indicate that incorporating genetic algorithms into Android malware detection is a valuable approach. Furthermore, to improve malware detection performance, it is useful to apply genetic algorithm-based feature selection to machine learning.

Lessons Learnt on Reproducibility in Machine Learning Based Android Malware Detection

A well-known curse of computer security research is that it often produces systems that, while technically sound, fail operationally. To overcome this curse, the community generally seeks to assess proposed systems under a variety of settings in order to make explicit every potential bias. In this respect, recently, research achievements on machine learning based malware detection are being considered for thorough evaluation by the community. Such an effort of comprehensive evaluation supposes first and foremost the possibility to perform an independent reproduction study in order to sharpen evaluations presented by approaches’ authors. The question Can published approaches actually be reproduced? thus becomes paramount despite the little interest such mundane and practical aspects seem to attract in the malware detection field. In this paper, we attempt a complete reproduction of five Android Malware Detectors from the literature and discuss to what extent they are “reproducible”. Notably, we provide insights on the implications around the guesswork that may be required to finalise a working implementation. Finally, we discuss how barriers to reproduction could be lifted, and how the malware detection field would benefit from stronger reproducibility standards—like many various fields already have.

Windows Malware Detector Using Convolutional Neural Network Based on Visualization Images

The evolution of malware is continuing at an alarming rate, despite the efforts made towards detecting and mitigating them. Malware analysis is needed to defend against its sophisticated behaviour. However, the manual heuristic inspection is no longer effective or efficient. To cope with these critical issues, behaviour-based malware detection approaches with machine learning techniques have been widely adopted as a solution. It involves supervised classifiers to appraise their predictive performance on gaining the most relevant features from the original features' set and the trade-off between high detection rate and low computation overhead. Though machine learning-based malware detection techniques have exhibited success in detecting malware, their shallow learning architecture is still deficient in identifying sophisticated malware. Therefore, in this paper, a Convolutional Neural Network (CNN) based Windows malware detector has been proposed that uses the execution time behavioural features of the Portable Executable (PE) files to detect and classify obscure malware. The 10-fold cross-validation tests were conducted to assess the proficiency of the proposed approach. The experimental results showed that the proposed approach was effective in uncovering malware PE files by utilizing significant behavioural features suggested by the Relief Feature Selection Technique. It attained detection accuracy of 97.968 percent.

A novel permission-based Android malware detection system using feature selection based on linear regression

With the developments in mobile and wireless technology, mobile devices have become an important part of our lives. While Android is the leading operating system in market share, it is the platform most targeted by attackers. Although many solutions have been proposed in the literature for the detection of Android malware, there is still a need for attribute selection methods to be used in Android malware detection systems. In this study, a machine learning-based malware detection system is proposed to distinguish Android malware from benign applications. At the feature selection stage of the proposed malware detection system, it is aimed to remove unnecessary features by using a linear regression-based feature selection approach. In this way, the dimension of the feature vector is reduced, the training time is decreased, and the classification model can be used in real-time malware detection systems. When the results of the study are examined, the highest 0.961 is obtained according to the F-measure metric by using at least 27 features.

A Framework for Enhancing Deep Neural Networks Against Adversarial Malware

Machine learning-based malware detection is known to be vulnerable to adversarial evasion attacks. The state-of-the-art is that there are no effective defenses against these attacks. As a response to the adversarial malware classification challenge organized by the MIT Lincoln Lab and associated with the AAAI-19 Workshop on Artificial Intelligence for Cyber Security (AICS'2019), we propose six guiding principles to enhance the robustness of deep neural networks. Some of these principles have been scattered in the literature, but the others are introduced in this paper for the first time. Under the guidance of these six principles, we propose a defense framework to enhance the robustness of deep neural networks against adversarial malware evasion attacks. By conducting experiments with the Drebin Android malware dataset, we show that the framework can achieve a 98.49\% accuracy (on average) against grey-box attacks, where the attacker knows some information about the defense and the defender knows some information about the attack, and an 89.14% accuracy (on average) against the more capable white-box attacks, where the attacker knows everything about the defense and the defender knows some information about the attack. The framework wins the AICS'2019 challenge by achieving a 76.02% accuracy, where neither the attacker (i.e., the challenge organizer) knows the framework or defense nor we (the defender) know the attacks. This gap highlights the importance of knowing about the attack.

The Classification and Detection of Malware Using Soft Relevance Evaluation

In recent years, researchers have made a great success on the automatic classification and detection of malware utilizing machine learning methods. However, most machine learning based approaches over rely on the training samples such that a new malware family not belonging to the training set cannot be identified. To address such issue, we propose a soft relevance value ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">s</i> -value), a new evaluating way of feature soft relevance that uses the mixed distance criterion to assess classified results. Specifically, we leverage the mixed distance criterion from pattern recognition to distinguish testing samples as a new family which is not labeled in training set. Finally, we evaluate how <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">s</i> -value can be used to distinguish and classify a new malware family with the malware datasets from the Research Prediction Competition of Microsoft Malware Classification Challenge and Windows (Kaggle). The experimental results show that, the train-ing time is approximately 12 hours, while the prediction time is only ∼0.5 second. Comparing against the Kaggle winner, our time costs for training and pprediction only occupy 16.7% and 3.8% of the winner.s time costs, respectively. The accuracy of classifying malware reaches 99.8%. Such results indicates that our proposed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">s</i> -value achieves a balance in accuracy, training and prediction time, and outperforms the state-of-the-art machine learning based malware detection approaches. Besides, our method is able to identify new malware families that are not included in the training set.