Malware Detection Method Research Articles

Android malware detection using the random forest algorithm

The rapid growth in Android device usage has resulted in a significant increase in malware targeting this platform, posing serious threats to user security and privacy. This research tackles the challenge of Android malware detection by leveraging advanced machine learning techniques, with a particular emphasis on the random forest (RF) algorithm. Our primary objective is to accurately identify and classify malicious applications to enhance the security of Android devices. In this study, we employed the RF algorithm to analyze a comprehensive dataset of Android applications, where the classification of each application as either malware or benign is known. The method was rigorously tested, yielding impressive results: an average accuracy of 98.47%, a sensitivity of 98.60%, and an F-score of 98.60%. These metrics underscore the effectiveness of our approach. Moreover, we conducted a comparative analysis of the RF algorithm against other malware detection methods. The results demonstrate that the RF algorithm outperforms these alternative methods, offering superior detection capabilities and contributing to more robust Android security measures.

Performance Comparative Study on Zero Day Malware Detection Using XGBoost and Random Forest Classifiers

Zero-day malware is a significant threat to cybersecurity as it is unknown to antivirus systems and can cause significant damage before being detected. Traditional malware detection methods rely on signatures and patterns specific to known malware, but these methods are ineffective against zero-day malware that has not been previously encountered. Machine learning has shown promise in detecting and classifying unknown threats, including zero-day malware. In this research, we propose using machine learning classifiers to detect and classify zero-day malware. The selected classifiers are Random Forest and XGBoost, well-known and widely used in machine learning. To evaluate the effectiveness of our approach, we first collect and pre-process a dataset of known malware. The dataset, Meraz'18, is used to train and test the selected classifiers. This dataset contains PEHeaders with static analysis performed with each section calculated on its entropy. The dataset contains a representation of benign and malicious files that has been use in previous studies for zero-day malware detection, namely during the payload injection phase. To prevent overfitting, 10-fold-cross-validation is utilized. The performance metrics of these classifiers such as F1-score, accuracy, Cohen’s kappa, precision and recall analyzed on the known malware dataset and evaluate their ability to detect and classify zero-day malware. Hyperparameter tuning is used to tune each model to give the best performance of each model. The results show that the proposed classifiers perform extremely well, both achieving up to almost 98% accuracy. Using machine learning classifiers for zero-day malware detection and classification can significantly improve cybersecurity by providing a way to detect and protect against unknown threats. This work is an essential step towards the development of more robust cybersecurity systems that can effectively protect against unknown threats.

Comparative Analysis of Feature Selection Methods with XGBoost for Malware Detection on the Drebin Dataset

Malware, or malicious software, continues to evolve alongside increasing cyberattacks targeting individual devices and critical infrastructure. Traditional detection methods, such as signature-based detection, are often ineffective against new or polymorphic malware. Therefore, advanced malware detection methods are increasingly needed to counter these evolving threats. This study aims to compare the performance of various feature selection methods combined with the XGBoost algorithm for malware detection using the Drebin dataset, and to identify the best feature selection method to enhance accuracy and efficiency. The experimental results show that XGBoost with the Information Gain method achieves the highest accuracy of 98.7%, with faster training times than other methods like Chi-Squared and ANOVA, which each achieved an accuracy of 98.3%. Information Gain yielded the best performance in accuracy and training time efficiency, while Chi-Squared and ANOVA offered competitive but slightly lower results. This study highlights that appropriate feature selection within machine learning algorithms can significantly improve malware detection accuracy, potentially aiding in real-world cybersecurity applications to prevent harmful cyberattacks.

Robust malicious software detection and classification using global whale optimization algorithm with deep learning approach

Software malware detection and classification leverage sophisticated procedures and methods from the cybersecurity domain for identifying and categorizing malicious software, generally called malware. This procedure analyses code behaviour, file structures, and other features to distinguish between benign and malicious programs. Machine learning (ML) and artificial intelligence (AI) are vital in this domain, allowing the progress of dynamic and adaptive systems that identify novel and developing malware attacks. By training on massive datasets of benign and malicious instances, these systems learn patterns and signatures indicative of malware. This lets them correctly categorize and respond to potential attacks in real-time. This study presents a Global Whale Optimization Algorithm with Neutrosophic Logic for Software Malware Detection and Classification (GWOANL-SMDC) technique. The GWOANL-SMDC technique secures the software via the Android malware recognition process. Primarily, the GWOANL-SMDC technique employs the Neutrosophic Cognitive Maps (NCM) model for the feature selection process. The GWOANL-SMDC technique uses a convolutional long short-term memory (ConvLSTM) model for software malware detection. At last, the GWOA-based parameter tuning is performed to improve the performance of the ConvLSTM model. The simulation values of the GWOANL-SMDC technique are examined on the malware dataset. The obtained results ensured that the GWOANL-SMDC technique improved capability in detecting software malware.

Static analysis framework for permission-based dataset generation and android malware detection using machine learning

Since Android is the popular mobile operating system worldwide, malicious attackers seek out Android smartphones as targets. The Android malware can be identified through a number of established detection techniques. However, the issues presented by modern malware cannot be met by traditional signature or heuristic-based malware detection methods. Previous research suggests that machine-learning classifiers can be utilised to analyse permissions, making it possible to differentiate between malicious and benign applications on the Android platform. There exist machine-learning methods that utilise permission-based attributes to build models for the detection of malware on Android devices. Nevertheless, the performance of these detection methods is dependent on the raw or feature datasets. Android malware research frequently faces a major obstacle due to the lack of adequate and up-to-date raw malware datasets. In this paper, we put forward a systematic approach to generate an Android permission-based dataset using static analysis. To create the dataset, we collect recent raw malware samples (APK files) and focus on the reverse engineering approach and permission-based features extraction. We also conduct a thorough feature analysis to determine the important Android permissions and present a machine-learning-based Android malware detection mechanism. The experimental result of our study demonstrates that with just 48 features, the random forest classifier-based Android malware detection model obtains the best accuracy of 97.5%.

Sparse attention with residual pyramidal depthwise separable convolutional based malware detection with optimization mechanism

Recent developments indicate that malware programs present a significant risk in the security and privacy of cloud systems. Existing research in malware detection encounters numerous significant challenges due to the constantly changing and advanced characteristics of malware. Malware detection systems frequently experience high rates of false positives and false negatives, where legitimate applications are incorrectly identified as malware or actual malware remains undetected, which results in operational inefficiencies. Traditional signature-based approaches struggle in recognizing new or modified malware. Additionally, sophisticated malware types, such as file less malware, ransom ware, and rootkits pose detection challenges as they integrate deeply into systems or alter their behaviour to evade detection. These challenges highlight the urgent need for ongoing advancements in this field. Existing methods of malware detection that rely on signatures have been found to be both inefficient and slow in the context of cloud environments. Also, Existing studies have focused on detecting malware by analysing input API calls. But, these models have encountered challenges such as limited accuracy and difficulties in effectively classifying malware types. In contrast, Deep Learning (DL) have shown success by analysing malware behaviour through API calls, which yields encouraging results. Additionally, the data produced by API calls necessitates more computational resources for training. To address these challenges, a new deep learning-based malware detection approach utilizes 2D grayscale images derived from API calls, along with an effective tuning strategy has been proposed. Initially, data are collected from cloud malware dataset. Then, API calls are converted into 2D gray scale images in order to construct gray scale image dataset. After getting the gray scale image, pre-processing is performed to reduce high level noise and to enhance the quality of image by weighted mean filter and anisotropic filter, which helps to improve the performance in classification. Next, these images are then passed into the feature extraction stage to extract sufficient features with an effective integrated densely connected squeeze MobileNet v2 (Ef-DeSMob2), which reduces the dimensionality issue and increase the computational complexity. Then, the collected features are passed into the classification phase to detect normal and malware classes from the samples using sparse attention with residual pyramidal depth wise separable convolutional neural networks (SA:ResPyDSC), which focus to enhance the security and reliability of the model. Finally, the hyper parameters in the classifier model like weights, bias are properly fine-tuned by utilizing hybrid white shark beluga optimization algorithm (Hy-WBeOp). The experimental findings illustrate that the proposed method attains accuracy of 98.06%, precision of 97.99%, recall of 97.05%, f1-score of 96.08% and error metrics like MSE AT 0.08, RMSE of 0.27 and MAE of 0.21, which shows that the proposed method helps to classify the malware classes accurately with less error rates. The proposed approach outperforms with the existing techniques because of its great efficiency. Overall, this approach establish a strong malware detection system classification and enhance the reliability and effectiveness of protection against malicious attacks.

Malware Detection using Deep Learning (DL)

The attack that occurred recently involved the utilization of malicious software, commonly referred to as malware, along with advanced techniques such as machine learning, specifically deep learning, code transformation, and polymorphism. This makes it harder for cyber experts to detect malware using traditional analysis methods. In view of the low accuracy and high false positive rate of traditional malware detection methods, this research proposes a fine-tuned deep learning model with a novel dataset that is compared with ANN, Support Vector Machine (SVM), random forest (RF), K-Nearest Neighbourhood (KNN) classifiers. Converting a malware code into an image could allow users to effectively identify the presence of malware, even if the original code is modified by the creator. This is due to the fact that the attributes of images remain unchanged, allowing for reliable identification. So, researchers used deep learning technology to detect malware, like detecting malaria from red blood cells. The deep learning model found that a more detailed analysis of malware data sets, focusing on RGB and greyscale images, is needed. These data sets currently rely on publicly available data, but the accuracy of the traditional model could be a lot higher. It also produces many false positive results. The main goal is to create a new data set and model using malware images to identify and categorize malware using deep learning without relying on existing image detection and transfer learning models. The researcher adjusted different hyper parameters, like the number of neurons, filters, stride, hidden units, layers, learning rate, batch size, activation function, optimizer, and epochs. Identifying and correcting issues in models, improving their clarity, stability, and fairness, as well as debugging and monitoring them, is a challenging task. To eliminate this obstacle, assess the model's behaviour and performance by employing various methods such as logging, profiling, testing, visualization, and model clarity. To overcome the challenges posed by the electricity shutdown, we utilized Google's cloud-based GPU and Python 3.7 language to conduct the experiment and train the model. Kali Linux is an operating system that can automatically encrypt file systems and has a lower chance of crashing the system due to malware in a virtual sandbox. The researcher used techniques like early stopping and cross-validation to prevent over fitting and assess generalization in addition to monitoring and evaluating the model's behaviour and performance. The researcher used different methods like L1 and L2 regularization, dropout, and batch normalization to improve the model's performance and avoid over fitting. The binary portable executable (PE) malware dataset is collected from Kaggle, Malimg, Virusshare, Malvis, MS Big2015, and VX-underground and finally converted to greyscale and RGB images to create a novel dataset to fill the lack of image dataset. The raw dataset was then rescaled to a 128x128 greyscale and RGB (red, green, blue) image and flattened to 1024-byte vector images input into convolution neural network (CNN) interpolation to extract features for malware detection. The newly acquired dataset is utilized as an input for the innovative DL algorithms in order to create a tailor-made model capable of accurately predicting malware. The findings in this customized CNN model by fine-tuning hyper-parameters with the three-channel RGB dataset outperformed a greyscale dataset with an accuracy of 98.7% and error rate of 1.3% in current malware compared with other artificial neural network (ANN), ML algorithms such as Support Vector Machine (SVM), K-Nearest Neighbour (KNN), and also Random Forest (RF) models. The proposed approach is compatible with all operating systems (Windows, Linux, and Mac) and can identify different types of malwares, such as packed, polymorphic, obfuscated, metamorphic, or variations of a malware family. There are some limitations to the shortage of malware image datasets and computational costs for fine-tuning hyper parameters in the whole model. Furthermore, the proposed approach detects malware and groups it into families without using common techniques like disassembly, decompilation, de-obfuscation, or running malicious code in a virtual environment.

Improving mobile security: A study on android malware detection using LOF

Abstract The ubiquity of smartphones in our daily lives has made them attractive targets for malicious actors seeking to compromise user data and device functionality. Android malware detection has become imperative to protect user privacy and device integrity. This paper presents a focused study on leveraging the Local Outlier Factor (LOF) method for Android malware detection using the DREBIN dataset. Our research addresses the need for accurate and efficient Android malware detection. We explore the LOF method, an anomaly-based detection technique, to assess its effectiveness in distinguishing malicious applications from benign ones within the Android ecosystem. Rigorous experiments using the extensive DREBIN dataset reveal LOF's superiority in accuracy, precision, recall, and False Positive Rate (FPR). We introduce additional metrics like Area Under the Curve (AUC), Matthews Correlation Coefficient (MCC), and True Negative Rate (TNR) to comprehensively evaluate LOF. Our findings highlight LOF's ability to balance false positives and false negatives, making it an ideal choice for Android malware detection. We emphasize the importance of representative datasets, such as DREBIN, for validation. In conclusion, this research positions LOF as a reliable tool for Android malware detection, offering robust protection against emerging threats. As mobile technology evolves, our study encourages further exploration of advanced techniques and real-world deployment scenarios.

A new adversarial malware detection method based on enhanced lightweight neural network

With the gradual expansion of Android systems from mobile phones to intelligent devices, a huge amount of malware has been found every year. To improve the malware detection performance and reduce its reliance on expert experience, deep learning technology has been widely used. However, as the complexity of deep learning models continues to increase, it rapidly increases the consumption of hardware resources. At the same time, anti-detection technology such as Generative Adversarial Networks (GANs) are widely used to evade Artificial Intelligence (AI)-based detection methods. In this paper, we propose a new classification model based on an improved lightweight neural network that can effectively improve the execution efficiency and detection performance of malware detection methods against adversarial malware samples. First, our method uses local-information-entropy-based image generation technology to construct effective image feature vectors. Then, the performance of the lightweight neural network model ESPNetV2 is improved from four aspects. Finally, a new adversarial malware generation model called Mal-WGANGP is proposed. It can automatically generate a large number of adversarial samples to robust our model. In order to evaluate our method, we construct several experiments and compare the detection performance of our method with 19 other novel efficient neural network detection models. Experimental results show that our image enhancement method and detection model have the highest detection accuracy of adversarial samples.

MDADroid: A novel malware detection method by constructing functionality-API mapping

As the Android ecosystem develops, malware also evolves to adapt to the changes. Consequently, malware remains a significant threat, posing a challenge in developing a low-resource consumption malware detection method that can adjust to updates in the Android API versions. We propose a novel method called MDADroid, which detects malware based on self-built Functionality-API mapping. We start by building a set of permission-related APIs using open-source knowledge. Then, we construct a Functionality-App-API heterogeneous graph based on collected data and establish a Functionality-API mapping from it. Finally, MDADroid transforms app features from the API level to the functionality level for malware detection, ensuring model resilience to API changes. We also design an API similarity calculation method that updates the Functionality-API mapping at a low cost. We evaluate MDADroid on multiple datasets, and the results show that MDADroid achieves an accuracy of 95.22%, 96.23%, 98.77%, and 99.56% on the AndroZoo, CICAndMal 2017, CICMalDroid 2020, and Drebin datasets, respectively, with training and testing times of 2 s, 0.6188 s, 1.34 s, and 1.02 s. Moreover, our method demonstrates excellent performance in the tests for resilience capabilities.

Android malware detection through centrality analysis of applications network

Android OS is a widely-used platform for mobile devices. However, with the increasing number of Android applications and ongoing advancements in application development, there is a need for flexible and scalable malware detection methods that can address the challenges posed by big data. Recently, researchers have developed methods based on complex network analysis that aim to reduce the complexity and enhance the scalability of malware detection. These methods have shown high accuracy in identifying Android malware. Our proposed method involves generating two weighted graphs that depict the relationships between applications in benign and malware states, respectively, by extracting the functions of each application. Network-analysis-based features are then extracted from the graphs and combined with static application features to distinguish malware applications from benign ones. Our approach demonstrated an increase in accuracy, achieving 99% and 98% accuracy on the DataMD and IntDroid datasets, respectively. We further demonstrate our proposed method’s superiority against state-of-the-art approaches.

Deep Image: A precious image based deep learning method for online malware detection in IoT environment

In this study, we address the challenge of online malware detection for IoT devices. We propose a method that monitors malware behavior, extracts dynamic features, and converts them into sparse binary images for analysis. The primary problem is to identify the most effective approach among clustering, probabilistic, and deep learning methods for analyzing this unique image dataset. We extract dynamic features from the monitored malware behavior, transforming them into binary images, which are then subjected to three different analysis methods. The clustering, probabilistic, and deep learning approaches are compared and evaluated in terms of various metrics. Our study contributes insights into the performance of various online malware detection approaches for IoT devices. We demonstrate that deep learning outperforms other methods, achieving the best results in seven out of eight metrics. The results of our analysis reveal that the deep learning approach exhibits the highest accuracy in seven of the eight evaluated metrics. We found that the lattice-based approach consistently returns the maximum maliciousness level, which can be instrumental in label flipping scenarios.

MDGraph: A novel malware detection method based on memory dump and graph neural network

Malware detection is of great importance to computer security. Although the malware detection approaches have made great progress in recent years, these methods are still limited in regard to identifying the advanced malware that conceals their malicious activities. To address this problem, we present a novel malware detection solution, MDGraph, which is based on the memory dump and graph neural network. MDGraph first dynamically grabs a memory dump file for a target process. Then, it applies the recursive disassembling technique to extract the program functions composed of assembly instruction sequences and the invocation relationship between functions from the memory dump. Next, the program functions are vectorized using the doc2vec model. Based on the vectorized functions and their connections, MDGraph leverages a graph neural network model for malware detection. The evaluation shows our method can identify unpacked and packed malware effectively, and it is superior to the recent malware detection methods based on the memory dump.

A COMPREHENSIVE REVIEW OF ARTIFICIAL INTELLIGENCE APPLICATIONS IN ENHANCING CYBERSECURITY THREAT DETECTION AND RESPONSE MECHANISMS

This literature review explores the transformative impact of artificial intelligence (AI) on enhancing cybersecurity measures across various domains. The study systematically examines the integration of AI in Intrusion Detection Systems (IDS), malware detection, phishing detection, threat intelligence, network security, and endpoint protection. Key findings reveal that AI-driven techniques significantly outperform traditional methods, particularly in real-time threat detection, accuracy, and adaptive response capabilities. Network-based IDS benefit from supervised and unsupervised learning algorithms, improving the identification of malicious network traffic and novel attack patterns. In malware detection, AI-enhanced static and dynamic analysis methods surpass signature-based approaches by detecting previously unknown malware and complex behaviors. Phishing detection has seen substantial improvements with AI applications in email filtering and URL analysis, reducing phishing incidents despite challenges like false positives. AI's role in threat intelligence is critical, automating data analysis to uncover hidden threats and employing predictive analytics to anticipate and mitigate cyber attacks. AI techniques in network security and endpoint protection enhance real-time monitoring and authentication processes, providing robust defenses against cyber intrusions. Despite these advancements, challenges such as handling high data volumes and the need for continuous learning to adapt to emerging threats remain. This review underscores the significant advancements, practical implementations, and ongoing challenges of leveraging AI in cybersecurity, highlighting its potential to fortify digital defenses and address the complexities of contemporary cyber threats.

A Survey on Malware Detection and Analysis

Malware, or malicious software, poses a significant threat to the security and functionality of computer systems globally. This survey provides a comprehensive analysis of current malware detection and analysis methods, focusing on data mining methodologies. The study categorizes malware detection techniques into signature-based and behaviour-based approaches, highlighting their respective strengths and weaknesses. It explores heuristic techniques enhanced by artificial intelligence, including neural networks and genetic algorithms, to improve detection accuracy. The literature review examines host-based and network-based intrusion detection systems, hybrid systems, and virtual machine introspection. The paper also discusses static and dynamic analysis methods, emphasizing the importance of analysing malware in controlled environments. Through detailed examination, this survey aims to present a thorough understanding of contemporary malware detection strategies and their applications, offering insights for future advancements in the field.

MALWARE IMAGE PREDICTION AND CLASSIFICATION USING CONVOLUTIONAL NEURAL NETWORK

Without users' permission, malware software can infect computers or other devices. Through these loopholes, criminals commit a range of illegal and criminal offences that violate the legitimate rights and interests of the nation. Traditional malware categorization techniques fall into two categories: static analysis and dynamic analysis. It is usually not necessary to execute malware binary samples in order to perform static analysis techniques, and disassembly makes it simple to recover important data such as text lists, routines, and hash values. The static analysis methods offer a high accuracy rate and a simple operation with a low consumption time. Static analysis tools, however, are limited to analyzing malware binary samples at the surface level, where they are readily influenced by deformation and other means of confusion. Furthermore, it is challenging to identify and categorize unknown malware. Methods of dynamic analysis are not impacted by obfuscation and can operate in a virtual environment. It has the ability to recognize newly discovered malware samples and track the dynamic alterations of malware binary samples over time. Nevertheless, it's an extremely intricate and time-consuming process. Malware has become one of the largest security threats in recent years due to its rapid growth. But feature engineering makes it difficult to handle large amounts of malware and readily limits the use of standard machine learning methods for malware categorization. However, dynamic analysis methodologies are not appropriate for efficiently categorizing large amounts of malware due to their complexity and high cost. In light of this, we propose a novel static malware detection method based on the convolutional neural network (CNN) employed in this work. Unlike existing methods, we use the data enhancement method to fix the unbalanced datasets, turn every viral byte into a colour image, and provide a better design.

Domain Adaptation-Based Deep Learning Framework for Android Malware Detection Across Diverse Distributions

This study addresses the challenge of Android malware detection, a critical issue due to the pervasive threats affecting mobile devices. As Android malware evolves, conventional detection methods struggle with novel or polymorphic malware that bypasses traditional defenses. This research leverages machine learning (ML) and deep learning (DL) techniques to overcome these limitations by adopting domain adaptation strategies that enhance model generalization across different distributions. The approach involves dividing a dataset into distinct distributions and applying domain adaptation techniques to ensure robustness and accuracy despite distribution shifts. Preliminary results demonstrate that domain adaptation significantly improves detection accuracy in target domains not represented in the training data. This paper showcases a domain adaptation-based method for Android malware detection, illustrating its potential to enhance security measures in dynamic environments. The findings suggest that integrating advanced ML and DL strategies with domain adaptation can substantially improve the efficacy of malware detection systems.

Approach to Detect Windows Malware Based on Malicious Tendency Image and ResNet Algorithm

Timely detection of self-replicating malware in the high market share Windows operating system can effectively prevent personal or corporate financial losses. The form and characteristics of malware are constantly evolving, leading to a concept drift issue that gradually decreases the effectiveness of traditional detection methods. Therefore, we propose WinMDet, a Windows malware detection method based on malicious tendency image and ResNet algorithm. First, to tackle the complexity and difficulty in accurately characterizing malware features, WinMDet retains detailed malware features and encodes them into malicious tendency images to better describe malware across different periods. Secondly, WinMDet utilizes previously generated malicious tendency images to train the initial detection model. Then, to alleviate the issue of malware concept drift, WinMDet employs Local Maximum Mean Discrepancy (LMMD) as the criterion for model transfer, enhancing the initial detection model’s ability to distinguish between malware and benign software. We conducted a comprehensive evaluation of WinMDet using common metrics such as accuracy, precision and recall. The results indicate that WinMDet performs remarkably well in terms of accuracy, exceeding 82%. Additionally, significant improvements were observed in precision and recall, surpassing 82.42% and 82.06%, respectively. After employing our LMMD-based transfer method, the initial detection model improved the detection accuracy of malware in 2021 and 2022 by approximately 4.22% to 8.06%. The false negative rate decreased by at most 4.34%, and the false positive rate decreased by at most 4.61%.

A New Malware Classification Framework Based on Deep Learning Algorithms

Recent advancements in computer technology have precipitated a shift towards virtual environments, accelerated by the COVID-19 pandemic. Cybercriminals have capitalized on this trend, transitioning their activities to exploit vulnerabilities in cyberspace. Malicious software (malware) has emerged as a preferred tool for launching cyber-attacks, continually evolving with sophisticated obfuscation and packing techniques to evade detection. Traditional machine learning (ML) algorithms, once effective in identifying malware, are now struggling to keep pace with these advancements. In response, deep learning (DL) algorithms offer a promising solution, leveraging their ability to discern intricate patterns and correlations within data. This study proposes a novel hybrid deep-learning-based architecture, integrating two pre-trained network models to enhance classification accuracy. Through extensive evaluation on datasets including Malimg, Microsoft BIG 2015, and Malevis, the proposed method demonstrates significant improvements in accuracy, outperforming existing ML-based malware detection methods in the literature. Specifically, the proposed method achieves an impressive accuracy of 97.78% on the Malimg dataset, underscoring its effectiveness in combating sophisticated malware variants. Keywords — Malware, malware classification, malware detection, malware variants, deep neural networks, transfer learning, deep learning.

Evolving malware detection through instant dynamic graph inverse reinforcement learning

The rapid development and increasing evolution of malware necessitate novel defensive techniques with high accuracy and minimal false positives to safeguard information systems against potential threats. Unfortunately, current malware detection methods, primarily relying on deep learning to identify malicious fingerprints from extensive training sets, prove ineffective against few-shot and evolving malware variants. To address these challenges, this paper introduces MalIRL, which designs a model-free inverse reinforcement learning (IRL) mechanism to automatically capture the evolving attack intent of malware. Specifically, MalIRL explores six representative categories of malware actions and employs sliding windows to organically divide the massive malware execution event stream into multiple attack stages, achieving a reduced action space and state space. To model dynamic malicious environments, MalIRL proposes an instant dynamic heterogeneous graph representation learning technique. This technique learns state representations of different malware attack stages, enhancing detection accuracy and efficiency by incrementally capturing the newly added contextual semantics of diverse malware entities and relations. In experiments with three real-world malware datasets, MalIRL surpasses existing state-of-the-art methods, particularly in few-shot malware detection scenarios, MalIRL exhibits performance benefits of up to 18.9%.