Extracting the Representative API Call Patterns of Malware Families Using Recurrent Neural Network

Abstract

With thousands of malware samples pouring out every day, how can we reduce malware analysis time and detect them effectively? Malware family classification provides one of good measures to predict characteristics of unknown malware since malware belonging to the same family can have similar features. Static analysis and dynamic analysis are techniques to obtain features to be used for classifying malware samples to their families. Static analysis performs analysis based on specific signatures included in the malware. Static analysis has the advantages that the scope of the analysis covers the entire code, and the analysis can be performed without executing the malware. However, it is very difficult to detect or classify malware variants with only the results of the static analysis, because malware developers use polymorphic or encryption techniques to avoid static analysis-based detection of anti-virus software. Dynamic analysis analyzes malware behaviors, so the results of dynamic analysis can be used to detect or classify malware variants. One of dynamic features that can be used to detect or classify malware variants is API call sequences. In this paper, we propose a novel method to extract representative API call patterns of malware families using Recurrent Neural Network (RNN). We conducted experiments with 787 malware samples belonging to 9 families. In our experiments, we extracted representative API call patterns of 9 malware families on 551 samples as a training set and performed classification on the 236 samples as a test set. Classification accuracy results using API call patterns extracted from RNN were measured as 71% on average. The results show the feasibility of our approach using RNN to extract representative API call pattern of malware families for malware family classification.