MalAnalyser: An effective and efficient Windows malware detection method based on API call sequences

Abstract

Continuously evolving malware and their variants pose severe threats to information systems. API call sequence based methodologies perform well in malware detection but often suffer from high-dimensional feature sets or loss of information. Malware developers take advantage of this situation and use sophisticated coding/ obfuscation techniques to add, remove, and replace redundant API calls in malware and evade existing detection mechanisms. To address these concerns, we propose a novel and lightweight API call sequence-based Windows malware detection system, MalAnalyser. Specifically, it first extracts frequent API call subsequences (patterns) from API call sequences because frequent patterns filter superfluous API calls and highlight useful information about malware samples. Thereafter, it applies Global Local Best Particle Swarm Optimization (GLBPSO) algorithm on frequent patterns and identifies a small set of significant features that contribute to malware detection. Finally, MalAnalyser enriches frequent malware patterns using Genetic Algorithm (GA) to unveil unseen malware behavior. Extensive experimentation was conducted to evaluate the performance of MalAnalyser on different datasets and scenarios. GLBPSO considered two feature sets as input during experimentation: (i) the entire feature set and (ii) 60% randomly selected features from the entire set. Experimentation results prove that MalAnalyser attains up to 99.7% accuracy using approximately 30% features of the entire feature set. It is important to note that MalAnalyser achieves up to 100% accuracy on GA enriched feature set. Further, we evaluate the performance of MalAnalyser on a benchmark dataset, and the results proved that MalAnalyser outperforms similar approaches by achieving 99.72% accuracy. Additionally, we determined the suitability of MalAnalyser against a specific type of malware i.e. ransomware, and results showed that MalAnalyser performed remarkably well on ransomware dataset as well.