MeMalDet: A memory analysis-based malware detection framework using deep autoencoders and stacked ensemble under temporal evaluations

Abstract

Malware attacks continue to evolve, making detection challenging for traditional static and dynamic analysis techniques. On the other hand, memory analysis provides valuable behavioral insights, but prior research lacks temporal evaluations which are critical for robust detection of new malware variants over time. This paper presents MeMalDet, a novel memory analysis-based malware detection technique using deep autoencoders and stacked ensemble learning. We introduce an improved dataset with temporal attributes enabling more realistic evaluations of memory-based malware detection techniques under concept drift (temporal data split). MeMalDet extracts optimal features from memory dumps using deep autoencoders in an unsupervised manner, avoiding manual feature engineering. A stacked ensemble of supervised classifiers then performs highly accurate malware detection. Extensive experiments on our improved large-scale public dataset demonstrate MeMalDet’s ability to maintain high performance when detecting obfuscated malware under temporal splits. We achieve up to 98.82% accuracy and 98.72% F1-score in detecting previously unseen advanced obfuscated malware, significantly improving upon state-of-the-art memory analysis-based malware detection techniques. The improved dataset enables temporally robust evaluations, which is a novel contribution. MeMalDet combines the benefits of representation learning and supervised machine learning ensemble classification for effective malware detection over time using memory analysis. This research provides a new capability for identifying evasive modern malware and combating evolving real-world threats.