Performance evaluations of AI‐based obfuscated and encrypted malicious script detection with feature optimization

Abstract

AbstractIn the digital security environment, the obfuscation and encryption of malicious scripts are primary attack methods used to evade detection. These scripts—easily spread through websites, emails, and file downloads—can be automatically executed on users' systems, posing serious security threats. To overcome the limitations of signature‐based detection methods, this study proposed a methodology for real‐time detection of obfuscated and encrypted malicious scripts using ML/DL models with feature optimization techniques. The obfuscated script datasets were analyzed to identify the unique characteristics, classified into 16 feature sets, to evaluate the optimal features for the best detection accuracy. Although the detection accuracy of these datasets was < 20%, when tested with commercial antivirus services, the experimental results using ML and DL models demonstrated that the proposed light gradient boosting model (LGBM) could achieve the best detection accuracy and processing speed. The LGBM outperformed other artificial intelligence models by achieving 97% accuracy and the minimum processing time in the decoded, obfuscated, and encrypted dataset cases.