Detecting malicious files using non-signature-based methods

Abstract

Malware or malicious code intends to harm the computer systems without the knowledge of system users. Malware are unknowingly installed by naive users while browsing the internet. Once installed, the malicious programs perform unintentional activities like: a) steal user name, password; b) install spy software to provide remote access to the attackers; c) flood spam messages; d) perform denial of service attacks, etc. With the emergence of metamorphic malware (that uses complex obfuscation techniques), signature-based detectors fail to identify new variants of malware. In this paper, we investigate non-signature techniques for malware detection and demonstrate methods of feature selection that are best suited for detection purposes. Features are produced using mnemonic n-grams and instruction opcodes (opcodes along with addressing modes). The redundant features are eliminated using class-wise document frequency, scatter criterion and principal component analysis (PCA). The experiments are conducted on the malware dataset collected from VX Heavens and benign executables (gathered from fresh installation of Windows XP operating system and other utility software's). The experiments also demonstrate that proposed methods that do not require signatures are effective in identifying and classifying morphed malware.