Enhanced capsule network‐based executable files malware detection and classification—deep learning approach

Abstract

SummaryMalware has considerably increased recently, posing a serious security danger to both people and enterprises. In order to distinguish and stop the negative effects of malware, a variety of machine and deep learning approaches have been used to detect it. However, while extracting malware features, the feature‐to‐feature spatial hierarchy is not taken into account by the existing techniques and as a result, information is lost during the pooling operation. Hence, a modified capsule deep neural network was developed in which discriminative features are extracted from three channel image derived from malware binary with considering feature‐to‐feature spatial hierarchy. Also, conventional capsule deep neural network is modified by adding a global average pooling layer before fully connected layer thereby classified the dataset as malicious or benign without any loss of information. Moreover, these malwares were not accurately classified based on their families using existing variants of convolutional neural network (CNN) since malware family variants can modify due to minute changes in malware binaries. Hence, a hybrid deep convolutional neural network (DCNN) and long‐short‐term memory (LSTM) has been utilized that determine minute changes in malware binaries using LSTM without vanishing gradient issue and effectively perform malware family classification using DCNN. As a result, the proposed approach successfully identifies malware in executable files and categorizes malware into families with 98.5% accuracy.