A system call refinement-based enhanced Minimum Redundancy Maximum Relevance method for ransomware early detection

Abstract

Ransomware is a special type of malicious software that encrypts the user's assets and makes it unavailable to the users until a ransom is paid to the ransomware author. Such attacks have become one of the most widespread malware that poses serious threat to both individuals and business organizations. Against this destructive malicious program, the dynamic analysis approach is the most popular approach for detecting such an attack. The majority of dynamic analysis relies on the system calls, as these provide an interface for programs to request service from the operating system. However, the redundancy and the irrelevant system calls that the ransomware authors inject in the actual execution flow of suspicious binaries generate a high noisy behavioural sequence that adversely impacts in the detection performance of anti-ransomware tools. To this end, we proposed a non-signature-based detection approach based on the effective windows API call sequences using supervised machine learning techniques. To achieve this objective, we propose an Enhanced Maximum-Relevance and Minimum-Redundancy (EmRmR) filter method to remove the noisy features and select the most relevant subset of features to characterize the real behaviour of the ransomware. Unlike the original mRmR, the EmRmR avoids unnecessary computations intrinsic in the original mRmR algorithms with a small number of evaluations. In addition, this work has introduced a refinement process to reduce the size of the program's call traces by removing those windows API calls that do not have a strong indication for describing the critical behaviour of the ransomware. After accomplishing extensive experimental evaluations, and comparing with existing behavioural-based detection approaches, the proposed method has shown to be effective for discriminating the behaviour of ransomware, and indicates a high detection accuracy with few false-positive rates.