Malware Clustering Research Articles

Scalable and Fast Algorithm for Constructing Phylogenetic Trees With Application to IoT Malware Clustering

With the development of IoT devices, there is a rapid increase in new types of IoT malware and variants, causing social problems. The malware’s phylogenetic tree has been used in many studies for malware clustering or better understanding of malware evolution. However, when dealing with a large-scale malware set, conventional methods for constructing a phylogenetic tree is very time-consuming or even cannot be done in a realistic time. To solve this problem, we propose a high-speed, scalable phylogenetic tree construction algorithm with a clustering algorithm to cluster it. The proposed method involves the following steps: (1) Calculating the similarity of the specimen pairs using the normalized compression distance. (2) Creating a phylogenetic tree containing all specimens, instead of calculating the similarity of all pairs of a specimen, our algorithm only calculates a small part of the similarity matrix. (3) Dividing the phylogenetic tree into clusters by applying the minimum description length criterion. In addition, we propose a new online processing algorithm to add new malware specimens into the existing phylogenetic tree sequentially. Our goal is to reduce the computational cost of constructing the phylogenetic tree and improve the clustering accuracy of our previous research. We evaluated our method’s clustering accuracy and scalability with 65,494 IoT malware specimens. The results showed that our algorithm reduced the computation by 97.52% compared with the conventional method. Our clustering algorithm achieved accuracies of 95.5% and 99.3% for clustering family name and architecture name, respectively.

Enhancing DNN-Based Binary Code Function Search With Low-Cost Equivalence Checking

Binary code function search has been used as the core basis of various security and software engineering applications, including malware clustering, code clone detection, and vulnerability audits. Recognizing logically similar assembly functions, however, remains a challenge. Most binary code search tools rely on program structure-level information, such as control flow and data flow graphs, that is extracted using program analysis techniques or deep neural networks (DNNs). However, DNN-based techniques capture lexical-, control structure-, or data flow-level information of binary code for representation learning, which is often too coarse-grained and does not accurately denote program functionality. Additionally, it may exhibit low robustness to a variety of challenging settings, such as compiler optimizations and obfuscations. This paper proposes a general solution for enhancing the top- <inline-formula><tex-math notation="LaTeX">$k$</tex-math></inline-formula> ranked candidates in DNN-based binary code function search. The key idea is to design a low-cost and comprehensive equivalence check that quickly exposes functionality deviations between the target function and its top- <inline-formula><tex-math notation="LaTeX">$k$</tex-math></inline-formula> matched functions. Functions that fail this equivalence check can be shaved from the top- <inline-formula><tex-math notation="LaTeX">$k$</tex-math></inline-formula> list, and functions that pass the check can be revisited to move ahead on the top- <inline-formula><tex-math notation="LaTeX">$k$</tex-math></inline-formula> ranked candidates, in a deliberate way. We design a practical and efficient equivalence check, named <monospace>BinUSE</monospace> , using <i>under-constrained</i> symbolic execution (USE). USE, a variant of symbolic execution, improves scalability by initiating symbolic execution directly from function entry points and relaxing constraints on function parameters. It eliminates the overhead incurred by path explosion and costly constraints. <monospace>BinUSE</monospace> is specifically designed to deliver an assembly function-level equivalence check, enhancing DNN-based binary code search by reducing its false alarms with low cost. Our evaluation shows that <monospace>BinUSE</monospace> can enable a general and effective enhancement of four state-of-the-art DNN-based binary code search tools when confronted with challenges posed by different compilers, optimizations, obfuscations, and architectures.

ADCAS: Adversarial Deep Clustering of Android Streams

The data sequences which are used in malware analysis can be attacked in many applications. However, adversarial attacks are rarely regarded in these types of data. In this paper, a deep learning-based malware clustering approach for sequential data was proposed and the impact of deploying adversarial attacks on it was investigated. An input data stream of android applications was considered as a sequence and the proposed method was tested with the extracted static features of android applications. Three android benchmark datasets, Drebin, Genome, and Contagio, are used to assess the proposed approach. In most experiments, the False Positive Rate (FPR) values of deep clustering algorithms increase to over 60% after the attack, according to the obtained results. Also, the accuracy rates drop to less than 83% in all cases. But by applying the proposed defense method the FPR values reduced while accuracy rates increased.

Semi-Supervised Malware Clustering Based on the Weight of Bytecode and API

With the rapid advances of anti-virus and anti-tracking technologies, three aspects in malware clustering need to be improved for effective clustering, i.e., the robustness of features, the accuracy of similarity measurements, and the effectiveness of clustering algorithms. In this paper, we propose a novel malware family clustering approach based on dynamic and static features with their weights. In this approach, we employ a new similarity measurement method based on EMD to improve the accuracy of feature similarities. In addition, to reduce convergence time and improve clustering purity, we design a novel semi-supervised clustering algorithm, termed as S-DBSCAN by involving supervision information into the original algorithm known as Density-Based Spatial Clustering of Applications with Noise (DBSCAN). The experimental results demonstrate that the proposed approach can correctly and accurately distinguish the samples among various families and achieve outperformed purity with 98.7%.

Familial Clustering for Weakly-Labeled Android Malware Using Hybrid Representation Learning

Labeling malware or malware clustering is important for identifying new security threats, triaging and building reference datasets. The state-of-the-art Android malware clustering approaches rely heavily on the raw labels from commercial AntiVirus (AV) vendors, which causes misclustering for a substantial number of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">weakly-labeled malware</i> due to the inconsistent, incomplete and overly generic labels reported by these closed-source AV engines, whose capabilities vary greatly and whose internal mechanisms are opaque (i.e., intermediate detection results are unavailable for clustering). The raw labels are thus often used as the only important source of information for clustering. To address the limitations of the existing approaches, this paper presents Andre, a new ANDroid Hybrid REpresentation Learning approach to clustering weakly-labeled Android malware by preserving heterogeneous information from multiple sources (including the results of static code analysis, the meta-information of an app, and the raw-labels of the AV vendors) to jointly learn a hybrid representation for accurate clustering. The learned representation is then fed into our outlier-aware clustering to partition the weakly-labeled malware into known and unknown families. The malware whose malicious behaviours are close to those of the existing families on the network, are further classified using a three-layer Deep Neural Network (DNN). The unknown malware are clustered using a standard density-based clustering algorithm. We have evaluated our approach using 5,416 ground-truth malware from Drebin and 9,000 malware from VirusShare (uploaded between Mar. 2017 and Feb. 2018), consisting of 3324 weakly-labeled malware. The evaluation shows that Andre effectively clusters weakly-labeled malware which cannot be clustered by the state-of-the-art approaches, while achieving comparable accuracy with those approaches for clustering ground-truth samples.

Static malware clustering using enhanced deep embedding method

SummaryMalware refers to any software, programs, or files that are intentionally utilised to compromise the system and cause unexpected losses to end‐users such as economical losses or privacy breaches. The rapid growth of malware makes it impossible to keep up with its progress merely via human interventions or manual analysis. One of the challenges for the human‐oriented approaches is they will cause backlog and inability to keep up with the development traces of the malware. Hence, an efficient method is needed urgently to analyse effectively and identify accurately the malware in their domain. Malware clustering has been extensively studied in the machine learning area with regards to distance functions, grouping algorithm and cluster validation. A large number of research studies have been done via behavioral analysis for clustering to achieve high performance of malware detections. However, there is a trade‐off for better detection performance between behaviorial approaches and high computational forces. Up to date, little work focuses on the deep learning representations for malware clustering. Therefore, in this paper, we propose an enhanced deep embedded clustering method to facilitate an effective and efficient malware clustering process. The new method takes advantage of linear dimensionality reduction and a customised deep neural network to learn malware representations in an orthogonal space and performs cluster assignments. Our experimental results demonstrate that the proposed clustering model outperforms the traditional K‐means method with regards to the enhanced features using various auto‐encoder, pre‐trained weight and principle component analysis (PCA).

Behavioral-based malware clustering and classification

Detection of malwares and security attacks is a complex process that can vary in its details, analysis activities, etc. As part of the detection process, malware scanners try to categorize a malware once it is detected under one of the known malware categories (e.g. worms, spywares, viruses, etc.). However, many studies and researches indicate problems with scanners categorizing or identifying a particular malware under different categories. There are different reasons for such challenges where different malware scanners, and sometime the same malware scanner, will categorize the same malware under different categories in different times or instances. In this paper, we evaluated this problem summarizing existing approaches on malware classification.

Spatially Clustered Autonomous Vehicle Malware: Producing New Urban Geographies of Inequity

Malicious software (malware) is both a hurdle to autonomous vehicle (AV) adoption and a serious threat to AV occupant safety. Yet, to date, the topic of how subnational transportation decision makers should handle this cybersecurity threat has been underexplored. This paper describes how malware can spread through vehicle-to-X networks and infect AVs to a non-technical planning audience. It then explains how AV malware can cluster spatially in linguistic, socioeconomic, and political enclaves of American cities. Since AVs can identify new indicators of safety imperceptible to humans, malware clusters can produce new geographies of accessibility, mobility, economic, and environmental inequity across a city. In provisioning the inherently localized technical malware prevention tools available to them, subnational transportation planners have the capacity to reproduce or overcome historical and current transportation system inequities.

A Spatio-Temporal malware and country clustering algorithm: 2012 IIJ MITF case study

A huge number of botnet malware variants can be downloaded by zombie personal computers as secondary injections and upgrades according to their botmasters to perform different distributed/coordinated cyber attacks such as phishing, spam e-mail, malicious Web sites, ransomware, DDoS. In order to generate a faster response to new threats and better understanding of botnet activities, grouping them based on their malicious behaviors has become extremely important. This paper presents a Spatio-Temporal malware clustering algorithm based on its (weekly-hourly-country) features. The dataset contains more than 32 million of malware download logs from 100 honeypots set up by Malware Investigation Task Force (MITF) of Internet Initiative Japan Inc. (IIJ) from 2011 to 2012. The Top-20 malware clustering results coincidentally correspond to Conficker.B and Conficker.C with relatively high precision and recall rates up to 100.0, 88.9 % and 91.7, 100.0 %, respectively. On the other hand, the resulting two clusters of Top-20 countries are comparable to those with high and low growth rates recently reported in 2015 by Asghari et al. Therefore, our approach can be validated and evaluated to yield precision and recall of up to 75.0 and 86.7 %, respectively.

Impeding behavior-based malware analysis via replacement attacks to malware specifications

As the underground market of malware flourishes, there is an exponential increase in the number and diversity of malware. A crucial question in malware analysis research is how to define malware specifications or signatures that faithfully describe similar malicious intent and also clearly stand out from other programs. Although the traditional malware specifications based on syntactic signatures are efficient, they can be easily defeated by various obfuscation techniques. Since the malicious behavior is often stable across similar malware instances, behavior-based specifications which capture real malicious characteristics during run time, have become more prevalent in anti-malware tasks, such as malware detection and malware clustering. This kind of specification is typically extracted from the system call dependence graph that a malware sample invokes. In this paper, we present replacement attacks to camouflage similar behaviors by poisoning behavior-based specifications. The key method of our attacks is to replace a system call dependence graph to its semantically equivalent variants so that the similar malware samples within one family turn out to be different. As a result, malware analysts have to put more efforts into reexamining the similar samples which may have been investigated before. We distil general attacking strategies by mining more than 5200 malware samples’ behavior specifications and implement a compiler-level prototype to automate replacement attacks. Experiments on 960 real malware samples demonstrate the effectiveness of our approach to impede various behavior-based malware analysis tasks, such as similarity comparison and malware clustering. In the end, we also discuss possible countermeasures in order to strengthen existing malware defense.

Clustering versus SVM for malware detection

Previous work has shown that cluster analysis can be used to effectively classify malware into meaningful families. In this research, we apply cluster analysis to the challenging problem of classifying previously unknown malware. We perform several experiments involving malware clustering. We compare our clustering results to those obtained when a support vector machine (SVM) is trained on the malware family. Using clustering, we are able to classify malware with an accuracy comparable to that of an SVM. An advantage of the clustering approach is that a new malware family can be classified before a model has been trained specifically for the family.

Scalable fine-grained behavioral clustering of HTTP-based malware

A large number of today’s botnets leverage the HTTP protocol to communicate with their botmasters or perpetrate malicious activities. In this paper, we present a new scalable system for network-level behavioral clustering of HTTP-based malware that aims to efficiently group newly collected malware samples into malware family clusters. The end goal is to obtain malware clusters that can aid the automatic generation of high quality network signatures, which can in turn be used to detect botnet command-and-control (C&C) and other malware-generated communications at the network perimeter.We achieve scalability in our clustering system by simplifying the multi-step clustering process proposed in [31], and by leveraging incremental clustering algorithms that run efficiently on very large datasets. At the same time, we show that scalability is achieved while retaining a good trade-off between detection rate and false positives for the signatures derived from the obtained malware clusters. We implemented a proof-of-concept version of our new scalable malware clustering system and performed experiments with about 65,000 distinct malware samples. Results from our evaluation confirm the effectiveness of the proposed system and show that, compared to [31], our approach can reduce processing times from several hours to a few minutes, and scales well to large datasets containing tens of thousands of distinct malware samples.