Clustering of Malicious Executable Files Based on the Sequence Analysis of System Calls

Abstract

The use of clustering algorithms to determine the types of malicious software files based on the analysis of the WinAPI function call sequences is investigated. The use of clustering algorithms such as k-means, EM-algorithm, hierarchical algorithm, and the affinity propagation method is considered. The quality of clustering is evaluated using the silhouette metrics, the Calinski–Harabasz index, and the Davies–Bouldin index.