Generalized anomaly detection model for windows-based malicious program behavior

Abstract

In this paper we demonstrate that it is possible in general to detect Windows-based malicious program behavior. Since S. Forrest et al. used the N-grams method to classify system call trace data, dynamic learning has become a promising research area. However, most research works have been done in the UNIX environment and have limited scope. In Forrest’s original model, “Self” is defined based on a normal process whereas “Non-Self” corresponds to one or two malicious processes. We extend this technique into the Windows environment. In our model, “Self” is defined to represent the general pattern of hundreds of Windows program behaviors; “Non-Self” is defined to represent all program behaviors that fall out of norm. Because of the difficulty in collecting program behavior, insufficient research results are available. We collected around 1000 system call traces of various normal and malicious programs in the Windows OS. A normal profile was built using a Hidden Markov Model (HMM). The evaluation was based on the entire trace. Our classification results are promising.