Combined Data and Execution Flow Host Intrusion Detection Using Machine Learning

Abstract

We present in this chapter a novel method for detecting intrusion into host systems that combines both data and execution flow of programs. To do this, we use sequences of system call traces produced by the host’s kernel, together with their arguments. The latter are further augmented with contextual information and domain-level knowledge in the form of signatures, and used to generate clusters for each individual system call, and for each application type. The argument-driven cluster models are then used to rewrite process sequences of system calls, and the rewritten sequences are fed to a naive Bayes classifier that builds class conditional probabilities from Markov modeling of system call sequences, thus capturing execution flow. The domain level knowledge augments our machine learning-based detection technique with capabilities of deep packet inspection capabilities usually found, until now, in network intrusion detection systems. We provide the results for the clustering phase, together with their validation using the Silhouette width, the cross-validation technique, and a manual analysis of the produced clusters on the 1999 DARPA data-set from the MIT Lincoln Lab.