System Call Anomaly Detection Using Multi-HMMs

Abstract

This paper focuses on techniques to detect anomalous behavior in system call sequences. Since profiling complex sequential data is still an open problem in anomaly detection, there is a need to explore new approaches. While previous research has used Hidden Markov Models (HMMs) for anomaly-based intrusion detection, the proposed models tend to increase rapidly in complexity in order to increase the detection rate while reducing the false detections. In this paper, we propose a multi-HMMapproach applied for anomaly detection in clustered system call sequences. We run our experiments using the well-known system call data set provided by the University of New Mexico (UNM). Our process trace clustering approach using HMMs for system call anomaly detection provides accurate results and reduces the complexity required to detect anomalies. In this paper, we show how system call traces processed with our HMM method can provide a path forward to improved intrusion detection techniques.