Anomaly Detection Method Based on HMMs Using System Call and Call Stack Information

Abstract

Anomaly detection has emerged as an important approach to computer security. In this paper, a new anomaly detection method based on Hidden Markov Models (HMMs) is proposed to detect intrusions. Both system calls and return addresses from the call stack of the program are extracted dynamically to train and test HMMs. The states of the models are associated with the system calls and the observation symbols are associated with the sequences of return addresses from the call stack. Because the states of HMMs are observable, the models can be trained with a simple method which requires less computation time than the classical Baum-Welch method. Experiments show that our method reveals better detection performance than traditional HMMs based approaches.