A graph-based approach to detect unexplained sequences in a log

Abstract

In this paper we challenge the issue of detecting anomalous events in computer systems log files, through a novel graph mining approach. The basic idea is to model log temporal sequences as a particular graph and event detection as a particular path finding problem. Thus, anomalous sequences correspond to log parts that can not be “explained” by any path in the graph. We propose a novel Iterative Partitioning Log Mining technique to parse any kind of logs and to model their temporal sequence as a probabilistic penalty graph. The approach has been implemented in a framework supporting both real time and batch processing realized on the top of the Apache Spark analytics engine for large-scale data processing. Experimental results show the advantages of the proposed framework in terms of effectiveness for different system configurations.