Event Block Identification and Analysis for Effective Anomaly Detection to Build Reliable HPC Systems

Abstract

System logs provide invaluable resources for understanding system behavior and detecting anomalies on high performance computing (HPC) systems. As HPC systems continue to grow in both scale and complexity, the sheer volume of system logs and the complex interaction among system components make the traditional manual problem diagnosis and even automated line-by-line log analysis infeasible or ineffective. In this paper, we present a System Log Event Block Detection (SLEBD) framework that identifies groups of log messages that follow certain sequence but with variations, and explore these event blocks for event-based system behavior analysis and anomaly detection. Compared with the existing approaches that analyze system logs line by line, SLEBD is capable of characterizing system behavior and identifying intricate anomalies at a higher (i.e., event) level. We evaluate the performance of SLEBD by using syslogs collected from production supercomputers. Experimental results show that our framework and mechanisms can process streaming log messages, efficiently extract event blocks and effectively detect anomalies, which enables system administrators and monitoring tools to understand and process system events in real time. Additionally, we use the identified event blocks and explore deep learning algorithms to model and classify event sequences.