Detecting Anomalous Events on Distributed Systems Using Convolutional Neural Networks

Abstract

Detection of anomalous events is very crucial for the maintenance and performance tuning in long-running distributed systems. System logs contain the complete information of system operation that can be used for describing the situations of the computing nodes. However, log messages are unstructured and difficult to utilize. In this work, we propose a novel anomaly detection framework in a Hadoop Distributed File System (HDFS) that transforms the log messages to structured data and automatically monitors the system operation logs using Convolutional Neural Networks (CNN). We evaluate the performance of anomaly detection in terms of precision, recall, and f-measure. The proposed framework can provide with precision = 94.76 ± 0.81%, recall = 99.53 ± 0.23%, and f-measure = 97.09 ± 0.49%. To apply the proposed framework in the practical application, we also concern about the training time and prediction productivity. From our experimental results, our proposed framework outperforms the existing models (i.e., LSTM and Bi-LSTM) with higher recall, lower training time, and higher prediction productivity.