An Approach to Cloud Platform Log Anomaly Detection Based on Natural Language Processing and LSTM

Abstract

Cloud platform logs record platform runtime information and are important data for cloud platform anomaly detection. Due to the complex log format and rich semantic information, simple statistical analysis methods cannot fully capture log information. And the cloud platform architecture is constantly being updated, log statements are constantly evolving, and new abnormal logs may appear. In addition, most of the existing methods only perform anomaly detection on log templates, and the information is relatively one-sided, which limits the types of anomalies they can detect. Aiming at the problems that most of the current methods will not be able to diagnose or misjudge the unknown log status and miss the abnormality, this paper proposes an anomaly detection method LogNL based on (Natural Language Processing, NLP) and LSTM (Long Short Term Memory, LSTM). LogNL first uses automatic analysis methods to extract log templates and parameters, uses TF-IDF (Term Frequency–Inverse Document Frequency, TF-IDF) to obtain template feature representations, and then constructs parameter value vectors for logs of different templates, and finally uses LSTM network-based construction of pattern anomaly detection models and parameter value anomaly detection models to achieve cloud Platform log anomaly detection. Experiments on two real cloud platform log data sets show that LogNL has higher accuracy than existing supervised learning methods and unsupervised learning methods.