Network anomaly detection based on keyword embedding log

Abstract

Log anomaly detection is an important and challenging task in the field of Artificial Intelligence for IT Operations (AIOps). Logs that record important runtime information are widely used for troubleshooting purposes. There have been many studies that use log data to construct deep learning methods for detecting system anomalies, which are usually based on log parsing. However, they ignore the effect of keywords that are promising for system status analysis. Here, we propose KELog (Keyword Embedding Log), a novel log anomaly detection approach that utilizes keyword information. We build a keyword library by keyword information extraction and fuse them into log representations. In this way, KELog can raise the reliability of anomaly detection. The experimental results on a real-world log dataset of a communications operator show that the F1 score of our proposed KELog method achieves a maximum increase of 0.341 compared with the commonly used machine learning algorithms (PCA, SVM, Invaiant Mining) and a maximum increase of 0.039 compared with deep learning algorithms (DeepLog, LogBERT) respectively. In 2021, ITU launched the second ITU AI/ML in 5G Challenge. We used KELog to participate in the thematic track of the Artificial Intelligence Innovation and Application Competition in the China Division, and won first place with a full F1 score.