Detection of anomalies in key performance indicator data by a convolutional long short-term memory prediction model

Abstract

The rapid development of Internet technology has led to increasingly complex web service systems. The resulting large number of component interactions pose a challenge to anomaly detection, which is realized primarily by operation and maintenance (OM) personnel through the deployment, management, and monitoring of various key performance indicators (KPIs). Anomalous behaviors during daily OM often cause problems in KPI data; these problems include high noise, high dimensionality, and large-scale data streams. In addition, anomalies in KPI data occur infrequently and are of various types. These factors are the cause of the very low accuracy currently observed in conventional machine learning methods for detecting anomalies in large OM systems. Hence, a convolutional long short-term memory (C-LSTM) neural network is presented in this study to detect anomalies in small datasets that contain a variety of anomalies. First, a sliding window is used to preprocess the KPI data. Then, a C-LSTM neural network, which combines the features of the convolutional neural network (CNN) and LSTM algorithms, is employed to effectively model the time and numerical information contained in the preprocessed KPI data. Finally, the C-LSTM algorithm is tested on the datasets used in the competition of the Artificial Intelligence for Information Technology Operations (AIOPs) Active Network Management (ANM) 2018 Fall Project. The results show that the C-LSTM prediction algorithm outperforms the conventional LSTM and CNN algorithms in terms of its capacity to detect anomalies in small datasets that contain various anomalies, with a 12.90% higher accuracy, 5.68% higher recall, and 9.58% higher F1-score.