Research on anomaly detection and real-time reliability evaluation with the log of cloud platform

Abstract

With the development of software, massive information systems generate large-scale data resources involving a large amount of log information, which is of lots of meaningful contents. Accurate analysis and anomaly prediction based on logs are particularly important for building safe and reliable systems. It is evident that current anomaly prediction is not accurate enough, and there are few applications to the study of real-time reliability evaluation. Therefore, this paper uses ensemble learning model to analyze and predict anomaly of the massive system logs based on the complete procedures of log processing, including log analysis, feature extraction, anomaly detection, prediction evaluation, and real-time reliability evaluation. Compared with the traditional machine learning methods, the proposed model is able to improve the accuracy, recall rate and F1 value of anomaly prediction. The evaluation results are used to correct the real-time reliability in view of the low predicted recall rate, which greatly improves the accuracy of real-time reliability and thus provides accurate data basis and anomaly location basis for Artificial Intelligence for IT Operations. In this paper we have made the following innovations and contributions in anomaly detection and reliability measurement. The whole process was established from original log to real-time reliability measurement. New methods were adopted to improve the accuracy, recall and F1 value of anomaly detection. A real-time reliability measurement method is proposed.