Anomaly detection on OpenStack logs based on an improved robust principal component analysis model and its projection onto column space

Abstract

AbstractWith the advent of technology and the development of more complex software systems, the size of logs generated by these systems has increasingly risen so that the anomaly detection for remediating common errors has been more difficult than ever. The cloud emergence in the information technology (IT) industry has led to the immigration of enterprises toward it, which has extended the application of cloud management stacks such as OpenStack. By using the OpenStack platform, users can access resource infrastructure and manage virtual machines (VMs). The anomaly detection in OpenStack logs is not realized conveniently due to the substantial size of logs, and it is required to automate this process. Since there is no appropriate open‐source dataset for OpenStack logs, we have generated 25,000 logs by injecting three types of anomalies to propose a more efficient technique in terms of performance and time in detecting anomalies in OpenStack logs relative to recent studies by proper OpenStack log parsing and analyzing these logs by data mining algorithms. To this end, compared to the previous research study, we could improve the anomaly detection performance in terms of F1 score, recall, and precision by 9%, 4%, and 14%, respectively, and decrease the running time relative to the log size by at least 30 s.