Research on Log Pre-processing for Exascale System Using Sparse Representation

Abstract

With system size and complexity is growing rapidly, traditional passive fault tolerance can no longer guarantee the reliability of system because of the high overhead and poor scalability of these methods. Active fault tolerance is believed to be the most important fault tolerant approach for exascale systems. Aiming at system failure prediction, this paper proposes a system logs pre-processing method using classification via sparse representation (SRCP). Adopting the idea of vectorization, SRCP removes the details of each log and generates the corresponding Vectors. It uses TF-IDF (term frequency-inverse document frequency) method to Weight each keyword which can reveal more precise information about correlation between log records. In order to improve the accuracy and flexibility of pre-processing method, log vectors are processed by sparse representation classification. For generalization purpose, SRCP does not adopt any expert system or domain knowledge. Experimental results show that, SRCP can not only achieve both outstanding precision and F-measure, but also provide a satisfactory compression ratio.