Log Template Research Articles

TPLAD: Template-Parsed Log Anomaly Detection for Electrical Database Systems

In recent years, with the increasing complexity of software systems, logs have become crucial for system maintenance. Log-based anomaly detection plays a vital role in automatically detecting system anomalies through log analysis. However, current log-based anomaly detection approaches encounter significant practical challenges. Supervised methods often require a large amount of manually labeled training data, which can be time-consuming and costly to obtain. On the other hand, unsupervised and semi-supervised approaches may suffer from subpar performance, as they do not leverage historical anomalies to improve their detection capabilities. These challenges underscore the necessity for the development of more efficient and effective log-based anomaly detection methods. Database anomaly access detection is critical for ensuring the stability and security of database systems. We present a survey of existing log anomaly detection models and propose a novel approach, Template-Parsed Log Anomaly Detection (TPLAD) model, for automated anomaly detection in massive database log files. The proposed model combines the original log template with template parsing using code and text semantic representation. Experimental results demonstrate the effectiveness of the proposed approach in detecting abnormal database access patterns, including runtime errors, unauthorized access, and data leaks. The findings indicate that TPLAD model shows promise in enhancing database security and stability in business systems.

XDrain: Effective log parsing in log streams using fixed-depth forest

Logs record rich information that can help operators diagnose system failure [1]. Analyzing logs in log streams can expedite the diagnostic process and effectively mitigate the impact of failures. Log parsing is a prerequisite for automated log analysis, which transforms semi-structured logs into structured logs. However, the effectiveness of existing parsers has only been evaluated on a limited set of logs, which lack sufficient log types. After conducting a more comprehensive evaluation of the existing log parser, we identified the following deficiencies: (1) Variable-starting logs can make some log parsers error-prone. (2) The order of logs in a log stream can have a great impact on the effectiveness. We proposes XDrain to satisfy these challenges by using fixed-depth forest. XDrain first shuffles the order of logs and the order of words within each log a few times. Secondly, XDrain will generate parsing forest for all the logs generated after the shuffling. Finally, the final log template is generated by voting. Evaluation results show that XDrain outperforms existing log parsers on two widely-used accuracy metrics and is immune to inappropriate log order. XDrain only takes about 97.89 s to parse one million logs on average.

DLLog: An Online Log Parsing Approach for Large-Scale System

Syslog is a critical data source for analyzing system problems. Converting unstructured log entries into structured log data is necessary for effective log analysis. However, existing log parsing methods demonstrate promising accuracy on limited datasets, but their generalizability and precision are uncertain when applied to diverse log data. Enhancements in these areas are necessary. This paper proposes an online log parsing method called DLLog, which is based on deep learning and has the longest common subsequence. DLLog utilizes the GRU neural network to mine template words and applies the longest common subsequence to parse log entries in real-time. In the offline stage, DLLog combines multiple log features to accurately extract the template words, creating a log template set to assist online log parsing. In the online stage, DLLog parses log entries by calculating the matching degree between the real-time log entry and the log template in the log template set. This method also supports the incremental update of the log template set to handle new log entries generated by systems. We summarized the previous works and validated DLLog using real log data collected from 16 systems. The results demonstrate that DLLog achieves high parsing accuracy, universality, and adaptability.

Polo: Adaptive Trie-Based Log Parser for Anomaly Detection

Automated log parsing is essential for many log-mining applications, as logs provide a vast range of information on events and variations within an operating system or software at runtime. Over the years, various methods have been proposed for log parsing. With improved log-parsing methods, log-mining applications can gain deeper insights into system behaviors and identify anomalies or failures promptly. However, current log parsers still face limitations, such as insufficient parsing of log templates and a lack of parallelism, as well as inaccurate log template parsing. To overcome these limitations, we have designed Polo, a parser that leverages a prefix forest composed of ternary search trees to mine templates from logs. We then conducted extensive experiments to evaluate the accuracy of Polo on nine representative system logs, achieving an average accuracy of 0.987. It is 9.93% to 40.95% faster than the state-of-the-art parsing methods. Furthermore, we evaluated our approach on a downstream log analysis task, specifically anomaly detection. The experimental results demonstrated that, in terms of F1-score, our parser outperformed Deeplog, LogAnomaly, CNN, and LogRobust by 11.5%, 4%, 1%, and 19.1%, respectively, exhibiting a promising recall score of 0.971. These results indicate the effectiveness of Polo for anomaly detection.

An Intelligent Framework for Log Anomaly Detection Based on Log Template Extraction

Log anomaly detection holds great significance in computer systems and network security. A large amount of log data is generated in the background of various information systems and equipment, so automated methods are required to identify abnormal behavior that may indicate security threats or system malfunctions. The traditional anomaly detection methods usually rely on manual statistical discovery, or match by regular expression which are complex and time-consuming. To prevent system failures, minimize troubleshooting time, and reduce service interruptions, a log template-based anomaly detection method has been proposed in this context. This approach leverages log template extraction, log clustering, and classification technology to timely detect abnormal events within the information system. The effectiveness of this method has been thoroughly tested and compared against traditional log anomaly detection systems. The results demonstrate improvements in log analysis depth, event recognition accuracy, and overall efficiency.

LPV: A Log Parsing Framework Based on Vectorization

Logs are pervasive in modern computing systems, and are valuable to service and system management. Nevertheless, with the rapidly growing size and complexity of computing systems, the log volume is exploding, which makes automatic log analysis imperative. Generally, in automatic log analysis, the first and fundamental step is log parsing, to which a lot of effort has been devoted. However, in most existing log parsing methods, log messages are merely treated as plain text. In natural language processing (NLP) area, it is a common practice to represent words and sentences with vectors, then the similarity between two words or sentences can be measured by the distance between their vectors. Inspired by these, we put forward a novel log parsing framework, named LPV (Log Parser based on Vectorization), which performs log parsing by converting log messages and log templates into vectors, with the help of a vectorization method in NLP. LPV incorporates offline and online log parsing. In the offline log parsing, the central idea is to first represent log messages with vectors, so that the similarity between two log messages can be measured by the distance between their vectors, then we cluster log messages via clustering the vectors, and finally we extract log templates from the resultant clusters. By the end of the offline log parsing, each log template is assigned with an average vector, so that in the online log parsing, the similarity between an incoming log message and each log template can also be measured by the distance between their vectors. Extensive experiments have been conducted based on several public log datasets to evaluate LPV with three different vectorization methods. The results demonstrate that, with a proper vectorization method, LPV performs competitive with state-of-the-art log parsing methods, in both effectiveness and efficiency.

Brain: Log Parsing With Bidirectional Parallel Tree

Automated log analysis can facilitate failure diagnosis for developers and operators using a large volume of logs. Log parsing is a prerequisite step for automated log analysis, which parses semi-structured logs into structured logs. However, existing parsers are difficult to apply to software-intensive systems, due to their unstable parsing accuracy on various software. Although neural network-based approaches are stable, their inefficiency makes it challenging to keep up with the speed of log production.We found that a logging statement always generate the same template words, thus, the word with the most frequency in each log is more likely to be constant. However, the identical constant and variable generated from different logging statements may break this rule Inspired by this key insight, we propose a new stable log parsing approach, called Brain, which creates initial groups according to the longest common pattern. Then a bidirectional tree is used to hierarchically complement the constant words to the longest common pattern to form the complete log template efficiently. Experimental results on 16 benchmark datasets show that our approach outperforms the state-of-the-art parsers on two widely-used parsing accuracy metrics, and it only takes around 46 seconds to process one million lines of logs.

PVE: A log parsing method based on VAE using embedding vectors

Log parsing is a critical task that converts unstructured raw logs into structured data for downstream tasks. Existing methods often rely on manual string-matching rules to extract template tokens, leading to lower adaptability on different log datasets. To address this issue, we propose an automated log parsing method, PVE, which leverages Variational Auto-Encoder (VAE) to build a semi-supervised model for categorizing log tokens. Inspired by the observation that log template tokens often consist of words, we choose common words and their combinations to serve as training data to enhance the diversity of structure features of template tokens. Specifically, PVE constructs two types of embedding vectors, the sum embedding and the n-gram embedding, for each word and word combination. The structure features of template tokens can be learned by training VAE on these embeddings. PVE categorizes a token as a template token if it is similar to the training data when log parsing. To improve efficiency, we use the average similarity between token embedding and VAE samples to determine the token type, rather than the reconstruction error. Evaluations on 16 real-world log datasets demonstrate that our method has an average accuracy of 0.878, which outperforms comparison methods in terms of parsing accuracy and adaptability.

LogPal: A Generic Anomaly Detection Scheme of Heterogeneous Logs for Network Systems

As a key resource for diagnosing and identifying problems, network syslog contains vast quantities of information. And it is the main source of data for anomaly detection of systems. Syslog presents the characteristics of large scale, diverse types and sources, data noise, and quick evolvement, which makes the detection methods not generic enough. To effectively address problem of log anomaly labelling caused by massive heterogeneous logs, we propose LogPal, a generic anomaly detection scheme of heterogeneous logs for network systems, which innovatively combines template sequences and raw log sequences to construct and generate log pattern events. By improving the self-attention mechanism of transformer, LogPal proactively synthesizes self-attention and handles log pattern events in a unique way. The model can make full use of log template and sequence semantic information, by automatically becoming aware of the pattern of logs. We implemented experiments to evaluate the performance of LogPal on publicly available datasets, and the outcome of the experiments shows that LogPal automatically adapts to log type changes and improves precision, recall, and F1 score to 99% on publicly available datasets.

MDFULog: Multi-Feature Deep Fusion of Unstable Log Anomaly Detection Model

Effective log anomaly detection can help operators locate and solve problems quickly, ensure the rapid recovery of the system, and reduce economic losses. However, recent log anomaly detection studies have shown some drawbacks, such as concept drift, noise problems, and fuzzy feature relation extraction, which cause data instability and abnormal misjudgment, leading to significant performance degradation. This paper proposes a multi-feature deep fusion of an unstable log anomaly detection model (MDFULog) for the above problems. The MDFULog model uses a novel log resolution method to eliminate the dynamic interference caused by noise. This paper proposes a feature enhancement mechanism that fully uses the correlation between semantic information, time information, and sequence features to detect various types of log exceptions. The introduced semantic feature extraction model based on Bert preserves the semantics of log messages and maps them to log vectors, effectively eliminating worker randomness and noise injection caused by log template updates. An Informer anomaly detection classification model is proposed to extract practical information from a global perspective and predict outliers quickly and accurately. Experiments were conducted on HDFS, OpenStack, and unstable datasets, showing that the anomaly detection method in this paper performs significantly better than available algorithms.

PatCluster: A Top-Down Log Parsing Method Based on Frequent Words

Logs are a combination of static message type fields and dynamic variable fields, and the accuracy of log parsing affects the result of subsequent log analysis tasks. In this regard, an offline log parsing method based on frequent words is introduced: PatCluster. This method first generates root nodes by preprocessing; secondly, the frequency of words is counted, and the word with the largest frequency is extracted as the segmentation condition to refine the template generated by the root node. So on recursively, pattern nodes are formed for all elements of the nodes, and corresponding templates are generated to finally achieve the purpose of log pattern mining. The mining process of the log patterns is from coarse to fine which is based on fewer assumptions, and the pattern fitting depth can be controlled by adjusting the termination condition. In optimized algorithm model, we also consider the maximum extent of the log template matching the token in the log message. The experimental results show that this method effectively improves the log parsing quality and has higher log parsing accuracy than other methods, and is more suitable for handling logs with complex structures.

Power monitoring system log analysis method based on K-Means clustering

The amount of log data of the power monitoring system is very large, and log analysis is difficult, which causes the power monitoring system to take more time to analyze the log, and there are problems such as low recall rate. Based on this, the design of the power monitoring system log analysis based on K-Means clustering method. For log mining and preprocessing, the K-Means clustering algorithm is used to measure the similarity and dissimilarity of the logs, clustering and locating the dominant number of similar logs in all log data, and compiling standardized documents for them, and compiling the After the data is stored, the log template is established to analyze the power monitoring system log, so as to realize the power monitoring system log analysis based on K-Means clustering. The experimental results show that the log analysis method studied effectively improves the analysis efficiency and recall rate, and can accurately analyze the log, find abnormal logs, reduce the number of false alarms, and prove the effectiveness of the log analysis method.

Amulog: A general log analysis framework for comparison and combination of diverse template generation methods*

SummaryOne of the ways to analyze unstructured log messages from large‐scale IT systems is to classify log messages with log templates generated by template generation methods. However, there is currently no common knowledge pertained to the comparison and practical use of log template generation methods because they are implemented on the basis of diverse environments. To this end, we design and implement amulog, a general log analysis framework for comparing and combining diverse log template generation methods. Amulog consists of three key functions: (1) parsing log messages into headers and segmented messages, (2) classifying the log messages using a scalable template‐matching method, and (3) storing the structured data in a database. This framework helps us easily utilize time‐series data corresponding to the log templates for further analysis. We evaluate amulog with a log dataset collected from a nation‐wide academic network and demonstrate that it classifies the log data in a reasonable amount of time even with over 100,000 log template candidates. The template‐matching method in amulog also reduces 75% processing time for template generation and keeps the accuracy when combined with an existing structure‐based template generation method. In order to show the effectiveness of amulog in comparing log template generation methods, we demonstrate that the appropriate template generation methods and accuracy metrics largely depend on the purpose of further analysis by comparing the accuracy of six existing log template generation methods with 10 different accuracy metrics on amulog.

Log Sequence Anomaly Detection Based on Local Information Extraction and Globally Sparse Transformer Model

Anomaly detection for log sequences is a necessary task for system intelligent operation and fault diagnosis. In a log sequence, adjacent logs have the property of local correlation, while long-distance logs have remote dependencies. It is helpful to fully mine these information during modeling for improving the performance of anomaly detection. Meanwhile, there are some redundant information or noise in the log sequence, which has no contribution to the detection, and may even bring negative impact. The existing methods for log sequence anomaly detection do not take the above problems into account when constructing models. In this paper, we propose LSADNET, an unsupervised log sequence anomaly detection network based on local information extraction and globally sparse Transformer model. LSADNET applies multi-layer convolution to capture the local correlation between adjacent logs, and utilizes Transformer to learn the global dependency among long-distance logs. Meanwhile, we propose a globally sparse Transformer model to improve the self-attention mechanism, which can help to retain important information adaptively and eliminate the irrelevant information in the log sequence. In addition, according to the co-occurrence mode of log templates, we put forward the calculation formula of log template transfer value, and apply it to log vectorization. Through sufficient experiments on two public datasets, it is confirmed that LSADNET has better performance than the state-of-art methods.

LTmatch: A Method to Abstract Pattern from Unstructured Log

Logs record valuable data from different software and systems. Execution logs are widely available and are helpful in monitoring, examination, and system understanding of complex applications. However, log files usually contain too many lines of data for a human to deal with, therefore it is important to develop methods to process logs by computers. Logs are usually unstructured, which is not conducive to automatic analysis. How to categorize logs and turn into structured data automatically is of great practical significance. In this paper, LTmatch algorithm is proposed, which implements a log pattern extracting algorithm based on a weighted word matching rate. Compared with our preview work, this algorithm not only classifies the logs according to the longest common subsequence(LCS) but also gets and updates the log template in real-time. Besides, the pattern warehouse of the algorithm uses a fixed deep tree to store the log patterns, which optimizes the matching efficiency of log pattern extraction. To verify the advantages of the algorithm, we applied the proposed algorithm to the open-source data set with different kinds of labeled log data. A variety of state-of-the-art log pattern extraction algorithms are used for comparison. The result shows our method is improved by 2.67% in average accuracy when compared with the best result in all the other methods.

HitAnomaly: Hierarchical Transformers for Anomaly Detection in System Log

Enterprise systems often produce a large volume of logs to record runtime status and events. Anomaly detection from system logs is crucial for service management and system maintenance. Most existing log-based anomaly detection methods use log event indexes parsed from log data to detect anomalies. Those methods cannot handle unseen log templates and lead to inaccurate anomaly detection. Some recent studies focused on the semantics of log templates but ignored the information of parameter values. Therefore, their approaches failed to address the abnormal logs caused by parameter values. In this article, we propose HitAnomaly, a log-based anomaly detection model utilizing a hierarchical transformer structure to model both log template sequences and parameter values. We designed a log sequence encoder and a parameter value encoder to obtain their representations correspondingly. We then use an attention mechanism as our final classification model. In this way, HitAnomaly is able to capture the semantic information in both log template sequence and parameter values and handle various types of anomalies. We evaluated our proposed method on three log datasets. Our experimental results demonstrate that HitAnomaly has outperformed other existing log-based anomaly detection methods. We also assess the robustness of our proposed model on unstable log data.

Log Template Extraction Algorithm Based on Normalized Feature Discrimination

Log Template Extraction Algorithm Based on Normalized Feature Discrimination

Priolog: Mining Important Logs via Temporal Analysis and Prioritization

Log analytics are a critical part of the operational management in today’s IT services. However, the growing software complexity and volume of logs make it increasingly challenging to mine useful insights from logs for problem diagnosis. In this paper, we propose a novel technique, Priolog, that can narrow down the volume of logs into a small set of important and most relevant logs. Priolog uses a combination of log template temporal analysis, log template frequency analysis, and word frequency analysis, which complement each other to generate an accurately ranked list of important logs. We have implemented this technique and applied to the problem diagnosis task of the popular OpenStack platform. Our evaluation indicates that Priolog can effectively find the important logs that hold direct hints to the failure cause in several scenarios. We demonstrate the concepts, design, and evaluation results using actual logs.

An efficient real-time data collection framework on petascale systems

High efficiency data collection remains a great challenge for HPC reliability and resilience, yet may pave the way to overcome the barrier before fault prediction. Not only for increasing scalability up to exascale systems but even for contemporary supercomputer architectures does it require substantial efforts to efficiently collect and analyze data that contains system fault information within the framework of faults prediction. In this term, the article mainly focuses on efficient data collection and data preprocessing, preferring to optimize an effective framework to improve the efficiency of data collection in petascale system. The core of our framework includes a data collection acceleration layer scheduled by H2FS, a further detailed information get by performance analysis tool, as well as a new method for log template extraction, which all attribute to a more efficient and convenient framework for the real-time data collection. Hereafter, we conducted extensive tests based on a petascale system to verify the solution, and the experimental results demonstrate to be effectiveness and scalability of our framework.

An online log template extraction method based on hierarchical clustering

The raw log messages record extremely rich system, network, and application running dynamic information that is a good data source for abnormal detection. Log template extraction is an important prerequisite for log sequence anomaly detection. The problems of the existing log template extraction methods are mostly offline, and the few online methods have insufficient F1-score in multi-source log data. In view of the shortcomings of the existing methods, an online log template extraction method called LogOHC is proposed. Firstly, the raw log messages are preprocessed, and the word distributed representation (word2vec) is used to vectorize the log messages online. Then, the online hierarchical clustering algorithm is applied, and finally, log templates are generated. The experimental analysis shows that LogOHC has a higher F1-score than the existing log template extraction methods, is suitable for multi-source log data sets, and has a shorter single-step execution time, which can meet the requirements of online real-time processing.