Anomalies In Logs Research Articles

Improving Log Anomaly Detection via Spatial Pooling: Combining SPClassifier with Ensemble Method

In the ever-updating field of software development, new bugs emerge daily, requiring significant time for analysis. As a result, research is being conducted on automating bug resolution using techniques such as anomaly detection through deep learning applied to text logs. This study focuses on anomaly detection using text logs and aims to address current challenges. Specifically, we aim to improve the accuracy of SPClassifier, a robust and lightweight AI model capable of handling dynamic log datasets through ad-hoc learning. We employ three ensemble learning methods to enhance the accuracy of SPClassifier. The method that achieved the greatest improvement was Improved Bagging, which combines the non-overlapping sampling of Pasting with the overlapping sampling of Bagging, resulting in a maximum F1-score improvement of 155%. Additionally, on certain datasets, the F1-score surpassed that of well-known DNN methods by 130%. Furthermore, the proposed method demonstrated lower variance compared to DNN methods, indicating its advantage, particularly in environments where datasets frequently fluctuate, such as development fields. These results highlight the clear superiority of the proposed method, which is lightweight in terms of computational resources and supports ad-hoc learning.

ContexLog: Non-Parsing Log Anomaly Detection With All Information Preservation and Enhanced Contextual Representation

ContexLog: Non-Parsing Log Anomaly Detection With All Information Preservation and Enhanced Contextual Representation

Neural Network-Based Log Anomaly Detection Algorithm for 6G Wireless Integrated Cyber-Physical System

Neural Network-Based Log Anomaly Detection Algorithm for 6G Wireless Integrated Cyber-Physical System

Few-Shot Log Anomaly Detection Based on Matching Networks

Few-Shot Log Anomaly Detection Based on Matching Networks

Abnormal User Behavior Detection for Industry Big Data Analysis

Abstract The security of user information and the precision of user services are paramount, necessitating effective detection of abnormal user behavior in vast datasets. This study introduces the QGAN-BDE algorithm, which leverages a quantum generative adversarial network combined with a novel approach for detecting and evaluating abnormal user behavior. Through a feature matching strategy, the algorithm ensures close data alignment between the generator and discriminator. At the same time, the integration of a classical convolutional neural network within the BDE network assesses user behavior abnormalities. Setting distinct thresholds for abnormal behavior and threats enables the differentiation between normal and abnormal activities. Utilizing a dataset and financial stock log data for simulation, the proposed method achieves an AUC value of approximately 0.912 with small negative data samples. Additionally, it records generator and discriminator loss values within the ranges of [1.05,1.55] and [0.49,0.61], respectively, and demonstrates over 80% accuracy in detecting financial stock log anomalies. This method’s reliance on comprehensive big data allows for an in-depth analysis of user behavior, facilitating the timely identification and management of abnormalities.

Unsupervised Strategies In Detecting Log Anomalies using AIOps Monitoring to Amplify Performance by PCA and ANN Systems

Unsupervised Strategies In Detecting Log Anomalies using AIOps Monitoring to Amplify Performance by PCA and ANN Systems

Unsupervised strategies in detecting log anomalies using AIOps monitoring to amplify performance by PCA and ANN systems

Unsupervised strategies in detecting log anomalies using AIOps monitoring to amplify performance by PCA and ANN systems

Landscape and Taxonomy of Online Parser-Supported Log Anomaly Detection Methods

Landscape and Taxonomy of Online Parser-Supported Log Anomaly Detection Methods

SigML++: Supervised Log Anomaly with Probabilistic Polynomial Approximation

Security log collection and storage are essential for organizations worldwide. Log analysis can help recognize probable security breaches and is often required by law. However, many organizations commission log management to Cloud Service Providers (CSPs), where the logs are collected, processed, and stored. Existing methods for log anomaly detection rely on unencrypted (plaintext) data, which can be a security risk. Logs often contain sensitive information about an organization or its customers. A more secure approach is always to keep logs encrypted (ciphertext). This paper presents “SigML++”, an extension of “SigML” for supervised log anomaly detection on encrypted data. SigML++ uses Fully Homomorphic Encryption (FHE) according to the Cheon–Kim–Kim–Song (CKKS) scheme to encrypt the logs and then uses an Artificial Neural Network (ANN) to approximate the sigmoid (σ(x)) activation function probabilistically for the intervals [−10,10] and [−50,50]. This allows SigML++ to perform log anomaly detection without decrypting the logs. Experiments show that SigML++ can achieve better low-order polynomial approximations for Logistic Regression (LR) and Support Vector Machine (SVM) than existing methods. This makes SigML++ a promising new approach for secure log anomaly detection.

LogBASA: Log Anomaly Detection Based on System Behavior Analysis and Global Semantic Awareness

System log anomaly detection is important for ensuring stable system operation and achieving rapid fault diagnosis. System log sequences include data on the execution paths and time stamps of system tasks in addition to a large amount of semantic information, which enhances the reliability and effectiveness of anomaly detection. At the same time, considering the correlation between system log sequences can effectively improve fault diagnosis efficiency. However, the existing system log anomaly detection methods mostly consider only the sequence patterns or semantic information on the logs, so their anomaly detection results show a high rate of missed and false alarms. To solve these problems, this paper proposed an unsupervised log anomaly detection model (LogBASA) based on the system behavior analysis and global semantic awareness, aiming to decrease the leakage rate and increase the log sequence anomaly detection accuracy. First, a system log knowledge graph was constructed based on massive, unstructured, and multilevel system log data to represent log sequence patterns, which facilitates subsequent anomaly detection and localization. Then, a self-attention encoder-decoder transformer model was developed for log spatiotemporal association analysis. This model combines semantic mapping and spatiotemporal features of log sequences to analyze system behavior and log semantics in multiple dimensions. Furthermore, a system log anomaly detection method that combines adaptive spatial boundary delineation and sequence reconstruction objective functions was proposed. This method uses special words to characterize the log sequence states, delineates anomaly boundaries automatically, and reconstructs log sequences through unsupervised training for anomaly detection. Finally, the proposed method was verified by numerous experiments on three real datasets. The results indicate that the proposed method can achieve an accuracy rate of 99.3%, 95.1%, and 97.2% on HDFS, BGL, and Thunderbird datasets, which proves the effectiveness and superiority of the LogBASA model.

An Intelligent Framework for Log Anomaly Detection Based on Log Template Extraction

Log anomaly detection holds great significance in computer systems and network security. A large amount of log data is generated in the background of various information systems and equipment, so automated methods are required to identify abnormal behavior that may indicate security threats or system malfunctions. The traditional anomaly detection methods usually rely on manual statistical discovery, or match by regular expression which are complex and time-consuming. To prevent system failures, minimize troubleshooting time, and reduce service interruptions, a log template-based anomaly detection method has been proposed in this context. This approach leverages log template extraction, log clustering, and classification technology to timely detect abnormal events within the information system. The effectiveness of this method has been thoroughly tested and compared against traditional log anomaly detection systems. The results demonstrate improvements in log analysis depth, event recognition accuracy, and overall efficiency.

Loader: A Log Anomaly Detector Based on Transformer

Detecting anomalies in logs is crucial for service and system management, since logs are widely used to record the runtime status, and are often the only data available for postmortem analysis. Since anomalies are usually rare in real-world services and systems, a common and feasible practice is to mine or learn normal patterns from logs, and deem those violating the normal patterns as anomalies. As log sequences are a kind of time series data, RNN (Recurrent Neural Network) and its variants have been extensively employed to capture the normal patterns. Nevertheless, the sequential nature of RNN and its variants makes them hard to parallelize and capture long-term dependencies, which may hinder their performance. To address this issue, in this paper we propose Loader, a novel semi-supervised <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><b>lo</b></u> g <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><b>a</b></u> nomaly <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><b>d</b></u> etector based on Transform <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><b>er</b></u> , because the Transformer architecture eschews recurrence and is able to draw global dependencies. Loader leverages the Transformer encoder to capture normal patterns from normal log sequences. When detecting, it gives a set of candidate log templates, that may appear after the input log substring under normal conditions. If the template of the actual next log message is not within the candidate set, this implies an anomaly. Previous similar methods select the most possible <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k$</tex-math></inline-formula> log templates as candidates in any case, so the performance is sensitive to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k$</tex-math></inline-formula> , and it is nontrivial to pick a proper <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k$</tex-math></inline-formula> . To alleviate this, we design a more flexible and robust ‘top- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$p$</tex-math></inline-formula> ’ algorithm, which determines the candidate set based on the cumulative probability of the most possible log templates. Extensive experiments are conducted based on three public log datasets, the experimental results validate the effectiveness and competitiveness of our approach.

MLog: Mogrifier LSTM-Based Log Anomaly Detection Approach Using Semantic Representation

MLog: Mogrifier LSTM-Based Log Anomaly Detection Approach Using Semantic Representation

Development of Anomaly Detection System Based on Distributed Log Tracing

Software system developers must respond quickly to failures in order to avoid reputational and financial losses for their customers. Therefore, it is important to detect behavioral anomalies in the operation of software systems in a timely manner. At the moment, various tools for automatic monitoring of systems are being actively developed, but logs are the main tool for analyzing failures. Logs contain information about the operation of the system at various points of execution. Modern systems often have a distributed microservice architecture, which significantly complicates the task of analyzing logs. Logs of such systems are collected centrally from different microservices, forming a huge flow of information that is very difficult to analyze manually. However, the problem of identifying logs related to a specific request to the system is solved by distributed tracing, the use of which opens up wide opportunities for the introduction of automatic analysis. There are already many solutions for detecting anomalies in logs, but they do not take advantage of distributed tracing. The article is considered to solving the problem of detecting behavioral anomalies in the work of distributed software systems based on automatic analysis of log traces. The solution is based on the synthesis of machine learning methods. Log traces are preprocessed and cleaned using process mining methods. Next, vectorization and clustering of log messages is performed. After that, a long short-term memory network (LSTM) is used to analyze deviations in the sequences of processed logs. As a result of the work performed, a prototype of the anomaly detection system was developed and tested.

Fracture identification in reservoirs using well log data by window sliding recurrent neural network

Detecting fractures using well logs can be difficult due to the complex response of conventional logs. To address this issue, a novel method called Fracture Identification by window sliding and recurrent neural network (FIsr) is proposed. FIsr uses window sliding to generate sequence image data for training a bidirectional recurrent neural network (BiLSTM) classifier, with columns selected from both conventional and reconstructed logs. Undersampling is applied to balance the data, as the number of fracture samples is much smaller than nonfracture samples. BiLSTM extracts features from the sequence data in two directions, considering label correlations and detecting local log anomalies caused by fractures. The prediction for each sample is based on multiple overlapping sequence images to reduce uncertainties. The proposed method is validated using a dataset from carbonate reservoirs of the Asmari Formation in the Middle East, with an accuracy of 95% and recall and precision metrics exceeding 90%. A blind well test shows that FIsr can detect all fracture zones, and the distribution of fractures along the well trajectory confirms previous knowledge of the area. The study also discusses the influence of factors in FIsr.

LogBD: A Log Anomaly Detection Method Based on Pretrained Models and Domain Adaptation

The log data generated during operation of a software system contain information about the system, and using logs for anomaly detection can detect system failures in a timely manner. Most existing log anomaly detection methods are specific to a particular system, have cold-start problems, and are sensitive to updates in log format. In this paper, we propose a log anomaly detection method LogBD based on pretrained models and domain adaptation, which uses the pretraining model BERT to learn the semantic information of logs. This method can solve problems caused by the multiple meaning of words and log statement updates. The distance to determine anomalies in LogBD is constructed on the basis of domain adaptation, using TCNs to extract common features of different system logs and mapping them to the same hypersphere space. Lastly, experiments were conducted on two publicly available datasets to evaluate the method. The experimental results showed that the method can better solve the log instability problem and exhibits some improvement in the cross-system log anomaly detection effect.

SSDLog: a semi-supervised dual branch model for log anomaly detection

With versatility and complexity of computer systems, warning and errors are inevitable. To effectively monitor system’s status, system logs are critical. To detect anomalies in system logs, deep learning is a promising way to go. However, abnormal system logs in the real world are often difficult to collect, and effectively and accurately categorize the logs is an even time-consuming project. Thus, the data incompleteness is not conducive to the deep learning for this practical application. In this paper, we put forward a novel semi-supervised dual branch model that alleviate the need for large scale labeled logs for training a deep system log anomaly detector. Specifically, our model consists of two homogeneous networks that share the same parameters, one is called weak augmented teacher model and the other is termed as strong augmented student model. In the teacher model, the log features are augmented with small Gaussian noise, while in the student model, the strong augmentation is injected to force the model to learn a more robust feature representation with the guidance of teacher model provided soft labels. Furthermore, to further utilize unlabeled samples effectively, we propose a flexible label screening strategy that takes into account the confidence and stability of pseudo-labels. Experimental results show favorable effect of our model on prevalent HDFS and Hadoop Application datasets. Precisely, with only 30% training data labeled, our model can achieve the comparable results as the fully supervised version.

Distributed system anomaly detection using deep learning‐based log analysis

AbstractAnomaly detection is a key step in ensuring the security and reliability of large‐scale distributed systems. Analyzing system logs through artificial intelligence methods can quickly detect anomalies and thus help maintenance personnel to maintain system security. Most of the current works only focus on the temporal or spatial features of distributed system logs, and they cannot sufficiently extract the global features of distributed system logs to achieve a good correct rate of anomaly detection. To further address the shortcomings of existing methods, this paper proposes a deep learning model with global spatiotemporal features to detect the presence of anomalies in distributed system logs. First, we extract semi‐structured log events from log templates and model them as natural language. In addition, we focus on the temporal characteristics of logs using the bidirectional long short‐term memory network and the spatial invocation characteristics of logs using the Transformer. Extensive experimental evaluations show the advantages of our proposed model for distributed system log anomaly detection tasks. The optimal F1‐Score on three open‐source datasets and our own collected distributed system datasets reach 98.04%, 94.34%, 88.16%, and 97.40%, respectively.

LogPal: A Generic Anomaly Detection Scheme of Heterogeneous Logs for Network Systems

As a key resource for diagnosing and identifying problems, network syslog contains vast quantities of information. And it is the main source of data for anomaly detection of systems. Syslog presents the characteristics of large scale, diverse types and sources, data noise, and quick evolvement, which makes the detection methods not generic enough. To effectively address problem of log anomaly labelling caused by massive heterogeneous logs, we propose LogPal, a generic anomaly detection scheme of heterogeneous logs for network systems, which innovatively combines template sequences and raw log sequences to construct and generate log pattern events. By improving the self-attention mechanism of transformer, LogPal proactively synthesizes self-attention and handles log pattern events in a unique way. The model can make full use of log template and sequence semantic information, by automatically becoming aware of the pattern of logs. We implemented experiments to evaluate the performance of LogPal on publicly available datasets, and the outcome of the experiments shows that LogPal automatically adapts to log type changes and improves precision, recall, and F1 score to 99% on publicly available datasets.

A New Distributed Log Anomaly Detection Method based on Message Middleware and ATT-GRU

A New Distributed Log Anomaly Detection Method based on Message Middleware and ATT-GRU