Runtime State Research Articles

TraStrainer: Adaptive Sampling for Distributed Traces with System Runtime State

TraStrainer: Adaptive Sampling for Distributed Traces with System Runtime State

PreLog: A Pre-trained Model for Log Analytics

Large-scale software-intensive systems often produce a large volume of logs to record runtime status and events for troubleshooting purposes. The rich information in log data enables a variety of system management and diagnosis tasks. Over the years, many approaches have been proposed for automated log analytics. However, these approaches usually design separate models for each specific task, which cannot be generalized to other tasks. They are also not robust when dealing with logs from heterogeneous sources. In this paper, we propose PreLog, a novel pre-trained model for log analytics. PreLog is pre-trained on a large amount of unlabelled log data to capture the semantic meaning of logs. We design two log-specific pre-training objectives, including entry-level and sequence-level objectives, which enable PreLog to better understand the hidden structure and semantics of logs. To perform downstream log analytics tasks, we leverage a prompt tuning paradigm to convert downstream tasks' objectives into a similar form as the pre-training stage. We have conducted extensive experiments on two main log analytics tasks (i.e., log parsing and log-based anomaly detection). Experimental results show that PreLog achieves better or comparable results in comparison with the state-of-the-art, task-specific approaches. PreLog is cost-effective and can be uniformly applied to many log analytics tasks through the prompt tuning paradigm.

Software Failure Prediction Based On Program State and First-Error Characteristics

Abstract Evaluating program reliability against software faults involves the work of analyzing program failure against software faults as behavior on failures is fundamental for assessing reliability. Similar failure behavior reflects similar reliability assessment against software faults of same time–space distribution. Since program failure is determined by fault manifestations, working on similarities on fault manifestations helps figuring out failures in similarity as well. In this paper, we propose a novel method to characterize program behavior by defining fine-grained runtime states at assembly-level of code, aiming to capture the very first abnormal manifestation—first-error—after software fault being activated during program run. Failure prediction model and measurement is presented based on similarities reflected by the runtime behaviors obtained. Fault injection experiments are conducted to verify the prediction measurement by utilizing software faults of Orthogonal Defect Classification to inject and MiBench programs to perform. Over 15 000+ times of fault injection by considering type and location of fault were conducted into each program in order to quantify and analyze the sensitivity on first-error and similarity on failure accordingly. The results show that programs having particular structural features (e.g. massive operations with regards to calculations, nested function-calls, case structures and loop structures, etc.) are well characterized toward first-error behaviors by extracting fine-grained states. Similarities on first-error sensitivity can be represented better for these programs as well. Same trend is seen on failure behavior and its prediction measurement.

A fine-grained robust performance diagnosis framework for run-time cloud applications

To maintain the required service quality of time-critical cloud applications, operators must continuously monitor their runtime status, detect potential performance anomalies, and diagnose the root causes of these anomalies effectively. However, existing performance diagnosis methods face challenges such as the need for high-quality labeled data, the low reusability and robustness of performance anomaly detection models, and the absence of real-time fine-grained root cause localization. These challenges make fixing performance issues quickly and developing effective adaptation decisions difficult. We provide a Fine-grained Robust Performance Diagnosis (FIRED) framework to tackle those challenges. The framework offers a metrics selection component to filter noise and improve detection efficiency, an anomaly detection component that assembles several well-selected base models with a deep neural network, and adopts weakly supervised learning considering fewer labels exist in reality. The framework also employs a real-time, fine-grained root cause localization component to locate dependent resource metrics of performance anomalies. Our experiments show that the framework can effectively reduce data noise and achieve the best accuracy and algorithm robustness for performance anomaly detection. In addition, the framework can accurately localize the first root causes, with an average accuracy higher than 0.7 for locating the first four root cause metrics.

NCCMF: Non-Collaborative Continuous Monitoring Framework for Container-Based Cloud Runtime Status

NCCMF: Non-Collaborative Continuous Monitoring Framework for Container-Based Cloud Runtime Status

AllInfoLog: Robust Diverse Anomalies Detection Based on All Log Features

Large-scale services are generating massive logs, which trace the runtime states and critical events. Anomaly detection via logs is critical for service maintenance and reliability assurance. Existing log-based anomaly detection methods make use of the limited information in log data, resulting in their incapability of detecting diverse anomalies related to unused log features. In this paper, we propose AllInfoLog, a robust log-based anomaly detection method taking advantage of all log information, to detect diverse types of anomalies. To capture all log features, AllInfoLog utilizes four encoders to extract semantic, parameter, time, and other feature embeddings, respectively. The embeddings of all log features are then combined to train an attention-based Bi-LSTM model to detect diverse anomalies. The experimental evaluations on real-world log datasets, synthetic datasets, and unstable log datasets demonstrate AllInfoLog outperforms the state-of-the-art log-based anomaly detection methods from aspects of performance and robustness, and has effectiveness to detect diverse types of anomalies.

Loader: A Log Anomaly Detector Based on Transformer

Detecting anomalies in logs is crucial for service and system management, since logs are widely used to record the runtime status, and are often the only data available for postmortem analysis. Since anomalies are usually rare in real-world services and systems, a common and feasible practice is to mine or learn normal patterns from logs, and deem those violating the normal patterns as anomalies. As log sequences are a kind of time series data, RNN (Recurrent Neural Network) and its variants have been extensively employed to capture the normal patterns. Nevertheless, the sequential nature of RNN and its variants makes them hard to parallelize and capture long-term dependencies, which may hinder their performance. To address this issue, in this paper we propose Loader, a novel semi-supervised <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><b>lo</b></u> g <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><b>a</b></u> nomaly <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><b>d</b></u> etector based on Transform <underline xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><b>er</b></u> , because the Transformer architecture eschews recurrence and is able to draw global dependencies. Loader leverages the Transformer encoder to capture normal patterns from normal log sequences. When detecting, it gives a set of candidate log templates, that may appear after the input log substring under normal conditions. If the template of the actual next log message is not within the candidate set, this implies an anomaly. Previous similar methods select the most possible <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k$</tex-math></inline-formula> log templates as candidates in any case, so the performance is sensitive to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k$</tex-math></inline-formula> , and it is nontrivial to pick a proper <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$k$</tex-math></inline-formula> . To alleviate this, we design a more flexible and robust ‘top- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$p$</tex-math></inline-formula> ’ algorithm, which determines the candidate set based on the cumulative probability of the most possible log templates. Extensive experiments are conducted based on three public log datasets, the experimental results validate the effectiveness and competitiveness of our approach.

Logformer: Cascaded Transformer for System Log Anomaly Detection

Modern large-scale enterprise systems produce large volumes of logs that record detailed system runtime status and key events at key points. These logs are valuable for analyzing performance issues and understanding the status of the system. Anomaly detection plays an important role in service management and system maintenance, and guarantees the reliability and security of online systems. Logs are universal semi-structured data, which causes difficulties for traditional manual detection and pattern-matching algorithms. While some deep learning algorithms utilize neural networks to detect anomalies, these approaches have an over-reliance on manually designed features, resulting in the effectiveness of anomaly detection depending on the quality of the features. At the same time, the aforementioned methods ignore the underlying contextual information present in adjacent log entries. We propose a novel model called Logformer with two cascaded transformer-based heads to capture latent contextual information from adjacent log entries, and leverage pre-trained embeddings based on logs to improve the representation of the embedding space. The proposed model achieves comparable results on HDFS and BGL datasets in terms of metric accuracy, recall and F1-score. Moreover, the consistent rise in F1-score proves that the representation of the embedding space with pre-trained embeddings is closer to the semantic information of the log.

An Improved Dynamic Smooth Weighted Round-robin Load-balancing Algorithm

Load-balancing is a key component of highly available network infrastructure. Weighted round-robin is one of the important strategies to achieve load-balancing, which is a classic research subject. In recent years, the weighted round-robin algorithm has developed from static weight to dynamic. The Dynamic weighted round-robin algorithm needs further research in time-sensitive of dynamic weights and the rationality of results. Based on Nginx, this paper improves the dynamic weighting algorithm. Firstly, it proposes a method of dynamically calculating weights. The method determines the weight of the server in the cluster according to its theoretical performance and runtime state. Secondly, making the request allocation result more balanced and avoiding continuous centralized scheduling by an improved weighted algorithm. Experiments show that the algorithm proposed in this paper is effective and can improve system performance and load-balancing efficiency.

Basic block coverage for search-based unit testing and crash reproduction

Search-based techniques have been widely used for white-box test generation. Many of these approaches rely on the approach level and branch distance heuristics to guide the search process and generate test cases with high line and branch coverage. Despite the positive results achieved by these two heuristics, they only use the information related to the coverage of explicit branches (e.g., indicated by conditional and loop statements), but ignore potential implicit branchings within basic blocks of code. If such implicit branching happens at runtime (e.g., if an exception is thrown in a branchless-method), the existing fitness functions cannot guide the search process. To address this issue, we introduce a new secondary objective, called Basic Block Coverage (BBC), which takes into account the coverage level of relevant basic blocks in the control flow graph. We evaluated the impact of BBC on search-based unit test generation (using the DynaMOSA algorithm) and search-based crash reproduction (using the STDistance and WeightedSum fitness functions). Our results show that for unit test generation, BBC improves the branch coverage of the generated tests. Although small (sim 1.5%), this improvement in the branch coverage is systematic and leads to an increase of the output domain coverage and implicit runtime exception coverage, and of the diversity of runtime states. In terms of crash reproduction, in the combination of STDistance and WeightedSum, BBC helps in reproducing 3 new crashes for each fitness function. BBC significantly decreases the time required to reproduce 43.5% and 45.1% of the crashes using STDistance and WeightedSum, respectively. For these crashes, BBC reduces the consumed time by 71.7% (for STDistance) and 68.7% (for WeightedSum) on average.

Benchmarking and Configuring Security Levels in Intermittent Computing

Intermittent computing derives its name from the intermittent character of the power source used to drive the computing, typically an energy harvester of ambient energy sources. Intermittent computing is characterized by frequent transitions between the powered and the non-powered state. To enable the processor to quickly recover from unexpected power loss, regular checkpoints store the run-time state of the program, including variables, control information, and machine state. In sensitive applications such as logged measurements, checkpoints must be secured against tamper and replay. We investigate the overhead of creating, securing, and restoring checkpoints with respect to the application. We propose a configurable checkpoint security setting that leverages application properties to reduce overhead of checkpoint security and implement the same using a secure checkpointing protocol. We discuss a prototype implementation for a FRAM-based micro-controller, and we characterize the cost of adding and configuring security to traditional checkpointing using a suite of embedded benchmark applications.

Time-travel Investigation: Toward Building a Scalable Attack Detection Framework on Ethereum

Ethereum has been attracting lots of attacks, hence there is a pressing need to perform timely investigation and detect more attack instances. However, existing systems suffer from the scalability issue due to the following reasons. First, the tight coupling between malicious contract detection and blockchain data importing makes them infeasible to repeatedly detect different attacks. Second, the coarse-grained archive data makes them inefficient to replay transactions. Third, the separation between malicious contract detection and runtime state recovery consumes lots of storage. In this article, we propose a scalable attack detection framework named EthScope , which overcomes the scalability issue by neatly re-organizing the Ethereum state and efficiently locating suspicious transactions. It leverages the fine-grained state to support the replay of arbitrary transactions and proposes a well-designed schema to optimize the storage consumption. The performance evaluation shows that EthScope can solve the scalability issue, i.e., efficiently performing a large-scale analysis on billions of transactions, and a speedup of around \( \text{2,300}\times \) when replaying transactions. It also has lower storage consumption compared with existing systems. Further analysis shows that EthScope can help analysts understand attack behaviors and detect more attack instances.

Development of an Anomaly Detection Algorithm in Distributed Systems based on Runtime and System Status

Computational grades have emerged as a new approach to solving large-scale problems in the fields of science, engineering, and commerce. The computing grid is a hardware and software infrastructure that provides affordable, reliable, comprehensive, and affordable access to the computational abilities of others. A computational grid is associated with a set of resources on a large scale. Computational grades have emerged as a new approach to solving large-scale problems in the fields of science, engineering, and commerce. The computing grid is a hardware and software infrastructure that provides affordable, reliable, comprehensive, and affordable access to the computational abilities of others. A computational grid is associated with a set of resources on a large scale. The purpose of this thesis is to provide a method that optimizes resource management and scheduling in at least one direction. The main focus of the research is on the time criterion, the deadline for doing things and receiving the response from the grid are parameters that can be examined. From a more general perspective, the aim of the research is to get the answer as quickly as possible from the calculation grid. The proposed algorithm improves the scheduling and resource management of the grid in the direction of improvement, and the structure and form of this problem have not yet been resolved. The proposed solution has been considered hypothesis and removed some of the definitions of grid scheduling, such as cost, quality of service, architectures, and others, but ultimately heed it and timed the grid in some ways.

TSAGen: Synthetic Time Series Generation for KPI Anomaly Detection

A key performance indicator (KPI) consists of critical time series data that reflect the runtime states of network systems (e.g., response time and available bandwidth). Despite the importance of KPI, datasets for KPI anomaly detection available to the public are very limited, due to privacy concerns and the high overhead in manually labelling the data. The insufficiency of public KPI data poses a great barrier for network researchers and practitioners to evaluate and test what-if scenarios in the development of artificial intelligence for IT operations (AIOps) and anomaly detection algorithms. To tackle the difficulty, we develop a univariate time series generation tool called TSAGen, which can generate KPI data with anomalies and controllable characteristics for KPI anomaly detection. Experiment results show that the data generated by TSAGen can be used for comprehensive evaluation of anomaly detection algorithms with diverse user-defined what-if scenarios.

Runtime verification in uncertain environment based on probabilistic model learning.

Runtime verification (RV) is a lightweight approach to detecting temporal errors of system at runtime. It confines the verification on observed trajectory which avoids state explosion problem. To predict the future violation, some work proposed the predictive RV which uses the information from models or static analysis. But for software whose models and codes cannot be obtained, or systems running under uncertain environment, these predictive methods cannot take effect. Meanwhile, RV in general takes multi-valued logic as the specification languages, for example the true, false and inconclusive in three-valued semantics. They cannot give accurate quantitative description of correctness when inconclusive is encountered. We in this paper present a RV method which learns probabilistic model of system and environment from history traces and then generates probabilistic runtime monitor to quantitatively predict the satisfaction of temporal property at each runtime state. In this approach, Hidden Markov Model (HMM) is firstly learned and then transformed to Discrete Time Markov Chain (DTMC). To construct incremental monitor, the monitored LTL property is translated into Deterministic Rabin Automaton (DRA). The final probabilistic monitor is obtained by generating the product of DTMC and DRA, and computing the probabilities for each state. With such a method, one can give early warning once the probability of correctness is lower than a pre-defined threshold, and have the chance to do adjustment in advance. The method has been implemented and experimented on real UAS (Unmanned Aerial Vehicle) simulation platform.

Optimization of Multi-Core Accelerator Performance Based on Accurate Performance Estimation

Multicore accelerators have emerged to efficiently execute recent applications with complex computational dimensions. Compared to a single-core accelerator, a multicore accelerator handles a larger amount of communication and computation simultaneously. Since the conventional performance estimation algorithm tailored to single-core accelerators cannot estimate the performance of multicore accelerators accurately, we propose a novel performance estimation algorithm for a multicore accelerator. The proposed algorithm predicts a dynamic communication bandwidth of each direct memory access controller (DMAC) based on the runtime state of DMACs, making it possible to estimate the communication amounts handled by DMACs accurately by taking into account the temporal intervals. The proposed algorithm is evaluated for convolutional neural networks and wireless communications. The experimental results using a pre-register transfer level (RTL) simulator shows that the proposed algorithm can estimate the performance of a multicore accelerator with the estimation error of up to 2.8%, regardless of the system communication bandwidth. These results were also verified by the hardware implementations on Xilinx ZYNQ. In addition, the proposed algorithm is used to explore a design space of accelerator core dimensions, and the resulting optimal core dimension provides performance gains of 10.8% and 31.2%, compared to the conventional multicore accelerator and single-core accelerator, respectively. The source code is available on the GitHub repository: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/SDL-KU/OptAccTile</uri> .

A machine learning assisted data placement mechanism for hybrid storage systems

Emerging applications produce massive files that show different properties in file size, lifetime, and read/write frequency. Existing hybrid storage systems place these files onto different storage mediums assuming that the access patterns of files are fixed. However, we find that the access patterns of files are changeable during their lifetime. The key to improve the file access performance is to adaptively place the files on the hybrid storage system using the run-time status and the properties of both files and the storage systems. In this paper, we propose a machine learning assisted data placement mechanism that adaptively places files onto the proper storage medium by predicting access patterns of files. We design a PMFS based tracer to collect file access features for prediction and show how this approach is adaptive to the changeable access pattern. Based on data access prediction results, we present a linear data placement algorithm to optimize the data access performance on the hybrid storage mediums. Extensive experimental results show that the proposed learning algorithm can achieve over 90% accuracy for predicting file access patterns. Meanwhile, this paper can achieve over 17% improvement of system performance for file accesses compared with the state-of-the-art linear-time data placement methods.

ConAnomaly: Content-Based Anomaly Detection for System Logs.

Enterprise systems typically produce a large number of logs to record runtime states and important events. Log anomaly detection is efficient for business management and system maintenance. Most existing log-based anomaly detection methods use log parser to get log event indexes or event templates and then utilize machine learning methods to detect anomalies. However, these methods cannot handle unknown log types and do not take advantage of the log semantic information. In this article, we propose ConAnomaly, a log-based anomaly detection model composed of a log sequence encoder (log2vec) and multi-layer Long Short Term Memory Network (LSTM). We designed log2vec based on the Word2vec model, which first vectorized the words in the log content, then deleted the invalid words through part of speech tagging, and finally obtained the sequence vector by the weighted average method. In this way, ConAnomaly not only captures semantic information in the log but also leverages log sequential relationships. We evaluate our proposed approach on two log datasets. Our experimental results show that ConAnomaly has good stability and can deal with unseen log types to a certain extent, and it provides better performance than most log-based anomaly detection methods.

Generative Adversarial Network-Based Transfer Reinforcement Learning for Routing With Prior Knowledge

With the incremental deployment of software defined networking, the routing algorithms have gained more power on observability and controllability. Deep reinforcement learning, as an experience-driven approach, shows considerable potential in routing problem with the help of the centralized controller. It is an adaptive, lightweight, and model-free approach to coping with dynamic runtime status, large-scale traffic, and heterogeneous objective of SDN routing. However, it is still not suitable for the variable and complex emerging networks, because the huge training cost prevents fast convergence in a varying or discrepant environment. In this paper, we propose a transfer reinforcement learning algorithm to improve the training efficiency, and handle the variation in network status and topology. Specifically, we leverage the generative adversarial network to learn domain-invariant features that is suitable for deep reinforcement learning-based routing in different network environments. This mechanism utilizes the previous model and accelerates the training process. We implement our routing algorithm in the production level software switches and controller, while evaluating it comprehensively with many topologies and network status distributions. The experimental results show that our work not only outperforms the state-of-the-art deep reinforcement learning-based routing frameworks, but also has more training efficiency than the naive transfer learning algorithm both on different topologies and network status distributions.

A Co-Scheduling Framework for DNN Models on Mobile and Edge Devices with Heterogeneous Hardware

With the emergence of more and more powerful chipsets and hardware and the rise of Artificial Intelligence of Things (AIoT), there is a growing trend for bringing Deep Neural Network (DNN) models to empower mobile and edge devices with intelligence such that they can support attractive AI applications in a real-time manner. To leverage heterogeneous computational resources (such as CPU, GPU, DSP, etc.) to effectively and efficiently support the concurrent inference of multiple DNN models on a mobile or edge device, we propose a novel online Co-Scheduling framework based on deep REinforcement Learning, called COSREL. COSREL has the following desirable features: 1) it achieves significant speedup over commonly-used methods by efficiently utilizing all the computational resources on heterogeneous hardware; 2) it leverages emerging Deep Reinforcement Learning (DRL) to make dynamic and wise online scheduling decisions based on system runtime state; 3) it is capable of making a good tradeoff among inference latency, throughput, and energy efficiency; and 4) it makes no changes to given DNN models, thus preserves their accuracies. To evaluate COSREL, we conduct extensive experiments on an off-the-shelf Android smartphone. The experimental results show that COSREL consistently outperforms other baselines in terms of throughput, latency, and energy efficiency.