Root Cause Of Anomalies Research Articles

Visual Diagnostics of Parallel Performance in Training Large-Scale DNN Models.

Diagnosing the cluster-based performance of large-scale deep neural network (DNN) models during training is essential for improving training efficiency and reducing resource consumption. However, it remains challenging due to the incomprehensibility of the parallelization strategy and the sheer volume of complex data generated in the training processes. Prior works visually analyze performance profiles and timeline traces to identify anomalies from the perspective of individual devices in the cluster, which is not amenable for studying the root cause of anomalies. In this article, we present a visual analytics approach that empowers analysts to visually explore the parallel training process of a DNN model and interactively diagnose the root cause of a performance issue. A set of design requirements is gathered through discussions with domain experts. We propose an enhanced execution flow of model operators for illustrating parallelization strategies within the computational graph layout. We design and implement an enhanced Marey's graph representation, which introduces the concept of time-span and a banded visual metaphor to convey training dynamics and help experts identify inefficient training processes. We also propose a visual aggregation technique to improve visualization efficiency. We evaluate our approach using case studies, a user study and expert interviews on two large-scale models run in a cluster, namely, the PanGu- α 13B model (40 layers), and the Resnet model (50 layers).

Multisensor Anomaly Detection and Interpretable Analysis for Linear Induction Motors

In this paper, a graph neural network anomaly detection framework is proposed to improve the safety of linear induction motors, a key component of high-speed maglev trains. In our framework, each sensor sequence is treated as a separate feature. The similarity and correlation between multi-dimensional features are learned as prior knowledge for graph structure learning. The spatial-temporal graph attention network incorporates prior knowledge to learn complex correlations between nodes. Furthermore, the framework optimises a joint model for anomaly detection, avoiding the trap of falling into either local or global optimisation and thus achieving the most stable detection. Experimental results of our method on four real-world datasets show that it is more accurate than other state-of-the-art methods in detecting anomalies and capturing inter-sensor correlations. Further analysis of graph attention weights and visualization subgraphs show that our framework is well interpretable and allowing users to locate the root cause of anomalies.

An unsupervised root cause analysis method for satellite on-orbit anomalies based on causal discovery

For a modern satellite, there are usually thousands of telemetry parameters. How to find the root cause of the satellite anomalies using telemetry data is an important but difficult task. This paper proposes an unsupervised root cause analysis method for satellite on-orbit anomalies based on causal discovery. The method includes a component impact analysis method based on multivariate-to-one transfer entropy, which can quickly identify components that cause system anomalies; and a pruning strategy based on the change of causality are proposed to screen and identify the parameters of the components to determine the root cause of the anomalies. Simulation experiments and real case verification show that the accuracy, recall, and F1-score of the method in this paper all reach more than 80%, which is greatly improved compared with the existing methods. The case study also shows that when anomalies occur, the root cause analysis method can detect the root cause using only 4 min of anomalous information, which helps to stop the spread of anomalies more effectively.

Novel E2E-QoE metric for PHY optimization: A cross-layered framework

Existing systems use key performance indicators (KPIs) as metrics for physical layer (PHY) optimization, which suffers from the problem of over-optimization, because some unnecessary PHY enhancements are imperceptible to terminal users and thus induce additional cost and energy waste. Therefore, it is necessary to utilize directly the quality of experience (QoE) of user as a metric of optimization, which can achieve the global optimum of QoE under cost and energy constraints. However, QoE is still a metric of application layer that cannot be easily used to design and optimize the PHY. To address this problem, we in this paper propose a novel end-to-end QoE (E2E-QoE) based optimization architecture at the user-side for the first time. Specifically, a cross-layer parameterized model is proposed to establish the relationship between PHY and E2E-QoE. Based on this, an E2E-QoE oriented PHY anomaly diagnosis method is further designed to locate the time and root cause of anomalies. Finally, we investigate to optimize the PHY algorithm directly based on the E2E-QoE. The proposed frameworks and algorithms are all validated using the data from real fifth-generation (5G) mobile system, which show that using E2E-QoE as the metric of PHY optimization is feasible and can outperform existing schemes.

ServiceRank: Root Cause Identification of Anomaly in Large-Scale Microservice Architectures

Nowadays, increasing business applications running in the cloud are embracing the microservice architecture. This article presents the challenges and implications of diagnosing root causes of anomalies in large-scale microservice architecture using real incidents in IBM Bluemix. We propose ServiceRank, a novel framework for anomaly detection and root cause identification in the microservice architecture to tackle these challenges. ServiceRank introduces an anomaly detector followed by a root cause analysis module, which detects the suspected abnormal service without pre-defined thresholds. To generalize our approach, we design a causal relationship extraction approach to construct impact graphs for root cause investigation according to specific anomalies. To eliminate cloud design-patterns' impact on anomaly diagnosis, we propose a correlation calibration mechanism in ServiceRank and present a calibration algorithm for the circuit breaker - A typical protection pattern in the microservice architecture. Finally, we design a heuristic investigation algorithm based on the second-order random walk to identify the anomaly's root cause. Experimental results in a simulated environment and the IBM Bluemix platform show that ServiceRank outperforms selected approaches in accuracy and offers fast identification of root cause service when an anomaly occurs. Moreover, we can deploy ServiceRank rapidly and easily in various systems without any pre-defined knowledge.

Deepnoise: Learning sensor and process noise to detect data integrity attacks in CPS

Cyber-physical systems (CPS) have been widely deployed in critical infrastructures and are vulnerable to various attacks. Data integrity attacks manipulate sensor measurements and cause control systems to fail, which are one of the prominent threats to CPS. Anomaly detection methods are proposed to secure CPS. However, existing anomaly detection studies usually require expert knowledge (e.g., system model-based) or are lack of interpretability (e.g., deep learning-based). In this paper, we present Deepnoise, a deep learning-based anomaly detection method for CPS with interpretability. Specifically, we utilize the sensor and process noise to detect data integrity attacks. Such noise represents the intrinsic characteristics of physical devices and the production process in CPS. One key enabler is that we use a robust deep autoencoder to automatically extract the noise from measurement data. Further, an LSTM-based detector is designed to inspect the obtained noise and detect anomalies. Data integrity attacks change noise patterns and thus are identified as the root cause of anomalies by Deepnoise. Evaluated on the SWaT testbed, Deep-noise achieves higher accuracy and recall compared with state-of-the-art model-based and deep learning-based methods. On average, when detecting direct attacks, the precision is 95.47%, the recall is 96.58%, and F is 95.98%. When detecting stealthy attacks, precision, recall, and F scores are between 96% and 99.5%.

Detecting and Localizing Anomalies in Container Clusters Using Markov Models

Detecting the location of performance anomalies in complex distributed systems is critical to ensuring the effective operation of a system, in particular, if short-lived container deployments are considered, adding challenges to anomaly detection and localization. In this paper, we present a framework for monitoring, detecting and localizing performance anomalies for container-based clusters using the hierarchical hidden Markov model (HHMM). The model aims at detecting and localizing the root cause of anomalies at runtime in order to maximize the system availability and performance. The model detects response time variations in containers and their hosting cluster nodes based on their resource utilization and tracks the root causes of variations. To evaluate the proposed framework, experiments were conducted for container orchestration, with different performance metrics being used. The results show that HHMMs are able to accurately detect and localize performance anomalies in a timely fashion.

Root Cause Analysis of Traffic Anomalies Using Uneven Diffusion Model

Detection and analysis of traffic anomalies are important for the development of intelligent transportation systems. In particular, the root causes of traffic anomalies in road networks as well as their propagation and influence to the surrounding areas are highly meaningful. The root cause analysis of traffic anomalies aims to identify those road segments, where the traffic anomalies are detected by the traffic statuses significantly deviating from the usual condition and are originated due to incidents occurring in those roads such as traffic accidents or social events. The existing methods for traffic anomaly root cause analysis detect all traffic anomalies first and then apply, implicitly or explicitly, specified causal propagation rules to infer the root cause. However, these methods require reliable detection techniques to accurately identify all traffic anomalies and extensive domain knowledge of city traffic to specify plausible causal propagation rules in road networks. In contrast, this paper proposes an innovative and integrated root cause analysis method. The proposed method is featured by 1) defining a visible outlier index as the probabilistic indicator of traffic anomalies/disturbances and 2) automatically learning spatiotemporal causal relationship from historical data to build an uneven diffusion model for root cause analysis. The accuracy and effectiveness of the proposed method have been demonstrated by experiments conducted on a trajectory dataset with 2.5 billion location records of 27 266 taxies in Shenzhen city.

Root Cause Failure Analysis of Corrosion in Wet Gas Piping

This paper discusses the root causes and operational mitigations of corrosion anomalies reported for the wet gas system of a floating production storage and offloading (FPSO) vessel. This system is classified in general into the flowing and dead leg sections and operated below 62 °C, where a protective FeCO3 film will not form. Corrosion which is temperature dependent is general. The root cause of the anomalies in the flowing wet gas system was found to be due to CaN and solid deposition in the LP separator, increasing levels of corrosive condensate, which drips down the pipe walls, resulting in wall losses of less than 50%. Evacuation of deposits in the LP separator and subsequent control of its formation rate would mitigate anomalies attributed to increasing levels of corrosive condensates. The corrosion pattern at elbows, modified by turbulent flow of emergent fluid streams through the complex-geometry elbows, is mitigated by velocity control. The root cause of the inspection anomalies at dead leg locations was due to the lifting of the PSV and/or infrequent flow through a bypass valve and directly to flare. Infrequent transportation of condensed, corrosive water thereby resulted in dead leg corrosion. Proposed mitigation entails replacement of defective PSVs and modifying flow frequencies. This paper summarizes all the identified root causes and proposed mitigations and provides a greater understanding of the internal degradation mechanisms operating in wet gas systems in general.

An Automatic Detection and Diagnosis Framework for Mobile Communication Systems

As the complexity of commercial cellular networks grows, there is an increasing need for automated methods detecting and diagnosing cells not only in complete outage but with degraded performance as well. Root cause analysis of the detected anomalies can be tedious and currently carried out mostly manually if at all; in most practical cases, operators simply reset problematic cells. In this paper, a novel integrated detection and diagnosis framework is presented that can identify anomalies and find the most probable root cause of not only severe problems but even smaller degradations as well. Detecting an anomaly is based on monitoring radio measurements and other performance indicators and comparing them to their usual behavior captured by profiles, which are also automatically built without the need for thresholding or manual calibration. Diagnosis is based on reports of previous fault cases by identifying and learning their characteristic impact on different performance indicators. The designed framework has been evaluated with proof-of-concept simulations including artificial faults in an LTE system. Results show the feasibility of the framework for providing the correct root cause of anomalies and possibly ranking the problems by their severity.

Internet quality abnormal analysis with k-means clustering

The anomaly detection using data mining techniques has widely aroused interests in recent years. This paper proposes a novel algorithm to do the Internet quality abnormal analysis. The algorithm is developed with k-means clustering method to analyze the real-time data of Internet quality acquired at the testbed of Beijing University of Posts and Telecommunications (BUPT) for the first time. The main objective of this paper is to construct a statistical Internet quality abnormal analysis system capable of classifying the Internet quality data as normal or abnormal. For this purpose the algorithm constructs a model in training process. The data of Internet quality are compared with the model and data with distinct deviation from the model is marked as abnormal. The algorithm can split the abnormal data which are applied as a guide to further investigation of Internet anomaly root cause. And the result proved to be credible, which will be useful for the network administrators to manage the network.

Reference-driven performance anomaly identification

Complex system software allows a variety of execution conditions on system configurations and workload properties. This paper explores a principled use of reference executions--those of similar execution conditions from the target--to help identify the symptoms and causes of performance anomalies. First, to identify anomaly symptoms, we construct change profiles that probabilistically characterize expected performance deviations between target and reference executions. By synthesizing several single-parameter change profiles, we can scalably identify anomalous reference-to-target changes in a complex system with multiple execution parameters. Second, to narrow the scope of anomaly root cause analysis, we filter anomaly-related low-level system metrics as those that manifest very differently between target and reference executions. Our anomaly identification approach requires little expert knowledge or detailed models on system internals and consequently it can be easily deployed. Using empirical case studies on the Linux I/O subsystem and a J2EE-based distributed online service, we demonstrate our approach's effectiveness in identifying performance anomalies over a wide range of execution conditions as well as multiple system software versions. In particular, we discovered five previously unknown performance anomaly causes in the Linux 2.6.23 kernel. Additionally, our preliminary results suggest that online anomaly detection and system reconfiguration may help evade performance anomalies in complex online systems.

Density Displays for Data Stream Monitoring

AbstractIn many business applications, large data workloads such as sales figures or process performance measures need to be monitored in real‐time. The data analysts want to catch problems in flight to reveal the root cause of anomalies. Immediate actions need to be taken before the problems become too expensive or consume too many resources. In the meantime, analysts need to have the “big picture” of what the information is about. In this paper, we derive and analyze two real‐time visualization techniques for managing density displays: (1) circular overlay d isplays which visualize large volumes of data without data shift movements after the display is full, thus freeing the analyst from adjusting the mental picture of the data after each data shift; and (2) variable resolution density displays which allow users to get the entire view without cluttering. We evaluate these techniques with respect to a number of evaluation measures, such as constancy of the display and usage of display space, and compare them to conventional d isplays with periodic shifts. Our real time data monitoring system also provides advanced interactions such as a local root cause analysis for further exploration. The applications using a number of real‐world data sets show the wide applicability and usefulness of our ideas.