Microservice Systems Research Articles

ServiceAnomaly: An anomaly detection approach in microservices using distributed traces and profiling metrics

Anomaly detection is an essential activity for identifying abnormal behaviours in microservice-based systems. A common approach is to model the system behaviour during normal operation using either distributed traces or profiling metrics. The model is then used to detect anomalies during system operation. In this paper, we present a new anomaly detection approach, called ServiceAnomaly, for anomaly detection in microservice systems that combines distributed traces and six profiling metrics to build an annotated directed acyclic graph that characterizes the normal behaviour of the system. Unlike existing techniques, our approach captures the context propagation provided by distributed traces as a graph that is annotated with functions characterizing both linear and non-linear relationships between profiling metrics. The final annotated graph is used to detect abnormal executions during system operation. The results of applying our approach to two open-source benchmarks show that our approach detects anomalies with an F1-score up to 86%. We also show how developers can use the annotated graph to reason about the causes of anomalies

Graph neural network based robust anomaly detection at service level in SDN driven microservice system

Software-Defined Network (SDN) has been widely employed in data center networks to host various software systems such as microservice systems. A microservice system is constructed by a microservice architecture composed of dozens of services running in different lightweight virtual machines (e.g., containers) and relying on overlay networks for connectivity. However, with the increase of the system scale and the network traffic volume, the network becomes more complex and common to behave anomalously. Therefore, it is error-prone to manually investigate anomalies. To tackle this problem, we propose a non-intrusive monitoring system to help network operators obtain deep insights from the network traffic generated by the running system. Based on the monitoring system, we propose an anomaly detection method to automatically detect network anomalies. The core idea is to learn behavior patterns of inter/intra-components by modeling spatial–temporal relationships amongst collected network metrics with the proposed model STCell-VAE. The pattern will be leveraged to reconstruct the input data. Subsequently, the reconstruction probability is used to determine anomalies. Additionally, STCell-VAE can assimilate network information from new components as the network evolves, eliminating the necessity for retraining. Experiments are conducted within a real networking system and four simulated large-scale networks. The results demonstrate that our system can detect anomalies with about 0.7 F1-score, which outperforms state-of-the-art methods.

DyCause: Crowdsourcing to Diagnose Microservice Kernel Failure

Today many web applications in the cloud (apps) are built based on microservices. However, as the anomaly propagates in a highly dynamic and complex way, troubleshooting them becomes full of challenges. Existing diagnostic methods are mostly designed based on monitoring metrics retrieved from the microservice system kernel. Therefore, application owners and even site reliability engineers (SREs) cannot effectively resort to those methods when the microservice systems lack such a comprehensive monitoring infrastructure. In this paper, we develop DyCause, a crowdsourcing solution to the asymmetric diagnostic information problem. Our solution collects the operational status of kernel services collaboratively from the user space and initiates diagnosis on demand. Without the requirement of any architectural or functional infrastructure, it is both fast and lightweight to deploy DyCause in a microservice system. In order to discover the fine-grained dynamic causalities between services during the anomaly, we also design an efficient algorithm based on statistical analysis. Based on this algorithm, we can also analyze the anomaly propagation paths within the microservice system and generate a better interpretable diagnosis. In our evaluation, we test DyCause in a controlled simulation environment and a real-world cloud system. Our results have shown that DyCause has the best accuracy and efficiency among several state-of-the-art methods and is more robust in terms of parameters.

Robust Failure Diagnosis of Microservice System Through Multimodal Data

Robust Failure Diagnosis of Microservice System Through Multimodal Data

Research on Fault Detection for Microservices Based on Log Information and Social Network Mechanism Using BiLSTM-DCNN Model

The microservice architecture breaks through the traditional cluster architecture mode based on virtual machines and uses containers as carriers to interact through lightweight communication mechanisms to reduce system coupling and provide more flexible system service support. With the expansion of the system scale, a large number of system logs with complex structures and chaotic relationships are generated. How to accurately analyze the system logs and make efficient fault prediction is particularly important for building a safe and reliable system. By studying neural network technology, this paper proposes an Attention-Based Bidirectional Long Short-Term Memory Network (Bi-LSTM). Combined with the dual channel convolutional neural network model (DCNN), it uses the attention mechanism to explore the differences between dimensional features, realizes multi-dimensional feature fusion, and establishes a BiLSTM-DCNN deep learning model that integrates the attention mechanism. From the perspective of social network analysis, a data preprocessing method is proposed to process fault redundant data and improve the accuracy of fault prediction under Microservices. Compare BiLSTM-DCNN with the mainstream system log analysis machine learning models SVM, CNN and Bi-LSTM, and explore the advantages of BiLSTM-DCNN in processing microservice system log text. The model is applied to simulation data and HDFS data set for experimental comparison, which proves the good generalization ability and universality of BiLSTM-DCNN.

Detection and filling of functional holes in microservice systems: Method and infrastructure support

With the widespread use of microservices technology, a growing number of Microservice Systems (MSS) have emerged. Monolithic applications are divided into several small and independent microservices that provide functions through APIs. These microservices can be orchestrated by service composition to satisfy various user requirements: microservice functionalities are aggregated into coarse-grained solutions to provide composite functions to users. However, although various service composition approaches have been presented in many works of literature, they failed to solve the situation that no feasible solutions can be found because of missing functions in MSS, named Functional Hole (FH). As a result, the system cannot satisfy user requirements and faces Quality of Service (QoS) declines. In this paper, to reduce the impact of missing functions, we defined the FH to describe the absence of functions for MSS when it cannot satisfy user requirements. Moreover, for the first time, we proposed the Detection and Filling Problem of FH (DFPFH) in a running MSS. A three-phase algorithm with supporting infrastructure was developed to solve DFPFH. It detects FHs based on hypergraphs, fills FHs with services from an external service system at runtime, and generates suggestions for thoroughly filling FH to developers. Plenty of experiments were conducted, and the results validate our approaches’ usability, effectiveness, and performance.

A Complex Behavioral Interaction Analysis Method for Microservice Systems With Bounded Buffers

A Complex Behavioral Interaction Analysis Method for Microservice Systems With Bounded Buffers

Catalog and detection techniques of microservice anti-patterns and bad smells: A tertiary study

Background:Various works investigated microservice anti-patterns and bad smells in the past few years. We identified seven secondary publications that summarize these, but they have little overlap in purpose and often use different terms to describe the identified anti-patterns and smells. Objective:This work catalogs recurring bad design practices known as anti-patterns and bad smells for microservice architectures, and provides a classification into categories as well as methods for detecting these practices. Method:We conducted a systematic literature review in the form of a tertiary study targeting secondary studies identifying poor design practices for microservices. Results:We provide a comprehensive catalog of 58 disjoint anti-patterns, grouped into five categories, which we derived from 203 originally identified anti-patterns for microservices. Conclusion:The results provide a reference to microservice developers to design better-quality systems and researchers who aim to detect system quality based on anti-patterns. It also serves as an anti-pattern catalog for development-aiding tools, which are not currently available for microservice system development but could mitigate quality degradation throughout system evolution.

Scalability and Performance of Microservices Architectures

The inevitability of continuous evolution and seamless integration of dynamic alterations remains a paramount consideration in the realm of software engineering. This concern is particularly pronounced within the context of contemporary microservices architectures embedded in heterogeneous and decentralized systems composed of numerous interdependent components. A pivotal focal point within such a software design paradigm is to sustain optimal performance quality by ensuring harmonious collaboration among autonomous facets within an intricate framework. The challenge of microservices evolution has predominantly revolved around upholding the harmonization of diverse microservices versions during updates, all while curbing the computational overhead associated with such validation. This study leverages previous research outcomes and tackles the evolution predicament by introducing an innovative formal model, coupled with a fresh exposition of microservices RESTful APIs. The amalgamation of Formal Concept Analysis and the Liskov Substitution Principle plays a pivotal role in this proposed solution. A series of compatibility constraints is delineated and subjected to validation through a controlled experiment employing a representative microservices system. The suggested approach is poised to enhance the development of more sustainable microservices applications and elevate the efficacy of DevOps practices engaged in the creation and upkeep of microservices architectures.

Graph-Based Root Cause Localization in Microservice Systems with Protection Mechanisms

Service anomalies are difficult to locate accurately due to their propagation through service dependencies in microservice systems. Besides, the protection mechanisms are introduced into the microservice systems to ensure the stable operation of services. However, the existing approaches ignore the impact of protection mechanisms on the root cause localization of abnormal services. Specifically, the circuit breaking and rate limiting mechanisms can refuse service requests and thus change the way of anomaly propagation. Moreover, the different service request frequencies and latency make service dependencies change dynamically, resulting in the different probabilities of anomaly propagation among services. In this paper, we propose a novel framework named MicroGBPM to locate the root cause of abnormal services. We model the anomaly propagation among services as a dynamically constructed service attributed graph with metrics and traces when a failure occurs. To eliminate the impact of the protection mechanisms, we design a two-stage dynamic calibration strategy to adjust the probability of anomaly propagation among services. Then, we propose a random walking approach to calculate the root cause results by using the PageRank algorithm. The experimental results show that MicroGBPM improves the accuracy of root cause localization compared to other approaches in the microservice systems with protection mechanisms.

Alarm reduction and root cause inference based on association mining in communication network

With the growing demand for data computation and communication, the size and complexity of communication networks have grown significantly. However, due to hardware and software problems, in a large-scale communication network (e.g., telecommunication network), the daily alarm events are massive, e.g., millions of alarms occur in a serious failure, which contains crucial information such as the time, content, and device of exceptions. With the expansion of the communication network, the number of components and their interactions become more complex, leading to numerous alarm events and complex alarm propagation. Moreover, these alarm events are redundant and consume much effort to resolve. To reduce alarms and pinpoint root causes from them, we propose a data-driven and unsupervised alarm analysis framework, which can effectively compress massive alarm events and improve the efficiency of root cause localization. In our framework, an offline learning procedure obtains results of association reduction based on a period of historical alarms. Then, an online analysis procedure matches and compresses real-time alarms and generates root cause groups. The evaluation is based on real communication network alarms from telecom operators, and the results show that our method can associate and reduce communication network alarms with an accuracy of more than 91%, reducing more than 62% of redundant alarms. In addition, we validate it on fault data coming from a microservices system, and it achieves an accuracy of 95% in root cause location. Compared with existing methods, the proposed method is more suitable for operation and maintenance analysis in communication networks.

Evaluation of a Smart Intercom Microservice System Based on the Cloud of Things

This research migrates a monolithic smart intercom system to a microservice architecture, making the system more secure, stable, and scalable. The security mechanisms of the instant messaging platform are combined with microservices to improve the security of the system. The stability and performance of microservices are shown to be better than those of monolithic services through experimental tests. Residents can use different instant messaging software instead of the handset to improve the convenience of using this system. This system also implements community broadcasts, platform broadcasts, unit broadcasts, and family broadcasts to exchange messages across different instant messaging platforms. This paper proposes OpenAPI for other smart intercoms to integrate this system’s services and resources. In addition, this study deploys two microservice architectures using a native load balancer with a kube proxy and a service mesh load balancer with an istio proxy. Experiments were conducted during which residents tested the kube proxy and the istio proxy using stress testing tools on two microservice architectures, and the results showed that the kube proxy was slightly better than the istio proxy.

TrinityRCL: Multi-Granular and Code-Level Root Cause Localization Using Multiple Types of Telemetry Data in Microservice Systems

The microservice architecture has been commonly adopted by large scale software systems exemplified by a wide range of online services. Service monitoring through anomaly detection and root cause analysis (RCA) is crucial for these microservice systems to provide stable and continued services. However, compared with monolithic systems, software systems based on the layered microservice architecture are inherently complex and commonly involve entities at different levels of granularity. Therefore, for effective service monitoring, these systems have a special requirement of multi-granular RCA. Furthermore, as a large proportion of anomalies in microservice systems pertain to problematic code, to timely troubleshoot these anomalies, these systems have another special requirement of RCA at the finest code-level. Microservice systems rely on telemetry data to perform service monitoring and RCA of service anomalies. The majority of existing RCA approaches are only based on a single type of telemetry data and as a result can only support uni-granular RCA at either application-level or service-level. Although there are attempts to combine metric and tracing data in RCA, their objective is to improve RCA's efficiency or accuracy rather than to support multi-granular RCA. In this paper, we propose a new RCA solution <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">TrinityRCL</i> that is able to localize the root causes of anomalies at multiple levels of granularity including application-level, service-level, host-level, and metric-level, with the unique capability of code-level localization by harnessing all three types of telemetry data to construct a causal graph representing the intricate, dynamic, and nondeterministic relationships among the various entities related to the anomalies. By implementing and deploying <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">TrinityRCL</i> in a real production environment, we evaluate <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">TrinityRCL</i> against two baseline methods and the results show that <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">TrinityRCL</i> has a significant performance advantage in terms of accuracy at the same level of granularity with comparable efficiency and is particularly effective to support large-scale systems with massive telemetry data.

MicroState: An Anomaly Localization Method in Heterogeneous Microservice Systems

MicroState: An Anomaly Localization Method in Heterogeneous Microservice Systems

Online Deployment Algorithms for Microservice Systems With Complex Dependencies

Cloud and edge computing have been widely adopted in many application scenarios. With the increasing demand of fast iteration and complexity of business logic, it is challenging to achieve rapid development and continuous delivery in such highly distributed cloud and edge computing environment. At present, microservice-based architecture has been the dominant deployment style, and a microservice system has to evolve agilely to offer stable Quality of Service (QoS) in the situation where user requirement changes frequently. Many research have been conducted to optimally re-deploy microservices to adapt to changing requirements. Nevertheless, complex dependencies between microservices and the existence of multiple instances of one single microservice in a microservice system have not been fully considered in existing works. This paper defines SPPMS, the Service Placement Problem in Microservice Systems that feature complex dependencies and multiple instances, as a Fractional Polynomial Problem (FPP) . Considering the high computation complexity of FPP, it is then transformed into a Quadratic Sum-of-Ratios Fractional Problem (QSRFP) which is further solved by the proposed greedy-based algorithms. Experiments demonstrate that our models and algorithms outperform existing approaches in both quality and computation speed.

Features of Building a Microservice System for Modeling Nonstationary States of Nonlinear Circuits

Features of Building a Microservice System for Modeling Nonstationary States of Nonlinear Circuits

TraceGra: A trace-based anomaly detection for microservice using graph deep learning

Trace is widely used to detect anomalies in distributed microservice systems because of the capability of precisely reconstructing user request paths. However, most existing trace-based anomaly detection approaches treat the trace as a sequence of microservice invocations with response time information, which ignores the graph structure of trace and abnormal resource consumption of the complex distributed deployment environment of microservice. In this paper, we propose TraceGra, an unsupervised encoder–decoder anomaly detection approach. TraceGra first provides a unified graph representation to combine traces and performance metrics of the container. Then, it introduces the graph neural network (GNN) and long short-term memory network (LSTM) to extract the topology and temporal features, respectively. Finally, it adds the two-part loss value with two hyperparameters as the anomaly score. The evaluation results on an open-source dataset and a local dataset collected from an ARM server cluster show that TraceGra achieves a high precision (0.97) and recall (0.93), outperforming some state-of-the-art approaches with an average increase of 0.1 in F1-score.

Microservice combination optimisation based on improved gray wolf algorithm

Microservices architecture is a new paradigm for application development. The problem of optimising the performance of microservice architectures from a non-functional perspective is a typical Nondeterministic Polynomial (NP) problem. Therefore, aiming to quantify the non-functional requirements of computing microservice systems, while solving the problem of latency in computing the best combination of services with the maximum QoS objective function value, this paper proposes a microservice combination approach based on the QoS model and a CGWO algorithm for optimisation computation for this model. The experimental results verify that the error rate of the method is only 0.528% on the non-functional combination optimisation problem, and the computational efficiency of the algorithm increases by 97.29% when the complexity of the problem search space increases, while CGWO improves 65.97% and 81.25% respectively in the accuracy of optimisation compared to the prototype of the algorithm (GWO), and has a stable optimisation performance, aspect. It proves that the research in this paper has a high advantage in automatically searching for the best QoS for the microservice combination problem.

A microservice regression testing selection approach based on belief propagation

Regression testing is required to assure the quality of each iteration of microservice systems. Test case selection is one of main techniques to optimize regression testing. Existing techniques mainly involve artifacts acquisition, processing and maintenance, thus hard to apply in microservice regression testing since it is difficult to obtain and process required artifacts from multiple development teams, which is normal in cases of microservice systems. This paper proposes a novel approach, namely MRTS-BP, which takes API gateway logs instead of artifacts as inputs. By mining service dependencies from API gateway logs, MRTS-BP analyzes service change impacts based on a propagation calculation, and selects test cases affected by changes based on impact degree values. To evaluate the effectiveness of MRTS-BP, empirical studies based on four real deployed systems are presented. Retest-all strategy and a regression testing selection approach based on control flow graphs called RTS-CFG are compared with MRTS-BP. The results show that, MRTS-BP can significantly reduce both the number of test cases and overall time cost while maintaining the fault detection capability of selected test suite, and that MRTS-BP can save more time cost than RTS-CFG with the similar safety and precision.

Microservice Security Metrics for Secure Communication, Identity Management, and Observability

Microservice architectures are increasingly being used to develop application systems. Despite many guidelines and best practices being published, architecting microservice systems for security is challenging. Reasons are the size and complexity of microservice systems, their polyglot nature, and the demand for the continuous evolution of these systems. In this context, to manually validate that security architecture tactics are employed as intended throughout the system is a time-consuming and error-prone task. In this article, we present an approach to avoid such manual validation before each continuous evolution step in a microservice system, which we demonstrate using three widely used categories of security tactics: secure communication, identity management, and observability. Our approach is based on a review of existing security guidelines, the gray literature, and the scientific literature, from which we derived Architectural Design Decisions (ADDs) with the found security tactics as decision options. In our approach, we propose novel detectors to detect these decision options automatically and formally defined metrics to measure the conformance of a system to the different options of the ADDs. We apply the approach to a case study data set of 10 open source microservice systems, plus another 20 variants of these systems, for which we manually inspected the source code for security tactics. We demonstrate and assess the validity and appropriateness of our metrics by performing an assessment of their conformance to the ADDs in our systems’ dataset through statistical methods.