User-facing Services Research Articles

A self-stabilizing and auto-provisioning orchestration for microservices in edge-cloud continuum

Modern user-facing services are progressively evolving from large monolithic applications to complex graphs of loosely-coupled microservices. While this shift provides opportunities to offload some microservices of a user-facing service to edge devices that are close to the end users, it also complicates the application deployment and resource provisioning in the edge-cloud continuum, due to complex relationships across microservices and unstable public networks. To reduce resource wastage and improve user experience, this paper presents SMO, a self-managed orchestration system for microservices. SMO leverages a self-stabilizing placement mechanism to optimally deploy microservices through perceiving both interferences and communication overheads. During runtime, it further tailors a multi-agent deep deterministic policy gradient (MADDPG)-based model in combination with attention and prioritized replay, which automatically provisions appropriate resources for each microservice subject to the differentiated tail latency Service Level Objectives (SLOs) of multiple user workloads. Experimental results demonstrate that SMO saves up to 39% of CPU cores and 47% of memory footprint while providing guarantees for heterogeneous tail latency SLOs.

AutoMan: Resource-efficient provisioning with tail latency guarantees for microservices

Modern user-facing services are progressively evolving from large monolithic applications to complex graphs of loosely-coupled microservices. While microservice architecture greatly improves the efficiency of development and operation, it also complicates resource allocation and performance guarantee due to complex dependencies across different microservices. To prevent resource wastage and ensure user satisfaction, we present AutoMan, a learning-driven resource manager for microservices that enables much more efficient resource provisioning while guaranteeing the end-to-end tail latency Service Level Objective (SLO). AutoMan leverages a multi-agent deep deterministic policy gradient (MADDPG)-based method to capture the dependencies across different microservices and to allocate a proper amount of resources to each microservice subject to the target end-to-end tail latency SLO. During runtime, it further proactively identifies the critical microservices responsible for performance anomaly by deriving partial SLOs mathematically, and performs dynamic reprovisioning to mitigate the potential SLO violations. Testbed experiments show that AutoMan can save CPU and memory resources by up to 49.6% and 29.1% on average, while guaranteeing the same end-to-end tail latency objective.

Adaptive Resource Efficient Microservice Deployment in Cloud-Edge Continuum

User-facing services are now evolving towards the microservice architecture where a service is built by connecting multiple microservice stages. Since the entire service is heavy, the microservice architecture shows the opportunity to only offload some microservice stages to the edge devices that are close to the end users. However, emerging techniques often result in the violation of Quality-of-Service (QoS) of microservice-based services in cloud-edge continuum, as they do not consider the communication overhead or the resource contention between microservices and external co-located tasks. We propose Nautilus, a runtime system that effectively deploys microservice-based user-facing services in cloud-edge continuum. Nautilus ensures the QoS of microservice-based user-facing services while minimizing the required computational resources, which is comprised of a communication-aware microservice mapper, a contention-aware resource manager and an IO-sensitive and load-aware microservice migration scheduler. The mapper divides the microservice graph into multiple partitions based on the communication overhead and maps the partitions to appropriate nodes. On each node, the resource manager determines the optimal resource allocation for its microservices based on reinforcement learning that may capture the complex contention behaviors. Once the microservices are suffered from external IO pressure, the IO-sensitive microservice scheduler migrates the critical one to idle nodes. Furthermore, when the load of microservices changes dynamically, the load-aware microservice scheduler migrates microservices from busy nodes to idle ones to ensure the QoS goal of the entire service. Our experimental results show that Nautilus can guarantee the required QoS target under external shared resources contention while the state-of-the-art suffers from QoS violations. Meanwhile, Nautilus reduces the computational resource usage by 23.9% and the network bandwidth usage by 53.4%, while achieving the required 99%-ile latency.

Data protection record of processing activities and privacy notice generator toolkit by EMBL’s European Bioinformatics Institute

EMBL-EBI, Europe's biomolecular data hub, is a world leader in managing and analysing big data in biology and making it freely available to scientists worldwide. Researchers can access the open data resources and related services of EMBL-EBI by submitting minimal personal data. In May 2018, following the enforcement of the European Data Protection Regulation (GDPR), EMBL adopted the EMBL Internal Policy no. 68 on General Data Protection. It reflects European data protection principles while remaining within the bounds of EMBL's international legal status. As a result of GDPR and EMBL's Internal Policy No. 68 coming into force, 190 EMBL-EBI user-facing services that processed personal data in 2018 were required to have Records of Processing Activities (RoPA) and Privacy Notices (PN). EMBL-EBI's solution was to develop a Data Protection Engine (DPE) that automatically generates RoPA and PN when a service owner answers a series of questions. In addition to maintaining a centrally located database for RoPAs and PNs, the DPE tracks changes to the documents, as well as providing versioning and time-stamped updates. It is the aim of this article to share the EMBL-EBI IT department’s experience with designing and implementing the DPE and providing a toolkit to let others develop a similar solution and benefit from our experience. Implementation steps, benefits, challenges, opportunities, and practices are discussed and critically analysed.

Autonomous Lifecycle Management for Resource-Efficient Workload Orchestration for Green Edge Computing

Edge computing is an important pillar for green computation by bringing the Cloud resources to the Edge, serving real-time applications, and reducing the computing and network resources required to transfer data for processing in the Cloud. 5G brings network densification and enables massive IoT and V2X applications, which triggers the need for edge computing to host network functions and user-facing services in a converged edge platform(s). Several edge computing deployments are being observed by ecosystem players (telco, ISVs, chip vendors, CSPs, … etc.) for IoT or V2X services, however, focusing on converged network functions and services. The point that is still in its early stages is the dynamic workload orchestration across the converged edge platforms running network functions and multi-tenant IoT services with different compute requirements and different Service Level Objectives (SLOs). This paper focuses on autonomous life cycle management for converged edge platform(s) to enable resource-efficient workload orchestration, contributing to the green goal. We present a solution for intelligent dynamic resources configuration on edge computing platforms hosting multi-tenant services while guaranteeing the SLO for each service and helping green communication goal. The presented solution has been deployed in a trial, and we present results on efficient resources configuration.

Toward QoS-Awareness and Improved Utilization of Spatial Multitasking GPUs

Datacenters use GPUs to provide the significant computing throughput required by emerging user-facing services. The diurnal user access pattern of user-facing services provides a strong incentive to co-located applications for better GPU utilization, and prior work has focused on enabling co-location on multicore processors and traditional non-preemptive GPUs. However, current GPUs are evolving towards spatial multitasking and introduce a new set of challenges to eliminate QoS violations. To address this open problem, we explore the underlying causes of QoS violation on spatial multitasking GPUs. In response to these causes, we propose C-Laius, a runtime system that carefully allocates the computation resource to co-located applications for maximizing the throughput of batch applications while guaranteeing the required QoS of user-facing services. C-Laius not only allows co-locating one user-facing application with multiple batch applications, but also supports the co-location of multiple user-facing applications with batch applications. In the case of a single co-located user-facing application, our evaluation on an Nvidia RTX 2080Ti GPU shows that C-Laius improves the utilization of spatial multitasking GPUs by 20.8 percent, while achieving the 99%-ile latency target for user-facing services. As to the case of multiple co-located user-facing applications, C-Laius ensures no violation of QoS while improving the accelerator utilization by 35.9 percent on average.

Every Timestamp Counts: Accurate Tracking of Network Latencies Using Reconcilable Difference Aggregator

User-facing services deployed in data centers must respond quickly to user actions. The measurement of network latencies is of paramount importance. Recently, a new family of compact data structures has been proposed to estimate one-way latencies. In order to achieve scalability, these new methods rely on timestamp aggregation. Unfortunately, this approach suffers from serious accuracy problems in the presence of packet loss and reordering, given that a single lost or out-of-order packet may invalidate a huge number of aggregated samples. In this paper, we unify the problem to detect lost and reordered packets within the set reconciliation framework. Although the set reconciliation approach and the data structures for aggregating packet timestamps are previously known, the combination of these two principles is novel. We present a space-efficient synopsis called reconcilable difference aggregator (RDA). RDA maximizes the percentage of useful packets for latency measurement by mapping packets to multiple banks and repairing aggregated samples that have been damaged by lost and reordered packets. RDA simultaneously obtains the average and the standard deviation of the latency. We provide a formal guarantee of the performance and derive optimized parameters. We further design and implement a user-space passive latency measurement system that addresses practical issues of integrating RDA into the network stack. Our extensive evaluation shows that compared with existing methods, our approach improves the relative error of the average latency estimation in 10–15 orders of magnitude, and the relative error of the standard deviation in 0.5–6 orders of magnitude.

Baymax

Modern warehouse-scale computers (WSCs) are being outfitted with accelerators to provide the significant compute required by emerging intelligent personal assistant (IPA) workloads such as voice recognition, image classification, and natural language processing. It is well known that the diurnal user access pattern of user-facing services provides a strong incentive to co-locate applications for better accelerator utilization and efficiency, and prior work has focused on enabling co-location on multicore processors. However, interference when co-locating applications on non-preemptive accelerators is fundamentally different than contention on multi-core CPUs and introduces a new set of challenges to reduce QoS violation. To address this open problem, we first identify the underlying causes for QoS violation in accelerator-outfitted servers. Our experiments show that queuing delay for the compute resources and PCI-e bandwidth contention for data transfer are the main two factors that contribute to the long tails of user-facing applications. We then present Baymax, a runtime system that orchestrates the execution of compute tasks from different applications and mitigates PCI-e bandwidth contention to deliver the required QoS for user-facing applications and increase the accelerator utilization. Using DjiNN, a deep neural network service, Sirius, an end-to-end IPA workload, and traditional applications on a Nvidia K40 GPU, our evaluation shows that Baymax improves the accelerator utilization by 91.3% while achieving the desired 99%-ile latency target for for user-facing applications. In fact, Baymax reduces the 99%-ile latency of user-facing applications by up to 195x over default execution.

Baymax

Modern warehouse-scale computers (WSCs) are being outfitted with accelerators to provide the significant compute required by emerging intelligent personal assistant (IPA) workloads such as voice recognition, image classification, and natural language processing. It is well known that the diurnal user access pattern of user-facing services provides a strong incentive to co-locate applications for better accelerator utilization and efficiency, and prior work has focused on enabling co-location on multicore processors. However, interference when co-locating applications on non-preemptive accelerators is fundamentally different than contention on multi-core CPUs and introduces a new set of challenges to reduce QoS violation. To address this open problem, we first identify the underlying causes for QoS violation in accelerator-outfitted servers. Our experiments show that queuing delay for the compute resources and PCI-e bandwidth contention for data transfer are the main two factors that contribute to the long tails of user-facing applications. We then present Baymax, a runtime system that orchestrates the execution of compute tasks from different applications and mitigates PCI-e bandwidth contention to deliver the required QoS for user-facing applications and increase the accelerator utilization. Using DjiNN, a deep neural network service, Sirius, an end-to-end IPA workload, and traditional applications on a Nvidia K40 GPU, our evaluation shows that Baymax improves the accelerator utilization by 91.3% while achieving the desired 99%-ile latency target for for user-facing applications. In fact, Baymax reduces the 99%-ile latency of user-facing applications by up to 195x over default execution.

Server Engineering Insights for Large-Scale Online Services

The rapid growth of online services in the last decade has led to the development of large data centers to host these workloads. These large-scale online, user-facing services have unique engineering and capacity provisioning design requirements. The authors explore these requirements, focusing on system balancing, the impact of technology trends, and the challenges of online service workloads.