Service Level Objectives Research Articles

Tao: Improving Resource Utilization while Guaranteeing SLO in Multi-tenant Relational Database-as-a-Service

It is an open challenge for cloud database service providers to guarantee tenants' service-level objectives (SLOs) and enjoy high resource utilization simultaneously. In this work, we propose a novel system Tao to overcome it. Tao consists of three key components: (i) tasklet-based DAG generator, (ii) tasklet-based DAG executor, and (iii) SLO-guaranteed scheduler. The core concept in Tao is tasklet, a coroutine-based lightweight execution unit of the physical execution plan. In particular, we first convert each SQL operator in the traditional physical execution plan into a set of fine-grained tasklets by the tasklet-based DAG generator. Then, we abstract the tasklet-based DAG execution procedure and implement the tasklet-based DAG executor using C++20 coroutines. Finally, we introduce the SLO-guaranteed scheduler for scheduling tenants' tasklets across CPU cores. This scheduler guarantees tenants' SLOs with a token bucket model and improves resource utilization with an on-demand core adjustment strategy. We build Tao on an open-sourced relational database, Hyrise, and conduct extensive experimental studies to demonstrate its superiority over existing solutions.

GenesisRM: A state-driven approach to resource management for distributed JVM web applications

Reducing resource waste while maintaining end-to-end latency service-level objective (SLO) by simultaneously managing CPU bandwidth, memory allocation, and pod number of web applications running on Java virtual machine (JVM) is challenging. The challenges stem from the complexity of the multi-type resource allocation optimization problem, the high sensitivity of JVM performance to resource scaling actions, and the lack of low-level resource scaling mechanisms. We present GenesisRM, a resource management framework with a novel state-driven architecture. Specifically, we design a state control model for JVM web applications that encompasses seven pod states. This model serves as an abstraction layer, decoupling the centralized resource management system into a global state manager and distributed pod managers. The state manager controls the state transitions of the pods based on the overall workload, while the pod managers dynamically allocate resources for each pod according to the state and local workload.Then, we propose a multi-frequency control model with two predictive state controllers and a reactive state controller to manage the state of pods based on the state control model. In addition, GenesisRM brings new mechanisms to scale JVM pods’ low-level resources without damaging their performance. We evaluate our work using a real-world JVM web application benchmark in three different scale server clusters of Pengcheng Laboratory Developer Cloud, and the 21-day experimental results show that GenesisRM saves 31.70% CPU and 17.60% memory compared to the best-performing state-of-the-art solutions while guaranteeing the SLO imposed on end-to-end latency.

OptScaler: A Collaborative Framework for Robust Autoscaling in the Cloud

Autoscaling is a critical mechanism in cloud computing, enabling the autonomous adjustment of computing resources in response to dynamic workloads. This is particularly valuable for co-located, long-running applications with diverse workload patterns. The primary objective of autoscaling is to regulate resource utilization at a desired level, effectively balancing the need for resource optimization with the fulfillment of Service Level Objectives (SLOs). Many existing proactive autoscaling frameworks may encounter prediction deviations arising from the frequent fluctuations of cloud workloads. Reactive frameworks, on the other hand, rely on realtime system feedback, but their hysteretic nature could lead to violations of stringent SLOs. Hybrid frameworks, while prevalent, often feature independently functioning proactive and reactive modules, potentially leading to incompatibility and undermining the overall decision-making efficacy. In addressing these challenges, we propose OptScaler, a collaborative autoscaling framework that integrates proactive and reactive modules through an optimization module. The proactive module delivers reliable future workload predictions to the optimization module, while the reactive module offers a self-tuning estimator for real-time updates. By embedding a Model Predictive Control (MPC) mechanism and chance constraints into the optimization module, we further enhance its robustness. Numerical results have demonstrated the superiority of our workload prediction model and the collaborative framework, leading to over a 36% reduction in SLO violations compared to prevalent reactive, proactive, or hybrid autoscalers. Notably, OptScaler has been successfully deployed at Alipay, providing autoscaling support for the world-leading payment platform.

Towards Millions of Database Transmission Services in the Cloud

Alibaba relies on its robust database infrastructure to facilitate realtime data access and ensure business continuity despite regional disruptions. To address these operational imperatives, Alibaba developed the Data Transmission Service (DTS), which has become critical for internal applications and public cloud services alike. This paper presents a comprehensive study of the architectural innovations, resource scheduling mechanisms, and performance optimization strategies that have been implemented within DTS to tackle the significant challenges of cross-network, heterogeneous data transmission in a cost-effective manner. We explore the novel Any-to-Any (A2A) architecture, which simplifies the complexity of data paths between diverse databases and mitigates network connectivity issues, thereby significantly reducing development overhead. Additionally, we examine a dynamic network bandwidth scheduling algorithm that effectively maintains Service-Level Objectives (SLOs), complemented by a serverless mechanism that ensures efficient resource utilization. Furthermore, DTS utilizes advanced strategies such as transaction dependency tracking, hot data consolidation, and batching to enhance synchronization performance and efficiency. DTS has distilled the lessons learned from years of serving our customer base and currently supports nearly 1 million public cloud instances annually. Our evaluation results show that DTS can effectively and efficiently handle real-time data transmission in both experimental and production environments.

Concurrent service auto-scaling for Knative resource quota-based serverless system

Serverless computing platforms currently provide application developers with two ways to control their services’ resource usage cost: resource usage-based and resource quota-based. In usage-based systems, resources are assigned to services based on the amount they consume. Meanwhile, a fixed resource amount is reserved and shared between services for quota-based systems. Several studies have been proposed to enhancing the default serverless auto- scaling algorithms to optimize resource usage and service quality. However, almost all previous works targeted usage-based systems and optimized auto-scaling performance for each separate service. In contrast, this work targets auto-scaling of concurrent services in serverless quota-based systems. In serverless quota-based systems, over-quota resource usage and latency Service Level Objective (SLO) violation may occur when auto-scaling new instances during burst traffic moments. Hence, we aim to find an optimal hybrid auto-scaling decision to minimize over-quota resource usage and latency SLO violation. To solve this problem, We applied deep reinforcement learning and traffic prediction techniques. Our solution is developed and implemented based on the characteristics of the most popular open-source serverless platform Knative. For evaluation, we compared our solution with the default Knative auto-scaler and our previously proposed usage-based hybrid Knative auto-scaler. We configured the default Knative auto-scaler with two different settings that can optimize the total resource usage of all services or minimize service latency. Compared to the optimized resource usage version, our solution traded a slightly higher resource usage for an 11.3% latency SLO violation duration reduction. Meanwhile, compared with the optimized latency version, our solution reduced the over-quota resource usage rate by 18.25% while achieving a similar latency SLO preservation performance. Compared to the usage-based hybrid auto-scaler, the latency SLO violation rate and over-quota resource usage were approximately reduced by half.

On the Analysis of Inter-Relationship between Auto-Scaling Policy and QoS of FaaS Workloads

A recent development in cloud computing has introduced serverless technology, enabling the convenient and flexible management of cloud-native applications. Typically, the Function-as-a-Service (FaaS) solutions rely on serverless backend solutions, such as Kubernetes (K8s) and Knative, to leverage the advantages of resource management for underlying containerized contexts, including auto-scaling and pod scheduling. To take the advantages, recent cloud service providers also deploy self-hosted serverless services by facilitating their on-premise hosted FaaS platforms rather than relying on commercial public cloud offerings. However, the lack of standardized guidelines on K8s abstraction to fairly schedule and allocate resources on auto-scaling configuration options for such on-premise hosting environment in serverless computing poses challenges in meeting the service level objectives (SLOs) of diverse workloads. This study fills this gap by exploring the relationship between auto-scaling behavior and the performance of FaaS workloads depending on scaling-related configurations in K8s. Based on comprehensive measurement studies, we derived the logic as to which workload should be applied and with what type of scaling configurations, such as base metric, threshold to maximize the difference in latency SLO, and number of responses. Additionally, we propose a methodology to assess the scaling efficiency of the related K8s configurations regarding the quality of service (QoS) of FaaS workloads.

Equilibrium in the Computing Continuum through Active Inference

Computing Continuum (CC) systems are challenged to ensure the intricate requirements of each computational tier. Given the system’s scale, the Service Level Objectives (SLOs), which are expressed as these requirements, must be disaggregated into smaller parts that can be decentralized. We present our framework for collaborative edge intelligence, enabling individual edge devices to (1) develop a causal understanding of how to enforce their SLOs and (2) transfer knowledge to speed up the onboarding of heterogeneous devices. Through collaboration, they (3) increase the scope of SLO fulfillment. We implemented the framework and evaluated a use case in which a CC system is responsible for ensuring Quality of Service (QoS) and Quality of Experience (QoE) during video streaming. Our results showed that edge devices required only ten training rounds to ensure four SLOs; furthermore, the underlying causal structures were also rationally explainable. The addition of new types of devices can be done a posteriori; the framework allowed them to reuse existing models, even though the device type had been unknown. Finally, rebalancing the load within a device cluster allowed individual edge devices to recover their SLO compliance after a network failure from 22% to 89%.

CARIn: Constraint-Aware and Responsive Inference on Heterogeneous Devices for Single- and Multi-DNN Workloads

The relentless expansion of deep learning (DL) applications in recent years has prompted a pivotal shift towards on-device execution, driven by the urgent need for real-time processing, heightened privacy concerns, and reduced latency across diverse domains. This paper addresses the challenges inherent in optimising the execution of deep neural networks (DNNs) on mobile devices, with a focus on device heterogeneity, multi-DNN execution, and dynamic runtime adaptation. We introduce CARIn , a novel framework designed for the optimised deployment of both single- and multi-DNN applications under user-defined service-level objectives (SLOs). Leveraging an expressive multi-objective optimisation (MOO) framework and a runtime-aware sorting and search algorithm ( RASS ) as the MOO solver, CARIn facilitates efficient adaptation to dynamic conditions while addressing resource contention issues associated with multi-DNN execution. Notably, RASS generates a set of configurations, anticipating subsequent runtime adaptation, ensuring rapid, low-overhead adjustments in response to environmental fluctuations. Extensive evaluation across diverse tasks, including text classification, scene recognition, and face analysis, showcases the versatility of CARIn across various model architectures, such as Convolutional Neural Networks (CNNs) and Transformers, and realistic use cases. We observe a substantial enhancement in the fair treatment of the problem’s objectives, reaching 1.92 × when compared to single-model designs, and up to 10.69 × in contrast to the state-of-the-art OODIn framework. Additionally, we achieve a significant gain of up to 4.06 × over hardware-unaware designs in multi-DNN applications. Finally, our framework sustains its performance while effectively eliminating the time overhead associated with identifying the optimal design in response to environmental challenges.

OptimML: Joint Control of Inference Latency and Server Power Consumption for ML Performance Optimization

Power capping is an important technique for high-density servers to safely oversubscribe the power infrastructure in a data center. However, power capping is commonly accomplished by dynamically lowering the server processors’ frequency levels, which can result in degraded application performance. For servers that run important machine learning (ML) applications with Service-Level Objective (SLO) requirements, inference performance such as recognition accuracy must be optimized within a certain latency constraint, which demands high server performance. In order to achieve the best inference accuracy under the desired latency and server power constraints, this paper proposes OptimML, a multi-input-multi-output (MIMO) control framework that jointly controls both inference latency and server power consumption, by flexibly adjusting the machine learning model size (and so its required computing resources) when server frequency needs to be lowered for power capping. Our results on a hardware testbed with widely adopted ML framework (including PyTorch, TensorFlow, and MXNet) show that OptimML achieves higher inference accuracy compared with several well-designed baselines, while respecting both latency and power constraints. Furthermore, an adaptive control scheme with online model switching and estimation is designed to achieve analytic assurance of control accuracy and system stability, even in the face of significant workload/hardware variations.

Decision Making on a Software Upgrade or Decommission with Data Mining and Machine Learning Techniques in Information Technology Industry

The Organizations have been investing more in Technology and Infrastructure spends like software upgrades, software renewals, software replacements, platform migrations etc., apart from investment in Business, People, and Processes. In this context, it is not an easy task for stakeholders to decide whether to go for a software upgrade or to replace it with another software. There is no unified approach or solution to consolidate data and relationships of Information Technology Assets, Software Upgrades, Software costs, Software defects, Software Performance Metrics, Security issues, IT system versions, service level objectives etc. Due to this, the decision making of software upgrades and software decommissioning is a tedious process and takes more time and effort. There is a need to build a solution that can integrate and validate the information like software assets, software upgrade success and failure likelihoods, cost benefit analysis of Cloud Computing, software metrics for fault prediction, software maintainability prediction results, Digital Transformation readiness and other related factors. There is an opportunity to apply Machine Learning techniques in defining and deriving the success likelihoods on the following data: Systems and data integration, software assets compatibility, operational service level agreement breaches, quality assurance metrics, security issues, number of open defects, number of defect fixes, number of priority incidents, mean time to resolve critical incidents, expected cost increase in software maintenance, potential cost reduction with the software or hardware replacement etc. This Research Proposal outlines the above mentioned to build a recommendation system aka decision tree namely Software Upgrades or Decommissions Life Cycle.

SLA-Adaptive Threshold Adjustment for a Kubernetes Horizontal Pod Autoscaler

Kubernetes is an open-source container orchestration system that provides a built-in module for dynamic resource provisioning named the Horizontal Pod Autoscaler (HPA). The HPA identifies the number of resources to be provisioned by calculating the ratio between the current and target utilisation metrics. The target utilisation metric, or threshold, directly impacts how many and how quickly resources will be provisioned. However, the determination of the threshold that would allow satisfying performance-based Service Level Objectives (SLOs) is a long, error-prone, manual process because it is based on the static threshold principle and requires manual configuration. This might result in underprovisioning or overprovisioning, leading to the inadequate allocation of computing resources or SLO violations. Numerous autoscaling solutions have been introduced as alternatives to the HPA to simplify the process. However, the HPA is still the most widely used solution due to its ease of setup, operation, and seamless integration with other Kubernetes functionalities. The present study proposes a method that utilises exploratory data analysis techniques along with moving average smoothing to identify the target utilisation threshold for the HPA. The objective is to ensure that the system functions without exceeding the maximum number of events that result in a violation of the response time defined in the SLO. A prototype was created to adjust the threshold values dynamically, utilising the proposed method. This prototype enables the evaluation and comparison of the proposed method with the HPA, which has the highest threshold set that meets the performance-based SLOs. The results of the experiments proved that the suggested method adjusts the thresholds to the desired service level with a 1–2% accuracy rate and only 4–10% resource overprovisioning, depending on the type of workload.

A self-stabilizing and auto-provisioning orchestration for microservices in edge-cloud continuum

Modern user-facing services are progressively evolving from large monolithic applications to complex graphs of loosely-coupled microservices. While this shift provides opportunities to offload some microservices of a user-facing service to edge devices that are close to the end users, it also complicates the application deployment and resource provisioning in the edge-cloud continuum, due to complex relationships across microservices and unstable public networks. To reduce resource wastage and improve user experience, this paper presents SMO, a self-managed orchestration system for microservices. SMO leverages a self-stabilizing placement mechanism to optimally deploy microservices through perceiving both interferences and communication overheads. During runtime, it further tailors a multi-agent deep deterministic policy gradient (MADDPG)-based model in combination with attention and prioritized replay, which automatically provisions appropriate resources for each microservice subject to the differentiated tail latency Service Level Objectives (SLOs) of multiple user workloads. Experimental results demonstrate that SMO saves up to 39% of CPU cores and 47% of memory footprint while providing guarantees for heterogeneous tail latency SLOs.

Inference serving with end-to-end latency SLOs over dynamic edge networks

While high accuracy is of paramount importance for deep learning (DL) inference, serving inference requests on time is equally critical but has not been carefully studied especially when the request has to be served over a dynamic wireless network at the edge. In this paper, we propose Jellyfish—a novel edge DL inference serving system that achieves soft guarantees for end-to-end inference latency service-level objectives (SLO). Jellyfish handles the network variability by utilizing both data and deep neural network (DNN) adaptation to conduct tradeoffs between accuracy and latency. Jellyfish features a new design that enables collective adaptation policies where the decisions for data and DNN adaptations are aligned and coordinated among multiple users with varying network conditions. We propose efficient algorithms to continuously map users and adapt DNNs at runtime, so that we fulfill latency SLOs while maximizing the overall inference accuracy. We further investigate dynamic DNNs, i.e., DNNs that encompass multiple architecture variants, and demonstrate their potential benefit through preliminary experiments. Our experiments based on a prototype implementation and real-world WiFi and LTE network traces show that Jellyfish can meet latency SLOs at around the 99th percentile while maintaining high accuracy.

Transient staffing at the beginning of work

Production and service systems often rely on stationary queueing formulas to determine the number of agents that need to be scheduled. However, this results in over-staffing decisions when the system starts empty. To address this issue, we propose a solution called transient staffing, which involves providing appointment times for agents to ensure that the expected queue length remains below a certain threshold. We model the system as a transient Erlang-A queue. To compute the appointment times, we employ a uniformized approximation that discretizes the elapsed time, which can be made as accurate as desired. We prove that the appointment times decrease with the arrival rate and traffic intensity, while increasing with the abandonment rate. Through numerical investigations, we find that adopting a transient staffing strategy leads to substantial cost savings, especially in scenarios where demand is high, customers are patient, traffic intensity is low, and the service level objective is intermediate. Furthermore, we use a fluid approximation to derive closed-form expressions for the agents' appointment times, and prove the impact of the system parameters. We show that this approximation performs well in predicting the first appointment times in contexts with high demand, low traffic intensity, and lax service level objective.

Database Native Model Selection: Harnessing Deep Neural Networks in Database Systems

The growing demand for advanced analytics beyond statistical aggregation calls for database systems that support effective model selection of deep neural networks (DNNs). However, existing model selection strategies are based on either training-based algorithms that deliver high-performing models at the expense of high computational cost, or training-free algorithms that enhance computational efficiency with reduced effectiveness. These strategies often disregard computational cost and response time Service-Level Objectives (SLOs), which are of concern to average or budget-conscious machine learning users. In addition, they lack a well-designed integration of the model selection algorithms with DBMSs, which hinders efficient in-database model selection. This paper presents TRAILS, a resource-efficient and SLO-aware in-database model selection system. To leverage the strengths of both training-free and training-based model selection, we first characterize nine state-of-the-art training-free model evaluation metrics and propose a more effective one named JacFlow, and then, restructure the conventional model selection procedure into two phases: filtering and refinement. A novel coordinator is also introduced to strike a balance between the high efficiency of train-free algorithms and the high effectiveness of training-based algorithms, ensuring high-performing model selection while adhering to target SLOs. Moreover, we incorporate the proposed algorithm into PostgreSQL to develop TRAILS, thereby both enhancing resource efficiency and reducing model selection latency. This integration establishes a foundation for declarative model definition and selection within DBMSs. Empirical results demonstrate that our TRAILS reduces model selection time and computational expenses considerably by up to 24.38x and 29.32x respectively compared to existing model selection systems.

Efficient computational strategies for a mathematical programming model for multi-echelon inventory optimization based on the guaranteed-service approach

This paper presents a Multi-Echelon Inventory Optimization (MEIO) framework, based on the Guaranteed-Service Model (GSM), to allocate safety stocks across a supply chain with several locations and products, minimizing costs while meeting service level objectives. Extending previous work by Achkar et al. (2023), this paper enhances the Mixed-Integer Quadratically Constrained Program (MIQCP) with a highly efficient solution approach. The model introduces a piecewise linear approximation, significantly improving computational efficiency and the accuracy of the approximation for the fill rate function. It also introduces a different and more efficient approach to account for stochastic lead times using a discrete function. Moreover, an extension of the approach to account for non-normally distributed demands is proposed. The model is applied to several instances of a real-world case study from a pharmaceutical company, with more than 7300 product-location combinations, showing that optimal solutions can be obtained within few seconds of computational time.

Efficient Resource Utilization in IoT and Cloud Computing

With the proliferation of IoT devices, there has been exponential growth in data generation, placing substantial demands on both cloud computing (CC) and internet infrastructure. CC, renowned for its scalability and virtual resource provisioning, is of paramount importance in e-commerce applications. However, the dynamic nature of IoT and cloud services introduces unique challenges, notably in the establishment of service-level agreements (SLAs) and the continuous monitoring of compliance. This paper presents a versatile framework for the adaptation of e-commerce applications to IoT and CC environments. It introduces a comprehensive set of metrics designed to support SLAs by enabling periodic resource assessments, ensuring alignment with service-level objectives (SLOs). This policy-driven approach seeks to automate resource management in the era of CC, thereby reducing the dependency on extensive human intervention in e-commerce applications. This paper culminates with a case study that demonstrates the practical utilization of metrics and policies in the management of cloud resources. Furthermore, it provides valuable insights into the resource requisites for deploying e-commerce applications within the realms of the IoT and CC. This holistic approach holds the potential to streamline the monitoring and administration of CC services, ultimately enhancing their efficiency and reliability.

A convertible neural processor supporting adaptive quantization for real-time neural networks

This paper presents Stimpack, a responsive approach toward adaptive neural processing unit (NPU), aiming to satisfy service-level objectives (SLOs) under highly massive loads of neural network (NN) inference. Initially, Stimpack operates in a base-mode processing and computes NNs similarly to a conventional NPU accelerator. During base-mode processing, if SLO violation is likely to occur soon, it changes the operation mode to a burst-mode processing. In the burst-mode processing, Stimpack computes quantized networks instead of the original ones, enabling computational throughput to be scaled up to two times higher than the base-mode processing. This switchable processing is facilitated by three hardware/software schemes. First, a reconfigurable core is adopted to support two different precisions and boost processing throughput for avoiding SLO violations. As computation resources can be shared between the two operation modes, the area overhead of a reconfigurable core is negligible. The second is an on-chip quantization unit that mitigates the data transfer overhead incurred by mode changing. It quantizes parameters stored in on-chip memory on the fly, instead of bringing quantized parameters from off-chip memory. Third, Stimpack leverages a scheduler that determines mode switching based on the amount of workloads in the server. By monitoring ongoing and queued requests of NPU, the scheduler conservatively activates burst-mode processing to minimize accuracy loss. Our analysis shows that, compared to a state-of-the-art NPU, Stimpack achieves 48.4% speedup and allows a 41.4% large load on average while satisfying SLO and near-ideal accuracy.

FireFace: Leveraging Internal Function Features for Configuration of Functions on Serverless Edge Platforms.

The emerging serverless computing has become a captivating paradigm for deploying cloud applications, alleviating developers' concerns about infrastructure resource management by configuring necessary parameters such as latency and memory constraints. Existing resource configuration solutions for cloud-based serverless applications can be broadly classified into modeling based on historical data or a combination of sparse measurements and interpolation/modeling. In pursuit of service response and conserving network bandwidth, platforms have progressively expanded from the traditional cloud to the edge. Compared to cloud platforms, serverless edge platforms often lead to more running overhead due to their limited resources, resulting in undesirable financial costs for developers when using the existing solutions. Meanwhile, it is extremely challenging to handle the heterogeneity of edge platforms, characterized by distinct pricing owing to their varying resource preferences. To tackle these challenges, we propose an adaptive and efficient approach called FireFace, consisting of prediction and decision modules. The prediction module extracts the internal features of all functions within the serverless application and uses this information to predict the execution time of the functions under specific configuration schemes. Based on the prediction module, the decision module analyzes the environment information and uses the Adaptive Particle Swarm Optimization algorithm and Genetic Algorithm Operator (APSO-GA) algorithm to select the most suitable configuration plan for each function, including CPU, memory, and edge platforms. In this way, it is possible to effectively minimize the financial overhead while fulfilling the Service Level Objectives (SLOs). Extensive experimental results show that our prediction model obtains optimal results under all three metrics, and the prediction error rate for real-world serverless applications is in the range of 4.25∼9.51%. Our approach can find the optimal resource configuration scheme for each application, which saves 7.2∼44.8% on average compared to other classic algorithms. Moreover, FireFace exhibits rapid adaptability, efficiently adjusting resource allocation schemes in response to dynamic environments.

A Novel Approach Based on Service Level Agreement (SLA) for Evaluating the Web Service Quality

Web service technology has gained a more important role in developing distributed applications and systems on the Internet. The rapid growth of published web services makes their discovery more and more difficult. Nowadays, most web service providers sign Services Level Agreement (SLA) contracts with their clients in order to guarantee the offered functionality of their services. This paper proposes an approach to monitor the Quality of Services (QoS) in web service according to Service Level Objectives (SLO) in SLA. Monitoring procedures are introduced to check variations in the pre-agreed metric values of SLAs. Then, the deviation between the actual quality and the acceptable quality level can be identified and analyzed. Finally, the weaknesses of the web service practices can be discovered and solved.