Service Level Objectives Violations Research Articles

SLA-Adaptive Threshold Adjustment for a Kubernetes Horizontal Pod Autoscaler

Kubernetes is an open-source container orchestration system that provides a built-in module for dynamic resource provisioning named the Horizontal Pod Autoscaler (HPA). The HPA identifies the number of resources to be provisioned by calculating the ratio between the current and target utilisation metrics. The target utilisation metric, or threshold, directly impacts how many and how quickly resources will be provisioned. However, the determination of the threshold that would allow satisfying performance-based Service Level Objectives (SLOs) is a long, error-prone, manual process because it is based on the static threshold principle and requires manual configuration. This might result in underprovisioning or overprovisioning, leading to the inadequate allocation of computing resources or SLO violations. Numerous autoscaling solutions have been introduced as alternatives to the HPA to simplify the process. However, the HPA is still the most widely used solution due to its ease of setup, operation, and seamless integration with other Kubernetes functionalities. The present study proposes a method that utilises exploratory data analysis techniques along with moving average smoothing to identify the target utilisation threshold for the HPA. The objective is to ensure that the system functions without exceeding the maximum number of events that result in a violation of the response time defined in the SLO. A prototype was created to adjust the threshold values dynamically, utilising the proposed method. This prototype enables the evaluation and comparison of the proposed method with the HPA, which has the highest threshold set that meets the performance-based SLOs. The results of the experiments proved that the suggested method adjusts the thresholds to the desired service level with a 1–2% accuracy rate and only 4–10% resource overprovisioning, depending on the type of workload.

A deep reinforcement learning-based optimization method for long-running applications container deployment

Unlike the short execution cycles of batch jobs, intelligent algorithmic applications typically run in long-cycle containers in the cloud(Long-Running Applications, LRA). Both need to meet strict SLO (service level objective) requirements, consider performance scaling to cope with peak load demands, and face issues such as I/O dependencies and resource contention and interference from coexisting containers. The above greatly complicates container deployment and can easily lead to performance bottlenecks. Therefore, the optimization of LRA-like container deployment is one of the key issues that cannot be avoided and needs to be addressed in the cloud computing model. This research uses deep reinforcement learning (DRL) to optimize the deployment of LRAs class containers. The proposed non-generic model is able to customize a dedicated model for each container group, providing high-quality placement and low training complexity; meanwhile, the proposed batch deployment scheme is able to optimize various scheduling objectives that are not directly supported by existing constraint-based schedulers, such as minimizing SLO violations. The experimental results show that the performance of the DRL deployment algorithm improves by 56.2% compared to the average RPS of the baseline, indicating that the manual deployment scheme can only meet the basic requirements but cannot cover the complex interactions between containers under constraints from a global perspective. This limitation severely limits the performance of the whole pod. Meanwhile, based on previous experience, the time consumption of a single deployment scheme is about 1 hour, while the time consumption of the DRL deployment algorithm may be less than 7.5 minutes.

AutoInfer: Self-Driving Management for Resource-Efficient, SLO-Aware Machine=Learning Inference in GPU Clusters

As Internet-of-Things (IoT) keeps growing, IoT-side intelligence services, such as intelligent personal assistant, healthcare surveillance, and smart home service, offload more and more complex machine learning (ML) inference workloads to cloud clusters. GPUs have been widely adopted to accelerate the execution of these ML inference workloads. However, current cluster management systems guarantee low tail latency for ML inferences using resource over-provisioning and small batch sizes, resulting in a serious waste of GPU resources and increasing the service costs greatly. To mitigate poor GPU utilization, we present AutoInfer, a self-driving cluster management system for ML inference serving in GPU clusters, where users express only the latency and accuracy requirements for their workloads without needing to specify the model variant, GPU provisioning strategy, and batching mechanism. AutoInfer extends the matrix factorization model to automatically recommend model variants for each new incoming ML inference workload with respect to latency and accuracy requirements, by identifying similarities to previously scheduled workloads. During runtime, AutoInfer leverages online telemetry data and deep reinforcement learning to adaptively adjust the GPU allocation and batch size to account for load variations while minimizing the effects on tail latency Service Level Objectives (SLOs). Testbed experiments show that AutoInfer is able to improve the average GPU utilization by up to 77% and keep the tail latency SLO violations under 5.5%.

An Instability-Resilient Renewable Energy Allocation System for a Cloud Datacenter

Renewable energy supply is a promising solution for datacenters' increasing electricity monetary cost, energy consumption and harmful gas emissions. However, due to the instability of renewable energy, insufficient renewable energy supply may lead to the use of stored energy or brown energy. To handle this problem, in this paper, we propose an instability-resilient renewable energy allocation system. We define a job's service-level-objective (SLO) as the successful running probability by only using supplied renewable energy. The system allocates jobs with the same SLO level to the same physical machine (PM) group, and powers each PM group with renewable energy generators that have probability no less than its SLO to produce the amount no less than its energy demand. We use a deep learning technique to predict the probability of producing the amount no less than each value of each renewable energy source, and predict the energy demands of each PM area. We formulate an optimization problem to match renewable energy resources with different instabilities to different PM groups for supply, and use reinforcement learning method and linear programming method to solve it. We further propose an energy-driven computing resource assignment method, which adjusts the amount of computing resource of each job based on job deadline and failure probability in each PM group, and a failure prediction based energy saving method. Real trace driven experiments show that our methods achieve much lower SLO violations, total energy monetary cost and total carbon emission compared to other methods and the effectiveness of individual methods.

AutoMan: Resource-efficient provisioning with tail latency guarantees for microservices

Modern user-facing services are progressively evolving from large monolithic applications to complex graphs of loosely-coupled microservices. While microservice architecture greatly improves the efficiency of development and operation, it also complicates resource allocation and performance guarantee due to complex dependencies across different microservices. To prevent resource wastage and ensure user satisfaction, we present AutoMan, a learning-driven resource manager for microservices that enables much more efficient resource provisioning while guaranteeing the end-to-end tail latency Service Level Objective (SLO). AutoMan leverages a multi-agent deep deterministic policy gradient (MADDPG)-based method to capture the dependencies across different microservices and to allocate a proper amount of resources to each microservice subject to the target end-to-end tail latency SLO. During runtime, it further proactively identifies the critical microservices responsible for performance anomaly by deriving partial SLOs mathematically, and performs dynamic reprovisioning to mitigate the potential SLO violations. Testbed experiments show that AutoMan can save CPU and memory resources by up to 49.6% and 29.1% on average, while guaranteeing the same end-to-end tail latency objective.

A Resource-Efficient Predictive Resource Provisioning System in Cloud Systems

In cloud systems, demand-prediction based resource provisioning schemes help assure the SLOs (service level objectives) of cloud tenants. We notice that if a provisioning scheme does not exclude bursts from historical resource demands in normal demand prediction or always uses a large padding to correct under-prediction, it will lead to resource over-provisioning and low resource utilization. To improve the previous schemes, in this paper, we present a Resource-efficient Predictive Resource Provisioning system in cloud systems (RPRP) that excludes bursts in demand prediction and has algorithms to specifically handle bursts to avoid resource over-provisioning. Rather than setting padding to a possibly high value, RPRP has a load-dependent padding algorithm that adaptively determines padding based on predicted demands. To handle bursts, RPRP has a burst-resilient shared padding algorithm that reserves resource shared by multiple co-located VMs rather than for individual VMs. It also embodies a responsive padding algorithm that adaptively adjusts padding to recover from both under-provisioning and over-provisioning. We implemented RPRP on top of Xen and conducted both trace-driven simulation and real-world testbed experiments. The experimental results show that RPRP achieves higher resource utilization, more accurate demand predictions, and fewer SLO violations than previous schemes.

Cushion : A proactive resource provisioning method to mitigate SLO violations for containerized microservices

Deploying microservices in container-based cloud environments increases the agility of resource scaling. However, the delay in autoscaling for microservices caused by container cold start results in response time service-level objectives (SLO) violations under burst workloads. This paper proposes Cushion, a proactive resource provisioning method for containerized microservices to mitigate SLO violations caused by burst workloads, which promptly schedules workloads to reserved container instances when workloads suddenly increase and meanwhile steadily scales the container instances according to the SLOs of the microservices. Cushion was evaluated through prototype-based experiments in a containerized testbed using a benchmark microservice and four web workload traces by comparing it against existing methods. The experimental results show 9.12x times lower SLO violations, 23.75% higher throughput with 24.35% more CPU usage, and 4.85% more memory usage compared to the baseline methods.

Burst-Aware Predictive Autoscaling for Containerized Microservices

Autoscaling methods are used for cloud-hosted applications to dynamically scale the allocated resources for guaranteeing Quality-of-Service (QoS). The public-facing application serves dynamic workloads, which contain bursts and pose challenges for autoscaling methods to ensure application performance. Existing State-of-the-art autoscaling methods are burst-oblivious to determine and provision the appropriate resources. For dynamic workloads, it is hard to detect and handle bursts online for maintaining application performance. In this article, we propose a novel burst-aware autoscaling method which detects burst in dynamic workloads using workload forecasting, resource prediction, and scaling decision making while minimizing response time service-level objectives (SLO) violations. We evaluated our approach through a trace-driven simulation, using multiple synthetic and realistic bursty workloads for containerized microservices, improving performance when comparing against existing state-of-the-art autoscaling methods. Such experiments show an increase of <inline-formula><tex-math notation="LaTeX">$\times $</tex-math><alternatives><mml:math><mml:mo>&#x00D7;</mml:mo></mml:math><inline-graphic xlink:href="iqbal-ieq1-2995937.gif"/></alternatives></inline-formula>1.09 in total processed requests, a reduction of <inline-formula><tex-math notation="LaTeX">$\times $</tex-math><alternatives><mml:math><mml:mo>&#x00D7;</mml:mo></mml:math><inline-graphic xlink:href="iqbal-ieq2-2995937.gif"/></alternatives></inline-formula>5.17 for SLO violations, and an increase of <inline-formula><tex-math notation="LaTeX">$\times $</tex-math><alternatives><mml:math><mml:mo>&#x00D7;</mml:mo></mml:math><inline-graphic xlink:href="iqbal-ieq3-2995937.gif"/></alternatives></inline-formula>0.767 cost as compared to the baseline method.

Blockchain-enabled real-time SLA monitoring for cloud-hosted services

Cloud computing is an important technology for businesses and individual users to obtain computing resources over the Internet on-demand and flexibly. Although cloud computing has been adopted across diverse applications, the owners of time-and-performance critical applications require cloud service providers’ guarantees about their services, such as availability and response times. Service Level Agreements (SLAs) are a mechanism to communicate and enforce such guarantees typically represented as service level objectives (SLOs), and financial penalties are imposed on SLO violations. Due to delays and inaccuracies caused by manual processing, an automatic method to periodically verify SLA terms in a transparent and trustworthy manner is fundamental to effective SLA monitoring, leading to the acceptance and credibility of such service to the customers of cloud services. This paper presents a blockchain-based distributed infrastructure that leverages fundamental blockchain properties to achieve immutable and trustworthy SLA monitoring within cloud services. The paper carries out an in-depth empirical investigation for the scalability of the proposed system in order to address the challenge of transparently enforcing real-time monitoring of cloud-hosted services leveraging blockchain technology. This will enable all the stakeholders to enforce accurate execution of SLA without any imprecisions and delays by maintaining an immutable ledger publicly across blockchain network. The experimentation takes into consideration several attributes of blockchain which are critical in achieving optimum performance. The paper also investigates key characteristics of these factors and their impact to the behaviour of the system for further scaling it up under various cases for increased service utilization.

DAVINCI: online and Dynamic Adaptation of eVolvable vIrtual Network services over Cloud Infrastructures

Service function chains, or more generally, virtual network services, evolve throughout their life-cycle due to traffic fluctuations, resource usage, and the stringent requirements of the SLO (Service Level Objective) that must be fulfilled. Network operators resort to elasticity mechanisms (e.g., vertical and horizontal scaling) and migration procedures to adapt the service function chains dynamically to meet their requirements constantly. However, most state-of-the-art solutions do not evaluate the overall impact of such mechanisms on the service function chains or the resulting penalty costs (e.g., migration downtime and SLO violation). To bridge this gap, we propose DAVINCI, a decision-making tool with different adaptation policies that allows the network operator to adapt the service function chains to minimize network changes (e.g., reallocating paths and VNFs) while keeping penalty costs at their lowest level. Moreover, DAVINCI allows for the adaptations (e.g., migration, vertical and horizontal scaling) to be expressed as a set of decisions and leverages the Min Cost Flow problem to estimate migration downtime. The analytical evaluation shows that DAVINCI outperforms the existing state-of-the-art solution (NFV-PEAR) in terms of migration, traffic costs, and the overhead induced by the migration and elasticity mechanisms while significantly reducing penalty costs.

DYVERSE: DYnamic VERtical Scaling in multi-tenant Edge environments

Multi-tenancy in resource-constrained environments is a key challenge in Edge computing. In this paper, we develop ‘DYVERSE: DYnamic VERtical Scaling in Edge’ environments, which is the first light-weight and dynamic vertical scaling mechanism for managing resources allocated to applications for facilitating multi-tenancy in Edge environments. To enable dynamic vertical scaling, one static and three dynamic priority management approaches that are workload-aware, community-aware and system-aware, respectively are proposed. This research advocates that dynamic vertical scaling and priority management approaches reduce Service Level Objective (SLO) violation rates. An online-game and a face detection workload in a Cloud-Edge test-bed are used to validate the research. The merit of DYVERSE is that there is only a sub-second overhead per Edge server when 32 Edge servers are deployed on a single Edge node. When compared to executing applications on the Edge servers without dynamic vertical scaling, static priorities and dynamic priorities reduce SLO violation rates of requests by up to 4% and 12% for the online game, respectively, and in both cases 6% for the face detection workload. Moreover, for both workloads, the system-aware dynamic vertical scaling method effectively reduces the latency of non-violated requests, when compared to other methods.

Reinforcement-Learning-Empowered MLaaS Scheduling for Serving Intelligent Internet of Things

Machine learning (ML) has been embedded in many Internet of Things (IoT) applications (e.g., smart home and autonomous driving). Yet it is often infeasible to deploy ML models on IoT devices due to resource limitation. Thus, deploying trained ML models in the cloud and providing inference services to IoT devices becomes a plausible solution. To provide low-latency ML serving to massive IoT devices, a natural and promising approach is to use parallelism in computation. However, existing ML systems (e.g., Tensorflow) and cloud ML-serving platforms (e.g., SageMaker) are service-level-objective (SLO) agnostic and rely on users to manually configure the parallelism at both request and operation levels. To address this challenge, we propose a region-based reinforcement learning (RRL)-based scheduling framework for ML serving in IoT applications that can efficiently identify optimal configurations under dynamic workloads. A key observation is that the system performance under similar configurations in a region can be accurately estimated by using the system performance under one of these configurations due to their correlation. We theoretically show that the RRL approach can achieve fast convergence speed at the cost of performance loss. To improve the performance, we propose an adaptive RRL algorithm based on Bayesian optimization to balance the convergence speed and the optimality. The proposed framework is prototyped and evaluated on the Tensorflow Serving system. Extensive experimental results show that the proposed approach can outperform state-of-the-art approaches by finding near-optimal solutions over eight times faster while reducing inference latency up to 88.9% and reducing SLO violation up to 91.6%.

Enforcing trustworthy cloud SLA with witnesses: A game theory-based model using smart contracts.

SummaryThere lacks trust between the cloud customer and provider to enforce traditional cloud SLA (Service Level Agreement) where the blockchain technique seems a promising solution. However, current explorations still face challenges to prove that the off‐chain SLO (Service Level Objective) violations really happen before recorded into the on‐chain transactions. In this paper, a witness model is proposed implemented with smart contracts to solve this trust issue. The introduced role, “Witness”, gains rewards as an incentive for performing the SLO violation report, and the payoff function is carefully designed in a way that the witness has to tell the truth, for maximizing the rewards. This fact that the witness has to be honest is analyzed and proved using the Nash Equilibrium principle of game theory. For ensuring the chosen witnesses are random and independent, an unbiased selection algorithm is proposed to avoid possible collusions. An auditing mechanism is also introduced to detect potential malicious witnesses. Specifically, we define three types of malicious behaviors and propose quantitative indicators to audit and detect these behaviors. Moreover, experimental studies based on Ethereum blockchain demonstrate the proposed model is feasible, and indicate that the performance, ie, transaction fee, of each interface follows the design expectations.

Towards efficiently supporting database as a service with QoS guarantees

Quality of Service (QoS) is at the core of the vision of Database as a Service (DBaaS), which provides guarantees to database users on the usability of database services, even when the underlying database infrastructure is shared by multiple users. When QoS guarantees are necessitated, traditional approaches in DBaaS often have to reserve computation resources (e.g. CPU and memory) for tenants according to their performance Service Level Objectives (SLOs), so that the database engine always possesses sufficient resources to accomplish the expected workloads under any circumstance. Such resource reservation schemes inevitably result in poor resource utilization, as the actual workloads of tenants are usually below their maximal workload expectation described in their SLOs.In this paper, we propose a novel scheme called FrugalDB to further improve resource utilization and thus reduce operational cost for DBaaS systems with QoS guarantees. FrugalDB accommodates two independent database engines, an in-memory engine for heavy workloads with tight SLOs, and a disk-based engine for light workloads with loose SLOs. The in-memory database is leveraged to migrate temporary heavy workloads from the disk-based database, when the latter itself does not suffice to handle all active tenants’ workloads, and thus it could be relieved of the tremendous data serving pressure yielded by those heavy workloads, and focus on processing massive numbers of light workloads. When the heavy workloads fade out, FrugalDB reversely migrates a tenant’s workload from the in-memory database into the disk-based database, so that the occupied memory resources could be recollected for subsequent workload migrations. By an effective workload estimation method and an efficient migration schedule algorithm, FrugalDB tries to minimize workload migration cost incurred in moving workloads between the two engines. By allocating each tenant’s workload to an appropriate engine via workload migration, this dual-engine scheme can substantially save computation resources, and thus consolidate more tenants on a single database server. We evaluate FrugalDB with extensive experiments, which show that it has a higher tenant consolidation rate with performance SLO guarantees and fewer performance SLO violations than the existing systems, as well as with acceptable response latency.

OnlineElastMan: self-trained proactive elasticity manager for cloud-based storage services

The pay-as-you-go pricing model and the illusion of unlimited resources in the Cloud initiate the idea to provision services elastically. Elastic provisioning of services allocates/de-allocates resources dynamically in response to the changes of the workload. It minimizes the service provisioning cost while maintaining the desired service level objectives (SLOs). Model-predictive control is often used in building such elasticity controllers that dynamically provision resources. However, they need to be trained, either online or offline, before making accurate scaling decisions. The training process involves tedious and significant amount of work as well as some expertise, especially when the model has many dimensions and the training granularity is fine, which is proved to be essential in order to build an accurate elasticity controller. In this paper, we present OnlineElastMan, which is a self-trained proactive elasticity manager for cloud-based storage services. It automatically evolves itself while serving the workload. Experiments using OnlineElastMan with Cassandra indicate that OnlineElastMan continuously improves its provision accuracy, i.e., minimizing provisioning cost and SLO violations, under various workload patterns.

Aggressive Resource Provisioning for Ensuring QoS in Virtualized Environments

Elasticity has now become the elemental feature of cloud computing as it enables the ability to dynamically add or remove virtual machine instances when workload changes. However, effective virtualized resource management is still one of the most challenging tasks. When the workload of a service increases rapidly, existing approaches cannot respond to the growing performance requirement efficiently because of either inaccuracy of adaptation decisions or the slow process of adjustments, both of which may result in insufficient resource provisioning. As a consequence, the Quality of Service (QoS) of the hosted applications may degrade and the Service Level Objective (SLO) will be thus violated. In this paper, we introduce SPRNT, a novel resource management framework, to ensure high-level QoS in the cloud computing system. SPRNT utilizes an aggressive resource provisioning strategy which encourages SPRNT to substantially increase the resource allocation in each adaptation cycle when workload increases. This strategy first provisions resources which are possibly more than actual demands, and then reduces the over-provisioned resources if needed. By applying the aggressive strategy, SPRNT can satisfy the increasing performance requirement in the first place so that the QoS can be kept at a high level. The experimental results show that SPRNT achieves up to 7.7 $\times$ speedup in adaptation time, compared with existing efforts. By enabling quick adaptation, SPRNT limits the SLO violation rate up to 1.3 percent even when dealing with rapidly increasing workload.

Responding rapidly to service level violations using virtual appliances

One of the key goals in the data center today is providing storage services with service-level objectives (SLOs) for performance metrics such as latency and throughput. Meeting such SLOs is challenging due to the dynamism observed in these environments. In this position paper, we propose dynamic instantiation of virtual appliances, that is, virtual machines with storage functionality, as a mechanism to meet storage SLOs efficiently. In order for dynamic instantiation to be realistic for rapidlychanging environments, it should be automated. Therefore, an important goal of this paper is to show that such automation is feasible. We do so through a caching case study. Specifically, we build the automation framework for dynamically instantiating virtual caching appliances. This framework identifies sets of interfering workloads that can benefit from caching, determines the cache-size requirements of workloads, non-disruptively migrates the application to use the cache, and warms the cache to quickly return to acceptable service levels. We show through an experiment that this approach addresses SLO violations while using resources efficiently.

A feedback control mechanism for adaptive SLO maintenance in dynamic service level management

With the increasing scale of information technology (IT) service system, traditional threshold-based static service level management (SLM) solution appears to be inadequate to meet current increasingly management requirement of SLM. Due to the stochastic service request rate, the random inherent failure and load surge of IT devices during service operating stage of large scaled IT system, service level objective (SLO) maintenance issue has become a realistic and important issue in dynamic SLM. This paper proposes a closed-loop feedback control mechanism to adaptively maintain SLO that service provider (SP) guaranteed at service operation stage. The mechanism can automatically tune the capacity of IT infrastructure according to service performance dispersion and reduce SLO violations. Considering that the tuning operations also affect service performance, fuzzy control is applied to alleviate the negative effect caused by tuning operations. In the dynamic SLM system that is applied with this mechanism compared with the traditional threshold-based solution, it is proved that the amount of SLO violations obviously decreases, the reliability of the service system increases relatively, and the resource utilization of IT infrastructure is optimized.