Quality Of Service Violations Research Articles

Intent-driven orchestration of serverless applications in the computing continuum

Following the emergence of Internet of Things (IoT) and edge computing technologies and the development of distributed applications with strict Quality of Service (QoS) requirements, a transition is underway from centralized computing models to distributed computing continuum systems. The latter provide abstractions of the available resources, while they lead to the development of orchestration mechanisms with increased distributed intelligence and autonomy characteristics. These characteristics make the computing continuum suitable for managing serverless computing applications, considering their loose coupling with the resources and the need for reactiveness and efficiency to serve dynamic workloads in the edge and cloud part of the infrastructure. This paper presents an intent-based approach for the orchestration of a function chain over an edge and cloud infrastructure. We propose a Reinforcement Learning (RL) based autoscaling method that incorporates a reward function that aims to provide optimal utilization of resources and timely autoscaling of the functions, taking advantage of methods for workload prediction. Based on a high-level intent, the proposed optimization problem dictates the scheduling of the functions, minimizing the communication overhead and transformation cost while also considering the energy efficiency of the infrastructure. The proposed solution is evaluated with other state-of-the-art techniques for autoscaling and scheduling of serverless functions in a small edge infrastructure. Our solution provides 1.52% QoS violations with a slight increase in the deployed resources, while also significantly reducing the total cost and power consumption for the deployment of the function chain.

NPU-Accelerated Imitation Learning for Thermal Optimization of QoS-Constrained Heterogeneous Multi-Cores

Thermal optimization of a heterogeneous clustered multi-core processor under user-defined QoS targets requires application migration and DVFS. However, selecting the core to execute each application and the VF levels of each cluster is a complex problem because (1) the diverse characteristics and QoS targets of applications require different optimizations, and (2) per-cluster DVFS requires a global optimization considering all running applications. State-of-the-art resource management for power or temperature minimization either relies on measurements that are commonly not available (such as power) or fails to consider all the dimensions of the optimization (e.g., by using simplified analytical models). To solve this, ML methods can be employed. In particular, IL leverages the optimality of an oracle policy, yet at low run-time overhead, by training a model from oracle demonstrations. We are the first to employ IL for temperature minimization under QoS targets. We tackle the complexity by training NN at design time and accelerate the run-time NN inference using NPU. While such NN accelerators are becoming increasingly widespread, they are so far only used to accelerate user applications. In contrast, we use for the first time an existing accelerator on a real platform to accelerate NN-based resource management. To show the superiority of IL compared to RL in our targeted problem, we also develop multi-agent RL-based management. Our evaluation on a HiKey 970 board with an Arm big.LITTLE CPU and NPU shows that IL achieves significant temperature reductions at a negligible run-time overhead. We compare TOP-IL against several techniques. Compared to ondemand Linux governor, TOP-IL reduces the average temperature by up to 17 ˆC at minimal QoS violations for both techniques. Compared to the RL policy, our TOP-IL achieves 63 % to 89 % fewer QoS violations while resulting similar average temperatures. Moreover, TOP-IL outperforms the RL policy in terms of stability. We additionally show that our IL-based technique also generalizes to different software (unseen applications) and even hardware (different cooling) than used for training.

RAaaS: Resource Allocation as a Service in multiple cloud providers

Cloud users require a set of specific computing needs for their applications, while cloud providers offer a variety of computing products and services on the Internet. These two cloud players make deals through the use of service level agreements (SLAs) where, for instance, prices and levels of quality of service (QoS) are defined. From the cloud user’s point of view, building a robust set of SLAs becomes a challenging problem when multiple cloud providers are present in the market. The allocation of resources in the cloud to run complex applications with guaranteed reliable, secure and acceptable response times is not an easy task, and this paper aims to tackle this problem. This work describes a resource allocation service that aims to optimize the user’s request of cloud resources (virtual machines — VMs) onto multiple Infrastructure-as-a-Service (IaaS) cloud providers. The Resource-Allocation-as-a-Service (RAaaS) proposed in this paper works as a standalone service between cloud users and cloud providers, and it relies on three different requirements: reliability, processing, and mutual trust. The proposed resource allocation service is carried out using the three very common types of VM billing models: on-demand, reserved and spot, where the spot cost model is employed to furnish low-cost resources for the application allocation to improve its reliability. The contributions of this paper are threefold: (i) a three-dimension SLA encompassing reliability, processing, and trust; (ii) an integer linear program (ILP) to schedule cloud-based VMs to applications considering the three-dimension SLA model, and (iii) a heuristic algorithm to mitigate possible QoS violations. Experimental results show that the proposed RAaaS procedure is capable of optimizing resource allocation considering multiple criteria in the SLA while mitigating the extra costs introduced by mutual trust between customers using redundant spot instances allocation.

Multi-Agent Deep Reinforcement Learning Based Resource Allocation for Ultra-Reliable Low-Latency Internet of Controllable Things

As a promising technology in the 5G era, the artificial intelligence (AI) enabled Internet of controllable things (IoCT) is expected to be an integral part of heterogeneous networks (HetNets) in the future. However, the realization of ultra-reliable low-latency communications (URLLC) in IoCT communications underlaid HetNet has stringent quality of service (QoS) requirements, resulting in unprecedented challenges for existing wireless resource allocation methods. In this paper, we first describe a cellular HetNet model with uplink IoCT communications, then formulate a dynamic mixed-integer nonlinear programming (MINLP) resource allocation problem for maximizing the long-term average energy efficiency under URLLC requirements including reliability, latency, and transmission rate. To solve the problem, we propose a decentralized MADRL-based resource allocation algorithm with a decentralized partially observable Markov decision process (dec-POMDP) and a mixed-centralized-decentralized (MCD) framework to address the partial observability and the scalability issues, respectively. In addition, we design a reward function featuring the objective decomposition, baseline-guided scaling, and QoS violation penalty so that the agents are coordinated. Extensive experiments demonstrate the convergence, scalability, and robustness of the proposed algorithm. Besides, the proposed algorithm substantially outperforms conventional resource allocation methods and different agent communication mechanisms in terms of maximizing energy efficiency.

MagicScaler: Uncertainty-Aware, Predictive Autoscaling

Predictive autoscaling is a key enabler for optimizing cloud resource allocation in Alibaba Cloud's computing platforms, which dynamically adjust the Elastic Compute Service (ECS) instances based on predicted user demands to ensure Quality of Service (QoS). However, user demands in the cloud are often highly complex, with high uncertainty and scale-sensitive temporal dependencies, thus posing great challenges for accurate prediction of future demands. These in turn make autoscaling challenging---autoscaling needs to properly account for demand uncertainty while maintaining a reasonable trade-off between two contradictory factors, i.e., low instance running costs vs. low QoS violation risks. To address the above challenges, we propose a novel predictive autoscaling framework MagicScaler , consisting of a Multi-scale attentive Gaussian process based predictor and an uncertainty-aware scaler. First, the predictor carefully bridges the best of two successful prediction methodologies---multi-scale attention mechanisms, which are good at capturing complex, multi-scale features, and stochastic process regression, which can quantify prediction uncertainty, thus achieving accurate demand prediction with quantified uncertainty. Second, the scaler takes the quantified future demand uncertainty into a judiciously designed loss function with stochastic constraints, enabling flexible trade-off between running costs and QoS violation risks. Extensive experiments on three clusters of Alibaba Cloud in different Chinese cities demonstrate the effectiveness and efficiency of MagicScaler , which outperforms other commonly adopted scalers, thus justifying our design choices.

Energy-Efficient and QoS-Aware Computation Offloading in GEO/LEO Hybrid Satellite Networks

Benefiting from advanced satellite payload technologies, edge computing servers can be deployed on satellites to achieve orbital computing and reduce the mission processing delay. However, geostationary Earth orbit (GEO) satellites are hindered by long-distance communication, whereas low Earth orbit (LEO) satellites are restricted by time windows. Relying solely on GEO or LEO satellites cannot meet the strict quality of service (QoS) requirements of on-board missions while conserving energy consumption. In this paper, we propose a computation offloading strategy for GEO/LEO hybrid satellite networks that minimizes total energy consumption while guaranteeing the QoS requirements of multiple missions. We first innovatively transform the on-board partial computation offloading problem, which is a mixed-integer nonlinear programming (MINLP) problem, into a minimum cost maximum flow (MCMF) problem. Then, the successive shortest path-based computation offloading (SSPCO) method is introduced to obtain the offloading decision in polynomial time. To evaluate the effectiveness and performance of SSPCO, we conduct a series of numerical experiments and compare SSPCO with other offloading methods. The experimental results demonstrate that our proposed SSPCO outperforms the reference methods in terms of total energy consumption, QoS violation degree, and algorithm running time.

Toward Optimal Load Prediction and Customizable Autoscaling Scheme for Kubernetes

Most enterprise customers now choose to divide a large monolithic service into large numbers of loosely-coupled, specialized microservices, which can be developed and deployed separately. Docker, as a light-weight virtualization technology, has been widely adopted to support diverse microservices. At the moment, Kubernetes is a portable, extensible, and open-source orchestration platform for managing these containerized microservice applications. To adapt to frequently changing user requests, it offers an automated scaling method, Horizontal Pod Autoscaler (HPA), that can scale itself based on the system’s current workload. The native reactive auto-scaling method, however, is unable to foresee the system workload scenario in the future to complete proactive scaling, leading to QoS (quality of service) violations, long tail latency, and insufficient server resource usage. In this paper, we suggest a new proactive scaling scheme based on deep learning approaches to make up for HPA’s inadequacies as the default autoscaler in Kubernetes. After meticulous experimental evaluation and comparative analysis, we use the Gated Recurrent Unit (GRU) model with higher prediction accuracy and efficiency as the prediction model, supplemented by a stability window mechanism to improve the accuracy and stability of the prediction model. Finally, with the third-party custom autoscaling framework, Custom Pod Autoscaler (CPA), we packaged our custom autoscaling algorithm into a framework and deployed the framework into the real Kubernetes cluster. Comprehensive experiment results prove the feasibility of our autoscaling scheme, which significantly outperforms the existing Horizontal Pod Autoscaler (HPA) approach.

An adaptive auto-scaling framework for cloud resource provisioning

Cloud computing emerged as a technology that offers scalable access to computing resources in conjunction with low maintenance costs. In this domain, cloud users utilize virtualized resources to benefit from on-demand and long-term pricing strategies. Although the latter consists of a more cost-efficient solution, it requires accurate estimations of future workload demands, which is a challenging task. Furthermore, clouds offer threshold-based auto-scaling rules that need to be manually controlled by the users according to application requirements. Still, tuning scaling parameters is not trivial, since it is mainly based on static scaling rules that may lead to unreasonable costs and quality of service violations. In this work we introduce ADA-RP, an adaptive auto-scaling framework for reliable resource provisioning in the cloud. ADA-RP uses historical time series data for training K-means and convolutional neural networks (CNN) to categorize future workload demands as High, Medium or Low based on CPU utilization. We auto-scale cloud resources in real-time based on the predicted workload demand to reduce costs and improve application performance. The experimental analysis is based on TPC-C runs on MySQL containers deployed on the Google Cloud Platform. Experimental results are prosperous, demonstrating the ability of ADA-RP (i) to reduce MySQL deployment costs by 48% in a single-tenant environment, and (ii) to double the executed queries per second in a multi-tenant environment considering user’s budget requirements.

A cost‐efficient and QoS‐aware adaptive placement of applications in fog computing

SummaryWith the rapid development of the Internet of Things (IoT), fog computing has emerged as a complementary solution to address the issues faced in cloud computing. However, it is a challenging issue to ensure both of the high Quality of Service (QoS) and low cost for different requests when dealing with computing resources. In this article, we propose a new approach on adaptive cost‐efficient and QoS‐aware application placement for fog computing called DATSS. Specifically, this approach consists of a QoS state driven strategy and a credibility rating mechanism. The algorithm divides the QoS into three different states by calculating the QoS violation rate and adaptively selects the best scheduling strategy. Then evaluates the stability of the node based on the task history, and prioritize the task on the node with a higher rating. Performance results show that the cost, delay and energy consumption of the proposed algorithm can be reduced by 20.9%, 15.7%, and 12.6% respectively compared with other algorithms at most. The proposed algorithm can effectively improve cost efficiency and stability under QoS constraints in fog computing environments.

PROMPT: Learning dynamic resource allocation policies for network applications

A growing number of service providers are exploring methods to improve server utilization and reduce power consumption by co-scheduling high-priority latency-critical workloads with best-effort workloads. This practice requires strict resource allocation between workloads to reduce contention and maintain Quality-of-Service (QoS) guarantees. Prior work demonstrated promising opportunities to dynamically allocate resources based on workload demand, but may fail to meet QoS objectives in more stringent operating environments due to the presence of resource allocation cliffs, transient fluctuations in workload performance, and rapidly changing resource demand. We therefore propose PROMPT, a novel resource allocation framework using proactive QoS prediction to guide a reinforcement learning controller. PROMPT enables more precise resource optimization, more consistent handling of transient behaviors, and more robust generalization when co-scheduling new best-effort workloads not encountered during policy training. Evaluation shows that the proposed method incurs 4.2x fewer QoS violations, reduces severity of QoS violations by 12.7x, improves best-effort workload performance, and improves overall power efficiency over prior work.

Game Theoretic Approach for Multipriority Data Transmission in 5G Vehicular Networks

The vehicle-to-vehicle (V2V) communication driven by the fifth generation (5G) cellular mobile network with the features of ultra-high reliability and low latency provides promising solutions to various applications in the intelligent transportation system (ITS). To improve the resource utilization and guarantee the quality-of-service (QoS), users in 5G vehicular networks have to select appropriate communication modes and control their own transmission power. However, the highly dynamic network topology and channel status pose challenges to the mode selection. In this paper, we propose a scheme for joint mode selection and power adaptation based on the game theoretic approach with the objective of maximizing the overall system throughput. We consider the transmission requirements of multi-priority packets of different vehicular applications, where packets with higher priority have more stringent latency constraints. The segmented auction method with reserve price is performed to select modes for the vehicular users (VUEs) and the Stackelberg gaming model is introduced to solve the problem of cochannel interference. We compare our approach with three existing methods in extensive simulations. The results show that our approach outperforms the existing methods in terms of network performance, including the network throughput, resource utilization and QoS violation rate.

SCOPE: Cost-Efficient QoS-Aware Switch and Controller Placement in Hybrid SDN

This article addresses the joint switch and controller placement problem in a hybrid software-defined network (SDN), which is an intermediate step of transforming a traditional backbone network into a pure SDN. For a hybrid SDN, the quality of service (QoS) is a primary concern for ensuring service guarantee of a traditional network, while providing additional benefits of softwarization. In the SDN, the positions of SDN controllers determine the QoS parameters, such as network throughput and flow processing delays. Many existing solutions determine the controller locations to optimize the latency of network links and control plane load. However, there is a need to address the problem considering additional aspects of the hybrid SDN, such as SDN switch placement and incremental upgrades. In this article, we propose a priority-based SDN switch placement approach and an incremental controller placement approach for the hybrid SDN. Simulation results depict that the proposed scheme increases programmable traffic by 9.90% compared to the existing MapFirst scheme when the switch upgrade budget is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">${\$}$</tex-math></inline-formula> 400 K. The proposed scheme decreases QoS violation by 20.89% compared to the Greedy approach with 0.3 million flows.

Adaptive Resource Efficient Microservice Deployment in Cloud-Edge Continuum

User-facing services are now evolving towards the microservice architecture where a service is built by connecting multiple microservice stages. Since the entire service is heavy, the microservice architecture shows the opportunity to only offload some microservice stages to the edge devices that are close to the end users. However, emerging techniques often result in the violation of Quality-of-Service (QoS) of microservice-based services in cloud-edge continuum, as they do not consider the communication overhead or the resource contention between microservices and external co-located tasks. We propose Nautilus, a runtime system that effectively deploys microservice-based user-facing services in cloud-edge continuum. Nautilus ensures the QoS of microservice-based user-facing services while minimizing the required computational resources, which is comprised of a communication-aware microservice mapper, a contention-aware resource manager and an IO-sensitive and load-aware microservice migration scheduler. The mapper divides the microservice graph into multiple partitions based on the communication overhead and maps the partitions to appropriate nodes. On each node, the resource manager determines the optimal resource allocation for its microservices based on reinforcement learning that may capture the complex contention behaviors. Once the microservices are suffered from external IO pressure, the IO-sensitive microservice scheduler migrates the critical one to idle nodes. Furthermore, when the load of microservices changes dynamically, the load-aware microservice scheduler migrates microservices from busy nodes to idle ones to ensure the QoS goal of the entire service. Our experimental results show that Nautilus can guarantee the required QoS target under external shared resources contention while the state-of-the-art suffers from QoS violations. Meanwhile, Nautilus reduces the computational resource usage by 23.9% and the network bandwidth usage by 53.4%, while achieving the required 99%-ile latency.

QoS-aware dynamic resource allocation with improved utilization and energy efficiency on GPU

Although GPUs have been indispensable in data centers, meeting the Quality of Service (QoS) under task consolidation on GPU is extremely challenging. Previous works mostly rely on the static task or resource scheduling and cannot handle the QoS violation during runtime. In addition, existing works fail to exploit the computing characteristics of batch tasks, and thus waste the opportunities to reduce power consumption while improving GPU utilization. To address the above problems, we propose a new runtime mechanism SMQoS that can dynamically adjust the resource allocation during runtime to meet the QoS of latency-sensitive (LS) tasks and determine the optimal resource allocation for batch tasks to improve GPU utilization and power efficiency. We implement the proposed mechanism on both simulator (SMQoS) and real GPU hardware (RH-SMQoS). The experimental results show that both SMQoS and RH-SMQoS can achieve better QoS for LS tasks and higher throughput for batch tasks compared to the state-of-the-art works. With hardware extension, the SMQoS can further reduce the power consumption by power gating idle computing resources.

A Novel Deep-Learning-Enabled QoS Management Scheme for Encrypted Traffic in Software-Defined Cellular Networks

Mobile users are served with over-the-top (OTT) services through their cellular networks. To ensure the privacy of users and confidentiality of content, most OTT service providers encrypt their traffic. When a cellular network has no information about the type of service, a default bearer may be created. However, the default bearer may not guarantee bandwidth to a service. Therefore, users may experience degraded service due to packet loss, delay, and reduced data rates. This article proposes a novel quality-of-service (QoS) management scheme for encrypted traffic in software-defined cellular networks. We introduce a deep-learning-enabled intelligent gateway to predict the service types of encrypted flows by considering statistical and QoS features. A QoS control manager maps the bearers to ongoing flows satisfying their QoS requirements. As a proof of concept, we implement a testbed considering encrypted traffic from the Tor network. Results indicate that the proposed scheme improves the network throughput by 41&#x0025;, decreases packet loss, delay, and QoS violations by 51&#x0025;, 21&#x0025;, and 52&#x0025;, respectively, and reduces the length and size of the queue at the base station compared to those of the conventional scheme. Moreover, the convolutional-neural-network-based classifier achieves higher accuracy, precision, recall, and <inline-formula><tex-math notation="LaTeX">$F1$</tex-math></inline-formula>-score, as well as lower loss values, compared to the multilayer perceptron classifier.

ReSQoV: A Scalable Resource Allocation Model for QoS-Satisfied Cloud Services

Dynamic resource provisioning is made more accessible with cloud computing. Monitoring a running service is critical, and modifications are performed when specific criteria are exceeded. It is a standard practice to add or delete resources in such situations. We investigate the method to ensure the Quality of Service (QoS), estimate the required resources, and modify allotted resources depending on workload, serialization, and parallelism due to resources. This article focuses on cloud QoS violation remediation using resource planning and scaling. A Resource Quantified Scaling for QoS Violation (ReSQoV) model is proposed based on the Universal Scalability Law (USL), which provides cloud service capacity for specific workloads and generates a capacity model. ReSQoV considers the system overheads while allocating resources to maintain the agreed QoS. As the QoS violation detection decision is Probably Violation and Definitely Violation, the remedial action is triggered, and required resources are added to the virtual machine as vertical scaling. The scenarios emulate QoS parameters and their respective resource utilization for ReSQoV compared to policy-based resource allocation. The results show that after USLbased Quantified resource allocation, QoS is regained, and validation of the ReSQoV is performed through the statistical test ANOVA that shows the significant difference before and after implementation.

Uplink Channel Allocation Scheme and QoS Management Mechanism for Cognitive Cellular-Femtocell Networks

Cognitive radio and femtocell are promising technologies which can satisfy the requirements of future mobile communications in terms of dynamic spectrum sharing and high user density areas. Providing quality-of-service (QoS) guaranteed realtime services is challenging issue of future cognitive cellular-femtocell mobile networks. In this paper, we introduce a user’s QoS management mechanism used to protect SINR of macro users from QoS violation caused by femtocell users. We design a novel uplink channel allocation scheme (denoted as “flexible scheme”) for real-time connections. The scheme uses the information of interference level and channel occupancy collected at cognitive femtocell access points and their covering macro base station (MBS) and apply relevant selection criteria to select an appropriate channel which causes the minimums interference to macro users of the covering MBS. Performance results prove that comparing with femtocell-access-point (FAP)-based and MBS-based uplink channel allocation schemes, the novel “flexible scheme” can provide lower unsuccessful probability of new connection requests.

Minimizing the monetary penalty and energy cost of server migration service

AbstractCurrent IaaS (infrastructure as a service) cloud service may not satisfy communication QoS (quality of service) requirements of delay‐sensitive network applications, if there is a significant physical distance between a server of the network application (NetApp server) at a data center and its network application clients (NetApp clients). In order to improve communication QoS of NetApp clients, we propose server migration service (SMS) in this article. SMS allows NetApp servers to migrate among different locations in the network (1) to optimally locate themselves in relation to NetApp clients and mitigate the QoS degradation caused by location‐related factors (ie, propagation delays on network links) and (2) to optimally distribute traffic load over routers and processing load over (physical) computers and decrease the energy consumption. We develop a mixed‐integer programming model that determines when and to which locations NetApp servers migrate to minimize the total operating cost of SMS, that is, the sum of the monetary penalty incurred due to QoS violation and energy cost incurred due to energy consumption, while preventing NetApp servers from excessively migrating and adversely impacting QoS of the non‐SMS service that share the resource of the substrate network with SMS. Simulation results show that the model developed in this article achieves up to 42% lower total operating cost of SMS compared to the model that only minimizes the monetary penalty of SMS without considering the energy cost of SMS.

Layerweaver+: A QoS-Aware Layer-Wise DNN Scheduler for Multi-Tenant Neural Processing Units

Many cloud service providers employ specialized hardware accelerators, called neural processing units (NPUs), to accelerate deep neural networks (DNNs). An NPU scheduler is responsible for scheduling incoming user requests and required to satisfy the two, often conflicting, optimization goals: maximizing system throughput and satisfying quality-of-service (QoS) constraints (e.g., deadlines) of individual requests. We propose Layerweaver+, a low-cost layer-wise DNN scheduler for NPUs, which provides both high system throughput and minimal QoS violations. For a serving scenario based on the industry-standard MLPerf inference benchmark, Layerweaver+ significantly improves the system throughput by up to 266.7% over the baseline scheduler serving one DNN at a time.

Intelligent Model for Predicting the Quality of Services Violation

Cloud computing is providing IT services to its customer based on Service level agreements (SLAs). It is important for cloud service providers to provide reliable Quality of service (QoS) and to maintain SLAs accountability. Cloud service providers need to predict possible service violations before the emergence of an issue to perform remedial actions for it. Cloud users’ major concerns; the factors for service reliability are based on response time, accessibility, availability, and speed. In this paper, we, therefore, experiment with the parallel mutant-Particle swarm optimization (PSO) for the detection and predictions of QoS violations in terms of response time, speed, accessibility, and availability. This paper also compares Simple-PSO and Parallel Mutant-PSO. In simulation results, it is observed that the proposed Parallel Mutant-PSO solution for cloud QoS violation prediction achieves 94% accuracy which is many accurate results and is computationally the fastest technique in comparison of conventional PSO technique.