Job Response Time Research Articles

Strongly Tail-Optimal Scheduling in the Light-Tailed M/G/1

We study the problem of scheduling jobs in a queueing system, specifically an M/G/1 with light-tailed job sizes, to asymptotically optimize the response time tail. This means scheduling to make \mathbfP [T > t], the chance a job's response time exceeds t, decay as quickly as possible in the t → ∞ limit. For some time, the best known policy was First-Come First-Served (FCFS), which has an asymptotically exponential tail: P [T > t] ∼ C e -γ t . FCFS achieves the optimal decay rate ~γ, but its tail constant C is suboptimal. We derive a closed-form expression for the optimal tail constant, and we introduce γ-Boost, a new policy that achieves this optimal tail constant. We also show via simulation that γ-Boost has excellent practical performance. This abstract summarizes our full paper.

Strongly Tail-Optimal Scheduling in the Light-Tailed M/G/1

We study the problem of scheduling jobs in a queueing system, specifically an M/G/1 with light-tailed job sizes, to asymptotically optimize the response time tail. This means scheduling to make \mathbfP [T > t], the chance a job's response time exceeds t, decay as quickly as possible in the t \to \infty limit. For some time, the best known policy was First-Come First-Served (FCFS), which has an asymptotically exponential tail: \mathbfP [T > t] \sim C e^-γ t . FCFS achieves the optimal *decay rate* γ, but its *tail constant* C is suboptimal. Only recently have policies that improve upon FCFS's tail constant been discovered. But it is unknown what the optimal tail constant is, let alone what policy might achieve it. In this paper, we derive a closed-form expression for the optimal tail constant C, and we introduce *γ-Boost*, a new policy that achieves this optimal tail constant. Roughly speaking, γ-Boost operates similarly to FCFS, but it pretends that small jobs arrive earlier than their true arrival times. This significantly reduces the response time of small jobs without unduly delaying large jobs, improving upon FCFS's tail constant by up to 50% with only moderate job size variability, with even larger improvements for higher variability. While these results are for systems with full job size information, we also introduce and analyze a version of γ-Boost that works in settings with partial job size information, showing it too achieves significant gains over FCFS. Finally, we show via simulation that γ-Boost has excellent practical performance.

Scheduling critical periodic jobs with selective partial computations along with gang jobs

One of the main issues with distributed systems, like clouds, is scheduling complex workloads, which are made up of various job types with distinct features. Gang jobs are one kind of parallel applications that these systems support. This paper examines the scheduling of workloads that comprise gangs and critical periodic jobs that can allow for partial computations when necessary to overcome gang job execution. The simulation's results shed important light on how gang performance is impacted by partial computations of critical jobs. The results also reveal that, under the proposed scheduling scheme, partial computations which take into account gangs’ degree of parallelism, might lower the average response time of gang jobs, resulting in an acceptable level of the average results precision of the critical jobs. Additionally, it is observed that as the deviation from the average partial computation increases, the performance improvement due to partial computations increases with the aforementioned tradeoff remaining significant.

Optimal configurations of Minimally Intelligent additive manufacturing machines for Makerspace production environments

Abstract Additive manufacturing (AM) has transformed job shop production and catalysed the growth of Makerspaces, FabLabs, Hackspaces, and Repair Cafés. AM has enabled the handling and manufacturing of a wide variety of components, and its accessibility has enabled more individuals to make. While smaller than their production-scale counterparts, the objectives of minimizing technician overhead, capital expenditure, and job response time remain the same. The typical First-Come First-Serve (FCFS) operating model, while functional, is not necessarily the most efficient and makes responding to a-typical or urgent demand profiles difficult. This article reports a study that investigated how AM machines configured with Minimally Intelligent agents can support production in these environments. An agent-based model that simulated 5, 10, 15, and 20 AM machines operating a 9 am−5 pm pattern and experiencing a diverse non-repeating demand profile was developed. Machines were configured with minimal intelligence – FCFS, First-Response First-Serve (FRFS), Longest Print Time (LPT), Shortest Print Time (SPT), and Random Selection logics – that governed the selection of jobs from the job pool. A full factorial simulation totaling 15,629 configurations was run until convergence to a ranked list of production performance – min Job Time-in-System. Performance changed as much as 200%. Performant configurations featured a variety of logics, while the least performant were dominated by FCFS and LPT. All FCFS (a proxy for today’s operations) was one of the least performant configurations. The results provide an optimal set of logics and performance bands that can be used to justify capital expenditure and AM operations in Makerspaces.

Constraint-Aware Federated Scheduling for Data Center Workloads

The use of data centers is ubiquitous, as they support multiple technologies across domains for storing, processing, and disseminating data. IoT applications utilize both cloud data centers and edge data centers based on the nature of the workload. Due to the stringent latency requirements of IoT applications, the workloads are run on hardware accelerators such as FPGAs and GPUs for faster execution. The introduction of such hardware alongside existing variations in the hardware and software configurations of the machines in the data center, increases the heterogeneity of the infrastructure. Optimal job performance necessitates the satisfaction of task placement constraints. This is accomplished through constraint-aware scheduling, where tasks are scheduled on worker nodes with appropriate machine configurations. The presence of placement constraints limits the number of suitable resources available to run a task, leading to queuing delays. As federated schedulers have gained prominence for their speed and scalability, we assess the performance of two such schedulers, Megha and Pigeon, within a constraint-aware context. We extend our previous work on Megha by comparing its performance with a constraint-aware version of the state-of-the-art federated scheduler Pigeon, PigeonC. The results of our experiments with synthetic and real-world cluster traces show that Megha reduces the 99th percentile of job response time delays by a factor of 10 when compared to PigeonC. We also describe enhancements made to Megha’s architecture to improve its scheduling efficiency.

Generic Edge Computing System for Optimization and Computation Offloading of Unmanned Aerial Vehicle

Several Unmanned Aerial Vehicles (UAVs) offer ground equipment low-latency edge computing services. On the other hand, low-Earth orbit satellites offer all-encompassing cloud computing services. The officially stated joint optimization problem is a mixed nonlinear programming problem, leading to the development of a two-layer optimization technique. It is suggested that a particle swarm optimization (PSO) algorithm can be used in the upper layer of the algorithm along with genetic algorithm (GA) operators to optimize the UAV's deployment position. The greedy algorithm is applied to optimize the offloading of processing jobs in the lowest layer of the algorithm. The feasibility and effectiveness of the proposed strategy have been validated by numerous numerical simulation tests. The findings show that it is similar to other benchmark algorithms. The proposed approach can also reduce the usual job response time of the system.

Cloud Configuration Optimization for Recurring Batch-Processing Applications

Recognizing the diversity of Big Data analytic jobs, cloud providers offer a wide range of VM instance types or even clusters to cater for different use cases. The choice of cloud configurations can have a significant impact on the response time and running cost of batch-processing applications, which may need to be re-run regularly with cloud-scale resources. However, identifying the best cloud configuration with a low search cost is quite challenging due to i) the large and high-dimensional configuration space, ii) the time-varying cloud service cost (e.g., AWS Spot instances), and iii) job response time variation even given the same configuration. To tackle these challenges, we design and implement <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Accordia</i> , a system that enables Adaptive Cloud Configuration Optimization for Recurring Data-Intensive Applications. By leveraging recent algorithmic advances in Gaussian Process UCB techniques, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Accordia</i> can unearth the cost-optimal configuration with a deadline constraint (i.e., maximum tolerated running time) under the time-varying cloud service cost. More importantly, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Accordia</i> manages to achieve a theoretical performance guarantee, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">sub-linearly increasing dynamic regret</i> of the job completion cost. Using extensive trace-driven simulations and empirical measurements of our Kubernetes-based implementation, we demonstrate that <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Accordia</i> can identify a near-cost-optimal configuration (i.e., within 10% of the optimum) after fewer than 20 runs from over 7000 candidate choices, which translates to a 2X-speedup and up to 17.9% cost-savings, when comparing to the state-of-the-art approach, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">CherryPick</i> .

Online Approximation Scheme for Scheduling Heterogeneous Utility Jobs in Edge Computing

Edge computing systems typically handle a wide variety of applications that exhibit diverse degrees of sensitivity to job latency. Therefore, a multitude of utility functions of the job response time need to be considered by the underlying job dispatching and scheduling mechanism. Nonetheless, previous studies in edge computing mainly focused on optimizing a single utility function across all jobs, e.g., linear, sigmoid, or the hard deadline. In this paper, we design online job dispatching and scheduling strategies in which different jobs can be categorized by different non-increasing utility functions. Our goal is to maximize the total utility of all scheduled jobs. We first prove that no online deterministic algorithm could achieve a competitive ratio better than the lower bound <inline-formula> <tex-math notation="LaTeX">$\Omega(\frac{1}{\sqrt{\epsilon}})$</tex-math> </inline-formula> under the <inline-formula> <tex-math notation="LaTeX">$(1+\epsilon)$</tex-math> </inline-formula>-speed augmentation model. We proceed to propose an online algorithm, named as, for handling jobs with heterogeneous utilities. We prove that is <inline-formula> <tex-math notation="LaTeX">$O(\frac{1}{\epsilon^2})$</tex-math> </inline-formula>-competitive. We also design its distributed version, i.e., . We implement and on an edge computing testbed running deep learning inference jobs. With the production trace from Google Cluster, our experimental and large-scale simulation results indicate that can increase the total utility by up to <inline-formula> <tex-math notation="LaTeX">$50\%$</tex-math> </inline-formula> compared with state-of-the-art methods. Besides, the performance loss of is only <inline-formula> <tex-math notation="LaTeX">$2\%$</tex-math> </inline-formula> compared with with a small communication overhead involved. Moreover, both of our algorithms are robust to estimation errors in job processing time and transmission delay.

Deep Adversarial Imitation Reinforcement Learning for QoS-Aware Cloud Job Scheduling

Although cloud computing is one of the promising technologies for online business services, how to schedule real-time cloud jobs with high quality of service (QoS) is still challenging current techniques. With the advancing of machine learning, deep reinforcement learning (DRL) has demonstrated its outstanding capability in dispatching time-sensitive tasks. However, the reinforced rewards in DRL are typically unavailable until the completion of the scheduling for all the jobs. Considering the fact that the trajectory of jobs in cloud is always long, current DRL-based solutions will meet challenges in finding the trajectories with high rewards, and thus would have issues such that the finally trained scheduling policy is suboptimal. To improve the problem, in this article, we propose a more advanced approach called a deep adversarial imitation reinforcement learning (AIRL) framework for scheduling time-sensitive cloud jobs. Specifically, we focus on scheduling user requests in a way to maximize job successful rate along with a significant reduction on job response time. We present the detailed design of our method, and our experimental results demonstrate that AIRL can generally outperform the existing cloud job scheduling approaches, including the DRL-based method, in the presence of different real-time workload and computing resource configurations.

Pricing-based resource allocation in three-tier edge computing for social welfare maximization

Edge computing is a promising computing paradigm for Internet of Everything and AI-driven applications where substantial computing resources are pushed to the edge of the network in close proximity to the end users. Unlike most of the existing works concentrating on system-side metrics such as job response time, we study how the entities in edge computing interact with each other. Specifically, we study a three-tier edge computing market that consists of edge servers, brokers, and edge users, where brokers are introduced to connect edge servers and edge users, and to facilitate resource deployment and maintenance for edge users. Our goal is to maximize social welfare. The uniqueness of this market, such as the agents’ private information and selfishness, prevents one from using standard optimization techniques. Therefore, we propose a pricing-based resource allocation mechanism via iterative bidding, called MECM, for the three-tier edge computing market. Our theoretical results show that MECM converges to the social optimum with a provable convergence rate of O(1k), where k is the number of iterations, and has desirable properties, i.e., budget balance and individual rationality. Our extensive simulations validate MECM’s performance and its properties in various scenarios.

Niyama: Node scheduling for cloud workloads with resource isolation

SummaryCloud providers place tasks from multiple applications on the same resource pool to improve the resource utilization of the infrastructure. The consequent resource contention has an undesirable effect on latency‐sensitive tasks. In this article, we present Niyama—a resource isolation approach that uses a modified version of deadline scheduling to protect latency‐sensitive tasks from CPU bandwidth contention. Conventionally, deadline scheduling has been used to schedule real‐time tasks with well‐defined deadlines. Therefore, it cannot be used directly when the deadlines are unspecified. In Niyama, we estimate deadlines in intervals and secure bandwidth required for the interval, thereby ensuring optimal job response times. We compare our approach with cgroups: Linux's default resource isolation mechanism used in containers today. Our experiments show that Niyama reduces the average delay in tasks by 3–20 when compared to cgroups. Since Linux's deadline scheduling policy is work‐conserving in nature, there is a small drop in the server‐level CPU utilization when Niyama is used naively. We demonstrate how the use of core reservation and oversubscription in the inter‐node scheduler can be used to offset this drop; our experiments show a 1.3–2.24 decrease in delay in job response time over cgroups while achieving high CPU utilization.

Efficient Scheduling of Jobs and Allocation of Resources in Cloud Computing

Due to the drastic utilization of clouds, a Proper and proficient allocation of resources in dynamically working environment of cloud systems turns into the challenging task. Different promising mechanisms have been created to work on the effectiveness of process of resource allocation. Yet at the same time there is some incompetency as far as resource allocation and job scheduling, when the systems become highly loaded. Hence, an effective algorithm for scheduling of jobs is needed to work on the proficiency of resource allocation activities. In this paper a advanced technique for scheduling of jobs is proposed for effective and unique process of allocation of resources in cloud computing. By making use of prediction-based techniques and mechanism of updating resource tables in dynamic manner, we achieve, better allocation of resources in the form of response time and completion of jobs. The experimental results demonstrate the effective outcomes compared to existing techniques, by achieving exactness in values for resource table updation.

Distributed Job Dispatching in Edge Computing Networks With Random Transmission Latency: A Low-Complexity POMDP Approach

Job dispatching is a fundamental problem in edge computing for load balancing among multiple edge servers. When implementing an edge computing system with distributed job dispatchers in a sizable network, such as a metropolitan area network (MAN), the highly dynamic transmission latency is nonnegligible, which could lead to outdated information being shared. Moreover, the fully observed system state is beyond reach as the reception of any broadcast is time consuming. In this article, we investigate the online distributed job dispatching problem in edge computing, where multiple access points (APs) collect jobs and then dispatch each job to an edge server. The distributed dispatcher on each AP would receive partially and outdated information exchanged via periodic broadcast. Hence, we formulate the distributed job dispatching problem by leveraging the partially observable Markov decision process (POMDP) and propose a novel approximate Markov decision process (MDP) solution framework, called <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DecMDP</monospace> , that bypasses the huge time complexity of conventional POMDP solutions. Both analytical and semi-analytical performance lower bounds are derived for the approximate MDP solution. Furthermore, we extend <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DecMDP</monospace> to handle a more general scenario where <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">a priori</i> knowledge of the system is absent. Finally, extensive simulations based on the Google Cluster traces show that our policy can achieve the best performance when compared with heuristic baselines, e.g., achieving 20.67% reduction in average job response time, and consistently performs well under various parameter settings.

Bounding Mean Slowdown in Multiserver Systems

Recent progress in queueing theory has made it possible to analyze the mean response time of multiserver queueing systems under advanced scheduling policies. However, this progress has so far been limited to the metric of mean response time. In practice, there are a wide variety of other metrics that can be more important. One such metric is mean slowdown, which is the average ratio between a job's response time and its size. While it is known that the "RS" policy minimizes mean slowdown in the single-server M/G/1, the problem is open for multiserver systems, including the M/G/k and load-balancing systems.

Scenario-based AI Benchmark Evaluation of Distributed Cloud/Edge Computing Systems

Distributed cloud/edge (DCE) platform has become popular in recent years. This paper proposes a new AI benchmark suite for assessing the performance of DCE platforms in machine learning (ML) and cognitive science applications. The benchmark suite is custom-designed to satisfy scenario-based performance requirements, namely the model training time, inference speed, model accuracy, job response time, quality of service, and system reliability. These metrics are substantiated by intensive experiments with real-life AI workloads. Our work is specially tailored for supporting massive AI multitasking across distributed resources in the networking environment. Our benchmark experiments were conducted on an AI-oriented AIRS cloud built at the Chinese University of Hong Kong, Shenzhen. We have tested a large number of ML/DL programs to narrow down the inclusion of ten representative AI kernel codes in the benchmark suite. Our benchmark results reveal the advantages of using the DCE systems cost-effectively in smart cities, healthcare, community surveillance, and transportation services. Our technical contributions are in the AIRS cloud architecture, benchmark design, testing, and distributed AI computing requirements. Our work will benefit computer system designers and AI application developers on clouds, edge, and mobile devices, that are supported by 5G mobile networks and AIoT resources.

P-ACOHONEYBEE: A Novel Load Balancer for Cloud Computing Using燤athematical Approach

Cloud computing is a collection of disparate resources or services, a web of massive infrastructures, which is aimed at achieving maximum utilization with higher availability at a minimized cost. One of the most attractive applications for cloud computing is the concept of distributed information processing. Security, privacy, energy saving, reliability and load balancing are the major challenges facing cloud computing and most information technology innovations. Load balancing is the process of redistributing workload among all nodes in a network; to improve resource utilization and job response time, while avoiding overloading some nodes when other nodes are underloaded or idle is a major challenge. Thus, this research aims to design a novel load balancing systems in a cloud computing environment. The research is based on the modification of the existing approaches, namely; particle swarm optimization (PSO), honeybee, and ant colony optimization (ACO) with mathematical expression to form a novel approach called P-ACOHONEYBEE. The experiments were conducted on response time and throughput. The results of the response time of honeybee, PSO, SASOS, round-robin, PSO-ACO, and P-ACOHONEYBEE are: 2791, 2780, 2784, 2767, 2727, and 2599 (ms) respectively. The outcome of throughput of honeybee, PSO, SASOS, round-robin, PSO-ACO, and P-ACOHONEYBEE are: 7451, 7425, 7398, 7357, 7387 and 7482 (bps) respectively. It is observed that P-ACOHONEYBEE approach produces the lowest response time, high throughput and overall improved performance for the 10 nodes. The research has helped in managing the imbalance drawback by maximizing throughput, and reducing response time with scalability and reliability.

Crucial File Selection Strategy (CFSS) for Enhanced Download Response Time in Cloud Replication Environments

Cloud Computing is a mass platform to serve high volume data from multi-devices and numerous technologies. Cloud tenants have a high demand to access their data faster without any disruptions. Therefore, cloud providers are struggling to ensure every individual data is secured and always accessible. Hence, an appropriate replication strategy capable of selecting essential data is required in cloud replication environments as the solution. This paper proposed a Crucial File Selection Strategy (CFSS) to address poor response time in a cloud replication environment. A cloud simulator called CloudSim is used to conduct the necessary experiments, and results are presented to evidence the enhancement on replication performance. The obtained analytical graphs are discussed thoroughly, and apparently, the proposed CFSS algorithm outperformed another existing algorithm with a 10.47% improvement in average response time for multiple jobs per round.

Optimal Job Scheduling With Resource Packing for Heterogeneous Servers

Jobs in modern computing clusters have highly diverse processing duration and heterogeneous resource requirements. In this paper, we consider the problem of job scheduling for a computing cluster comprised of multiple servers with heterogeneous computation resources, while taking the different resource demands of the jobs into account. Our focus is to achieve a low overall job response time for the system (which is also referred to as the job flowtime) while providing fairness between small and large jobs. Since the job flowtime minimization problem under multiple (even homogeneous) servers are known to be NP-hard, we propose an approximation algorithm to tackle the original online scheduling problem by adopting the recently-proposed notion of fractional job flowtime as a surrogate objective for minimization. For the general online job arrival case with multi-dimensional resource requirements, we apply Online Convex Optimization (OCO) techniques to design the corresponding scheduling algorithm with performance guarantees. In the single-dimensional resource setting, we show that the dynamic fit of the online version of our approximate algorithm grows only sublinearly with respect to time and derive a bound for its dynamic regret when comparing to its offline counterpart. While the baseline version of our proposed scheduling algorithm assumes the possibilities of job preemption and job migration across different servers, we show that the extent of job preemption and migration can be well controlled by augmenting the objective function with the corresponding switching costs.

Frequency scaling in multilevel queues

In this paper, we study a variant of PS+PS multilevel scheduling, which we call the PS+IS queue. Specifically, we use Processor Sharing (PS) at both queues, but with linear frequency scaling on the second queue, so that the latter behaves like an Infinite Server (IS) queue. The goals of the system are low response times for small jobs in the first queue, and reduced power consumption for large jobs in the second queue. The novelty of our model includes the frequency scaling at the second queue, and the batch arrival process at the second queue induced by the busy period structure of the first queue which has strictly higher priority. We derive a numerical solution for the PS+IS queueing system in steady-state, and then study its properties under workloads obtained from fitting of TCP flow traces. The simulation results confirm the e

Preemptive Parallel Job Scheduling for Heterogeneous Systems Supporting Urgent Computing

Dedicated infrastructures are commonly used for urgent computations. However, using dedicated resources is not always affordable due to budget constraints. As a result, utilizing shared infrastructures becomes an alternative solution for urgent computations. Since the infrastructures are meant to serve many users, the urgent jobs may arrive when regular jobs are using the necessary resources. In such a case, it is necessary to preempt the regular jobs so that urgent jobs can be executed immediately. Most conventional methods for job scheduling have focused on reducing the response times and waiting times of all jobs. However, these methods can delay urgent jobs and hinder them from being completed within a stipulated deadline. Furthermore, in heterogeneous systems with coprocessors, preemption becomes more difficult because coprocessors rely on several system software functionalities provided by the host processor. In this paper, we propose a parallel job scheduling method to effectively use shared heterogeneous systems for urgent computations. Our method employs an in-memory process swapping mechanism to preempt jobs running on the coprocessor devices. The results of our simulations show that our method can achieve a significant reduction in the response time and slowdown of regular jobs without substantial delays of urgent jobs.