Job Size Distribution Research Articles

Energy-Efficient Job-Assignment Policy With Asymptotically Guaranteed Performance Deviation

We study a job-assignment problem in a large-scale server farm system with geographically deployed servers as abstracted computer components (e.g., storage, network links, and processors) that are potentially diverse. We aim to maximize the energy efficiency of the entire system by effectively controlling carried load on networked servers. A scalable, near-optimal job-assignment policy is proposed. The optimality is gauged as, roughly speaking, energy cost per job. Our key result is an upper bound on the deviation between the proposed policy and the asymptotically optimal energy efficiency, when job sizes are exponentially distributed and blocking probabilities are positive. Relying on Whittle relaxation and the asymptotic optimality theorem of Weber and Weiss, this bound is shown to decrease exponentially as the number of servers and the arrival rates of jobs increase arbitrarily and in proportion. In consequence, the proposed policy is asymptotically optimal and, more importantly, approaches asymptotic optimality quickly (exponentially). This suggests that the proposed policy is close to optimal even for relatively small systems (and indeed any larger systems), and this is consistent with the results of our simulations. Simulations indicate that the policy is effective, and robust to variations in job-size distributions.

Minimizing Mean Response Time in Nonobservable Distributed Processing Systems with Nodes Operating under Egalitarian Processor-Sharing Policy

Abstract—Consideration is given to the dispatching problem in an almost nonobservable distributed processing system with M, M ≥ 2, single-server queues operating in parallel, each under the processor-sharing discipline. Jobs having the same job size distribution arrive one-by-one to the dispatcher, which immediately routes it to one of the queues. When making a routing decision, the dispatcher has no online information about the system (like current queues sizes, size of arriving job, etc.). The only information available to the dispatcher is job-size distribution, job-interarrival-time distribution, server speeds, time instants of previously arrived jobs, and previous routing decisions. Under these conditions, one is interested in the routing policies that minimize the job long-run mean response time. A new class of dispatching policies is proposed that, according to the numerical experiments, may significantly outperform all classical dispatching policies available for such system: (optimal) probabilistic policy and the round-robin policy.

Simple Near-Optimal Scheduling for the M/G/1

We consider the problem of preemptively scheduling jobs to minimize mean response time of an M/G/1 queue. When we know each job's size, the shortest remaining processing time (SRPT) policy is optimal. Unfortunately, in many settings we do not have access to each job's size. Instead, we know only the job size distribution. In this setting the Gittins policy is known to minimize mean response time, but its complex priority structure can be computationally intractable. A much simpler alternative to Gittins is the shortest expected remaining processing time (SERPT) policy. While SERPT is a natural extension of SRPT to unknown job sizes, it is unknown whether or not SERPT is close to optimal for mean response time. We present a new variant of SERPT called monotonic SERPT (M-SERPT) which is as simple as SERPT but has provably near-optimal mean response time at all loads for any job size distribution. Specifically, we prove the mean response time ratio between M-SERPT and Gittins is at most 3 for load ρ łeq 8/9$ and at most 5 for any load. This makes M-SERPT the only non-Gittins scheduling policy known to have a constant-factor approximation ratio for mean response time.

Performance Analysis of Workload Dependent Load Balancing Policies

Load balancing plays a crucial role in achieving low latency in large distributed systems. Recent load balancing strategies often rely on replication or use placeholders to further improve latency. However assessing the performance and stability of these strategies is challenging and is therefore mostly simulation based. In this paper we introduce a unified approach to analyze the performance and stability of a broad class of workload dependent load balancing strategies. This class includes many replication policies, such as replicate below threshold, delayed replication and replicate only small jobs, as well as strategies for fork-join systems. We consider systems with general job size distributions where jobs may experience server slowdown. We show that the equilibrium workload distribution of the cavity process satisfies a functional differential equation (FDE) and conjecture that the cavity process captures the limiting behavior of the system as its size tends to infinity. The cavity process method assumes that the workload at the queues is independent and identically distributed. This way, one may focus on the behaviour of one queue, this method was introduced in [1] and is shown to yield exact results for SQ(d) with job size distribution that have a decreasing hazard rate and LL(d) with general job sizes.

Mean Field Analysis of Join-Below-Threshold Load Balancing for Resource Sharing Servers

Load balancing plays a crucial role in many large scale computer systems. Much prior work has focused on systems with First-Come-First-Served (FCFS) servers. However, servers in practical systems are more complicated. They serve multiple jobs at once, and their service rate can depend on the number of jobs in service. Motivated by this, we study load balancing for systems using Limited-Processor-Sharing (LPS). Our model has heterogeneous servers, meaning the service rate curve and multiprogramming level (limit on the number of jobs sharing the processor) differs between servers. We focus on a specific load balancing policy: Join-Below-Threshold (JBT), which associates a threshold with each server and, whenever possible, dispatches to a server which has fewer jobs than its threshold. Given this setup, we ask: how should we configure the system to optimize objectives such as mean response time? Configuring the system means choosing both a load balancing threshold and a multiprogramming level for each server. To make this question tractable, we study the many-server mean field regime. In this paper we provide a comprehensive study of JBT in the mean field regime. We begin by developing a mean field model for the case of exponentially distributed job sizes. The evolution of our model is described by a differential inclusion, which complicates its analysis. We prove that the sequence of stationary measures of the finite systems converges to the fixed point of the differential inclusion, provided a unique fixed point exists. We derive simple conditions on the service rate curves to guarantee the existence of a unique fixed point. We demonstrate that when these conditions are not satisfied, there may be multiple fixed points, meaning metastability may occur. Finally, we give a simple method for determining the optimal system configuration to minimize the mean response time and related metrics. While our theoretical results are proven for the special case of exponentially distributed job sizes, we provide evidence from simulation that the system becomes insensitive to the job size distribution in the mean field regime, suggesting our results are more generally applicable.

Simple Near-Optimal Scheduling for the M/G/1

We consider the problem of preemptively scheduling jobs to minimize mean response time of an M/G/1 queue. When the scheduler knows each job's size, the shortest remaining processing time (SRPT) policy is optimal. Unfortunately, in many settings we do not have access to each job's size. Instead, we know only the job size distribution. In this setting, the Gittins policy is known to minimize mean response time, but its complex priority structure can be computationally intractable. A much simpler alternative to Gittins is the shortest expected remaining processing time (SERPT) policy. While SERPT is a natural extension of SRPT to unknown job sizes, it is unknown how close SERPT is to optimal.

Combining Size-Based Load Balancing with Round-Robin for Scalable Low Latency

When dispatching jobs to parallel servers, or queues, the highly scalable round-robin (RR) scheme reduces the variance of interarrival times at all queues to a great extent but has no impact on the variances of service processes. Contrariwise, size-interval task assignment (SITA) routing has little impact on the variances of interarrival times but makes the service processes as deterministic as possible. In this paper, we unify both ‘static’ approaches to design a scalable load balancing framework able to control the variances of the arrival and service processes jointly. It turns out that the resulting combination significantly improves performance and is able to drive the mean job delay to zero in the large-system limit; it is known that this property is not achieved when both approaches are considered separately. Within realistic parameters, we show that the optimal number of size intervals that partition the support of the job size distribution is small with respect to the system size. This enhances the applicability of the proposed load balancing scheme at a large scale. In fact, we find that adding a little bit of information about job sizes to a dispatcher operating under RR improves performance a lot. Under the optimal scaling of size intervals and assuming highly variable job sizes, numerical simulations indicate that the proposed algorithm is competitive with the (less scalable) join-the-shortest-workload algorithm even when the system size grows large.

Randomized Work Stealing Versus Sharing in Large-Scale Systems With Non-Exponential Job Sizes

Work sharing and work stealing are two scheduling paradigms to redistribute work when performing distributed computations. In work sharing, processors attempt to migrate pending jobs to other processors in the hope of reducing response times. In work stealing, on the other hand, underutilized processors attempt to steal jobs from other processors. Both paradigms generate a certain communication overhead and the question addressed in this paper is which of the two reduces the response time the most given that they use the same amount of communication overhead. Prior work presented explicit bounds, for large scale systems, on when randomized work sharing outperforms randomized work stealing in case of Poisson arrivals and exponential job sizes and indicated that work sharing is best when the load is below φ-1 ≈ 0.6180, with φ being the golden ratio. In this paper we revisit this problem and study the impact of the job size distribution using a mean field model. We present an efficient method to determine the boundary between the regions where sharing or stealing is best for a given job size distribution, as well as bounds that apply to any (phase-type) job size distribution. The main insight is that work stealing benefits significantly from having more variable job sizes and work sharing may become inferior to work stealing for loads as small as 1/2 + ε for any ε > 0.

On the impact of job size variability on heterogeneity-aware load balancing

Load balancing is one of the key components in many distributed systems as it heavily impacts performance and resource utilization. We consider a heterogeneous system where each server belongs to one of K classes and the speed of the server depends on its class. Two types of load balancing strategies are considered: arriving jobs are either immediately dispatched to a server class in a randomized manner, i.e., with probability $$p_k$$ a job is assigned to class k, or are dispatched based on their size, i.e., jobs with a size in $$[T_{k-1},T_k)$$ are assigned to class k. Within each class a power of d choices rule is used to select the server that executes the job. For large systems and exponential job size durations the optimal probabilities $$p_k$$ to minimize the mean response time can be determined easily via convex optimization. In this paper we develop a mean field model (validated by simulation) to investigate how the optimal probabilities $$p_k$$ are affected by the higher moments and in particular by the variability of the job size distribution when the service discipline at each server is first-come-first-served. In addition, we make use of the cavity method to study the optimal thresholds $$T_k$$ in case the dispatching is based on the job size.

Performance Analysis of Workload Dependent Load Balancing Policies

Load balancing plays a crucial role in achieving low latency in large distributed systems. Recent load balancing strategies often rely on replication or use placeholders to further improve latency. However assessing the performance and stability of these strategies is challenging and is therefore often simulation based. In this paper we introduce a unified approach to analyze the performance and stability of a broad class of workload dependent load balancing strategies. This class includes many replication policies, such as replicate below threshold, delayed replication and replicate only small jobs, as well as strategies for fork-join systems. We consider systems with general job size distributions where jobs may experience server slowdown. We show that the equilibrium workload distribution of the cavity process satisfies a functional differential equation and conjecture that the cavity process captures the limiting behavior of the system as its size tends to infinity. We study this functional differential equation in more detail for a variety of load balancing policies and propose a numerical method to solve it. The numerical method relies on a fixed point iteration or a simple Euler iteration depending on the type of functional differential equation involved. We further show that additional simplifications can be made if certain distributions are assumed to be phase-type. Various numerical examples are included that validate the numerical method and illustrate its strength and flexibility.

Dynamic load balancing with tokens

Efficiently exploiting the resources of data centers is a complex task that requires efficient and reliable load balancing and resource allocation algorithms. The former are in charge of assigning jobs to servers upon their arrival in the system, while the latter are responsible for sharing the server resources between their assigned jobs. These algorithms should adapt to various constraints, such as data locality, that restrict the feasible job assignments. In this paper, we propose a token-based algorithm that efficiently balances the load between the servers without requiring any knowledge on the job arrival rates and the server capacities. Assuming a balanced fair sharing of the server resources, we show that the resulting dynamic load balancing is insensitive to the job size distribution. Its performance is compared to that obtained under the best static load balancing and in an ideal system that would constantly optimize the resource utilization. We also make the connection with other token-based algorithms such as Join-Idle-Queue.

On the Power-of-d-choices with Least Loaded Server Selection

Motivated by distributed schedulers that combine the power-of-d-choices with late binding and systems that use replication with cancellation-on-start, we study the performance of the LL(d) policy which assigns a job to a server that currently has the least workload among d randomly selected servers in large-scale homogeneous clusters. We consider general job size distributions and propose a partial integro-differential equation to describe the evolution of the system. This equation relies on the earlier proven ansatz for LL(d) which asserts that the workload distribution of any finite set of queues becomes independent of one another as the number of servers tends to infinity. Based on this equation we propose a fixed point iteration for the limiting workload distribution and study its convergence. For exponential job sizes we present a simple closed form expression for the limiting workload distribution that is valid for any work-conserving service discipline as well as for the limiting response time distribution in case of first-come-first-served scheduling. We further show that for phase-type distributed job sizes the limiting workload and response time distribution can be expressed via the unique solution of a simple set of ordinary differential equations. Numerical and analytical results that compare response time of the classic power-of-d-choices algorithm and the LL(d) policy are also presented and the accuracy of the limiting response time distribution for finite systems is illustrated using simulations.

On the Power-of-d-choices with Least Loaded Server Selection

Motivated by distributed schedulers that combine the power-of-d-choices with late binding and systems that use replication with cancellation-on-start, we study the performance of the LL(d) policy which assigns a job to a server that currently has the least workload among d randomly selected servers in large-scale homogeneous clusters. We consider general job size distributions and propose a partial integro-differential equation to describe the evolution of the system. This equation relies on the earlier proven ansatz for LL(d) which asserts that the workload distribution of any finite set of queues becomes independent of one another as the number of servers tends to infinity. Based on this equation we propose a fixed point iteration for the limiting workload distribution and study its convergence.

Improving routing decisions in parallel non-observable queues

We revisit the well-known problem of scheduling in $$N\ge 2$$ non-observable parallel single-server queues with one dispatcher having no queue to store the incoming jobs. The dispatcher does not observe the current states of the queues and servers and the sizes of the incoming jobs. The only available information to it is the job size distribution, job’s inter-arrival time distribution and server’s speeds. For this problem setting it is known that if the dispatcher can memorize the sequence of its previous decisions, then a deterministic policy is better than the random choice policy with respect to the job’s mean waiting and mean sojourn time. In this paper we address the following question: can the deterministic policy be improved if the dispatcher, in addition to the decisions made, can memorize also the times between the decisions? We describe the new policy and numerically show that it is almost always possible to do better with it than with the deterministic policy. Two algorithms for choosing decisions under the new policy are given. Both are based on the Lindley recursion and are approximate in nature, because utilize the discretization of the underlying distributions. Our findings show that the relative gain of the new policy may reach $$10\%$$ , when minimizing job’s mean sojourn time, and more than $$50\%$$ for the job’s mean waiting time.

Towards Optimality in Parallel Scheduling

To keep pace with Moore's law, chip designers have focused on increasing the number of cores per chip rather than single core performance. In turn, modern jobs are often designed to run on any number of cores. However, to effectively leverage these multi-core chips, one must address the question of how many cores to assign to each job. Given that jobs receive sublinear speedups from additional cores, there is an obvious tradeoff: allocating more cores to an individual job reduces the job's runtime, but in turn decreases the efficiency of the overall system. We ask how the system should schedule jobs across cores so as to minimize the mean response time over a stream of incoming jobs. To answer this question, we develop an analytical model of jobs running on a multi-core machine. We prove that EQUI, a policy which continuously divides cores evenly across jobs, is optimal when all jobs follow a single speedup curve and have exponentially distributed sizes. EQUI requires jobs to change their level of parallelization while they run. Since this is not possible for all workloads, we consider a class of "fixed-width" policies, which choose a single level of parallelization, k, to use for all jobs. We prove that, surprisingly, it is possible to achieve EQUI's performance without requiring jobs to change their levels of parallelization by using the optimal fixed level of parallelization, k*. We also show how to analytically derive the optimal k* as a function of the system load, the speedup curve, and the job size distribution. In the case where jobs may follow different speedup curves, finding a good scheduling policy is even more challenging. In particular, we find that policies like EQUI which performed well in the case of a single speedup function now perform poorly. We propose a very simple policy, GREEDY*, which performs near-optimally when compared to the numerically-derived optimal policy.

An Efficient All Shapes Busy List Processor Allocation Algorithm for 3D Mesh Multicomputers

Contiguous processor allocation is useful for security and accounting reasons. This is due to the allocated jobs are separated from one another, where each sub-mesh of processors is allocated to an exclusive job request, and the allocated sub-mesh has the same size and shape of the requested job. The size and shape constraint leads to high processor fragmentation. Most recent contiguous allocation strategies suggested for 3D mesh-connected multiconputers try all possible orientations of an allocation request when allocation fails for the requested orientation, which reduces processor fragmentation and hence improves system performance. However, none of them considers all shapes of the request in the process of allocation. To generalize this restricted rotation, we propose, in this paper, a new contiguous allocation strategy for 3D mesh-connected multicomputers, referred to as All Shapes Busy List (ASBL for short), which takes into consideration all possible contiguous request shapes when attempting allocation for a job request. ASBL depends on the list of allocated sub-meshes, in the method suggested in (Bani-Mohammad et al., 2006), for selecting an allocated sub-mesh. The performance of the proposed ASBL allocation strategy has been evaluated considering several important scheduling strategies under a variety of system loads based on different job size distributions. The simulation results have shown that the ASBL allocation strategy improves system performance in terms of parameters such as the average turnaround time of jobs and system utilization under all scheduling strategies considered.

Asymptotically Optimal Job Assignment for Energy-Efficient Processor-Sharing Server Farms

We study the problem of job assignment in a large-scale realistically dimensioned server farm comprising multiple processor-sharing servers with different service rates, energy consumption rates, and buffer sizes. Our aim is to optimize the energy efficiency of such a server farm by effectively controlling carried load on networked servers. To this end, we propose a job assignment policy, called Most energy-efficient available server first Accounting for Idle Power (MAIP), which is both scalable and near optimal. MAIP focuses on reducing the productive power used to support the processing service rate. Using the framework of semi-Markov decision process, we show that, with exponentially distributed job sizes, MAIP is equivalent to the well-known Whittle’s index policy. This equivalence and the methodology of Weber and Weiss enable us to prove that, in server farms where a loss of jobs happens if and only if all buffers are full, MAIP is asymptotically optimal, as the number of servers tends to infinity under certain conditions associated with the large number of servers, as we have in a real server farm. Through extensive numerical simulations, we demonstrate the effectiveness of MAIP and its robustness to different job-size distributions, and observe that significant improvement in energy efficiency can be achieved by utilizing the knowledge of energy consumption rate of idle servers.

A Better Model for Task Assignment in Server Farms

An age-old problem in the design of server farms is the choice of the task assignment policy. This is the algorithm that determines how to assign incoming jobs to servers. Popular policies include Round-Robin assignment, Join-the-Shortest-Queue, Join-Queue-with-Least-Work, and so on. While much research has studied assignment policies, little has taken into account server-side variability -- the fact that the server we choose might be temporarily and unpredictably slow. We show that when server-side variability dominates runtime, replication of jobs can be very beneficial. We introduce the Replication-d algorithm that replicates each arrival to d servers chosen at random, where the job is considered "done" as soon as the first replica completes. We provide an exact closed-form analysis of Replication-d. We next introduce a much more general model, one which takes both the inherent job size distribution and the server-side variability into account. This is a departure from traditional queueing models which only allow for one "size" distribution. We propose and analyze a new task assignment policy, Replicate-Idle-Queue (RIQ), which is designed to perform well given these dual sources of variability.

Analysis of Randomized Join-the-Shortest-Queue (JSQ) Schemes in Large Heterogeneous Processor-Sharing Systems

In this paper, we investigate the stability and performance of randomized dynamic routing schemes for jobs based on the Join-the-Shortest Queue (JSQ) criterion in a heterogeneous system of many parallel servers. In particular, we consider servers that use processor sharing but with different server rates, and jobs are routed to the server with the smallest occupancy among a finite number of randomly sampled servers. We focus on the case of two servers that is often referred to as a Power-of-Two scheme. We first show that in the heterogeneous setting, uniform sampling of servers can cause a loss in the stability region and thus such randomized dynamic schemes need not outperform static randomized schemes in terms of mean delay in opposition to the homogeneous case of equal server speeds where the stability region is maximal and coincides with that of the static randomized routing. We explicitly characterize the stationary distributions of the server occupancies and show that the tail distribution of the server occupancy has a super-exponential behavior as in the homogeneous case as the number of servers goes to infinity. To overcome the stability issue, we show that it is possible to combine the static state-independent scheme with a randomized JSQ scheme that allows us to recover the maximal stability region combined with the benefits of JSQ, and such a scheme is preferable in terms of average delay. The techniques are based on a mean field analysis where we show that the stationary distributions coincide with those obtained under asymptotic independence of the servers and, moreover, the stationary distributions are insensitive to the job-size distribution.

Energy-Efficient Heuristics for Insensitive Job Assignment in Processor-Sharing Server Farms

Energy efficiency of server farms is an important design consideration of the green datacenter initiative. One effective approach is to optimize power consumption of server farms by controlling the carried load on the networked servers. In this paper, we propose a robust heuristic policy called E* for stochastic job assignment in a server farm, aiming to improve the energy efficiency by maximizing the ratio of job throughput to power consumption. Our model of the server farm considers a parallel system of finite-buffer processor-sharing queues with heterogeneous server speeds and energy consumption rates. We devise E* as an insensitive policy so that the stationary distribution of the number of jobs in the system depends on the job size distribution only through its mean. We provide a rigorous analysis of E* and compare it with a baseline approach, known as most energy-efficient server first (MEESF), that greedily chooses the most energy-efficient servers for job assignment. We show that E* has always a higher job throughput than that of MEESF, and derive realistic conditions under which E* is guaranteed to outperform MEESF in energy efficiency. Extensive numerical results are presented and demonstrate that E* can improve the energy efficiency by up to 100%.