Join The Shortest Queue Research Articles

Exponential Tail Bounds on Queues: A Confluence of Non- Asymptotic Heavy Traffic and Large Deviations

In general, obtaining the exact steady-state distribution of queue lengths is not feasible. Therefore, we focus on establishing bounds for the tail probabilities of queue lengths. We examine queueing systems under Heavy Traffic (HT) conditions and provide exponentially decaying bounds for the probability P(∈q > x), where ∈ is the HT parameter denoting how far the load is from the maximum allowed load. Our bounds are not limited to asymptotic cases and are applicable even for finite values of ∈, and they get sharper as ∈ - 0. Consequently, we derive non-asymptotic convergence rates for the tail probabilities. Furthermore, our results offer bounds on the exponential rate of decay of the tail, given by -1/2 log P(∈q > x) for any finite value of x. These can be interpreted as non-asymptotic versions of Large Deviation (LD) results. To obtain our results, we use an exponential Lyapunov function to bind the moment-generating function of queue lengths and apply Markov's inequality. We demonstrate our approach by presenting tail bounds for a continuous time Join-the-shortest queue (JSQ) system.

On the design and performance of scheduling policies exploiting spatial diversity for URLLC

In this paper, we study the performance of packet scheduling schemes for Ultra-Reliable Low-Latency Communications (URLLC) services. We exploit spatial diversity (i.e., redundant coverage of users) guaranteed in numerous 5G radio access network scenarios to examine the impact of multi-connectivity. Thus, we consider a set of URLLC users connected to two frequency layers or Radio Access Technologies (RATs) to ensure minimal queuing time. We review four packet scheduling and redundancy schemes, namely Join-the-Shortest-Queue (JSQ), the shortest expected delay (SED), systematic Redundancy (RED), and redundancy with Cancellation upon completion (CAN). We choose the outage probability as a metric, defined as the packet’s probability of arriving after some given target delay. We show that RED performs well at low load, whereas JSQ and SED are better when the load rises. Besides, CAN outperforms all other schemes. We then discuss the trade-off between performance and implementation complexity.

Load balancing policies without feedback using timed replicas

Dispatching policies such as join the shortest queue (JSQ), join the queue with smallest workload (JSW), and their power of two variants are used in load balancing systems where the instantaneous queue length or workload information at all queues or a subset of them can be queried. In situations where the dispatcher has an associated memory, one can minimize this query overhead by maintaining a list of idle servers to which jobs can be dispatched. Recent alternative approaches that do not require querying such information include the cancel-on-start and cancel-on-complete replication policies. The downside of such policies however is that the servers must communicate either the start or the completion time instant of each service to the dispatcher and must allow the coordinated and instantaneous cancellation of all redundant replicas. In practice, the requirements of query messaging, memory, and replica cancellation pose challenges in their implementation and their advantages are not clear. In this work, we consider load-balancing policies that do not need to query load information, do not need memory, and do not need to cancel replicas. Our policies allow the dispatcher to append a timer to each job or its replica. A job or a replica is discarded if its timer expires before it starts receiving service. We analyze several variants of this policy which are novel and simple to implement. We numerically observe that the variants of the proposed policy outperform popular feedback-based policies for low arrival rates, despite no feedback from servers to the dispatcher.

Exponential Tail Bounds on Queues

A popular approach to computing performance measures of queueing systems (such as delay and queue length) is studying the system in an asymptotic regime. However, these results are only valid in the limit and often provide bounds for the pre-limit systems that are not optimized and, hence, give loose bounds for the tail probabilities. In this paper, we provide optimized bounds for the tail probabilities of the scaled total queue length in a load-balancing system under Join the Shortest Queue (JSQ). Our bounds characterize the rate of convergence of the tail probabilities to the corresponding heavy traffic values. For the tail probability of the JSQ system, our bounds yield a multiplicative error that arises from three factors: pre-limit tail, pre-exponent error, and State-Space Collapse (SSC). As an immediate corollary of our main theorem, we provide a bound to the tail probabilities of a single-server queue. In this case, the multiplicative error only consists of pre-limit tail and pre-exponent error, since there is no state-space collapse.

Dual core processors: Coupled queues: Transient performance evaluation

High-Performance Computing (HiPC) systems routinely employ multi/many – core processors. Specifically, dual – core processors find many applications in pervasive computing devices. Dual–core processors employ buffers for queueing the incoming jobs. Traditionally, the queues at the processors are assumed to be independent and the queueing system is analyzed in equilibrium for tractability purposes. Queues are modeled using Continuous Time Markov Chains (CTMC's) and the equilibrium performance measures are determined to analyze as well as design the queueing systems. In most interesting cases, the incoming jobs are routed to the queues using the Join the Shortest Queue (JSQ) policy. Thus, with such an adaptive routing algorithm, the two queues are evidently coupled and are not statistically independent. Hence traditional equilibrium performance evaluation doesn't provide realistic performance measures. In this research paper, the two queues associated with buffers in dual-core processors are considered to be coupled. The Coupled Queues are modeled using a Quasi – Birth – and – Death (QBD) process. Using traditional results related to QBD processes, equilibrium performance measures are determined. More interestingly, we demonstrate the tractability of the computation of transient probability distribution of a QBD process. In the research literature, transient analysis of the QBD process was shown to be tractable in the Laplace transform domain. But in this research paper, we prove that the matrix exponential eQt arising in transient analysis (where Q is the generator matrix of the QBD process) can be computed directly in the time domain rendering efficient transient analysis of QBD process. Using the transient probability mass function of queue length, estimation of transient performance measures such as expected queue length, average delay, and tail distribution can be determined. Further, optimal adaptive routing algorithms for coupled queues can be designed.

The Join-the-Shortest-Queue System in the Halfin-Whitt Regime: Rates of Convergence to the Diffusion Limit

We bound the rate at which the steady-state distribution of the join-the-shortest-queue (JSQ) system converges, in the Halfin-Whitt regime, to its diffusion limit. Our proof uses Stein’s method and, specifically, the recently proposed prelimit generator comparison approach. The JSQ system is nontrivial and high-dimensional and has a state-space collapse component; our analysis may serve as a helpful example to readers wishing to apply the approach to their own setting.

A 3-queue polling system with join the shortest-serve the longest policy

In 1987, J.W. Cohen analyzed the so-called Serve the Longest Queue (SLQ) queueing system, where a single server attends two non-symmetric M/G/1-type queues, exercising a non-preemptive priority switching policy. Cohen further analyzed in 1998 a non-symmetric 2-queue Markovian system, where newly arriving customers follow the Join the Shortest Queue (JSQ) discipline. The current paper generalizes and extends Cohen’s works by studying a combined JSQ–SLQ model, and by broadening the scope of analysis to a non-symmetric 3-queue system, where arriving customers follow the JSQ strategy and a single server exercises the preemptive priority SLQ discipline. The system states’ multi-dimensional probability distribution function is derived while applying a non-conventional representation of the underlying process’s state-space. The analysis combines both Probability Generating Functions and Matrix Geometric methodologies. It is shown that the joint JSQ–SLQ operating policy achieves extremely well the goal of balancing between queue sizes. This is emphasized when calculating the Gini Index associated with the differences between mean queue sizes: the value of the coefficient is close to zero. Extensive numerical results are presented.

Asymptotic optimality of speed-aware JSQ for heterogeneous service systems

The Join-the-Shortest-Queue (JSQ) load-balancing scheme is known to minimise the average delay of jobs in homogeneous systems consisting of identical servers. However, it performs poorly in heterogeneous systems where servers have different processing rates. Finding a delay optimal scheme remains an open problem for heterogeneous systems. In this paper, we consider a speed-aware version of the JSQ scheme for heterogeneous systems and show that it achieves delay optimality in the fluid limit. One of the key issues in establishing this optimality result for heterogeneous systems is to show that the sequence of steady-state distributions indexed by the system size is tight in an appropriately defined space. The usual technique for showing tightness by coupling with a suitably defined dominant system does not work for heterogeneous systems. To prove tightness, we devise a new technique that uses the drift of exponential Lyapunov functions. Using the non-negativity of the drift, we show that the stationary queue length distribution has an exponentially decaying tail — a fact we use to prove tightness. Another technical difficulty arises due to the complexity of the underlying state-space and the separation of two time-scales in the fluid limit. Due to these factors, the fluid-limit turns out to be a function of the invariant distribution of a multi-dimensional Markov chain which is hard to characterise. By using some properties of this invariant distribution and using the monotonicity of the system, we show that the fluid limit is has a unique and globally attractive fixed point.

Scalable Load Balancing in Networked Systems: A Survey of Recent Advances

In this survey we provide an overview of recent advances on scalable load balancing schemes which provide favorable delay performance and yet require minimal implementation overhead. The basic load balancing scenario involves a single dispatcher where tasks arrive that must immediately be forwarded to one of $N$ single-server queues. The join-the-shortest-queue (JSQ) policy yields vanishing delays as $N$ grows large, as in a centralized queuing arrangement, but involves a prohibitive communication burden. In contrast, JSQ($d$) schemes that assign an incoming task to a server with the shortest queue among $d$ servers selected uniformly at random require little communication, but lead to constant delays. In order to examine this fundamental trade-off between delay performance and implementation overhead, we discuss a body of recent research on JSQ($d(N)$) schemes in which the diversity parameter $d(N)$ depends on $N$ and investigate the growth rate of $d(N)$ required to match the optimal JSQ performance on fluid and diffusion scales. Stochastic coupling techniques and scaling limits play an instrumental role in establishing this asymptotic optimality. We demonstrate how this methodology carries over to infinite-server settings, finite buffers, multiple dispatchers, servers arranged on graph topologies, and token-based load balancing schemes such as join-the-idle-queue (JIQ), thus providing a broad overview of the main trends in the field.

A load balancing system in the many-server heavy-traffic asymptotics

We study the load balancing system operating under Join the Shortest Queue (JSQ) in the many-server heavy-traffic regime. If $N$ is the number of servers, we let the difference between the total service rate and the total arrival rate be $N^{1-\alpha}$ with $\alpha>0$. We show that for $\alpha>4$ the average queue length behaves similarly to the classical heavy-traffic regime. Specifically, we prove that the distribution of the average queue length multiplied by $N^{1-\alpha}$ converges to an exponential random variable. Moreover, we show a result analogous to state space collapse. We provide two proofs for our result: one using the one-sided Laplace transform, and one using Stein's method. We additionally obtain the rate of convergence in the Wasserstein's distance.

Steady‐state analysis of load balancing with Coxian‐2 distributed service times

AbstractThis paper studies load balancing for many‐server (N servers) systems. Each server has a buffer of size b − 1, and can have at most one job in service and b − 1 jobs in the buffer. The service time of a job follows the Coxian‐2 distribution. We focus on steady‐state performance of load balancing policies in the heavy traffic regime such that the normalized load of system is λ = 1 − N−α for 0 < α < 0.5. We identify a set of policies that achieve asymptotic zero waiting. The set of policies include several classical policies such as join‐the‐shortest‐queue (JSQ), join‐the‐idle‐queue (JIQ), idle‐one‐first (I1F) and power‐of‐d‐choices (Po d) with d = O(Nα log N). The proof of the main result is based on Stein's method and state space collapse. A key technical contribution of this paper is the iterative state space collapse approach that leads to a simple generator approximation when applying Stein's method.

Generalized Cost-Based Job Scheduling in Very Large Heterogeneous Cluster Systems

We study job assignment in large, heterogeneous resource-sharing clusters of servers with finite buffers. This load balancing problem arises naturally in today's communication and big data systems, such as Amazon Web Services, Network Service Function Chains, and Stream Processing. Arriving jobs are dispatched to a server, following a load balancing policy that optimizes a performance criterion such as job completion time. Our contribution is a randomized Cost-Based Scheduling (CBS) policy in which the job assignment is driven by general cost functions of the server queue lengths. Beyond existing schemes, such as the Join the Shortest Queue (JSQ), the power of d or the SQ(d) and the capacity-weighted JSQ, the notion of CBS yields new application-specific policies such as hybrid locally uniform JSQ. As today's data center clusters have thousands of servers, exact analysis of CBS policies is tedious. In this article, we derive a scaling limit when the number of servers grows large, facilitating a comparison of various CBS policies with respect to their transient as well as steady state behavior. A byproduct of our derivations is the relationship between the queue filling proportions and the server buffer sizes, which cannot be obtained from infinite buffer models. Finally, we provide extensive numerical evaluations and discuss several applications including multi-stage systems.

Fundamental Limits of Volume-based Network DoS Attacks

Volume-based network denial-of-service (DoS) attacks refer to a class of cyber attacks where an adversary seeks to block user traffic from service by sending adversarial traffic that reduces the available user capacity. In this paper, we explore the fundamental limits of volume-based network DoS attacks by studying the minimum required rate of adversarial traffic and investigating optimal attack strategies. We start our analysis with single-hop networks where user traffic is routed to servers following the Join-the-Shortest- Queue (JSQ) rule. Given the service rates of servers and arrival rates of user traffic, we first characterize the feasibility region of the attack and show that the attack is feasible if and only if the rate of the adversarial traffic lies in the region.We then design an attack strategy that is (i). optimal: it guarantees the success of the attack whenever the adversarial traffic rate lies in the feasibility region and (ii). oblivious: it does not rely on knowledge of service rates or user traffic rates. Finally, we extend our results on the feasibility region of the attack and the optimal attack strategy to multi-hop networks that employ Back-pressure (Max-Weight) routing. At a higher level, this paper addresses a class of dual problems of stochastic network stability, i.e., how to optimally de-stabilize a network.

Stability of JSQ in queues with general server-job class compatibilities

We consider Poisson streams of exponentially distributed jobs arriving at each edge of a hypergraph of queues. Upon arrival, an incoming job is routed to the shortest queue among the corresponding vertices. This generalizes many known models such as power-of-d load balancing and JSQ (join the shortest queue) on generic graphs. We prove that stability in this model is achieved if and only if there exists a stable static routing policy. This stability condition is equivalent to that of the JSW (join the shortest workload) policy. We show that some graph topologies lead to a loss of capacity, implying more restrictive stability conditions than in, for example, complete graphs.

Fundamental Limits of Volume-based Network DoS Attacks

Volume-based network denial-of-service (DoS) attacks refer to a class of cyber attacks where an adversary seeks to block user traffic from service by sending adversarial traffic that reduces the available user capacity. In this paper, we explore the fundamental limits of volume-based network DoS attacks by studying the minimum required rate of adversarial traffic and investigating optimal attack strategies. We start our analysis with single-hop networks where user traffic is routed to servers following the Join-the-Shortest-Queue (JSQ) rule. Given the service rates of servers and arrival rates of user traffic, we first characterize the feasibility region of the attack and show that the attack is feasible if and only if the rate of the adversarial traffic lies in the region. We then design an attack strategy that is (i).optimal: it guarantees the success of the attack whenever the adversarial traffic rate lies in the feasibility region and (ii).oblivious: it does not rely on knowledge of service rates or user traffic rates. Finally, we extend our results on the feasibility region of the attack and the optimal attack strategy to multi-hop networks that employ Back-pressure (Max-Weight) routing. At a higher level, this paper addresses a class of dual problems of stochastic network stability, i.e., how to optimally de-stabilize a network.

Smart Dispatching in Heterogeneous Systems

In multi-server systems, selecting to which server to dispatch an arriving job is a key factor influencing system response time. One of the most widely studied policies is Join-the-Shortest-Queue (JSQ), which is known to minimize mean response time in certain settings [7]. Many variants on JSQ have been proposed, including JSQ-d, under which a job is dispatched to the shortest queue among d servers selected uniformly at random [3, 5]; Join-Idle-Queue (JIQ), under which the dispatcher knows which servers are idle but not the queue lengths of non-idle servers [2]; and others. The vast majority of work analyzing JSQ and related policies makes a key assumption: that the system is homogeneous, meaning that all servers have the same speed. This assumption is inaccurate in most modern computer systems. Server heterogeneity can arise, e.g., when a server farm consists of several generations of hardware, or when many virtual machines contend for resources on the same physical machine. Unfortunately, the wealth of results about how best to dispatch in homogeneous systems does not translate well to heterogeneous systems. Policies like JSQ-d and JIQ, which can achieve near-optimal performance in homogeneous systems, can lead to unacceptably high response times and even instability in heterogeneous systems [4, 8].

On Achieving Zero Delay with Power-of-$d$-Choices Load Balancing

Power-of- $d$ -choices is a popular load balancing algorithm for many-server systems such as large-scale data centers. For each incoming job, the algorithm probes $d$ servers, chosen uniformly at random from a total of $N$ servers ( $N$ is the number of servers in the system), and routes the job to the least loaded one. It is well known that power-of- $d$ -choices reduces queueing delays by orders of magnitude compared to the policy that routes each incoming job to a randomly selected server. The question to be addressed in this paper is how large $d$ needs to be so that power-of- $d$ -choices achieves asymptotic zero delay like the join-the-shortest-queue (JSQ) algorithm, which is a special case of power-of- $d$ -choices with $d=N.$ We are interested in the heavy-traffic regime where the load of the system, denoted by $\lambda, $ approaches to one as $N$ increases, and assume $\lambda =1-\gamma N^{-\alpha }$ for $0 This paper establishes that when $d=\Omega \left(\frac{\log N}{1-\lambda }\right)$ , finite buffer size $b$ and $0\leq \alpha the probability that an incoming job is routed to a busy server is asymptotically zero, i.e., a job experiences zero queueing delay with probability one asymptotically; and when $d=O\left(\frac{1}{1-\lambda }\right)$ and infinite buffer size $b=\infty$ , the probability that a job is routed to a busy server is lower bounded by a positive constant independent of $N.$ Therefore, our results show that $d=\Omega \left(\frac{\log N}{1-\lambda }\right)$ is sufficient and almost necessary for achieving zero delay with the power-of- $d$ -choices policy.

Open Problem—Load Balancing Using Delayed Information

Load-balancing algorithms are important for efficiently routing jobs in systems of parallel queues; however, there has been relatively little attention devoted to developing algorithms in the presence of messaging and/or routing delays. Given a system of parallel queues with infinite capacity buffers; first-come, first-serve service discipline; and a single stream of incoming jobs that are routed by a dispatcher upon arrival, it has been shown that join the shortest queue (JSQ) satisfies certain optimality properties, including minimizing mean wait time when the job sizes are exponentially distributed ( Winston 1977 , Ephremides et al. 1980 ) and state space collapse of the queue lengths under heavy traffic scaling for general service distributions ( Reiman 1984 ). However, implementation of JSQ uses up-to-date information about the state of the buffers, which requires instantaneous exchange of multiple messages between the dispatcher and the queues. This challenge has led to the development of efficient algorithms that require fewer messages, including join the shortest of d queues (JSQ(d); Vvedenskaya et al. 1996 , Mitzenmacher 2001 ), join the idle queue (JIQ; Badonnel and Burgess 2008 , Lu et al. 2011 ), and persistent idle load distribution (PILD; Atar et al. 2019a , b ); see van der Boor et al. (2018) for a survey of some recent results in the many-server limit. Although these algorithms do not use full information about the system, they still use up-to-date information about the state of some of the queues. Because of the physical separation between the dispatcher and the queues, processing effects, or periodic updates from the queues, the dispatcher may have access only to information about the delayed states of the queues. (There may also be routing delays because of the time it takes a job to travel between the dispatcher and its assigned server, for which these algorithms also do not account.) In such settings, it has been shown that JSQ can perform quite poorly and lead to sustained oscillations in queue lengths ( Mitzenmacher 2000 ).

Simulation and performance evaluation of mission critical dispatching systems

Dispatching systems, where jobs are routed to servers immediately upon arrival, appear frequently in parallel computing systems of different scales. With a dynamic dispatching policy, the system is generally analytically intractable and performance evaluation is carried out by means of Monte Carlo simulations. A typical performance metric is the mean response time which is often easy to estimate. In contrast, we consider systems where events generating costs are extremely rare. In our reference system, jobs have deadlines for waiting time. When deadlines are loose when compared to the system’s load, novel rare event simulation techniques must be applied. We consider both conditioning and importance sampling to this end. In numerical examples, the proposed techniques decrease the simulation time by several orders of magnitude. We also discover interesting performance relationships among the classical dispatching policies; Random split (RND), Round-robin (RR), Join-the-shortest-queue (JSQ) and Least-work-left (LWL).

Join-the-Shortest Queue diffusion limit in Halfin–Whitt regime: Sensitivity on the heavy-traffic parameter

Consider a system of $N$ parallel single-server queues with unit-exponential service time distribution and a single dispatcher where tasks arrive as a Poisson process of rate $\lambda(N)$. When a task arrives, the dispatcher assigns it to one of the servers according to the Join-the-Shortest Queue (JSQ) policy. Eschenfeldt and Gamarnik (Math. Oper. Res., 43(3):867-886, 2018) identified a novel limiting diffusion process that arises as the weak-limit of the appropriately scaled occupancy measure of the system under the JSQ policy in the Halfin-Whitt regime, where $(N - \lambda(N)) / \sqrt{N} \to \beta > 0$ as $N \to \infty$. The analysis of this diffusion goes beyond the state of the art techniques, and even proving its ergodicity is non-trivial, and was left as an open question. Recently, exploiting a generator expansion framework via the Stein's method, Braverman (arXiv:1801.05121, 2018) established its exponential ergodicity, and adapting a regenerative approach, Banerjee and Mukherjee (Ann. Appl. Probab., 29(2):1262-1309, 2018) analyzed the tail properties of the stationary distribution and path fluctuations of the diffusion. However, the analysis of the bulk behavior of the stationary distribution, viz., the moments, remained intractable until this work. In this paper, we perform a thorough analysis of the bulk behavior of the stationary distribution of the diffusion process, and discover that it exhibits different qualitative behavior, depending on the value of the heavy-traffic parameter $\beta$. Moreover, we obtain precise asymptotic laws of the centered and scaled steady state distribution, as $\beta$ tends to 0 and $\infty$. Of particular interest, we also establish a certain intermittency phenomena in the $\beta\to \infty$ regime and a surprising distributional convergence result in the $\beta\to 0$ regime.