Idle Servers Research Articles

Asymptotically optimal routing of a many-server parallel queueing system with long-run average criterion

This paper considers a parallel queueing system with multiple stations, each of which contains many statistically identical servers and has a dedicated queue. Upon each customer arrival, the system manager must decide to which station the customer should be routed, with the objective of minimizing the system’s long-run average delay cost. One feature of this paper is that a customer’s delay cost depends not only on his/her delay, but also on the routed station. Considering this heterogeneity across stations, we propose a routing policy, which can be regarded as an extension of the MED–FSF policy. Under this policy, any arriving customer will be routed to: (i) the station with the minimum value, which depends on the station’s expected delay and the station index when servers in all stations are fully occupied; or otherwise (ii) the station with a fastest idle server. Using asymptotic analysis, we derive diffusion limits of queue-length processes and their stationary distributions under the proposed policy in the Halfin–Whitt regime. Combined with an asymptotic lower bound result for the long-run average delay cost, we show that the proposed routing policy is asymptotically optimal under the considered objective. Finally, we provide numerical experiments to validate the accuracy of our diffusion approximation, and we compare the performance metrics under the proposed policy with those under other commonly used routing policies.

Asymptotically optimal energy consumption and inventory control in a make-to-stock manufacturing system

We study a make-to-stock manufacturing system in which a single server makes the production. The server consumes energy, and its power consumption depends on the server state: a busy server consumes more power than an idle server, and an idle server consumes more power than a turned-off server. When a server is turned on, it completes a costly set-up process that lasts a while. We jointly control the finished goods inventory and the server’s energy consumption. The objective is to minimize the long-run average inventory holding, backorder, and energy consumption costs by deciding when to produce, when to idle or turn off the server, and when to turn on a turned-off server. Because the exact analysis of the problem is challenging, we consider the asymptotic regime in which the server is in the conventional heavy-traffic regime. We formulate a Brownian control problem (BCP) with impulse and singular controls. In the BCP, the impulse control appears due to server shutdowns, and the singular control appears due to server idling. Depending on the system parameters, the optimal BCP solution is either a control-band or barrier policy. We propose a simple heuristic control policy from the optimal BCP solution that can easily be implemented in the original (non-asymptotic) system. Furthermore, we prove the asymptotic optimality of the proposed control policy in a Markovian setting. Finally, we show that our proposed policy performs close to optimal in numerical experiments.

A study on N-policy BMAP/G/1 queueing system

This study presents a simple approach for analyzing the BMAP/G/1 queueing system under N policy. In this policy, the server goes idle at the end of a busy period and the queue length is checked at every arrival moment. The idle server starts serving customers when the queue length reaches or above a predetermined number, namely N, and keeps doing so until the system is completely empty. We first use a simple sequential substitution approach to derive the system length distribution at departure epoch. A comparative study is also conducted to highlight the benefits and strength of our straightforward approach to that of the RG-factorization technique and the roots finding method. We extract the distribution of system length at a random time point by utilizing the remaining service time of a customer who is currently being served as the supplementary variable. The probability density function of the sojourn time distribution for an arbitrary customer of an incoming batch is also computed. We propose an expected linear cost function to estimate the optimal value of N at minimum cost. The validity of our analytic technique has been shown through a variety of numerical examples involving different service time distributions.

On the performance evaluation of distributed join-idle-queue load balancing with and without token withdrawals

Distributed Join-Idle-Queue load balancing is known to achieve vanishing waiting times in the large-scale limit provided that the number of dispatchers remains fixed, while the number of servers tends to infinity. When the number of dispatchers m scales to infinity together with the number of servers n, such that r=n/m remains fixed, the large-scale performance of Join-Idle-Queue load balancing is less clear as waiting times no longer vanish.In this paper we first discuss some existing mean field models for distributed Join-Idle-Queue load balancing with r=n/m fixed and explain why the well-known model introduced in Lu et al. (2011) is not exact in the large-scale limit. The inexactness is caused by mixing two variants of distributed Join-Idle-Queue load balancing: a variant with and one without token withdrawals. Next we introduce mean field models for Join-Idle-Queue load balancing with and without token withdrawals, where an idle server places a token at a dispatcher with the shortest among d randomly chosen dispatchers.The introduced mean field models in case of token withdrawals imply that for phase type distributed service times and a total job arrival rate of λn<n, the response time of a job corresponds to that in a standard M/PH/1 queue with load λq0. The value of q0 can be determined numerically and depends on λ,r and d, but not on the job size distribution (apart from its mean). This simple behavior is lost if token withdrawals do not take place. For the models without withdrawals we develop fast numerical algorithms to determine the performance. We present simulation experiments that suggest that the unique fixed point of the introduced mean field models provides exact results in the large-scale limit.

Heavy traffic analysis of multi-class bipartite queueing systems under FCFS

This paper examines the performance of multi-class, multi-server bipartite queueing systems, where each arriving customer is compatible with only a subset of servers. We focus on the system’s performance under a first-come, first-served-assign longest idle server service discipline. In this discipline, an idle server is matched with the compatible customer who has been waiting the longest, and a customer who can be served by multiple idle servers is routed to the server that has been idle for the longest period. We analyse the system under conventional heavy-traffic conditions, where the traffic intensity approaches one from below. Building upon the formulation and results of Afèche et al. (Oper Res 70(1):363–401, 2022), we generalize the model by allowing the vector of arrival rates to approach the heavy-traffic limit from an arbitrary direction. We characterize the steady-state waiting times of the various customer classes and demonstrate that a much wider range of waiting time outcomes is achievable. Furthermore, we establish that the matching probabilities, i.e. the probabilities of different customer classes being served by different servers, do not depend on the direction along which the system approaches heavy traffic. We also investigate the design of compatibility between customer classes and servers, finding that a service provider who has complete control over the matching can design a delay-minimizing matching by considering only the limiting arrival rates. When some constraints on the compatibility structure exist, the direction of convergence to heavy-traffic affects which compatibility structure minimizes delay. Additionally, we discover that the bipartite matching queueing system exhibits a form of Braess’s paradox, where adding more connectivity to an existing system can lead to higher average waiting times, despite the fact that neither customers nor servers act strategically.

Optimal Rate-Matrix Pruning For Heterogeneous Systems

We consider large-scale load balancing systems where processing time distribution of tasks depend on both task and server types. We analyze the system in the asymptotic regime where the number of task and server types tend to infinity proportionally to each other. In such heterogeneous setting, popular policies like Join Fastest Idle Queue (JFIQ), Join Fastest Shortest Queue (JFSQ) are known to perform poorly and they even shrink the stability region. Moreover, to the best of our knowledge, in this setup, finding a scalable policy with provable performance guarantee has been an open question prior to this work. In this paper, we propose and analyze two asymptotically delay-optimal dynamic load balancing approaches: (a) one that efficiently reserves the processing capacity of each server for "good" tasks and route tasks under the Join Idle Queue policy; and (b) a speed-priority policy that increases the probability of servers processing tasks at a high speed. Introducing a novel analytical framework and using the mean-field method and stochastic coupling arguments, we prove that both policies above achieve asymptotic zero queueing, whereby the probability that a typical task is assigned to an idle server tends to 1 as the system scales.

An efficient reducing mechanism for energy consumption in data center using hybrid consolidation techniques

The rise in internet user demand is a major factor in the expansion of infrastructure and the upsurge in energy use in cloud, colocation, and some business data centres. The advent of 5G has compounded the situation, as it substantially gives room for many new types of digital services, resulting in a need for richer consume a lot of energy when no scaling method is applied. Services such as mail, data storage and retrieval and other cloud services also require a lot of high energy consumption which eventually result into carbon(IV) oxide (CO2) emissions to the environment. This research therefore, focuses on lowering the energy usage of a data centre with heterogeneous power awareness either in an idle server state or high-performance state using a novel hybridized algorithm called “DyVoFesLoReMu”, comprising Dynamic Voltage Frequency Scaling (Dvfs) and a modified Local Regression Minimum Utilization (LrMu). A real dataset (workload) obtained online from PlanetLab consisting of hosts and Virtual Machines (VM) was simulated on a data center in CloudSim 3.0.3. Tool kit with preset parameters consisting of VM Allocation Policy and VM Selection Policy was used. The tool kit was utilised to create cloud infrastructure and simulate the essential features of a cloud environment. The Cloudsim was installed on Eclipse Integrated Development Environment (IDE) 2019 version on Windows 10 operating system. The hybridized algorithm was compared with other five (5) existing energy reducing algorithms and it was found to be more efficient with a range of 41-90% reduction in energy usage from the ten days workload traces and in comparison with the existing algorithms used for the simulation.

Performance Analysis of Work Stealing Strategies in Large-Scale Multithreaded Computing

Distributed systems use randomized work stealing to improve performance and resource utilization. In most prior analytical studies of randomized work stealing, jobs are considered to be sequential and are executed as a whole on a single server. In this article, we consider a homogeneous system of servers where parent jobs spawn child jobs that can feasibly be executed in parallel. When an idle server probes a busy server in an attempt to steal work, it may either steal a parent job or multiple child jobs. To approximate the performance of this system, we introduce a Quasi-Birth-Death Markov chain and express the performance measures of interest via its unique steady state. We perform simulation experiments that suggest that the approximation error tends to zero as the number of servers in the system becomes large. To further support this observation, we introduce a mean field model and show that its unique fixed point corresponds to the steady state of the Quasi-Birth-Death Markov chain. Using numerical experiments, we compare the performance of various simple stealing strategies as well as optimized strategies.

Load balancing policies without feedback using timed replicas

Dispatching policies such as join the shortest queue (JSQ), join the queue with smallest workload (JSW), and their power of two variants are used in load balancing systems where the instantaneous queue length or workload information at all queues or a subset of them can be queried. In situations where the dispatcher has an associated memory, one can minimize this query overhead by maintaining a list of idle servers to which jobs can be dispatched. Recent alternative approaches that do not require querying such information include the cancel-on-start and cancel-on-complete replication policies. The downside of such policies however is that the servers must communicate either the start or the completion time instant of each service to the dispatcher and must allow the coordinated and instantaneous cancellation of all redundant replicas. In practice, the requirements of query messaging, memory, and replica cancellation pose challenges in their implementation and their advantages are not clear. In this work, we consider load-balancing policies that do not need to query load information, do not need memory, and do not need to cancel replicas. Our policies allow the dispatcher to append a timer to each job or its replica. A job or a replica is discarded if its timer expires before it starts receiving service. We analyze several variants of this policy which are novel and simple to implement. We numerically observe that the variants of the proposed policy outperform popular feedback-based policies for low arrival rates, despite no feedback from servers to the dispatcher.

Performance Study of an M/M/1 Retrial Queueing System with Balking, Dissatisfied Customers, and Server Vacations

This study considers a single server retrial queueing system with dissatisfied customers, server vacations, and balking customers. When new arrivals clients see an idle server available, they immediately enter service. If a new arrival sees the server busy, that arrival may balk and leave the system. Otherwise, the new arrival will enter a retrial orbit and try again later. Customers who are dissatisfied with the service will be served again until the service is completed successfully. The server begins a vacation when the current customer completes service. Birth and death state transitions are given. Steady-state distributions for queue length are found. Furthermore, a recursive technique is used to evaluate probabilities. Several performance measures of the system are identified and effects of changes to parameters are observed.

Technical Note—Asymptotically Optimal Control of Omnichannel Service Systems with Pick-up Guarantees

Both walk-in and online customers appear in many service systems such as restaurants. Online customers are often given a target time for pick up, and the quality of the food/beverage can degrade if it is ready before the arrival of online customers. This distinctive feature brings essential difficulties in the analysis and control of such systems. In “Technical Note: Asymptotically Optimal Control of Omnichannel Service Systems with Pick-up Guarantees,” Gao, Huang, and Zhang study the optimal control problem of such service systems based on a two-class single server queueing model. The paper proposes a nearly-optimal policy that keeps the server idle when the queue of online customers drops below a threshold and there are no walk-in customers.

Dispatcher I-queue Optimization in Join Idle Queue Algorithm for Improved Load Balancing in Cloud Computing

Objectives: Modern data centers serving web and mobile applications employ distributed load balancers. The Join Idle Queue (JIQ) algorithm is ideally suited for load balancing in a distributed setup. It attains fast response time by directing service requests to idle servers which can immediately process them. However, JIQ is not optimal in tracking idle servers leaving room for improvement.&#x0D; Methods: We observed that JIQ assigns idle servers non-uniformly to load balancers leaving some load balancers with no access to idle servers. We propose a variant of the JIQ algorithm, Join Idle Queue Dispatcher I-queue Optimization (JIQ-DIO), which increases the probability of load balancers having access to idle servers without additional communication overhead leading to improved response time.&#x0D; Findings: We simulated JIQ-DIO on CloudSim Plus 3.0 and compared it with standard JIQ and its different variants. JIQ-DIO was found to increase the probability of incoming requests being directed to idle servers and lead to more than two-fold improvement in response time across a broad range of parameters.

Prioritized Energy Efficient Resource Scheduling in Cloud Computing

Resource scheduling in cloud computing is one of the impactful area of research. Cloud service providers maintain its efficacy through proper resource management schemes. Users experience seamless cloud services when cloud service provider manages its resources efficiently. Another aspect in resource management is energy efficient schemes. Energy efficiency largely depends on employment of minimum number of resource servers. This article discusses energy efficient resource scheduling mechanism on multi-layer prioritized resource requests. Resource requests are segregated into three categories and those are scheduled on resource servers based on their fast availability. The work focuses on scheduling resource requests on previously active servers as much as possible before activating any idle servers. Proposed scheduling scheme has been tested on a standard benchmark datasets and performance of the proposed method establishes its efficacy.*

Erlang loss systems with shortest idle server first service discipline: Maintenance considerations

We consider a variation of an Erlang loss system in which jobs are routed to servers according to the Shortest Idle Server First service discipline. Specifically, we consider a system in which idle servers are arranged in a stack; servers are returned to the top of the stack upon service completion; and arriving jobs are assigned to the server currently at the top of the stack. When busy, servers accumulate age and incur an age-dependent operating cost. For such systems, we (i) formulate a continuous-time Markov chain model to characterize the system’s transient behavior, and (ii) develop maintenance policies consisting of two possible actions: server group replacement and stack inversion. The stack inversion may be performed at any time prior to group replacement to achieve a more evenly distributed utilization among servers. We develop an optimization model to determine the optimal inversion and replacement times so as to minimize the long-run expected cost rate. Because the model is nonlinear and non-convex, we develop a set of algorithms to solve for the optimal replacement and inversion time. Lastly, we establish a lower bound for the inversion cost threshold below which it is optimal to invert the stack of servers before their replacement.

ON THE M/G/1 FEEDBACK RETRIAL QUEUEING WITH ORBITAL SEARCH OF CUSTOMERS

We consider a single server feedback retrial queueing system with orbital search. An arriving customer finding the server idle enters into service immediately; otherwise, the customer enters into an orbit. The service times are supposed to be arbitrarily distributed. An orbiting customer competes for service, the inter-retrial times are exponentially distributed. Upon completion of a service, the server stays idle for a while, waiting for either a new job or a job from the buffer. After the idle period, the server starts searching for blocked customers, according to the exponential distribution. During the searching time, the server cannot serve a customer. After the searching time the server gets a customer from the orbit if any, otherwise it stays idle again. After completing, the customer has an option to join the orbit as a feedback , customer or to leave the system forever. Various performance measures are derived, and numerical results are given.

Uniform stability of some large-scale parallel server networks

In this paper we study the uniform stability properties of two classes of parallel server networks with multiple classes of jobs and multiple server pools of a tree topology. These include a class of networks with a single nonleaf server pool, such as the ‘N’ and ‘M’ models, and networks of any tree topology with class-dependent service rates. We show that with \(\sqrt{n}\) safety staffing, and no abandonment, in the Halfin–Whitt regime, the diffusion-scaled controlled queueing processes are exponentially ergodic and their invariant probability distributions are tight, uniformly over all stationary Markov controls. We use a unified approach in which the same Lyapunov function is used in the study of the prelimit and diffusion limit. A parameter called the spare capacity (safety staffing) of the network plays a central role in characterizing the stability results: the parameter being positive is necessary and sufficient that the limiting diffusion is uniformly exponentially ergodic over all stationary Markov controls. We introduce the concept of “system-wide work conserving policies," which are defined as policies that minimize the number of idle servers at all times. This is stronger than the so-called joint work conservation. We show that, provided the spare capacity parameter is positive, the diffusion-scaled processes are geometrically ergodic and the invariant distributions are tight, uniformly over all “system-wide work conserving policies." In addition, when the spare capacity is negative we show that the diffusion-scaled processes are transient under any stationary Markov control, and when it is zero, they cannot be positive recurrent.

Increasing social welfare with delays: Strategic customers in the M/G/1 orbit queue

Strategic customers typically patronize service systems at a higher rate than the socially optimal one. Much literature has focused on inducing customers to join such systems at this latter rate. This entire literature considers nonidling policies that are the focus of queueing theory. We demonstrate that strategically imposing delays into service systems can improve the social welfare using the M/G/1 queue with orbit. This versatile queueing model has been extensively studied from a performance evaluation perspective. In this system, customers who arrive and find the server idle begin immediately their service. However, strategic customers who find the server busy decide whether to balk or join a virtual queue, that is, an orbit. Then, each time the server finishes a service, he begins to retrieve a customer from the orbit and the corresponding retrieving time is not negligible. These retrieving times function as extra delays that are imposed on customers that find a busy server. We show that when customers are strategic, there are certain ranges of the parameters where delaying the orbit customers can increase the welfare of a system or even maximize it. To this end, we characterize and compute the equilibrium strategies for the customers' joining/balking dilemma. We consider both the unobservable and observable versions of the system, and provide some insight on the optimal delay and level of information in such systems. We further show that the welfare for this system is higher than the corresponding standard M/G/1 queue with the same delay.

Improve the Energy Efficiency of Datacenters With the Awareness of Workload Variability

In modern datacenters, huge energy consumption is a significant problem that remains to be solved. Previous works reduce the system energy consumption by switching the idle servers to a low-power state. However, the workload demands on servers change dynamically and mainly depend on the real-time workload status. To maintain the system energy efficiency, when providing servers with some servers reserved, the status of workloads should be carefully considered. Generally, the status of workloads is characterized by some key factors sampled from the workloads. However, under dynamic workload demands, the accuracy of these sampled values varies. To accurately assess the status of server workloads, in this paper, we propose a Dynamic Time Scale based server Provision (DTSP) method, which takes the variability of workloads into consideration when providing servers for workload demands. To obtain accurate factor values indicating real-time workload, DTSP samples several key workload factors, including the coefficient of variation of arrival intervals, the request arrival rates of current workload and previous workload, and the mean service time of current requests, with a dynamic compatible rate. With these sampled factors, DTSP can accurately estimate the demands of workloads on servers and provide appropriate numbers of servers for the dynamic workloads. Extensive experiments demonstrate that taking the workload variability into consideration, DTSP can significantly promote the energy efficiency of a datacenter.

Balanced routing with partial information in a distributed parallel many-server queueing system

We consider a queueing system that is comprised of multiple stations serving a single class of customers. Each station consists of many statistically identical servers, and has its own dedicated queue. In this paper, we propose a family of generalized load-balancing routing policies parameterized by vectors c and d with ξ-choice (abbreviated as LB(ξ,c,d)), that are suitable for dynamic routing with partial state information. Under the proposed policy, upon an arrival, a subset of ξ stations will be randomly collected, with their state information being retrieved, where ξ is a positive integer-valued random variable satisfying P{ξ≥2}>0; then, this new customer will be routed to: (i) the station with minimum expected delay modified by c when all the collected stations are fully occupied; or otherwise (ii) the station with maximum idleness ratio modified by d among the collected stations with idle servers. The parameters c and d are employed to control the relative expected delay and utilization levels across stations. Using asymptotic analysis, we derive diffusion limits of queue-length processes as well as their stationary distributions under the LB(ξ,c,d) policy in the Halfin–Whitt regime. Based on these stationary diffusion limit results, we develop a solution procedure to determine policy parameters under different optimization criteria. Finally, we provide numerical experiments to validate the accuracy of our diffusion approximation, and compare the performances under the LB(ξ,c,d) policies with other routing policies.

Probability-Based Online Algorithm for Switch Operation of Energy Efficient Data Center

The huge amount of energy consumed by the data centers around the world every year motivates the cloud service providers to operate data centers in a more energy efficient way. A promising solution is to turn off the idle servers, which, however, may be turned on later, incurring a significant startup cost. The problem turns to dynamically provisioning the workload, and cutting down the energy cost which includes the power to support the running of data center and the startup cost. Different from previous studies that usually consider the worst case performance guarantee when designing online algorithms, this paper considers the average case which is more practical. We propose a simple online algorithm based on the expectation of job interval of workload, which is proven to be optimal for exponential and uniform distributions and achieves tight competitive ratio $\frac{e}{e-1}$ and $\frac{4}{3}$ respectively for them. Simulations using the synthetic data verify our theoretical analysis. Numerical results employing Google's data center workload trace demonstrate that the proposed algorithm outperforms the worst case based algorithm in terms of operation cost reduction.