Load Balancing Policies Research Articles

Scalable Load Balancing in the Presence of Heterogeneous Servers

In large-scale computer systems, deciding how to dispatch arriving jobs to servers is a primary factor affecting system performance. Consequently, there is a wealth of literature on designing, analyzing, and evaluating the performance of load balancing policies. For analytical tractability, most existing work on dispatching in large-scale systems makes a key assumption: that the servers are homogeneous, meaning that they all have the same speeds, capabilities, and available resources. But this assumption is not accurate in practice. Modern computer systems are instead heterogeneous: server farms may consist of multiple generations of hardware, servers with varied resources, or even virtual machines running in a cloud environment. Given the ubiquity of heterogeneity in today's systems, it is critically important to develop load balancing policies that perform well in heterogeneous environments. In this paper, we focus on systems in which server speeds are heterogeneous.

Disruption‐Free Load Balancing for Aerial Access Network

A fundamental issue of 6G networks with aerial access networks (AAN) as a core component is that user devices will send high‐volume traffic via AAN to backend servers. As such, it is critical to load balance such traffic such that it will not cause network congestion or disruption and affect users’ experience in 6G networks. Motivated by the success of software‐defined networking‐based load balancing, this paper proposes a novel system called Tigris, to load balance high‐volume AAN traffic in 6G networks. Different from existing load balancing solutions in traditional networks, Tigris tackles the fundamental disruption-resistant challenge in 6G networks for avoiding disruption of continuing flows and the control-path update challenge for limiting the throughput of updating load balancing instructions. Tigris achieves disruption‐free and low‐control‐path‐cost load balancing for AAN traffic by developing an online algorithm to compute disruption‐resistant, per‐flow load balancing policies and a novel bottom‐up algorithm to compile the per‐flow policies into a highly compact rule set, which remains disruption‐resistant and has a low control‐path cost. We use extensive evaluation to demonstrate the efficiency and efficacy of Tigris to achieve zero disruption of continuing AAN flows and an extremely low control‐path update overhead, while existing load balancing techniques in traditional networks such as ECMP cause high load variance and disrupt almost 100% continuing AAN flows.

A lower bound on the stability region of redundancy-[formula omitted] with FIFO service discipline

Due to the complex dynamics of systems with redundancy and cancellations, existing results on the stability region of the Redundancy-d (R(d)) load balancing policy are scarce and are limited to very special cases with respect to the joint service time distribution of tasks. We provide a non-trivial, closed form lower bound on the stability region the R(d) for a general joint service time distribution. While simulation results indicate our bound is not tight, it provides an easy-to-check performance guarantee.

Generalized Cost-Based Job Scheduling in Very Large Heterogeneous Cluster Systems

We study job assignment in large, heterogeneous resource-sharing clusters of servers with finite buffers. This load balancing problem arises naturally in today's communication and big data systems, such as Amazon Web Services, Network Service Function Chains, and Stream Processing. Arriving jobs are dispatched to a server, following a load balancing policy that optimizes a performance criterion such as job completion time. Our contribution is a randomized Cost-Based Scheduling (CBS) policy in which the job assignment is driven by general cost functions of the server queue lengths. Beyond existing schemes, such as the Join the Shortest Queue (JSQ), the power of d or the SQ(d) and the capacity-weighted JSQ, the notion of CBS yields new application-specific policies such as hybrid locally uniform JSQ. As today's data center clusters have thousands of servers, exact analysis of CBS policies is tedious. In this article, we derive a scaling limit when the number of servers grows large, facilitating a comparison of various CBS policies with respect to their transient as well as steady state behavior. A byproduct of our derivations is the relationship between the queue filling proportions and the server buffer sizes, which cannot be obtained from infinite buffer models. Finally, we provide extensive numerical evaluations and discuss several applications including multi-stage systems.

Scalable load balancing in the presence of heterogeneous servers

Heterogeneity is becoming increasingly ubiquitous in modern large-scale computer systems. Developing good load balancing policies for systems whose resources have varying speeds is crucial in achieving low response times. Indeed, how best to dispatch jobs to servers is a classical and well-studied problem in the queueing literature. Yet the bulk of existing work on large-scale systems assumes homogeneous servers; unfortunately, policies that perform well in the homogeneous setting can cause unacceptably poor performance in heterogeneous systems. We adapt the “power-of-d” versions of both the Join-the-Idle-Queue and Join-the-Shortest-Queue policies to design two corresponding families of heterogeneity-aware dispatching policies, each of which is parameterized by a pair of routing probabilities. Unlike their heterogeneity-unaware counterparts, our policies use server speed information both when choosing which servers to query and when probabilistically deciding where (among the queried servers) to dispatch jobs. Both of our policy families are analytically tractable: our mean response time and queue length distribution analyses are exact as the number of servers approaches infinity, under standard assumptions. Furthermore, our policy families achieve maximal stability and outperform well-known dispatching rules – including heterogeneity-aware policies such as Shortest-Expected-Delay – with respect to mean response time.

Load-Balancing Algorithm for Multiple Gateways in Fog-Based Internet of Things

This article investigates the performance of multicriteria-based load-balancing scheme among gateways in fog-assisted Internet of Things (IoT). We employ a queueing model of the IoT system to calculate the latency of data streams from the IoT devices to the applications. However, a gateway node may easily become congested since all the traffic to IP networks are directed toward the gateway. A congested point can affect system performance and may cause reliability problems in the system. Thus, we employ multiple gateways to alleviate the performance degradation of the network, along with a multicriteria decision-making (MCDM)-based load-balancing policy among the gateways to achieve a global load fairness. The proposed model is evaluated in single-hop IPv6 over low-power wireless personal area networks (6LoWPANs). The evaluation results show the effectiveness of the proposed load-balancing model in providing fast and reliable responses to user queries.

Constrained Load-Balancing Policies for Parallel Single-Server Queue Systems

Flow-control policies that balance server loads are well known for improving performance of queueing systems with multiple nodes. However, although load balancing benefits the system overall, it may negatively impact some of the queueing nodes. For example, it may reduce throughput rates or engender unfairness with respect to some performance measures. For queueing systems with multiple single-server nodes, we propose a set of constrained load-balancing policies that ensures the expected arrival rate to each queueing node is not reduced, and we show that such policies provide multiple benefits for each queueing node: stochastically fewer customers and lower variance of the number of customers at each queueing node. These results imply performance improvement as measured by multiple general objective functions, including but not limited to the expected number of customers at a queueing node, probability of having a high number of customers, variance of the number of customers, and expected number of customers conditional on exceeding a threshold defined by a fixed service level. We demonstrate numerically that our proposed policies capture a large portion of the potential maximal improvement. This paper was accepted by Noah Gans, stochastic models and simulation.

Mean Field Analysis of Join-Below-Threshold Load Balancing for Resource Sharing Servers

Load balancing plays a crucial role in many large scale computer systems. Much prior work has focused on systems with First-Come- First-Served (FCFS) servers. However, servers in practical systems are more complicated. They serve multiple jobs at once, and their service rate can depend on the number of jobs in service. Motivated by this, we study load balancing for systems using Limited- Processor-Sharing (LPS). Our model has heterogeneous servers, meaning the service rate curve and multiprogramming level (limit on the number of jobs sharing the processor) differs between servers. We focus on a specific load balancing policy: Join-Below-Threshold (JBT), which associates a threshold with each server and, whenever possible, dispatches to a server which has fewer jobs than its threshold. Given this setup, we ask: how should we configure the system to optimize objectives such as mean response time? Configuring the system means choosing both a load balancing threshold and a multiprogramming level for each server. To make this question tractable, we study the many-server mean field regime.

LSQ: Load Balancing in Large-Scale Heterogeneous Systems With Multiple Dispatchers

Nowadays, the efficiency and even the feasibility of traditional load-balancing policies are challenged by the rapid growth of cloud infrastructure and the increasing levels of server heterogeneity. In such heterogeneous systems with many load-balancers, traditional solutions, such as $JSQ$ , incur a prohibitively large communication overhead and detrimental incast effects due to herd behavior. Alternative low-communication policies, such as $JSQ(d)$ and the recently proposed $JIQ$ , are either unstable or provide poor performance. We introduce the Local Shortest Queue ( $LSQ$ ) family of load balancing algorithms. In these algorithms, each dispatcher maintains its own, local, and possibly outdated view of the server queue lengths, and keeps using $JSQ$ on its local view. A small communication overhead is used infrequently to update this local view. We formally prove that as long as the error in these local estimates of the server queue lengths is bounded in expectation , the entire system is strongly stable. Finally, in simulations, we show how simple and stable $LSQ$ policies exhibit appealing performance and significantly outperform existing low-communication policies, while using an equivalent communication budget. In particular, our simple policies often outperform even $JSQ$ due to their reduction of herd behavior. We further show how, by relying on smart servers ( i.e., advanced pull-based communication), we can further improve performance and lower communication overhead.

Dynamic Management of Resources in Cloud Computing

It has already been proven that VMs are over-utilized in the initial stages and are underutilized in the later stages. Due to the random utilization of the CPU, resources are sometimes heavily loaded whereas other resources are idle. Load imbalance causes service level agreement (SLA) violations resulting in poor quality of service (QoS) aided by the imperfect management of resources. An effective load balancing mechanism helps to achieve balanced utilization, which maximizes the throughput, availability, and reliability and reduces the response and migration time. The proposed algorithm can effectively minimize the response and the migration time and maximize reliability, and throughput. This research also helps to understand the load balancing policies and analysis of other research works.

Mean Field Analysis of Join-Below-Threshold Load Balancing for Resource Sharing Servers

Load balancing plays a crucial role in many large scale computer systems. Much prior work has focused on systems with First-Come-First-Served (FCFS) servers. However, servers in practical systems are more complicated. They serve multiple jobs at once, and their service rate can depend on the number of jobs in service. Motivated by this, we study load balancing for systems using Limited-Processor-Sharing (LPS). Our model has heterogeneous servers, meaning the service rate curve and multiprogramming level (limit on the number of jobs sharing the processor) differs between servers. We focus on a specific load balancing policy: Join-Below-Threshold (JBT), which associates a threshold with each server and, whenever possible, dispatches to a server which has fewer jobs than its threshold. Given this setup, we ask: how should we configure the system to optimize objectives such as mean response time? Configuring the system means choosing both a load balancing threshold and a multiprogramming level for each server. To make this question tractable, we study the many-server mean field regime. In this paper we provide a comprehensive study of JBT in the mean field regime. We begin by developing a mean field model for the case of exponentially distributed job sizes. The evolution of our model is described by a differential inclusion, which complicates its analysis. We prove that the sequence of stationary measures of the finite systems converges to the fixed point of the differential inclusion, provided a unique fixed point exists. We derive simple conditions on the service rate curves to guarantee the existence of a unique fixed point. We demonstrate that when these conditions are not satisfied, there may be multiple fixed points, meaning metastability may occur. Finally, we give a simple method for determining the optimal system configuration to minimize the mean response time and related metrics. While our theoretical results are proven for the special case of exponentially distributed job sizes, we provide evidence from simulation that the system becomes insensitive to the job size distribution in the mean field regime, suggesting our results are more generally applicable.

Proposing a load balancing algorithm with an integrative approach to reduce response time and service process time in data centers

Goal: Load balancing policies often map workloads on virtual machines, and are being sought to achieve their goals by creating an almost equal level of workload on any virtual machine. In this research, a hybrid load balancing algorithm is proposed with the aim of reducing response time and processing time.
 Design / Methodology / Approach: The proposed algorithm performs load balancing using a table including the status indicators of virtual machines and the task list allocated to each virtual machine. The evaluation results of response time and processing time in data centers from four algorithms, ESCE, Throttled, Round Robin and the proposed algorithm is done.
 Results: The overall response time and data processing time in the proposed algorithm data center are shorter than other algorithms and improve the response time and data processing time in the data center. The results of the overall response time for all algorithms show that the response time of the proposed algorithm is 12.28%, compared to the Round Robin algorithm, 9.1% compared to the Throttled algorithm, and 4.86% of the ESCE algorithm.
 Limitations of the investigation: Due to time and technical limitations, load balancing has not been achieved with more goals, such as lowering costs and increasing productivity.
 Practical implications: The implementation of a hybrid load factor policy can improve the response time and processing time. The use of load balancing will cause the traffic load between virtual machines to be properly distributed and prevent bottlenecks. This will be effective in increasing customer responsiveness. And finally, improving response time increases the satisfaction of cloud users and increases the productivity of computing resources.
 Originality/Value: This research can be effective in optimizing the existing algorithms and will take a step towards further research in this regard.

Gossip-based fault-tolerant load balancing algorithm with low communication overhead

In large-scale distributed-computing environments, numerous factors may affect the predictable performance of load balancing policies causing variable load distribution in the network and bottlenecks. This paper presents a two-level load balancing algorithm for solving the dynamic load balancing problem considering the loss of processors and connectivity of the network. The proposed method privileges local load balancing over global load balancing thus reduces the communication cost over the global network. To further minimize communication costs and to handle loss of connectivity, the algorithm proposes a new communication model for global communication among the processors. The performance, correctness, and scalability of the algorithm are analyzed which show that our algorithm requires O(log m) rounds and O(m log m) messages for m clusters and achieves minimum cluster and global load deviation. Simulation results also support our analytical studies.

Latency- and capacity-aware placement of chained Virtual Network Functions in FMC metro networks

Most of today's cloud applications are delivered by Cloud Service Providers (CSPs) on top of a physical network managed by one or multiple Infrastructure Providers (InPs). This new way of delivering services is impacting InPs' revenues, as InPs are only responsible for transporting data to users. Network Function Virtualization (NFV) was proposed to help InPs gain more flexibility in provisioning new services over their networks, hence achieving lower capital and operational costs, keeping stable revenue margins, and resisting the competition of CSPs (e.g., the “Over-The-Top”players). NFV aims at moving from the traditional approach of network functions running over dedicated hardware (e.g., firewall, NAT, etc.) into virtualized software modules running on top of Commercial Off The Shelf (COTS) equipment. However, deploying NFV in an operational network requires addressing two fundamental problems. The first consists on determining the locations where Virtual Network Functions (VNFs) will be hosted (i.e., VNF placement) and the second on how to properly steer network traffic to traverse the required VNFs in the right order (i.e, routing), thus provisioning network services in the form of Service Function Chains (SFCs). In this work we try to solve both problems focusing our analysis on a metro-regional scenario, where link bandwidth and COTS node processing capacity is inherently limited and where the current trend consists on moving towards a Fixed and Mobile Convergence (FMC) network infrastructure. We propose and compare different heuristic strategies for SFC provisioning, characterized by latency and/or capacity awareness (i.e., able to best exploit latency of links and/or processing capacity of COTS nodes for an effective placement of VNFs) and by the adoption of a load balancing policy for traffic routing, with the aim of maximally consolidating VNFs. We assess the benefits of our strategies against a state-of-the-art algorithm, both in terms of number of required COTS nodes in the metro/access network and of SFC acceptance ratio. Our findings indicate that combining latency and capacity awareness in the VNF placement process with a load-balancing routing strategy brings high benefits in terms of VNF consolidation and SFC acceptance ratio.

Performance Analysis of Workload Dependent Load Balancing Policies

Load balancing plays a crucial role in achieving low latency in large distributed systems. Recent load balancing strategies often rely on replication or use placeholders to further improve latency. However assessing the performance and stability of these strategies is challenging and is therefore often simulation based. In this paper we introduce a unified approach to analyze the performance and stability of a broad class of workload dependent load balancing strategies. This class includes many replication policies, such as replicate below threshold, delayed replication and replicate only small jobs, as well as strategies for fork-join systems. We consider systems with general job size distributions where jobs may experience server slowdown. We show that the equilibrium workload distribution of the cavity process satisfies a functional differential equation and conjecture that the cavity process captures the limiting behavior of the system as its size tends to infinity. We study this functional differential equation in more detail for a variety of load balancing policies and propose a numerical method to solve it. The numerical method relies on a fixed point iteration or a simple Euler iteration depending on the type of functional differential equation involved. We further show that additional simplifications can be made if certain distributions are assumed to be phase-type. Various numerical examples are included that validate the numerical method and illustrate its strength and flexibility.

A generic formal model for the comparison and analysis of distributed job-scheduling algorithms in grid environment

Nowadays, high-end Systems consist of thousands of individual devices which are in general heterogeneous. Grid computing environments are an example of high-end systems which are composed of many diverse and heterogeneous resources distributed within multiple geographical areas. The performance of such systems depends considerably on job scheduling and resource allocation algorithms. Indeed, to improve the global throughput of these environments, effective and efficient load balancing algorithms are fundamentally important. In this paper, we propose a completely distributed formal model for the description of grid architecture. Our model, is then parameterized to describe different job-scheduling algorithms in a completely parallel architecture. In particular, we focus on the modeling and description of distributed load-balancing algorithms. To reach this purpose, we define a set of parallel schedulers communicating together to achieve a given load-balancing policy. To show the applicability of our approach, we propose to specify and compare different well-known job-scheduling policies in grid environments. The formal verification of different properties of the studied protocols has been performed automatically using Model-checking and a set of performance analysis results are also provided.

Replicate to the shortest queues

This paper introduces a load-balancing policy that interpolates between two well-known policies, namely join the shortest queue (JSQ) and join the least workload (JLW), and studies it in heavy traffic. This policy, which we call replicate to the shortest queues (RSQ(d)), routes jobs from a stream of arrivals into buffers attached to N servers by replicating each arrival into $$1\le d\le N$$ tasks and sending the replicas to the d shortest queues. When the first of the tasks reaches a server, its $$d-1$$ replicas are canceled. Clearly, RSQ(1) is equivalent to JSQ, and it has been shown that RSQ(N) is equivalent to JLW; intermediate values of d provide a trade-off between good performance measures of JSQ and those of JLW. In heavy traffic, a key property underlying asymptotic analysis of load-balancing policies is state space collapse (SSC). Unlike policies such as JSQ, where SSC is well understood, the treatment of SSC under RSQ(d) requires addressing the massive cancellations that highly complicate the queue length dynamics. Our first main result is that SSC holds under RSQ(d) for possibly heterogeneous servers. Based on this result, we obtain diffusion limits for the queue lengths in the form of one-dimensional reflected Brownian motion, asymptotic characterization of the short-time-averaged delay process and a version of Reiman’s snapshot principle. We illustrate using simulations that as d increases the server workloads become more balanced, and the delay distribution’s tail becomes lighter. We also discuss the implementation complexity of the policy as compared to that of the redundancy routing policy, to which it is closely related.

Flexible Load Balancing with Multi-dimensional State-space Collapse

Heavy traffic delay analysis for load balancing policies has relied heavily on a condition called state-space collapse onto a single-dimensional line. In this paper, via Lyapunov driftbased method, we rigorously prove that even under a multidimensional state-space collapse, steady-state heavy-traffic delay optimality can still be achieved for a general load balancing system. This result directly implies that achieving steady-state heavy-traffic delay optimality simply requires that no server is kept idling while others are busy at heavy loads, thus complementing and extending the result obtained by diffusion approximations. Further, we explore the greater flexibility provided by allowing a multidimensional state-space collapse in designing new load balancing policies that are both throughput optimal and heavytraffic delay optimal in steady state.

The Mean-field Behavior of Processor Sharing Systems with General Job Lengths Under the SQ(d) Policy

In this paper, we derive the mean-field behavior of empirical distributions of large systems that consist of N (large) identical parallel processor sharing servers with Poisson arrival process having intensity Nλ and generally distributed job lengths under the randomized SQ(d) load balancing policy. Under this policy, an arrival is routed to the server with the least number of progressing jobs among d randomly chosen servers. The mean-field is then used to approximate the statistical properties of the system. In particular, we show that in the limit as N grows, individual servers are statistically independent of others (propagation of chaos) and more importantly, the equilibrium point of the mean-field is insensitive to the job length distributions. This has important engineering relevance for the robustness of such routing policies that are often used in web server farms. We use a measure-valued process approach and martingale techniques to obtain our results. We also provide numerical results to support our analysis.

Optimization of the Load Balancing Policy for Tiled Many-Core Processors

Tiled many-core processors (i.e., KNL and the TILE-Gx72 processor), on which processing cores are fitted onto a single chip and cores are interconnected via mesh-based networks, are different from the traditional many-core systems. Their operating system (OS) should be optimized to take into account the unique characteristics (for instance, cores are integrated into a single chip) of tiled many-core processors. This is because these characteristics were not taken into consideration when OSes designed for the traditional multicore (many-core) systems were deployed on tiled many-core processors. In this paper, we propose an optimized load balancing policy to improve the performance of multi-threaded applications. Making a thread select an appropriate idle (lightweight) tile (processing core) across all tiles on the single chip rather than a portion of tiles is able to reduce the overhead triggered by the load balancing policy, the penalty of cache misses because of the scheduling and more threads sharing the same tile (processing core), and the contention for memory controllers due to cache misses. The experimental results demonstrate that the optimized load balancing policy can provide up to 2.7× performance improvement on KNL and mitigate the performance degradation to separate extents on the TILE-Gx72 processor.