Load Balancing Policies Research Articles

Load balancing for heterogeneous serverless edge computing: A performance-driven and empirical approach

Serverless edge systems simplify the deployment of real-time AI-based Internet of Things (IoT) applications at the edge. However, the heterogeneity of edge computing nodes – in terms of both hardware and software – makes load balancing challenging in these systems. In this paper, we propose a performance-driven, empirical weight-tuning approach to achieve effective load balancing based on the characteristics and capabilities of the nodes. By extensively profiling the nodes, we gather knowledge on performance metrics such as throughput, energy efficiency, response time, AI accuracy, and cost. Using this acquired knowledge, we introduce a weighted round-robin strategy to optimize the performance metrics according to their observed significance. To address multiple objectives, we introduce a multi-objective method that aims to strike a balance between any arbitrary set of performance objectives simultaneously. Additionally, we explore a coordinated distributed approach to overcome the limitations of centralized load balancing. Next, we introduce Hedgi, a heterogeneous serverless edge architecture designed to efficiently configure and utilize the derived load balancing policies, validated empirically. To demonstrate the practicality of Hedgi, we containerize and serverlessize a real-time object detection application. Extensive empirical studies are conducted using Hedgi to evaluate the performance of the proposed load balancing approach. The results provide valuable insights into the design trade-offs of various load balancing policies and system designs in the heterogeneous serverless edge.

Load balancing policies without feedback using timed replicas

Dispatching policies such as join the shortest queue (JSQ), join the queue with smallest workload (JSW), and their power of two variants are used in load balancing systems where the instantaneous queue length or workload information at all queues or a subset of them can be queried. In situations where the dispatcher has an associated memory, one can minimize this query overhead by maintaining a list of idle servers to which jobs can be dispatched. Recent alternative approaches that do not require querying such information include the cancel-on-start and cancel-on-complete replication policies. The downside of such policies however is that the servers must communicate either the start or the completion time instant of each service to the dispatcher and must allow the coordinated and instantaneous cancellation of all redundant replicas. In practice, the requirements of query messaging, memory, and replica cancellation pose challenges in their implementation and their advantages are not clear. In this work, we consider load-balancing policies that do not need to query load information, do not need memory, and do not need to cancel replicas. Our policies allow the dispatcher to append a timer to each job or its replica. A job or a replica is discarded if its timer expires before it starts receiving service. We analyze several variants of this policy which are novel and simple to implement. We numerically observe that the variants of the proposed policy outperform popular feedback-based policies for low arrival rates, despite no feedback from servers to the dispatcher.

Performance of Load Balancers With Bounded Maximum Queue Length in Case of Non-Exponential Job Sizes

In large-scale distributed systems, balancing the load in an efficient way is crucial in order to achieve low latency. Recently, some load balancing policies have been suggested which are able to achieve a bounded maximum queue length in the large-scale limit. However, these policies have thus far only been studied in case of exponential job sizes. As job sizes are more variable in real systems, we investigate how the performance of these policies (and in particular the value of these bounds) is impacted by the job size distribution. We present a unified analysis which can be used to compute the bound on the queue length in case of phase-type distributed job sizes for four load balancing policies. We find that in most cases, the bound on the maximum queue length can be expressed in closed form. In addition, we obtain job size (in)dependent bounds on the expected response time. Our methodology relies on the use of the cavity process. That is, we conjecture that the cavity process captures the behaviour of the real system as the system size grows large. For each policy, we illustrate the accuracy of the cavity process by means of simulation.

Load Balancing in Compute Clusters With Delayed Feedback

Load balancing arises as a fundamental problem, underlying the dimensioning and operation of many computing and communication systems, such as job routing in data center clusters, multipath communication, Big Data and queueing systems. In essence, the decision-making agent maps each arriving job to one of the possibly heterogeneous servers while aiming at an optimization goal such as load balancing, low average delay or low loss rate. One main difficulty in finding optimal load balancing policies here is that the agent only partially observes the impact of its decisions, e.g., through the delayed acknowledgements of the served jobs. In this paper, we provide a partially observable (PO) model that captures the load balancing decisions in parallel buffered systems under limited information of delayed acknowledgements. We present a simulation model for this PO system to find a load balancing policy in real-time using a scalable Monte Carlo tree search algorithm. We numerically show that the resulting policy outperforms other limited information load balancing strategies such as variants of Join-the-Most-Observations and has comparable performance to full information strategies like: Join-the-Shortest-Queue, Join-the-Shortest-Queue(d) and Shortest-Expected-Delay. Finally, we show that our approach can optimise the real-time parallel processing by using network data provided by Kaggle.

Dynamic Weighted Round Robin Approach in Software-Defined Networks Using Pox Controller

Load balancing is important in solving over-load traffic problems in the network. Therefore, it has been among the first appealing applications in Software Defined Networking (SDN) networks. Numerous SDN-based load-balancing approaches have been recommended to enhance the performance of SDN networks. However, network control could be more manageable in large networks with hundreds of switches and routers. The SDN is a unique way of building, controlling, and developing networks to modify this unpleasant situation. The major concept of SDN contains logically centralizing network management in an SDN controller, which manages and observes the behaviour of the network. Numerous load-balancing approaches are known, such as Round Robin (RR), random policy, Weighted randomized policy (WRP), etc. Every load-balancing policy approach has some benefits and detriments. This paper developed an advanced load-balancing algorithm, a dynamic weighted round-robin (DWRR), and ran it on the top of the SDN controller. Then we calculate the result of our proposed load-balancing approach by comparing it with the current round-robin (RR) and weighted round-robin (WRR) approaches. Mininet tool is utilized for the investigation, and the controller utilized as the control plane is named the POX controller.

Learning-based joint service caching and load balancing for MEC blockchain networks

Integrating the blockchain technology into mobile-edge computing (MEC) networks with multiple cooperative MEC servers (MECS) providing a promising solution to improving resource utilization, and helping establish a secure reward mechanism that can facilitate load balancing among MECS. In addition, intelligent management of service caching and load balancing can improve the network utility in MEC blockchain networks with multiple types of workloads. In this paper, we investigate a learning-based joint service caching and load balancing policy for optimizing the communication and computation resources allocation, so as to improve the resource utilization of MEC blockchain networks. We formulate the problem as a challenging long-term network revenue maximization Markov decision process (MDP) problem. To address the highly dynamic and high dimension of system states, we design a joint service caching and load balancing algorithm based on the double-dueling Deep Q network (DQN) approach. The simulation results validate the feasibility and superior performance of our proposed algorithm over several baseline schemes.

On the Analysis and Evaluation of Proximity-based Load-balancing Policies

Distributed load balancing is the act of allocating jobs among a set of servers as evenly as possible. The static interpretation of distributed load balancing leads to formulating the load-balancing problem as a classical balls-and-bins problem with jobs (balls) never leaving the system and accumulating at the servers (bins). While most of the previous work in the static setting focus on studying the maximum number of jobs allocated to a server or maximum load , little importance has been given to the implementation cost , or the cost of moving a job/data to/from its allocated server, for such policies. This article designs and evaluates server proximity aware static load-balancing policies with a goal to reduce the implementation cost . We consider a class of proximity aware Power of Two (POT) choice-based assignment policies for allocating jobs to servers, where both jobs and servers are located on a two-dimensional Euclidean plane. In this framework, we investigate the tradeoff between the implementation cost and load-balancing performance of different allocation policies. To this end, we first design and evaluate a Spatial Power of two (sPOT) policy in which each job is allocated to the least loaded server among its two geographically nearest servers. We provide expressions for the lower bound on the asymptotic expected maximum load on the servers and prove that sPOT does not achieve classical POT load-balancing benefits. However, experimental results suggest the efficacy of sPOT with respect to expected implementation cost. We also propose two non-uniform server sampling-based POT policies that achieve the best of both implementation cost and load-balancing performance. We then extend our analysis to the case where servers are interconnected as an n -vertex graph G(S, E) . We assume each job arrives at one of the servers, u, chosen uniformly at random from the vertex set S. We then assign each job to the server with minimum load among servers u and v where v is chosen according to one of the following two policies: (i) Unif-POT( k ): Sample a server v uniformly at random from k -hop neighborhood of u; (ii) InvSq-POT( k ): Sample a server v from k -hop neighborhood of u with probability proportional to the inverse square of the distance between u and v . An extensive simulation over a wide range of topologies validates the efficacy of both the policies. Our simulation results show that both policies consistently produce a load distribution that is much similar to that of a classical POT. Depending on topology, we observe the total variation distance to be of the order of 0.002–0.08 for both the policies while achieving a 8%–99% decrease in implementation cost as compared to the classical POT.

Utility Maximizing Load Balancing Policies

Consider a service system where incoming tasks are instantaneously dispatched to one out of many heterogeneous server pools. Associated with each server pool is a concave utility function that depends on the class of the server pool and its current occupancy. We derive an upper bound for the mean normalized aggregate utility in stationarity and introduce two load balancing policies that achieve this upper bound in a large-scale regime. Furthermore, the transient and stationary behavior of these asymptotically optimal load balancing policies is characterized on the scale of the number of server pools in the same large-scale regime. Funding: This work was supported by the Netherlands Organization for Scientific Research (NWO) through [Gravitation Grant NETWORKS-024.002.003] and [Gravitation Grant Vici 202.068]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/stsy.2022.0103 .

Power-of-two sampling in redundancy systems: The impact of assignment constraints

A classical sampling strategy for load balancing policies is power-of-two, where any server pair is sampled with equal probability. This does not cover practical settings with assignment constraints which force non-uniform sampling. While intuition suggests that non-uniform sampling adversely impacts performance, this was only supported through simulations, and rigorous statements have remained elusive. Building on product-form distributions for redundancy systems, we prove the stochastic dominance of uniform sampling for a four-server system as well as arbitrary-size systems in light traffic.

Multi-Link Operation in IEEE 802.11be WLANs

The multi-link operation (MLO) is a new feature proposed to be part of the IEEE 802.11be Extremely High Throughput (EHT) amendment. Such feature represents a paradigm shift towards multi-link communications, as nodes will be allowed to transmit and receive data over multiple radio interfaces concurrently. To make it possible, the 802.11be Task Group has proposed different modifications in regards to nodes' architecture, transmission operation, and management functionalities. This article reviews such changes and tackles the question of how traffic should be distributed over multiple links, as it is still unresolved. To that end, we evaluate different load balancing strategies over the active links. Results show that in high load, dense and complex scenarios, implementing congestion aware load balancing policies to significantly enhance next-generation WLAN performance using MLO is a must.

Joint Parallel Offloading and Load Balancing for Cooperative-MEC Systems With Delay Constraints

Mobile-edge computing (MEC) has been recognized as a promising solution to provide efficient communication and computation capabilities for mobile users (MUs). However, the problem relating to parallel offloading and load balancing with multiple cooperative MEC servers and massive delay-sensitive execution workloads remains to be investigated. In this paper, we study a joint parallel offloading and load balancing policy for such an MEC system. We formulate a long-term system average-cost (i.e., weighted sum of energy consumption and execution delays) stochastic programming problem under MU battery level stability and delay constraints. Our aims include optimizing the data sizes for uplink offloading transmissions for optimized communication and computation resource allocation among multi-users, and maximizing the computation capability utilization of cooperative MEC servers by load balancing. To solve this problem, we first design a Lyapunov-based centralized cost management algorithm (LYP-CCMA) to obtain the optimal system average-cost under the battery level stability constraints. Further, we propose two algorithms based on alternating direction method of multipliers (ADMM) to implement distributed resources allocation. Simulation results verify our analysis and demonstrate the superior performance of our proposed schemes over several baseline schemes.

Improved Load Balancing in Large Scale Systems Using Attained Service Time Reporting

Our interest lies in load balancing jobs in large scale systems consisting of multiple dispatchers and FCFS servers. In the absence of any information on job sizes, a popular load balancing method is the SQ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$d$ </tex-math></inline-formula> ) policy, which uses queue length information reported by the servers to assign incoming jobs. When job sizes are highly variable, using only queue length information is clearly suboptimal and performance can be improved if some indication can be provided to the dispatcher about the size of an ongoing job. In a FCFS server measuring the attained service time of the ongoing job is easy and servers can therefore report this attained service time together with the queue length when queried by a dispatcher. In this paper, we propose and analyse a variety of load balancing policies which improve the SQ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$d$ </tex-math></inline-formula> ) policy by exploiting both the queue length and attained service time to assign jobs, as well as policies for which only the attained service time of the job in service is used. We present a unified analysis for all these policies in a large scale system under the usual asymptotic independence assumptions. The accuracy of the proposed analysis is illustrated using simulation. We present extensive numerical experiments which clearly indicate that a significant improvement in waiting (and thus also in response) time may be achieved by using the attained service time information on top of the queue length of a server. Moreover, the policies which do not make use of the queue length still provide an improved waiting time for moderately loaded systems.

Effectively Handling Network Congestion and Load Balancing in Software-Defined Networking

The concept of Software-Defined Networking (SDN) evolves to overcome the drawbacks of the traditional networks with Internet Protocol (I.P.) packets sending and packets handling. The SDN structure is one of the critical advantages of efficiently separating the data plane from the control plane to manage the network configurations and network management. Whenever there are multiple sending devices inside the SDN network, the OpenFlow switches are programmed to handle the limited number of requests for their interface. When the recommendations are exceeded from the specific threshold, the load on the switches also increases. This research article introduces a new approach named LBoBS to handle load balancing by adding the load balancing server to the SDN network. Besides, it is used to maximize SDN’s reliability and efficiency. It also works in coordination with the controller to effectively handle the load balancing policies. The load balancing server is implemented to manage the switches load effectively. Results are evaluated on the NS-3 simulator for packet delivery, bandwidth utilization, latency control, and packet decision ratios on the OpenFlow switches. It has been found that the proposed method improved SDN’s load balancing by 70% compared to the previous state-of-the-art methods.

Performance analysis of load balancing policies with memory

Joining the shortest or least loaded queue among d randomly selected queues are two fundamental load balancing policies. Under both policies the dispatcher does not maintain any information on the queue length or load of the servers. In this paper we analyze the performance of these policies when the dispatcher has some memory available to store the ids of some of the idle servers. We consider methods where the dispatcher discovers idle servers as well as methods where idle servers inform the dispatcher about their state.We focus on large-scale systems and our analysis uses the cavity method. The main insight provided is that the performance measures obtained via the cavity method for a load balancing policy with memory reduce to the performance measures for the same policy without memory provided that the arrival rate is properly scaled. Thus, we can study the performance of load balancers with memory in the same manner as load balancers without memory. In particular this entails closed form solutions for joining the shortest or least loaded queue among d randomly selected queues with memory in case of exponential job sizes. Moreover, we obtain a simple closed form expression for the (scaled) expected waiting time as the system tends towards instability.We present simulation results that support our belief that the approximation obtained by the cavity method becomes exact as the number of servers tends to infinity.

Proposing a Load Balancing Algorithm For Cloud Computing Applications

At the current time, one of the fields that are quickly developing in computing is clouding computing (CC). There are vital services needed by individuals as well as organizations that are offered in CC, namely IaaS, SaaS, and PaaS. Through the creation of the cloud, the development of the application has become easier, as well as providing services to end-users through virtualization over the internet. In CC, we have cloud service providers who deal with large computing structures completely defined on usage as they offer their services in a dependable mode. However; CC is facing the issue of load balancing, which impacts the performance and makes it weak if we overload the system. Developing an effective load balancing algorithm while utilizing CC efficiently is one of the providers’ ultimate goal. Virtualization and scalability are other dynamic techniques which entail developing VMs in CC. Data traffic and Web services provision is growing each day; thus, load balancing is a major challenge in the field of CC and is increasing the importance of task scheduling as well. At this point in time, evolvement in CC necessitates the greater need for infrastructure and resource development. The load balancing policy guarantees the efficient usage of the resource through the provision of services to the cloud subscribers. Load balancing also prioritizes subscribers by using an appropriate schedule. As a contribution in research, this paper proposes an algorithm for load balancing in CC. Proposed algorithm enhance the performance of Cloud Computing applications

Mean Waiting Time in Large-Scale and Critically Loaded Power of d Load Balancing Systems

Mean field models are a popular tool used to analyse load balancing policies. In some exceptional cases the waiting time distribution of the mean field limit has an explicit form. In other cases it can be computed as the solution of a differential equation or a recursive relation. In our paper [6] we study the limit of the mean waiting time E[W&amp;955;] as the arrival rate &amp;955; approaches 1 for a number of load balancing policies when job sizes are exponential with mean 1 (i.e. when the system gets close to instability). As E[W_&amp;955;] diverges to infinity, we scale with -log(1-&amp;955;) and present a method to compute the limit lim&amp;955;-&gt; 1- -E[W&amp;955;]/log(1-&amp;955;). We show that this limit has a surprisingly simple form for the load balancing algorithms considered. In addition, we propose an alternate scaling -log(p&amp;955;) instead of -log(1-&amp;955;), where p&amp;955; is adapted to the policy at hand. This scaling allows us to obtain good approximations of the mean waiting time when &amp;955; is close to one or zero.

Mean Waiting Time in Large-Scale and Critically Loaded Power of d Load Balancing Systems

Mean field models are a popular tool used to analyse load balancing policies. In some exceptional cases the waiting time distribution of the mean field limit has an explicit form. In other cases it can be computed as the solution of a set of differential equations. In this paper we study the limit of the mean waiting time E[Wλ] as the arrival rate λ approaches 1 for a number of load balancing policies in a large-scale system of homogeneous servers which finish work at a constant rate equal to one and exponential job sizes with mean 1 (i.e. when the system gets close to instability). As E[Wλ] diverges to infinity, we scale with -log(1-λ) and present a method to compute the limit limλ-&gt; 1- -E[Wλ]/l(1-λ). We show that this limit has a surprisingly simple form for the load balancing algorithms considered. More specifically, we present a general result that holds for any policy for which the associated differential equation satisfies a list of assumptions. For the well-known LL(d) policy which assigns an incoming job to a server with the least work left among d randomly selected servers these assumptions are trivially verified. For this policy we prove the limit is given by 1/d-1. We further show that the LL(d,K) policy, which assigns batches of K jobs to the K least loaded servers among d randomly selected servers, satisfies the assumptions and the limit is equal to K/d-K. For a policy which applies LL(di) with probability pi, we show that the limit is given by 1/ ∑i pi di - 1. We further indicate that our main result can also be used for load balancers with redundancy or memory. In addition, we propose an alternate scaling -l(pλ) instead of -l(1-λ), where pλ is adapted to the policy at hand, such that limλ-&gt; 1- -E[Wλ]/l(1-λ)=limλ-&gt; 1- -E[Wλ]/l(pλ), where the limit limλ-&gt; 0+ -E[Wλ]/l(pλ) is well defined and non-zero (contrary to limλ-&gt; 0+ -E[Wλ]/l(1-λ)). This allows to obtain relatively flat curves for -E[Wλ]/l(pλ) for λ ∈ [0,1] which indicates that the low and high load limits can be used as an approximation when λ is close to one or zero. Our results rely on the earlier proven ansatz which asserts that for certain load balancing policies the workload distribution of any finite set of queues becomes independent of one another as the number of servers tends to infinity.

Complete resource pooling of a load-balancing policy for a network of battery swapping stations

To reduce carbon emission in the transportation sector, there is currently a steady move taking place to an electrified transportation system. This brings about various issues for which a promising solution involves the construction and operation of a battery swapping infrastructure rather than in-vehicle charging of batteries. In this paper, we study a closed Markovian queueing network that allows for spare batteries under a dynamic arrival policy. We propose a provisioning rule for the capacity levels and show that these lead to near-optimal resource utilization, while guaranteeing good quality-of-service levels for electric vehicle users. Key in the derivations is to prove a state-space collapse result, which in turn implies that performance levels are as good as if there would have been a single station with an aggregated number of resources, thus achieving complete resource pooling.

A power and thermal-aware virtual machine management framework based on machine learning

Energy consumption in data centers grows rapidly in recent years. As a widely-applied energy-efficient method, workload consolidation also has its own limitations that may bring some negative effects, such as performance degradation, QoS violation, localized hotspots and so on, which is especially true when optimal objectives are inherently conflict. In this paper, we present a power and thermal-aware VM management framework called PTM-ML, which relies on machine learning technique to find optimal host configuration based on workload characteristics and cooling system’s working state. Based on such an optimal host configuration, it then makes VM migration and consolidation decisions by enforcing an efficient load-balancing policy, with aiming at achieving a better trade-off between energy efficiency and performance. The prototype of PTM-ML framework is deployed and evaluated in a real-world cloud data center. Extensive experiments are conducted by using different workload traces with distinctive characteristics, and the results are compared with four similar approaches in terms of total energy consumption, real-time power consumption, average latency and etc. Experimental results show that the proposed PTM-ML outperforms the existing approaches in terms of multiple metrics, and it also exhibits better robustness and adaptability in presence of dynamic workloads.

Steady‐state analysis of load balancing with Coxian‐2 distributed service times

AbstractThis paper studies load balancing for many‐server (N servers) systems. Each server has a buffer of size b − 1, and can have at most one job in service and b − 1 jobs in the buffer. The service time of a job follows the Coxian‐2 distribution. We focus on steady‐state performance of load balancing policies in the heavy traffic regime such that the normalized load of system is λ = 1 − N−α for 0 &lt; α &lt; 0.5. We identify a set of policies that achieve asymptotic zero waiting. The set of policies include several classical policies such as join‐the‐shortest‐queue (JSQ), join‐the‐idle‐queue (JIQ), idle‐one‐first (I1F) and power‐of‐d‐choices (Po d) with d = O(Nα log N). The proof of the main result is based on Stein's method and state space collapse. A key technical contribution of this paper is the iterative state space collapse approach that leads to a simple generator approximation when applying Stein's method.