Online Resource Allocation Problems Research Articles

Dynamic and efficient resource allocation for 5G end‐to‐end network slicing: A multi‐agent deep reinforcement learning approach

SummaryThe rapid evolution of user equipment (UE) and 5G networks drives significant transformations, bringing technology closer to end‐users. Managing resources in densely crowded areas such as airports, train stations, and bus terminals poses challenges due to diverse user demands. Integrating mobile edge computing (MEC) and network function virtualization (NFV) becomes vital when the service provider's (SP) primary goal is maximizing profitability while maintaining service level agreement (SLA). Considering these challenges, our study addresses an online resource allocation problem in an MEC network where computing resources are limited, and the SP aims to boost profit by securely admitting more UE requests at each time slot. Each UE request arrival rate is unknown, and the requirement is specific resources with minimum cost and delay. The optimization problem objective is achieved by allocating resources to requests at the MEC network in appropriate cloudlets, utilizing abandoned instances, reutilizing idle and soft slice instances to shorten delay and reduce costs, and immediately scaling inappropriate instances, thus minimizing the instantiation of new instances. This paper proposes a deep reinforcement learning (DRL) method for request prediction and resource allocation to mitigate unnecessary resource waste. Simulation results demonstrate that the proposed approach effectively accepts network slice requests to maximize profit by leveraging resource availability, reutilizing instantiated resources, and upholding goodwill and SLA. Through extensive simulations, we show that our proposed DRL‐based approach outperforms other state‐of‐the‐art techniques, namely, MaxSR, DQN, and DDPG, by 76%, 33%, and 23%, respectively.

The Online Knapsack Problem with Departures

The online knapsack problem is a classic online resource allocation problem in networking and operations research. Its basic version studies how to pack online arriving items of different sizes and values into a capacity-limited knapsack. In this paper, we study a general version that includes item departures, while also considering multiple knapsacks and multi-dimensional item sizes. We design a threshold-based online algorithm and prove that the algorithm can achieve order-optimal competitive ratios. Beyond worst-case performance guarantees, we also aim to achieve near-optimal average performance under typical instances. Towards this goal, we propose a data-driven online algorithm that learns within a policy-class that guarantees a worst-case performance bound. In trace-driven experiments, we show that our data-driven algorithm outperforms other benchmark algorithms in an application of online knapsack to job scheduling for cloud computing.

Technical Note—Greedy Algorithm for Multiway Matching with Bounded Regret

A Unified View of Online Matching and Resource Allocation Problems Whereas small regret online algorithms for applications as diverse as network revenue management (NRM), assemble-to-order (ATO) systems, and online stochastic bin packing (SBP) are known in the literature, the design of the existing algorithms are tailored to the specific application and often use the strategy of resolving a planning linear program. In the paper “Greedy Algorithm for Multiway Matching with Bounded Regret,” Gupta proposes a unified model for studying such online matching/allocation problems. In the unified model, resources of three types—off-line (e.g., inventory in NRM), online-queueable (e.g., orders or resources in ATO systems), or online-nonqueueable (e.g., requests in NRM, items in SBP)—must be combined to create feasible configurations. Leveraging the unified framework, the author gives one simple greedy algorithm that gives small regret (bounded or logarithmic in time horizon) for these diverse applications under a mild nondegeneracy condition on the off-line planning problem.

Reinforcement learning based monotonic policy for online resource allocation

This research aims to design an optimal and strategyproof mechanism for online resource allocation problems. In such problems, consumers randomly arrive with their resource requests in an arbitrary manner. As a result, there is uncertainty in the future resource demands. In addition, the allocation and payment decisions depend on the providers’ past experiences. To address these challenges, we propose a novel reinforcement learning algorithm for optimising the resource allocation policy. The proposed algorithm adopts a novel monotonic reward shaping function that uses a dominant-resource multi-label classification technique. Finally, a critical payment value is calculated in order to maintain the strategyproofness in the online environment. The experimental evaluations show that the proposed mechanism achieves results that are within 96% of the optimal social welfare while outperforming the other mechanisms that use fixed pricing.

Approximation schemes for the joint inventory selection and online resource allocation problem

In this paper, we introduce and study the joint inventory selection and online resource allocation problem, which is characterized by two sequential sets of decisions that are irrevocably linked. First, a decision maker (DM) must select starting inventory levels for a set of available resources. Subsequently, the DM must match arriving customers to available resources in an online fashion so as to maximize expected reward. We first study the problem in its most general form, before focusing on a specific version that arises at Anheuser Busch InBev (ABI). This particular application of our general setting is referred to as the ABI Trailer Problem, and it considers how ABI ships its beer to vendors via third‐party delivery trucks. In this problem, ABI must select the weights of preloaded trailers of beer, which are then matched in an online fashion to the arriving third‐party delivery trucks. For the general setting, we develop simple and easy‐to‐implement approaches that come with robust worst‐case performance guarantees. For the ABI setting, we reveal a simplifying structural property related to the optimal matching policy, which gives rise to a natural adaptation of our original approach. We test the efficacy of these policies through extensive numerical experiments, where we find that our approaches are either near‐optimal or improve upon state‐of‐the‐art benchmarks. In particular, using a data set from ABI, we are able to generate instances of the ABI Trailer Problem, on which our algorithm has the potential to yield revenue improvements in the range of millions of dollars per year.

DoSRA: A Decentralized Approach to Online Edge Task Scheduling and Resource Allocation

With the proliferation of novel Internet of Things (IoT) mobile applications and advanced communication technologies, nowadays we are surrounded by ubiquitous sensors and smart devices. These smart IoT devices generate a large volume of data day and night at the edge of the network, create a huge demand for edge computing resources, and thus, promote the emergence of the multiaccess edge computing (MEC) paradigm. In MEC environments, IoT devices or mobile users are allowed to offload their computational tasks to nearby edge servers to overcome the limitation of local computing resources. Though edge servers could provide low-latency service with high-responsible computing capabilities, they are still facing many challenges posed by the limited hardware resources and diverse offloading requests. However, traditional approaches are usually based on the centralized architecture and batch-processing scheduling mode, which might lead to low efficiency and high communication overhead. Besides, they also lack the consideration of task diversity and priorities, which are crucial in real-world application scenarios. Thus, smart task scheduling and resource provision strategies with a high real-time property are urgently needed for better user experience and higher resource utilization. In this article, we target the online edge IoT task scheduling and resource allocation problem and propose a decentralized approach (DoSRA). The experiments based on real-world edge environments have demonstrated that the proposed approach could achieve at most a 35.34% reduction on the average weighted offloading response time.

Online Risk-Averse Resource Allocation in Queuing Networks

In this article, we address the online resource allocation problem in service queuing systems under uncertainty. In particular, the optimal control policy is derived by using the real-time data, i.e., without fully knowing information about the system state. We employ the conditional value-at-risk to achieve the minimum long-run average cost subject to the constraints on the risk of instability. Then, we can ensure the service ability of the system at a high level under such on-the-fly uncertainty. We show that the proposed model naturally leads to a minmax saddle point optimization problem. We first present an intuitive offline primal–dual learning method, which can achieve a desirable convergence rate. Then, we further improve the algorithm by learning the optimal Lagrange multiplier concerning the instantaneous system state, i.e., online primal–dual learning algorithm, to achieve the same convergence rate as the offline algorithm. Besides, we demonstrate that the proposed method outperforms classical models in providing the lower average delay under uncertainty, which means it can improve the stability and provide a better quality of service under the ambiguity of uncertainty. A real case of inpatient bed allocation for hospital operations is presented to show the performance in applications. The results show that our model can reduce the average wait with certain total available beds under uncertain arrivals.

Online Resource Allocation Under Partially Predictable Demand

Real-Time Resource Allocation: Beyond Stochastic Demand Modeling

Approximation Schemes for the Joint Inventory Selection and Online Resource Allocation Problem

In this paper, we introduce and study the joint inventory and online resource allocation problem, which is characterized by two sequential sets of decisions that are irrevocably linked. First, a decision maker must select starting inventory levels for a set of available resources. Subsequently, the decision maker must match arriving customers to available resources in an online fashion so as to maximize expected reward. We fi rst study the problem in its most general form before focusing on a specific version that arises at Anheuser Busch InBev (ABI). This particular application of our general setting is referred to as the ABI Trailer Problem, and it considers how ABI ships its beer to vendors via third party delivery trucks. Specifically, in this problem, ABI must select the weights and inventory levels of preloaded trailers of beer, which are then matched in an online fashion to arriving third party delivery trucks. For the general setting, we develop simple and easy-to-implement approaches that come with robust worst-case performance guarantees. For the ABI setting, we reveal a simplifying structural property related to the optimal matching policy, which gives rise to a natural adaptation of our original approach. We test the efficacy of these policies through extensive numerical experiments, where we find that our approaches are either near-optimal or improve upon state-of-the-art benchmarks. In particular, using a data set from ABI, we are able to generate instances of the ABI Trailer Problem, on which our algorithm has the potential to yield revenue improvements in the range of millions of dollars per year.

Online Resource Allocation With Machine Variability: A Bandit Perspective

Approximation jobs that allow partial execution of their many tasks to achieve valuable results have played an important role in today's large-scale data analytics. This fact can be utilized to maximize the system utility of a big data computing cluster by choosing proper tasks in scheduling for each approximation job. A fundamental challenge herein, however, is that the machine service capacity may fluctuate substantially during a job's lifetime, which makes it difficult to assign valuable tasks to well-performing machines. In addition, the cluster scheduler needs to make online scheduling decisions without knowing future job arrivals according to machine availabilities. In this paper, we tackle this online resource allocation problem for approximation jobs in parallel computing clusters. In particular, we model a cluster with heterogeneous machines as a multi-armed bandit where each machine is treated as an arm. By making estimations on machine service rates while balancing the exploration-exploitation trade-off, we design an efficient online resource allocation algorithm from a bandit perspective. The proposed algorithm extends existing online convex optimization techniques and yields a sublinear regret bound. Moreover, we also examine the performance of the proposed algorithm via extensive trace-driven simulations and demonstrate that it outperforms the baselines substantially.

Optimal Posted Prices for Online Resource Allocation with Supply Costs

We study a general online resource allocation problem, where a provider sells multiple types of capacity-limited resources to heterogeneous customers that arrive in a sequential and arbitrary manner. The provider charges payment from customers who purchase a bundle of resources but must pay an increasing supply cost with respect to the total resource allocated. The goal is to maximize the social welfare, namely, the total valuation of customers for their purchased bundles, minus the total supply cost of the provider for all the resources that have been allocated. We adopt the competitive analysis framework and provide an optimal posted-pricing mechanism (PPM). Our PPM is optimal in the sense that no other online algorithms can achieve a better competitive ratio.

Measurement-based efficient resource allocation with demand-side adjustments

The problem of efficient resource allocation has drawn significant attention in many scientific disciplines due to its direct societal benefits, such as energy savings. Traditional approaches in addressing online resource allocation problems neglect the potential benefit of feedback information available from the running tasks/loads as well as the potential flexibility of a task to adjust its operation/service-level in order to increase efficiency. The present paper builds upon recent developments in the area of bandwidth allocation in computing systems and proposes a generalized design approach for resource allocation when only performance measurements of the running tasks are available, possibly corrupted by noise. We demonstrate through analysis and simulations the potential of the proposed scheme in providing fair and efficient allocation of resources in a large class of resource allocation problems.

Online Advance Scheduling with Overtime: A Primal-Dual Approach

Problem definition: We study a fundamental online resource allocation problem in service operations in which a heterogeneous stream of arrivals that varies in service times and rewards makes service requests from a finite number of servers/providers. This is an online adversarial setting in which nothing more is known about the arrival process of customers. Each server has a finite regular capacity but can be expanded at the expense of overtime cost. Upon arrival of each customer, the system chooses both a server and a time for service over a scheduling horizon subject to capacity constraints. The system seeks easy-to-implement online policies that admit a competitive ratio (CR), guaranteeing the worst-case relative performance. Academic/practical relevance: On the academic side, we propose online algorithms with theoretical CRs for the problem described above. On the practical side, we investigate the real-world applicability of our methods and models on appointment-scheduling data from a partner health system. Methodology: We develop new online primal-dual approaches for making not only a server-date allocation decision for each arriving customer, but also an overtime decision for each server on each day within a horizon. We also derive a competitive analysis to prove a theoretical performance guarantee. Results: Our online policies are (i) robust to future information, (ii) easy-to-implement and extremely efficient to compute, and (iii) admitting a theoretical CR. Comparing our online policy with the optimal offline policy, we obtain a CR that guarantees the worst-case performance of our online policy. Managerial implications: We evaluate the performance of our online algorithms by using real appointment scheduling data from a partner health system. Our results show that the proposed online policies perform much better than their theoretical CR, and outperform the pervasive First-Come-First-Served (FCFS) and nested threshold policies (NTPO) by a large margin.

Online Advance Scheduling with Overtime: A Primal-Dual Approach

We study a fundamental online resource allocation problem in service operations in which a heterogeneous stream of arrivals that vary in service times and rewards make service requests from a finite number of servers/providers. This is an online adversarial setting in which nothing more is known about the arrival process of customers. Each server has a finite regular capacity but can be expanded at the expense of overtime cost. Upon arrival of each customer, the system chooses both a server and a time for service over a scheduling horizon subject to capacity constraints. The system seeks easy-to-implement online policies that admit a competitive ratio (CR), guaranteeing the worst-case relative performance. On the academic side, we propose online algorithms with theoretical CRs for the problem described above. On the practical side, we investigate the real-world applicability of our methods and models on appointment-scheduling data from a partner health system. We develop new online primal-dual approaches for making not only a server-date allocation decision for each arriving customer but also an overtime decision for each server on each day within a horizon. We also derive a competitive analysis to prove a theoretical performance guarantee. Our online policies are (i) robust to future information, (ii) easy-to-implement and extremely efficient to compute, and (iii) admitting a theoretical CR. Comparing our online policy with the offline optimal policy, we obtain a CR which guarantees the worst-case performance of our online policy. We evaluate the performance of our online algorithms by using real appointment scheduling data from the UMHS. Our results show that the proposed online policies perform much better than their theoretical CR, and outperform the pervasive First-Come-First-Served (FCFS) and nested threshold policies (NTPO) by a large margin.

Online Resource Allocation Under Partially Predictable Demand

For online resource allocation problems, we propose a new demand arrival model where the sequence of arrivals contains both an adversarial component and a stochastic one. Our model requires no demand forecasting; however, due to the presence of the stochastic component, we can partially predict future demand as the sequence of arrivals unfolds. Under the proposed model, we study the problem of the online allocation of a single resource to two types of customers, and design online algorithms that outperform existing ones. Our algorithms are adjustable to the relative size of the stochastic component, and our analysis reveals that as the portion of the stochastic component grows, the loss due to making online decisions decreases. This highlights the value of (even partial) predictability in online resource allocation. We impose no conditions on how the resource capacity scales with the maximum number of customers. However, we show that using an adaptive algorithm — which makes online decisions based on observed data — is particularly beneficial when capacity scales linearly with the number of customers. Our work serves as a first step in bridging the long-standing gap between the two well-studied approaches to the design and analysis of online algorithms based on (1) adversarial models and (2) stochastic ones. Using novel algorithm design, we demonstrate that even if the arrival sequence contains an adversarial component, we can take advantage of the limited information that the data reveals to improve allocation decisions. We also study the classical secretary problem under our proposed arrival model, and we show that randomizing over multiple stopping rules may increase the probability of success.

Competitive online algorithms for resource allocation over the positive semidefinite cone

We consider a new and general online resource allocation problem, where the goal is to maximize a function of a positive semidefinite (PSD) matrix with a scalar budget constraint. The problem data arrives online, and the algorithm needs to make an irrevocable decision at each step. Of particular interest are classic experiment design problems in the online setting, with the algorithm deciding whether to allocate budget to each experiment as new experiments become available sequentially. We analyze two greedy primal-dual algorithms and provide bounds on their competitive ratios. Our analysis relies on a smooth surrogate of the objective function that needs to satisfy a new diminishing returns (PSD-DR) property (that its gradient is order-reversing with respect to the PSD cone). Using the representation for monotone maps on the PSD cone given by L\"owner's theorem, we obtain a convex parametrization of the family of functions satisfying PSD-DR. We then formulate a convex optimization problem to directly optimize our competitive ratio bound over this set. This design problem can be solved offline before the data start arriving. The online algorithm that uses the designed smoothing is tailored to the given cost function, and enjoys a competitive ratio at least as good as our optimized bound. We provide examples of computing the smooth surrogate for D-optimal and A-optimal experiment design, and demonstrate the performance of the custom-designed algorithm.

Measurement-based Efficient Resource Allocation with Demand-Side Adjustments

The problem of efficient resource allocation has drawn significant attention in many scientific disciplines due to its direct societal benefits, such as energy savings. Traditional approaches in addressing online resource allocation problems neglect the potential benefit of feedback information available from the running tasks/loads as well as the potential flexibility of a task to adjust its operation/service-level in order to increase efficiency. The present paper builds upon recent developments in the area of bandwidth allocation in computing systems and proposes a generalized design approach for resource allocation when only performance measurements of the running tasks are available, possibly corrupted by noise. We demonstrate through analysis and simulations the potential of the proposed scheme in providing fair and efficient allocation of resources in a large class of resource allocation problems.

A stochastic algorithm for online bipartite resource allocation problems

This paper deals with online resource allocation problems whereby buyers with a limited total budget want to purchase items which become available one at a time and which consume some amount of various limited resources upon allocation. A central resource allocation platform is in charge of allocating the items to the potential buyers, with the goal of maximizing the total revenue subject to budget and resource constraints. Sponsored search advertising is a typical example: in order to maximize revenue, search engines try to choose the best available advertisement to display on a web page resulting from a search query.Two main approaches have been proposed to address such online problems, depending on the assumptions made about the input sequence: one is trying to guarantee a performance against a worst case scenario (sometimes called the adversarial model); the other one, based on specific probabilistic assumptions about the input, is concerned with expected performance guarantee. However, combining the strengths of these two approaches could potentially outperform both in some settings. In this paper we propose such a practical method which goes beyond the adversarial model but requires only a limited amount of information about the future. We provide extensive computational results which demonstrate settings under which the performance of the proposed algorithm becomes attractive.

Optimal energy allocation policies for a high altitude flying wireless access point

AbstractInspired by recent industrial efforts towards high altitude flying wireless access points powered by renewable energy, an online resource allocation problem for a mobile access point travelling at high altitude is formulated. The access point allocates its resources (available energy) to maximise the total utility (reward) provided to a sequentially observed set of users demanding service. The problem is formulated as a 0/1 dynamic knapsack problem with incremental capacity over a finite time horizon, and the solution of which is quite open in the literature. We address the problem through deterministic and stochastic formulations followed by a model where the statistics of the underlying processes are not known and learned through rule‐based and neural network approaches. For the deterministic problem, several online approximations including optimisation via genetic algorithm and rule‐based approach are proposed based on an instantaneous threshold that can adapt to short‐time‐scale dynamics. For the stochastic model, after showing the optimality of a threshold‐based solution on a dynamic programming formulation, an approximate threshold‐based policy is obtained. The performances of proposed policies are compared with that of the optimal solution obtained through dynamic programming. Copyright © 2016 John Wiley & Sons, Ltd.

Online Resource Allocation with Limited Flexibility

We consider a class of online resource allocation problems in which there are n types of resources with limited initial inventory and n demand classes. The resources are flexible in that each type of resources can serve more than one demand class. In this paper, we focus on a special class of structures with limited flexibility, the long chain design, which was proposed by Jordan and Graves (1995) and has been an important concept in the design of sparse flexible processes. We study the long chain design in an online stochastic environment where the requests are drawn repeatedly and independently from a known probability distribution over the different demand classes. Also, the decision on how to address each request must be made immediately upon its arrival. We show the effectiveness of the long chain design in mitigating supply-demand mismatch under a simple myopic online allocation policy. In particular, we provide an upper bound on the expected total number of lost sales that is irrespective of how large the market size is.