Sequential Resource Allocation Research Articles

Beta Upper Confidence Bound Policy for the Design of Clinical Trials

The multi-armed bandit problem is a classic example of the exploration-exploitation trade-off well suited to model sequential resource allocation under uncertainty. One of its typical motivating applications is the adaptive designs in clinical trials which modify the trial's course in accordance with the pre-specified objective by utilizing results accumulating in the trial. Since the response to a procedure in clinical trials is not immediate, the multi-armed bandit policies require adaptation to delays to retain their theoretical guarantees. In this work, we show the importance of such adaptation by evaluating policies using the publicly available datasetThe International Stroke Trial of a randomized trial of aspirin and subcutaneous heparin among 19,435 patients with acute ischaemic stroke. In addition to adapted policies, we analyze the Upper Confidence Bound policy with the beta feedback to mitigate delays when the certainty evidence of successful treatment is available in a relatively short-term period after the procedure.

On fairness and efficiency in nonprofit operations: Dynamic resource allocations

We study a sequential resource allocation problem balancing fairness and efficiency for nonprofit operations. (Un)fairness is measured by the expected maximum demand shortfall among all communities, and (in)efficiency is measured by the expected remaining resources after allocation. We characterize the optimal allocation policy as a two‐threshold policy in which the optimal allocation quantities are spoon‐shaped in terms of the current maximum demand shortfall. We further show that the thresholds and optimal allocation quantity for each community are nondecreasing in resource levels, realized demand from the current community, and weight of the efficiency objective. Based on these results, we propose a simple heuristic policy and numerically show that it performs well and generates fair allocations in a stochastic majorization order. The numerical results show that adding a small weight to the fairness objective significantly improves the system's fairness at a small efficiency cost. Moreover, the optimal initial capacity level is increasing (decreasing) in demand variance if the efficiency weight is small (large). Our theoretical analysis can be extended to the fill rate–based fairness metric.

A Novel Multi-Objective Sequential Resource Allocation Optimization for UAV-Assisted VLC

Unmanned aerial vehicle (UAV)-assisted visible light communication (VLC) constitutes a promising approach for providing illumination and communication simultaneously. In this paper, we consider the problem of joint UAV deployment, cell association, and power allocation, to meet the illumination and data rate demands of a set of users in a geographical area. Towards this goal, we develop <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SOUVLC</monospace> , a multi-objective <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">s</monospace> equentially <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">o</monospace> ptimized <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">U</monospace> AV-assisted <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">VLC</monospace> framework that aims at minimizing the required number of UAVs <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">first</i> and minimizing the total transmitted power <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">second</i> . <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SOUVLC</monospace> mitigates inter-cell interference and accounts for illumination interference. We derive a mixed-integer linear formulation of <monospace xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SOUVLC</monospace> , numerically evaluate it under different user demands and distributions, and demonstrate its superiority over several benchmarks.

Correlated Combinatorial Bandits for Online Resource Allocation

We study a sequential resource allocation problem where, at each round, the decision-maker needs to allocate its limited budget among different available entities. In doing so, the decision-maker obtains the reward for each entity in that round. The goal of the decision-maker is to maximize the expected cumulative reward or equivalently minimize cumulative regret over a total of T rounds. Sequential resource allocation can be modeled as a combinatorial bandit by viewing the allocation of a budget to an entity as a base arm. In the context of resource allocation, the rewards received under different budget allocations are likely to be correlated. We propose a novel correlated combinatorial bandit framework that explicitly models such correlations. We develop a novel Correlated-UCB algorithm for online resource allocation, which yields significantly reduced regret relative to correlation-agnostic algorithms. In certain cases, our proposed algorithm even achieves bounded regret, which is an order-wise reduction in the regret relative to the correlation-agnostic approach which incurs logarithmic regret under all scenarios. We validate these performance gains through experiments on several applications such as online power allocation across wireless channels, job scheduling in multi-server systems and online access point assignment.

Multi-armed bandit problem with online clustering as side information

We consider the sequential resource allocation problem under the multi-armed bandit model in the non-stationary stochastic environment. Motivated by many real applications, where information can naturally be grouped, we consider a variation of the contextual multi-armed bandit with online clustering representing side information. We assume a stochastic environment in which the reward of each action, conditioned on a cluster, follows a Bernoulli distribution with unknown parameters. Additionally, we assume that the nature of the problem changes over time and the clusters drift incrementally, making the reward process non-stationary. In this setting, we propose a new algorithm based on a two-stage approach. The first stage is a sequential modification of the traditional k-means clustering algorithm, in which the algorithm deals with the continuous data stream and acts on a subset of data rather than a single batch. In the second stage, we incorporate the current information about clusters into the Thompson Sampling policy with discounting mechanism to track changes in the underlying reward and account for a potential cluster misclassification.

An application of approximate dynamic programming in multi-period multi-product advertising budgeting

&lt;p style='text-indent:20px;'&gt;Advertising has always been considered a key part of marketing strategy and played a prominent role in the success or failure of products. This paper investigates a multi-product and multi-period advertising budget allocation, determining the amount of advertising budget for each product through the time horizon. Imperative factors including life cycle stage, &lt;inline-formula&gt;&lt;tex-math id="M1"&gt;\begin{document}$ BCG $\end{document}&lt;/tex-math&gt;&lt;/inline-formula&gt; matrix class, competitors' reactions, and budget constraints affect the joint chain of decisions for all products to maximize the total profits. To do so, we define a stochastic sequential resource allocation problem and use an approximate dynamic programming (&lt;inline-formula&gt;&lt;tex-math id="M2"&gt;\begin{document}$ ADP $\end{document}&lt;/tex-math&gt;&lt;/inline-formula&gt;) algorithm to alleviate the huge size of the problem and multi-dimensional uncertainties of the environment. These uncertainties are the reactions of competitors based on the current status of the market and our decisions, as well as the stochastic effectiveness (rewards) of the taken action. We apply an approximate value iteration (&lt;inline-formula&gt;&lt;tex-math id="M3"&gt;\begin{document}$ AVI $\end{document}&lt;/tex-math&gt;&lt;/inline-formula&gt;) algorithm on a numerical example and compare the results with four different policies to highlight our managerial contributions. In the end, the validity of our proposed approach is assessed against a genetic algorithm. To do so, we simplify the environment by fixing the competitor's reaction and considering a deterministic environment.&lt;/p&gt;

Sequential Fair Allocation: Achieving the Optimal Envy-Efficiency Trade-off Curve

Optimizing Mobile Food Pantry Operations Under Demand Uncertainty Managing complex systems often involves making trade-offs between different objectives. A common example is seeking fairness guarantees in sequential resource allocation problems. For example, mobile food pantries are tasked with allocating resources under demand uncertainty with the goal of simultaneously minimizing inefficiency (leftover resources) and envy (deviations in allocations). In this work, we tackle a problem established from a partnership with the Food Bank of the Southern Tier in optimizing their mobile food-pantry operations. We provide an exact characterization of the achievable (envy, efficiency) pairs, showing that any algorithm achieving low envy must suffer from poor inefficiency and vice versa. We complement this exact characterization with a simple algorithm capable of achieving any desired point along the trade-off curve.

Adaptive Resource Allocation for Diverse Safety Message Transmissions in Vehicular Networks

In this paper, we propose a two-level adaptive resource allocation (TARA) framework to support vehicular safety message transmissions. In particular, three types of safety messages are considered in urban vehicular networks, i.e., event-triggered messages for urgent condition warnings, periodic messages for vehicular status notifications, and messages for environmental perception. Roadside units are deployed for network management, and thus messages can be transmitted through either vehicle-to-infrastructure or vehicle-to-vehicle connections. To satisfy the requirements of different message transmissions, TARA framework consists of a group-level resource reservation module and a vehicle-level resource allocation module. Particularly, the resource reservation module is designed to allocate resources to support different types of message transmissions for each vehicle group at the first level. To learn the implicit relationship between the resource demand and message transmission requests, a supervised learning model is devised in the resource reservation module, where to obtain the training data we further propose a sequential resource allocation (SRA) scheme. Based on historical network information, SRA scheme offline optimizes the allocation of sensing resources, i.e., choosing vehicles to provide perception data, and communication resources. With resources reserved for each group, the vehicle-level resource allocation module is then devised to distribute specific resources for each vehicle to satisfy the differential requirements in real-time. Extensive simulation results demonstrate the effectiveness of TARA framework in terms of the high packet delivery ratio and low latency for message transmissions, and the high quality of collective environmental perception.

Conditions for indexability of restless bandits and an algorithm to compute Whittle index

AbstractRestless bandits are a class of sequential resource allocation problems concerned with allocating one or more resources among several alternative processes where the evolution of the processes depends on the resources allocated to them. Such models capture the fundamental trade-offs between exploration and exploitation. In 1988, Whittle developed an index heuristic for restless bandit problems which has emerged as a popular solution approach because of its simplicity and strong empirical performance. The Whittle index heuristic is applicable if the model satisfies a technical condition known as indexability. In this paper, we present two general sufficient conditions for indexability and identify simpler-to-verify refinements of these conditions. We then revisit a previously proposed algorithm called the adaptive greedy algorithm which is known to compute the Whittle index for a sub-class of restless bandits. We show that a generalization of the adaptive greedy algorithm computes the Whittle index for all indexable restless bandits. We present an efficient implementation of this algorithm which can compute the Whittle index of a restless bandit withKstates in$\mathcal{O}\!\left(K^3\right)$computations. Finally, we present a detailed numerical study which affirms the strong performance of the Whittle index heuristic.

Maximal Linear Deadlock Avoidance Policies for Sequential Resource Allocation Systems: Characterization, Computation, and Approximation

The problem of maximally permissive deadlock avoidance for sequential resource allocation systems (RAS) is a well-defined problem in the current controls literature. The corresponding supervisor is known as the maximally permissive deadlock avoidance policy (DAP), and it can be perceived as a classifier effecting the dichotomy of the underlying state space into its “safe” and “unsafe” subspaces. In the deployment of the maximally permissive DAP, an important issue is the selection of an effective and computationally efficient representation of the aforementioned dichotomy. A popular such representation is the “linear classifier,” where the admissibility of any given RAS state is resolved based on its ability to satisfy a given set of linear inequalities. However, linear classifiers cannot provide effective representation of the maximally permissive DAP for all RAS instantiations. Hence, this article provides a methodology for synthesizing linear DAPs for any given RAS instance that might not be maximally permissive in the original sense of this term, but observe a more relaxed notion of “maximality” that is defined within the particular space of the DAPs that admit a linear representation. The presented developments formally define this new DAP class, and provide the necessary algorithms for the synthesis or the systematic approximation of maximal linear DAPs for any given RAS instance.

Послiдовний розподiл ресурсiв устохастичному середовищi: загальнийопис, аналiз та чисельнi експерименти

In this paper, we consider policies for the sequential resource allocation under the multi-armed bandit problem in a stochastic environment. In this model, an agent sequentially selects an action from a given set and an environment reveals a reward in return. In the stochastic setting, each action is associated with a probability distribution with parameters that are not known in advance. The agent makes a decision based on the history of the chosen actions and obtained rewards. The objective is to maximize the total cumulative reward, which is equivalent to the loss minimization. We provide a brief overview of the sequential analysis and an appearance of the multi-armed bandit problem as a formulation in the scope of the sequential resource allocation theory. Multi-armed bandit classification is given with an analysis of the existing policies for the stochastic setting. Two different approaches are shown to tackle the multi-armed bandit problem. In the frequentist view, the confidence interval is used to express the exploration-exploitation trade-off. In the Bayesian approach, the parameter that needs to be estimated is treated as a random variable. Shown, how this model can be modelled with help of the Markov decision process. In the end, we provide numerical experiments in order to study the effectiveness of these policies.

Variables Reduction in Sequential Resource Allocation Problems

This paper presents a general framework to address diverse notoriously difficult problems arising in the area of optimal resource management, exploitation of natural reserves, pension fund valuation, environmental protection, and storage operation. Using some common abstract features of this problem class, we present a technique which provides a significant reduction of decision variables. As an application, we discuss a battery storage control to show how a decision problem, which is practically unsolvable in the original formulation, can be treated by our method.

A branch and bound algorithm for dynamic resource allocation in population disease management

We embed an approximate dynamic program into a branch-and-bound framework to solve sequential resource allocation problems in population disease management. The proposed algorithm is capable of providing an optimality guarantee and getting bounds on the optimality gap of healthcare interventions. A numerical study on screening and treatment policy implementation for chronic hepatitis C virus (HCV) infection provides useful insights regarding HCV elimination for baby boomers.

Urgency-Based Fair Scheduling for LTE to Improve Packet Loss and Fairness: Design and Evaluation

Classic scheduling algorithms for the LTE uplink in the literature may cause severe packet loss, poor fairness, and inefficient resource allocation. To alleviate packet loss, delay budget rather than channel condition plays a key factor in our proposed algorithm that allows the user equipment (UE) with the least left delay budget to be scheduled first. Besides, the UEs with fewer allocated resources can be served earlier to improve fairness. For reaching efficient resource utilization, our proposed algorithm further poses a constraint in sequential resource allocation. Via extensive simulations, we successfully demonstrate that our proposed algorithm significantly outperforms the closely related algorithms in terms of packet loss ratio and fairness index at the cost of raised end-to-end delay but still within the delay bound. For our proposed algorithm, it also gains improvement in throughput due to its reduction in the packet loss ratio and bounded relaxation in the end-to-end delay.

Approximate dynamic programming for the aeromedical evacuation dispatching problem: Value function approximation utilizing multiple level aggregation

Sequential resource allocation decision-making for the military medical evacuation of wartime casualties consists of identifying which available aeromedical evacuation (MEDEVAC) assets to dispatch in response to each casualty event. These sequential decisions are complicated due to uncertainty in casualty demand (i.e., severity, number, and location) and service times. In this research, we present a Markov decision process model solved using a hierarchical aggregation value function approximation scheme within an approximate policy iteration algorithmic framework. The model seeks to optimize this sequential resource allocation decision under uncertainty of how to best dispatch MEDEVAC assets to calls for service. The policies determined via our approximate dynamic programming (ADP) approach are compared to optimal military MEDEVAC dispatching policies for two small-scale problem instances and are compared to a closest-available MEDEVAC dispatching policy that is typically implemented in practice for a large-scale problem instance. Results indicate that our proposed approximation scheme provides high-quality, scalable dispatching policies that are more easily employed by military medical planners in the field. The identified ADP policies attain 99.8% and 99.5% optimal for the 6- and 12-zone problem instances investigated, as well as 9.6%, 9.2%, and 12.4% improvement over the closest-MEDEVAC policy for the 6-, 12-, and 34-zone problem instances investigated.

Learning Proportionally Fair Allocations with Low Regret

This paper addresses a generic sequential resource allocation problem, where in each round a decision maker selects an allocation of resources (servers) to a set of tasks consisting of a large number of jobs. A job of task i assigned to server j is successfully treated with probability θ_ij $ in a round, and the decision maker is informed on whether this job is completed at the end of the round. The probabilities θ_ij $'s are initially unknown and have to be learned. The objective of the decision maker is to sequentially assign jobs of various tasks to servers so that it rapidly learns and converges to the Proportionally Fair (PF) allocation (or other similar allocations achieving an appropriate trade-off between efficiency and fairness). We formulate the problem as a multi-armed bandit (MAB) optimization problem, and devise sequential assignment algorithms with low regret (defined as the difference in utility achieved by an oracle algorithm aware of the θ_ij $'s and by the proposed algorithm over a given number of slots). We first provide the properties of the so-called Restricted-PF (RPF) allocation, obtained by assuming that each task can only use a single server, and in particular show that it is very close to the PF allocation. We devise ES-RPF, an algorithm that learns the RPF allocation with regret no greater than $\mathcal O \bigl(m^3øver θ_\min Δ_\min łog(T)\big)$ after T slots, where m , θ_\min $, and Δ_\min $ represent the number of tasks, the minimum success rate $\min_i,j θ_ij $, and an appropriately defined notion of gap, respectively. We further provide regret lower bounds satisfied by any algorithm targeting the RPF allocation. Finally, we present ES-PF, an algorithm directly learning the PF allocation, and prove that its regret does not exceed $\mathcal O \bigl(\fracm^2s θ_\min \sqrtT łog(T)\big)$ after T slots, where s denotes the number of servers.

State Space Characterization of Disjunctive Single-Unit Resource Allocation Systems

A deluge of approaches has been proposed to implement and deploy a computationally effective and maximally permissive liveness-enforcing supervisor for sequential resource allocation systems (RASs). However, they are stalled by the computational complexity of a large state space that grows exponentially with respect to the size of an underlying RAS. The attention of this work is restricted to a special class of RASs: Disjunctive/Single-unit (D/SU)-RASs. Given an initial resource configuration of a D/SU-RAS, a complete state enumeration can be obtained through pure algebraic operations on this configuration with the pre-computed initial basis state space. When an explicit and complete enumeration of reachable states is impossible (due to the limited memory and storage space), we propose the exact number of reachable states. In this case, given a state vector, its reachability can be decided by an algebraic method. Experimental studies demonstrate the efficiency of the proposed approach.

On Bayesian index policies for sequential resource allocation

This paper is about index policies for minimizing (frequentist) regret in a stochastic multi-armed bandit model, that are inspired by a Bayesian view on the problem. Our main contribution is to prove the asymptotic optimality of Bayes-UCB, an algorithm based on quantiles of posterior distributions, when the rewards distributions belong to a one-dimensional exponential family, for a large class of prior distributions. We also show that the Bayesian literature gives new insight on what kind of exploration rates could be used in frequentist, UCB-type algorithms. Indeed, approximations of the Bayesian optimal solution or the Finite Horizon Gittins indices suggest the introduction of two algorithms, KL-UCB + and KL-UCB-H + , whose asymptotic optimality is also established.

Robust Deadlock Avoidance for Sequential Resource Allocation Systems With Resource Outages

While the supervisory control (SC) problem of (maximally permissive) deadlock avoidance for sequential resource allocation systems (RASs) has been extensively studied in the literature, the corresponding results that are able to address potential resource outages are quite limited, both, in terms of their volume and their control capability. This paper leverages the recently developed SC theory for switched discrete event systems (s-DES) in order to provide a novel systematic treatment of this more complicated version of the RAS deadlock avoidance problem. Following the modeling paradigm of s-DES, both the operation of the considered RAS and the corresponding maximally permissive SC policy are decomposed over a number of operational modes that are defined by the running sets of the failing resources. In particular, the target supervisor must be decomposed to a set of “localized predicates,” where each predicate is associated with one of the operational modes. A significant part, and a primary contribution, of this paper concerns the development of these localized predicates that will enable the formal characterization and the effective computation of the sought supervisor. With these predicates available, a distributed representation for the sought supervisor that is appropriate for real-time implementation is eventually obtained through an adaptation of the relevant distributed algorithm that is provided by the current s-DES SC theory. Note to Practitioners —This paper extends the existing theory of deadlock avoidance for buffer-space allocation in flexibly automated production systems so that it accounts for disruptive effects due to potential temporary outages of some of the system servers. The set of the failing servers at any time instant defines the corresponding operational mode for the underlying resource allocation system. The primary problem that is addressed by this paper is the synthesis of a resource allocation policy that will ensure the ability of all process instances that do not require the failing resources in a particular mode, to execute repetitively and complete successfully while the system remains in that mode. In line with some past literature on this problem, we call the corresponding supervisory control problem as “robust deadlock avoidance,” and we leverage results from the recently emerged theory for modeling and control of switched discrete event systems in order to characterize and compute a maximally permissive solution for it.

Utility Maximizing Sequential Sensing Over a Finite Horizon

We consider the problem of optimally utilizing $N$ resources, each in an unknown binary state. The state of each resource can be inferred from state-dependent noisy measurements. Depending on its state, utilizing a resource results in either a reward or a penalty per unit time. The objective is a sequential strategy governing the decision of sensing and exploitation at each time to maximize the expected utility (i.e., total reward minus total penalty and sensing cost) over a finite horizon $L$. We formulate the problem as a Partially Observable Markov Decision Process (POMDP) and show that the optimal strategy is based on two time-varying thresholds for each resource and an optimal selection rule for which resource to sense. Since a full characterization of the optimal strategy is generally intractable, we develop a low-complexity policy that is shown by simulations to offer near optimal performance. This problem finds applications in opportunistic spectrum access, marketing strategies and other sequential resource allocation problems.