Whittle Index Research Articles

Whittle index approach to the multi-class queueing systems with convex holding costs and IHR service times

AbstractWe consider the dynamic scheduling problem in a single-server multi-class queueing system, where the accumulated holding costs of a customer is a convexly increasing function of the time that the customer spends in the system. This problem was already addressed by van Mieghem in his seminal paper (Ann Appl Probab 5:808–833, 1995). He launched the so-called generalized $$c\mu $$ c μ rule and proved its asymptotic optimality in the heavy traffic limit. The optimal scheduling problem with convex holding costs have also been studied in many other papers. However, the holding costs in these papers are typically count-convex, i.e., the aggregate holding cost rate related to a class of customers increases convexly as a function of the current number of customers in this class, which is clearly different from the time-convex holding costs introduced in van Mieghem (Ann Appl Probab 5:808–833, 1995) and studied also in the present paper. We utilize the Whittle index approach to develop a novel heuristic policy for the dynamic scheduling problem with time-convex holding costs. Instead of the original open version of the continuous-time scheduling problem, we apply the Whittle index approach to the corresponding discrete-time closed version of the problem. As the main theoretical contribution, we prove that, for IHR service times, the discounted discrete-time problem is indexable and also give an explicit expression for the Whittle index itself. A special challenge in our model is the two-dimensional state space with an infinite number of possible states. By utilizing the discrete-time results, we develop a novel index policy for the original continuous-time problem and demonstrate, by numerical simulations, that this Whittle index policy outperforms the generalized $$c\mu $$ c μ rule. Thus, while the generalized $$c\mu $$ c μ rule is asymptotically optimal in the heavy traffic limit, it can still be improved in normal traffic conditions.

Tabular and Deep Learning for the Whittle Index

The Whittle index policy is a heuristic that has shown remarkably good performance (with guaranteed asymptotic optimality) when applied to the class of problems known as Restless Multi-Armed Bandit Problems (RMABPs). In this article, we present QWI and QWINN, two reinforcement learning algorithms, respectively tabular and deep, to learn the Whittle index for the total discounted criterion. The key feature is the use of two time-scales, a faster one to update the state-action Q -values, and a relatively slower one to update the Whittle indices. In our main theoretical result, we show that QWI, which is a tabular implementation, converges to the real Whittle indices. We then present QWINN, an adaptation of QWI algorithm using neural networks to compute the Q -values on the faster time-scale, which is able to extrapolate information from one state to another and scales naturally to large state-space environments. For QWINN, we show that all local minima of the Bellman error are locally stable equilibria, which is the first result of its kind for DQN-based schemes. Numerical computations show that QWI and QWINN converge faster than the standard Q -learning algorithm, neural-network based approximate Q-learning, and other state-of-the-art algorithms.

Whittle index approach to multiserver scheduling with impatient customers and DHR service times

We consider the optimal scheduling problem in a multiserver queue with impatient customers belonging to multiple classes. We assume that each customer has a random abandonment time, after which the customer leaves the system if its service has not been completed before that. In addition, we assume that the scheduler is not able to anticipate the expiration of the abandonment times but only knows their distributions and how long each customer has been in the system. Many papers consider this scheduling problem under Poisson arrivals and linear holding costs assuming further that both the service times and the abandonment times have exponential distributions. Even with these additional assumptions, the exact solution is known only in very few special cases. To tackle this tricky problem, we apply the Whittle index approach. Unlike the earlier papers, which were restricted to exponential service times, we allow the service time distributions for which the hazard rate is decreasing. The Whittle index approach is applied to the discrete-time multiserver queueing problem with discounted costs. As our main theoretical result, we prove that the related relaxed optimization problem is indexable and derive the corresponding Whittle index explicitly. Based on this discrete-time result, we develop a reasonable heuristic for the original continuous-time multiserver scheduling problem. The performance of the resulting policy is evaluated in the M/G/M setup by numerical simulations, which demonstrate that it, indeed, gives better performance than the other policies included in the comparison.

On learning Whittle index policy for restless bandits with scalable regret

On learning Whittle index policy for restless bandits with scalable regret

A Neural Network-Based Whittle Index Policy for Beam Resource Allocation in Multitarget Tracking

A Neural Network-Based Whittle Index Policy for Beam Resource Allocation in Multitarget Tracking

Two families of indexable partially observable restless bandits and Whittle index computation

We consider the restless bandits with general finite state space under partial observability with two observational models: first, the state of each bandit is not observable at all, and second, the state of each bandit is observable when it is selected. Under the assumption that the models satisfy a restart property, we prove that both models are indexable. For the first model, we derive a closed-form expression for the Whittle index. For the second model, we propose an efficient algorithm to compute the Whittle index by exploiting the qualitative properties of the optimal policy. We present detailed numerical experiments for multiple instances of machine maintenance problem. The result indicates that the Whittle index policy outperforms myopic policy and can be close to optimal in different setups.

Age of Information Cost Minimization with No Buffers, Random Arrivals and Unreliable Channels: A PCL-Indexability Analysis

Over the last decade, the Age of Information has emerged as a key concept and metric for applications where the freshness of sensor-provided data is critical. Limited transmission capacity has motivated research on the design of tractable policies for scheduling information updates to minimize Age of Information cost based on Markov decision models, in particular on the restless multi-armed bandit problem (RMABP). This allows the use of Whittle’s popular index policy, which is often nearly optimal, provided indexability (index existence) is proven, which has been recently accomplished in some models. We aim to extend the application scope of Whittle’s index policy in a broader AoI scheduling model. We address a model with no buffers incorporating random packet arrivals, unreliable channels, and nondecreasing AoI costs. We use sufficient indexability conditions based on partial conservation laws previously introduced by the author to establish the model’s indexability and evaluate its Whittle index in closed form under discounted and average cost criteria. We further use the index formulae to draw insights on how scheduling priority depends on model parameters.

Reinforcement Learning for Dynamic Dimensioning of Cloud Caches: A Restless Bandit Approach

We study the dynamic cache dimensioning problem, where the objective is to decide how much storage to place in the cache to minimize the total costs with respect to the storage and content delivery latency. We formulate this problem as a Markov decision process, which turns out to be a restless multi-armed bandit problem and is provably hard to solve. For given dimensioning decisions, it is possible to develop solutions based on the celebrated Whittle index policy. However, Whittle index policy has not been studied for dynamic cache dimensioning, mainly because cache dimensioning needs to be repeatedly solved and jointly optimized with content caching. To overcome this difficulty, we propose a low-complexity fluid Whittle index policy, which jointly determines dimensioning and content caching. We show that this policy is asymptotically optimal. We further develop a lightweight reinforcement learning augmented algorithm dubbed fW-UCB when the content request and delivery rates are unavailable. fW-UCB is shown to achieve a sub-linear regret as it fully exploits the structure of the near-optimal fluid Whittle index policy and hence can be easily implemented. Extensive simulations using real traces support our theoretical results.

Scalable Decision-Focused Learning in Restless Multi-Armed Bandits with Application to Maternal and Child Health

This paper studies restless multi-armed bandit (RMAB) problems with unknown arm transition dynamics but with known correlated arm features. The goal is to learn a model to predict transition dynamics given features, where the Whittle index policy solves the RMAB problems using predicted transitions. However, prior works often learn the model by maximizing the predictive accuracy instead of final RMAB solution quality, causing a mismatch between training and evaluation objectives. To address this shortcoming, we propose a novel approach for decision-focused learning in RMAB that directly trains the predictive model to maximize the Whittle index solution quality. We present three key contributions: (i) we establish differentiability of the Whittle index policy to support decision-focused learning; (ii) we significantly improve the scalability of decision-focused learning approaches in sequential problems, specifically RMAB problems; (iii) we apply our algorithm to a previously collected dataset of maternal and child health to demonstrate its performance. Indeed, our algorithm is the first for decision-focused learning in RMAB that scales to real-world problem sizes.

Optimistic Whittle Index Policy: Online Learning for Restless Bandits

Restless multi-armed bandits (RMABs) extend multi-armed bandits to allow for stateful arms, where the state of each arm evolves restlessly with different transitions depending on whether that arm is pulled. Solving RMABs requires information on transition dynamics, which are often unknown upfront. To plan in RMAB settings with unknown transitions, we propose the first online learning algorithm based on the Whittle index policy, using an upper confidence bound (UCB) approach to learn transition dynamics. Specifically, we estimate confidence bounds of the transition probabilities and formulate a bilinear program to compute optimistic Whittle indices using these estimates. Our algorithm, UCWhittle, achieves sublinear O(H \sqrt{T log T}) frequentist regret to solve RMABs with unknown transitions in T episodes with a constant horizon H. Empirically, we demonstrate that UCWhittle leverages the structure of RMABs and the Whittle index policy solution to achieve better performance than existing online learning baselines across three domains, including one constructed from a real-world maternal and childcare dataset.

Conditions for indexability of restless bandits and an algorithm to compute whittle index – CORRIGENDUM

AbstractThis note corrects an error in the formula to obtain the Whittle index using the Sherman–Morrison formula in Akbarzadeh and Mahajan (2022). Also, some other minor typos are highlighted.

Exponential asymptotic optimality of Whittle index policy

Exponential asymptotic optimality of Whittle index policy

On Two Sensors Scheduling for Remote State Estimation With a Shared Memory Channel in a Cyber-Physical System Environment.

This article studies two sensors scheduling with a shared memory channel for remote state estimation in cyber-physical systems (CPSs). We consider that each sensor monitors a plant and sends its local estimate to the remote estimator over a shared memory communication channel, of which the packet reception results between two successive time instants are correlated. This article focuses on how the two sensors are scheduled to minimize the total estimation errors at the remote side. The problem is formulated as the Markov decision process (MDP) and the optimal policy is derived. Moreover, the threshold structure of the optimal policy is given to reduce computation overhead. After proving the Whittle indexability of the overall system under a given condition, the Whittle index policy is adopted to further reduce the computation overhead. Numerical simulations are given to illustrate the theoretical results.

Dynamic ordering and buyer selection policies when service affects future demand

Make-to-stock suppliers selling to regular buyers must balance the cost of overstocking against the cost of buyers’ reactions when items become unavailable. In selecting which buyers to satisfy when shortages occur, they must weigh the current revenue from the satisfied buyers against the loss in future demand from the dissatisfied buyers. To provide insight and decision support on these trade-offs, we (i) develop a novel newsvendor model of a firm with heterogeneous buyers with service-dependent demand, (ii) provide properties of the optimal ordering and buyer selection decisions of the firm, and (iii) derive a novel Lagrangian Relaxation (LR)-based index policy for selecting buyers and compare it with two other LR-based index policies. The model concerns a firm that orders items for a group of repeat buyers who generate different revenues and visit the firm with different average rates that depend on whether they are satisfied or dissatisfied with their last visit. The firm selects which buyers to serve if the demand exceeds the order quantity (current capacity). For two buyers, we show that a fixed order quantity (FOQ) policy and an index buyer selection policy are optimal. For more buyers, the optimal policy involves overstocking (understocking) when the overall buyer satisfaction is high (low) and selecting buyers that maximize the current revenue (future demand) when the overall buyer satisfaction after the demand is high (low). To tackle the problem, we derive three LR-based index policies: a Lagrangian index policy that uses a uniform capacity price, a Whittle index policy that uses a discriminatory capacity price and is myopically optimal, and a novel “active-constraint” index policy that uses a discriminatory capacity price when the capacity constraint is active. Numerical results indicate that the latter policy is near-optimal and outperforms the other two and that combining it with the right FOQ policy can be very efficient.

Deep Reinforcement Learning Heterogeneous Channels for Poisson Multiple Access

This paper proposes a medium access control (MAC) protocol based on deep reinforcement learning (DRL), i.e., multi-channel transmit deep-reinforcement learning multi-channel access (MCT-DLMA) in heterogeneous wireless networks (HetNets). The work concerns practical unsaturated channel traffic that arrives following a Poisson distribution instead of saturated traffic that arrives before.By learning the access mode from historical information, MCT-DLMA can well fill the spectrum holes in the communication of existing users. In particular, it enables the cognitive user to multi-channel transmit at a time, e.g., via the multi-carrier technology. Thus, the spectrum resource can be fully utilized, and the sum throughput of the HetNet is maximized. Simulation results show that the proposed algorithm provides a much higher throughput than the conventional schemes, i.e., the whittle index policy and the DLMA algorithms for both the saturated and unsaturated traffic, respectively. In addition, it also achieves a near-optimal result in dynamic environments with changing primary users, which proves the enhanced robustness to time-varying communications.

Whittle Index Based Age-of-Information Aware Scheduling for Markovian Channels

The focus of this work is on minimizing the time average of the weighted sum of the Age-of-Information in a multi-sensor system in the setting where all sensors report their measurements to a central monitoring station using a shared communication channel. We consider multiple information settings including complete CSI/delayed CSI/no CSI and two stochastic channel evolution models, i.e., i.i.d. channels and Markovian channels. In all settings considered, we prove the indexability of the scheduling problem. In addition, we compute the Whittle Index in closed form for some of the settings. Indexability for Markovian channels for the objective of minimizing AoI is a key contribution of this work. Further, under i.i.d. channels, we propose a novel efficient implementation for the Whittle Index-based policy. We use simulations to show that Whittle Index-based scheduling policies either outperform or match the performance of the state-of-the-art policies for all the settings considered.

Age-of-Information Oriented Scheduling for Multichannel IoT Systems With Correlated Sources

Age-of-information (AoI) based minimization problems have been widely considered in Internet-of-Things (IoT) networks with the settings of multi-source single-channel systems and multi-source multi-channel systems. Most existing works are limited to either the case of identical multi-channel or independent sources. In this paper, we study this problem under the identical and non-identical multi-channel, as well as the correlated sources setting. This correlation defines the case when updating a source’s AoI; others correlated to this one will also reveal partial information. To tackle this AoI-based minimization problem, we formulate it as a correlated restless multi-armed bandit (CRMAB) problem. By decoupling the CRMAB problem into <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$N$ </tex-math></inline-formula> independent single-armed bandit problems, we derive the closed-form expressions of the generalized Whittle index (GWI) and the generalized partial Whittle index (GPWI) under the identical channel and the non-identical channel settings, respectively. Then, we put forth the GWI-based and GPWI-based scheduling policies to solve this AoI-based minimization problem. In addition, we provide two lower numerical performance bounds for the proposed policies by solving the relaxed Lagrange problem of the decoupled CRMAB. Numerical results show that the proposed policies can achieve these lower bounds and outperform the state-of-the-art scheduling policies. Compared with the case of independent sources, the performance of the proposed policies in the case of correlated sources improves significantly, especially in high-density networks.

Uncertainty-of-Information Scheduling: A Restless Multiarmed Bandit Framework

This paper proposes using the uncertainty of information (UoI), measured by Shannon’s entropy, as a metric for information freshness. We consider a system in which a central monitor observes M binary Markov processes through m communication channels (m<M). The UoI of a Markov process corresponds to the monitor’s uncertainty about its state. At each time step, only m Markov processes can be selected to update its state to the monitor; hence there is a tradeoff among the UoIs of the processes that depend on the scheduling policy used to select the processes to be updated. The age of information (AoI) of a process corresponds to the time since its last update. In general, the associated UoI can be a non-increasing function, or even an oscillating function, of its AoI, making the scheduling problem particularly challenging. This paper investigates scheduling policies that aim to minimize the average sum-UoI of the processes over the infinite time horizon. We formulate the problem as a restless multi-armed bandit (RMAB) problem, and develop a Whittle index policy that is near-optimal for the RMAB after proving its indexability. We further provide an iterative algorithm to compute the Whittle index for the practical deployment of the policy. Although this paper focuses on UoI scheduling, our results apply to a general class of RMABs for which the UoI scheduling problem is a special case. Specifically, this paper’s Whittle index policy is valid for any RMAB in which the bandits are binary Markov processes and the penalty is a concave function of the belief state of the Markov process. Numerical results demonstrate the excellent performance of the Whittle index policy for this class of RMABs.

Scalable Operator Allocation for Multirobot Assistance: A Restless Bandit Approach

In this article, we consider the problem of allocating human operators in a system with multiple semiautonomous robots. Each robot is required to perform an independent sequence of tasks, subject to a chance of failing and getting stuck in a fault state at every task. If and when required, a human operator can assist or teleoperate a robot. Conventional dynamic programming-based techniques used to solve such problems face scalability issues due to an exponential growth of state and action spaces with the number of robots and operators. In this article, we derive conditions under which the operator allocation problem satisfies a technical condition called indexability, thereby enabling the use of the Whittle index heuristic. The conditions are easy to check, and we show that they hold for a wide range of problems of interest. Our key insight is to leverage the structure of the value function of individual robots, resulting in conditions that can be verified separately for each state of each robot. We apply these conditions to two types of transitions commonly seen in remote robot supervision systems. Through numerical simulations, we demonstrate the efficacy of Whittle index policy as a near-optimal and scalable approach that outperforms existing scalable methods.

User Association in Dense mmWave Networks as Restless Bandits

In this article, we study the problem of user association, i.e., determining which base station (BS) a user should associate with, in a dense millimeter wave (mmWave) network. In our system model, in each time slot, a user arrives with some probability in a region with a relatively small geographical area served by a dense mmWave network. Our goal is to devise an association policy under which, in each time slot in which a user arrives, it is assigned to exactly one BS so as to minimize the weighted average amount of time that users spend in the system. The above problem is a restless multi-armed bandit problem and is provably hard to solve. We prove that the problem is Whittle indexable, and based on this result, propose an association policy under which an arriving user is associated with the BS having the smallest Whittle index. Using simulations, we show that our proposed policy outperforms several user association policies proposed in prior work.