Marginal Productivity Index Research Articles

Multi-Gear Bandits, Partial Conservation Laws, and Indexability

This paper considers what we propose to call multi-gear bandits, which are Markov decision processes modeling a generic dynamic and stochastic project fueled by a single resource and which admit multiple actions representing gears of operation naturally ordered by their increasing resource consumption. The optimal operation of a multi-gear bandit aims to strike a balance between project performance costs or rewards and resource usage costs, which depend on the resource price. A computationally convenient and intuitive optimal solution is available when such a model is indexable, meaning that its optimal policies are characterized by a dynamic allocation index (DAI), a function of state–action pairs representing critical resource prices. Motivated by the lack of general indexability conditions and efficient index-computing schemes, and focusing on the infinite-horizon finite-state and -action discounted case, we present a verification theorem ensuring that, if a model satisfies two proposed PCL-indexability conditions with respect to a postulated family of structured policies, then it is indexable and such policies are optimal, with its DAI being given by a marginal productivity index computed by a downshift adaptive-greedy algorithm in AN steps, with A+1 actions and N states. The DAI is further used as the basis of a new index policy for the multi-armed multi-gear bandit problem.

A Verification Theorem for Threshold-Indexability of Real-State Discounted Restless Bandits

The Whittle index, which characterizes optimal policies for controlling certain single restless bandit projects (a Markov decision process with two actions: active and passive) is the basis for a widely used heuristic index policy for the intractable restless multiarmed bandit problem. Yet two roadblocks need to be overcome to apply such a policy: the individual projects in the model at hand must be shown to be indexable, so that they possess a Whittle index; and the index must be evaluated. Such roadblocks can be especially vexing when project state spaces are real intervals, as in recent sensor scheduling applications. This paper presents sufficient conditions for indexability (relative to a generalized Whittle index) of general real-state discrete-time restless bandits under the discounted criterion, which are not based on elucidating properties of the optimal value function and do not require proving beforehand optimality of threshold policies as in prevailing approaches. The main contribution is a verification theorem establishing that, if project performance metrics under threshold policies and an explicitly defined marginal productivity (MP) index satisfy three conditions, then the project is indexable with its generalized Whittle index being given by the MP index, and threshold policies are optimal for dynamic project control.

Scarce water resources and priority irrigation schemes from agronomic crops

Global environmental change places unavoidable pressure on water resources and agronomic crop production systems. Irrigation development is a credible measure to alleviate the challenge of food safety under water shortages, but it needs sufficient basis. The aim of this study is to address the problem of balancing water scarcity with food requirements, which are the key components of water security in regions with population growth. Marginal water productivity (MWP) indices for irrigation water performance and productivity evaluation were established in the current study. Based on the analysis of the regional water-crop relationship and spatial differences of MWP in China, the priorities for developing irrigation areas in different types of regions are discussed in this study. The results show that high MWPs are mainly in semi-arid regions with precipitation (P) between 500 and 1000 mm, while low MWPs mostly occur in areas with P more than 1000 and less than 500 mm. The significance and spatial distribution patterns of MWP are different than those of conventional irrigation water use efficiency evaluation indices, so its role cannot be replaced for the real production capacity of irrigation water evaluation. The strategies for global environmental change adaptation suggested in this study are taking MWP for irrigation water productivity evaluation and the priority irrigation schemes for agronomic crop determination; increasing MWP by means of irrigation efficiency and crop variety improvement worldwide; and raising global food production through the expansion of irrigation area in the regions hold high MWP and abundant water resources.

Linear programming relaxations and marginal productivity index policies for the buffer sharing problem

We study the dynamic admission control for a finite shared buffer with support of multiclass traffic under Markovian assumptions. The problem is often referred to as buffer sharing in the literature. From the linear programming (LP) formulation of the continuous-time Markov decision process (MDP), we construct a hierarchy of increasingly stronger LP relaxations where the hierarchy levels equal the number of job classes. Each relaxation in the hierarchy is obtained by projecting the original achievable performance region onto a polytope of simpler structure. We propose a heuristic policy for admission control, which is based on the theory of Marginal Productivity Index (MPI) and the Lagrangian decomposition of the first order LP relaxation. The dual of the relaxed buffer space constraint in the first order LP relaxation is used as a proxy to the cost of buffer space. Given that each of the decomposed queueing admission control problems satisfies the indexability condition, the proposed heuristic accepts a new arrival if there is enough buffer space left and the MPI of the current job class is greater than the incurred cost of buffer usage. Our numerical examples for the cases of two and eight job classes show the near-optimal performance of the proposed MPI heuristic.

A Faster Index Algorithm and a Computational Study for Bandits with Switching Costs

We address the intractable multi-armed bandit problem with switching costs, for which an index that partially characterizes optimal policies was introduced (Asawa, M., D. Teneketzis. 1996. Multi-armed bandits with switching penalties. IEEE Trans. Automatic Control 41 328–348), attaching to each project state a “continuation index” (its Gittins index) and a “switching index.” Asawa and Teneketzis proposed to jointly compute both as the Gittins index of a project with 2n states—when the original project has n states—resulting in an eightfold increase in O(n3) arithmetic operations relative to those to compute the continuation index. We present a faster decoupled computation method, which in a first stage computes the continuation index and then, in a second stage, computes the switching index an order of magnitude faster in at most n2 + O(n) arithmetic operations, achieving overall a fourfold reduction in arithmetic operations and substantially reduced memory operations. The analysis exploits the fact that the Asawa and Teneketzis index is the marginal productivity index of the project in its restless reformulation, using methods introduced by the author. Extensive computational experiments are reported, which demonstrate the dramatic runtime speedups achieved by the new algorithm, as well as the near optimality of the resultant index policy and its substantial gains against the benchmark Gittins index policy across a wide range of randomly generated two- and three-project instances.

Comments on: Dynamic priority allocation via restless bandit marginal productivity indices

This paper is a survey of recent results by the author on the design and computation of priority-index policies for some classes of problems, based on the unifying concept of marginal productivity index (MPI) introduced by the author himself. The paper is very interesting, reasonably self-contained, reader-friendly, and together with the comments by the other discussants gives a fairly complete overview of what is known nowadays on dynamic priority allocation to multiple projects. I believe there is not much to add to those comments and, therefore, I will concentrate on related open questions.

Dynamic priority allocation via restless bandit marginal productivity indices

This paper surveys recent work by the author on the theoretical and algorithmic aspects of restless bandit indexation as well as on its application to a variety of problems involving the dynamic allocation of priority to multiple stochastic projects. The main aim is to present ideas and methods in an accessible form that can be of use to researchers addressing problems of such a kind. Besides building on the rich literature on bandit problems, our approach draws on ideas from linear programming, economics, and multi-objective optimization. In particular, it was motivated to address issues raised in the seminal work of Whittle (Restless bandits: activity allocation in a changing world. In: Gani J. (ed.) A Celebration of Applied Probability, J. Appl. Probab., vol. 25A, Applied Probability Trust, Sheffield, pp. 287–298, 1988) where he introduced the index for restless bandits that is the starting point of this work. Such an index, along with previously proposed indices and more recent extensions, is shown to be unified through the intuitive concept of “marginal productivity index” (MPI), which measures the marginal productivity of work on a project at each of its states. In a multi-project setting, MPI policies are economically sound, as they dynamically allocate higher priority to those projects where work appears to be currently more productive. Besides being tractable and widely applicable, a growing body of computational evidence indicates that such index policies typically achieve a near-optimal performance and substantially outperform benchmark policies derived from conventional approaches.

Time-Constrained Restless Bandits and the Knapsack Problem for Perishable Items (Extended Abstract)

Motivated by a food promotion problem, we introduce the Knapsack Problem for Perishable Items (KPPI) to address a dynamic problem of optimally filling a knapsack with items that disappear randomly. The KPPI naturally bridges the gap and elucidates the relation between the PSPACE-hard restless bandit problem and the NP-hard knapsack problem. Our main result is a problem decomposition method resulting in an approximate transformation of the KPPI into an associated 0-1 knapsack problem. The approach is based on calculating explicitly the marginal productivity indices in a generic finite-horizon restless bandit subproblem.

Marginal productivity index policies for scheduling a multiclass delay-/loss-sensitive queue

This paper addresses the problem of scheduling a Markovian multiclass queue with a finite dedicated buffer for each class, where class-dependent linear holding and rejection cost rates model differing levels of tolerance to delay and loss. The goal is to design well-grounded and tractable scheduling policies that nearly minimize expected total discounted or long-run average cost. New dynamic index policies are introduced, awarding higher priority to classes with larger index values, where a class' index measures the marginal productivity of work at its current state. The results are obtained by deploying the work-cost analysis approach to marginal productivity indices (MPIs) for restless bandits developed by the author, which is extended to the bias criterion. The MPI furnishes new insights: for a loss-sensitive class, it is a decreasing function of the number of empty buffer spaces, independent of the buffer size; for a delay-sensitive class, it is a decreasing function of the queue length. Such opposite orderings show that preventive work is more valuable than reactive work for the latter classes, whereas the opposite holds for the former. The results of a computational study on two-class instances are reported, shedding light on how the MPI policy's relative performance varies with each parameter. Parameter ranges are thus identified where the MPI policy is near optimal, and substantially outperforms conventional benchmark policies.

Restless Bandit Marginal Productivity Indices, Diminishing Returns, and Optimal Control of Make-to-Order/Make-to-Stock M/G/1 Queues

This paper presents a framework grounded on convex optimization and economics ideas to solve by index policies problems of optimal dynamic allocation of effort to a discrete-state (finite or countable) binary-action (work/rest) semi-Markov restless bandit project, elucidating issues raised by previous work. Its contributions include: (i) the concept of a restless bandit’s marginal productivity index (MPI), characterizing optimal policies relative to general cost and work measures; (ii) the characterization of indexable restless bandits as those satisfying diminishing marginal returns to work, consistently with a nested family of threshold policies; (iii) sufficient indexability conditions via partial conservation laws (PCLs); (iv) the characterization of the MPI as an optimal marginal productivity rate relative to feasible active-state sets; (v) application to semi-Markov bandits under several criteria, including a new mixed average-bias criterion; and (vi) PCL-indexability analyses and MPIs for optimal service control of make-to-order/make-to-stock queues with convex holding costs, under discounted and average-bias criteria.