Average Cost Criterion Research Articles

Ergodic Control of Bipartite Matching Queues with Class Change and Matching Failure

Motivated by transplant applications, we study a bipartite matching queue with multiclass customers and multitype resources. Customers may change their classes or abandon the system while waiting in queue, and they may decline the offered resource units which results in matching failure. We are interested in designing efficient instantaneous matching policies that allocate resources upon arrival to waiting customers. Our objective is bicriteria and formulated as a cost functional that linearly combines the long-run average expected reward due to successful matches and the long-run average expected cost from customer waiting and abandonment. We first develop a stability condition on the class change and abandonment rates, which requires at least one customer queue with abandonment and that any queue without abandonment have a class transition path to a queue with abandonment. Under this condition, we construct a simple linear program, referred to as the fluid control problem (FCP), which serves as a lower bound for the original stochastic control problem under any admissible policy. We then propose a randomized matching policy based on the solution of the FCP and show that the proposed policy is asymptotically optimal under both the long-run average and ergodic cost criteria. In addition, we apply our method to study two X matching models with two customer classes and two resource types to provide insights on how the class change and matching failure impact the optimal policies.

Zero-sum stochastic games in continuous-time with risk-sensitive average cost criterion on a countable state space

We consider zero-sum stochastic games in continuous time with controlled Markov chains and with risk-sensitive average cost criterion. Here the transition and the cost rates may be unbounded. We prove the existence of the value of the game and a saddle-point equilibrium in the class of all stationary strategies under a Lyapunov stability condition. This is accomplished by establishing the existence of a principal eigenpair for the corresponding Hamilton-Jacobi-Isaacs (HJI) equation. This, in turn, is established by using a nonlinear version of Krein-Rutman theorem. We then obtain a characterization of the saddle-point equilibrium in terms of the corresponding HJI equation. Finally, we use a controlled population system to illustrate our results.

Optimal policy for two-stage Brownian inventory systems under the long-run average cost criterion

Optimal policy for two-stage Brownian inventory systems under the long-run average cost criterion

On Strategic Measures and Optimality Properties in Discrete-Time Stochastic Control with Universally Measurable Policies

This paper concerns discrete-time infinite-horizon stochastic control systems with Borel state and action spaces and universally measurable policies. We study optimization problems on strategic measures induced by the policies in these systems. The results are then applied to risk-neutral and risk-sensitive Markov decision processes to establish the measurability of the optimal value functions and the existence of universally measurable, randomized or nonrandomized, ϵ-optimal policies, for a variety of average cost criteria and risk criteria. We also extend our analysis to a class of minimax control problems and establish similar optimality results under the axiom of analytic determinacy. Funding: This work was supported by grants from DeepMind, the Alberta Machine Intelligence Institute (AMII), and Alberta Innovates-Technology Futures (AITF).

Age of Information Cost Minimization with No Buffers, Random Arrivals and Unreliable Channels: A PCL-Indexability Analysis

Over the last decade, the Age of Information has emerged as a key concept and metric for applications where the freshness of sensor-provided data is critical. Limited transmission capacity has motivated research on the design of tractable policies for scheduling information updates to minimize Age of Information cost based on Markov decision models, in particular on the restless multi-armed bandit problem (RMABP). This allows the use of Whittle’s popular index policy, which is often nearly optimal, provided indexability (index existence) is proven, which has been recently accomplished in some models. We aim to extend the application scope of Whittle’s index policy in a broader AoI scheduling model. We address a model with no buffers incorporating random packet arrivals, unreliable channels, and nondecreasing AoI costs. We use sufficient indexability conditions based on partial conservation laws previously introduced by the author to establish the model’s indexability and evaluate its Whittle index in closed form under discounted and average cost criteria. We further use the index formulae to draw insights on how scheduling priority depends on model parameters.

Optimal Linear-Quadratic Regulator for a Stochastic System under Mutually Inverse Time Preferences in the Cost

We investigate a long-run behavior of a linear stochastic system. It is assumed that the quadratic cost includes a time-varying function and its multiplicative inverse. Such a specification reflects the fact that time preferences used by agents to assess different types of losses evolve in opposite directions. We consider the case when priority is set for the losses associated with state deviations. The optimal control law is derived with respect to extended long-run average cost criteria. We provide conditions for the existence of an alternative control strategy, which is also optimal and is based on a solution of an algebraic Riccati equation.

Inventory dispensation and restocking in multiclass inventory systems: A decoupling approach

We consider an inventory system with Poisson demand arrivals arising from multiple classes. Restocking epochs are specified by a given sequence. At each epoch, we can place an order for any number of items for immediate delivery. We have to decide the inventory dispensation policy, which determines whether we should satisfy a demand, or reject it, or backlog it until the next restocking epoch. We allow class‐dependent rejection and backlogging costs. We also incorporate holding costs and procurement costs. We obtain the optimal restocking and inventory dispensation policies under the total discounted cost criterion as well as the long‐run average cost criterion. In particular, we develop efficient algorithms to compute the optimal policies (OPs) by decoupling the inventory dispensation policy from the restocking policy. We show that the OP operates as follows: At the restocking epochs, we order up to a fixed base‐stock level. We satisfy the demands from a class if the inventory is above a critical level that depends on the class and time to the next restocking epoch; else we reject or backlog it depending on whether the next restocking epoch is later or earlier than a class‐dependent critical number. The numerical experiments for the stochastic and deterministic interstocking periods are conducted to compare the OPs and costs.

Discrete-time zero-sum games for Markov chains with risk-sensitive average cost criterion

We study zero-sum stochastic games for controlled discrete time Markov chains with risk-sensitive average cost criterion with countable/compact state space and Borel action spaces. The payoff function is nonnegative and possibly unbounded for countable state space case and for compact state space case it is a real-valued and bounded function. For countable state space case, under a certain Lyapunov type stability assumption on the dynamics we establish the existence of the value and a saddle point equilibrium. For compact state space case we establish these results without any Lyapunov type stability assumptions. Using the stochastic representation of the principal eigenfunction of the associated optimality equation, we completely characterize all possible saddle point strategies in the class of stationary Markov strategies. Also, we present and analyze an illustrative example.

A survey of average cost problems in deterministic discrete-time control systems

This paper concerns optimal control problems for infinite-horizon discrete-time deterministic systems with the long-run average cost (AC) criterion. This optimality criterion can be traced back to a paper by Bellman [6] for a class of Markov decision processes (MDPs). We present a survey of some of the main approaches to study the AC problem, namely, the AC optimality (or dynamic programming) equation, the steady state approach, and the vanishing discount approach, emphasizing the difference between the deterministic control problem and the corresponding (stochastic) MDP. Several examples illustrate these approaches and related results. We also state some open problems.

Technical Note—Average Cost Optimality in Partially Observable Lost-Sales Inventory Systems

In many real-life situations, the inventory record may not match the actual stock perfectly. This can happen due to distortion of inventory data, such as transaction errors, misplaced inventories, and spoilage. In these cases, because the decision maker only has incomplete information about the inventory levels, many well-known inventory policies are not even admissible, and our understanding of the optimal policies, even their existence, is very limited. In “Average Cost Optimality in Partially Observable Lost-Sales Inventory Systems,” Bai et al. consider the classical lost-sales inventory model, in which the inventory level is only observed when it becomes zero. They formulate the cost-minimization problem as a partially observable Markov decision process. By exploiting the vanishing discount factor approach, they provide a way to verify the existence of optimal policies under the average cost criterion. The key step in their analysis is the construction of a valid policy, which, in a certain sense, copies the actions of another policy for the process starting from another initial state.

On structural properties of optimal average cost functions in Markov decision processes with Borel spaces and universally measurable policies

We consider Markov decision processes (MDPs) with Borel state and action spaces and universally measurable policies. For several long-run average cost criteria and two classes of MDPs, we prove sufficient conditions for the optimal average cost functions to be constant almost everywhere with respect to certain σ-finite measures. Besides suitable boundedness conditions on the positive parts of the one-stage costs, the key condition here is that each subset of states with positive measure be reachable with probability one under some policy. Our proofs exploit an inequality for the optimal average cost functions and its connection with submartingales, and, in a special case that involves stationary policies, also use the theory of recurrent Markov chains.

On Linear Programming for Constrained and Unconstrained Average-Cost Markov Decision Processes with Countable Action Spaces and Strictly Unbounded Costs

We consider the linear programming approach for constrained and unconstrained Markov decision processes (MDPs) under the long-run average-cost criterion, where the class of MDPs in our study have Borel state spaces and discrete countable action spaces. Under a strict unboundedness condition on the one-stage costs and a recently introduced majorization condition on the state transition stochastic kernel, we study infinite-dimensional linear programs for the average-cost MDPs and prove the absence of a duality gap and other optimality results. Our results do not require a lower-semicontinuous MDP model. Thus, they can be applied to countable action space MDPs where the dynamics and one-stage costs are discontinuous in the state variable. Our proofs make use of the continuity property of Borel measurable functions asserted by Lusin’s theorem.

Average Cost Optimality in Partially Observed Lost-Sales Inventory Systems

We consider a partially observable inventory system in which the inventory level can only be observed when it reaches zero, the unmet demand is lost, and replenishment orders must be decided so as to minimize the long-run average cost. This problem has an infinite-dimensional state space and is therefore difficult to analyze. We prove the existence of an average-cost optimal policy, using the so-called vanishing discount factor approach. As a main methodological contribution, we provide a way to verify the key condition -- the uniform boundedness of the relative discounted value function -- for the partially observed system. To accomplish that we construct -- for the partially observed system -- a valid policy which in a certain sense copies the actions of another policy for the process with a different initial state. To our best knowledge, this paper is the first dealing with inventory models with partially observable inventory levels under the long-run average cost criterion.

Nonzero-sum Risk-Sensitive Average Stochastic Games: The Case of Unbounded Costs

In this paper, we study discrete-time nonzero-sum stochastic games under the risk-sensitive average cost criterion. The state space is a denumerable set, the action spaces of players are Borel spaces, and the cost functions are unbounded. Under suitable conditions, we first introduce the risk-sensitive first passage payoff functions and obtain their properties. Then, we establish the existence of a solution to the risk-sensitive average cost optimality equation of each player for the case of unbounded cost functions and show the existence of a randomized stationary Nash equilibrium in the class of randomized history-dependent strategies. Finally, we use a controlled population system to illustrate the main results.

Optimal preventive maintenance for repairable weighted k-out-of-n systems

Weighted k-out-of-n systems form an important class of redundant systems with a wide range of applications in reliability engineering. In this paper, we introduce novel optimal preventive maintenance models for the class of weighted k-out-of-n systems based on average cost and availability criteria. Depending on the average number of failed components, we set up a cost (availability) function corresponding to the time that the total weights of the working components become less than a predetermined threshold m, or the age of the system reaches TPM. The form of the cost (and availability) function relies on the mixture representation of the system reliability based on the notion of the signature of the system. Some examples of weighted k-out-of-n systems are presented to demonstrate the proposed models numerically and graphically.

Value iteration algorithm for mean-field games

In the literature, existence of mean-field equilibria has been established for discrete-time mean field games under both the discounted cost and the average cost optimality criteria. In this paper, we provide a value iteration algorithm to compute stationary mean-field equilibrium for both the discounted cost and the average cost criteria, whose existence proved previously. We establish that the value iteration algorithm converges to the fixed point of a mean-field equilibrium operator. Then, using this fixed point, we construct a stationary mean-field equilibrium. In our value iteration algorithm, we use Q -functions instead of value functions.

Policy Iteration Approach to the Infinite Horizon Average Optimal Control of Probabilistic Boolean Networks

This article studies the optimal control of probabilistic Boolean control networks (PBCNs) with the infinite horizon average cost criterion. By resorting to the semitensor product (STP) of matrices, a nested optimality equation for the optimal control problem of PBCNs is proposed. The Laurent series expression technique and the Jordan decomposition method derive a novel policy iteration-type algorithm, where finite iteration steps can provide the optimal state feedback law, which is presented. Finally, the intervention problem of the probabilistic Ara operon in E. coil, as a biological application, is solved to demonstrate the effectiveness and feasibility of the proposed theoretical approach and algorithms.

Infinite-Horizon Optimal Control of Switched Boolean Control Networks With Average Cost: An Efficient Graph-Theoretical Approach.

This study investigates the infinite-horizon optimal control (IHOC) problem for switched Boolean control networks with an average cost criterion. A primary challenge of this problem is the prohibitively high computational cost when dealing with large-scale networks. We attempt to develop a more efficient approach from a novel graph-theoretical perspective. First, a weighted directed graph structure called the optimal state transition graph (OSTG) is established, whose edges encode the optimal action for each admissible state transition between states reachable from a given initial state subject to various constraints. Then, we reduce the IHOC problem into a minimum-mean cycle (MMC) problem in the OSTG. Finally, we develop an algorithm that can quickly find a particular MMC by resorting to Karp's algorithm in the graph theory and construct an optimal switching control law based on state feedback. The time complexity analysis shows that our algorithm, albeit still running in exponential time, can outperform all the existing methods in terms of time efficiency. A 16-state-3-input signaling network in leukemia is used as a benchmark to test its effectiveness. Results show that the proposed graph-theoretical approach is much more computationally efficient and can reduce the running time dramatically: it runs hundreds or even thousands of times faster than the existing methods. The Python implementation of the algorithm is available at https://github.com/ShuhuaGao/sbcn_mmc.

Scheduling Queues with Customer Transfers

We consider a multiclass single-server system wherein customers of each class form their own queue and can either abandon the system or transfer to other queues while waiting. A system manager makes dynamic scheduling decisions to minimize system-wide waiting cost. We solve the stochastic control problem under both infinite-horizon discounted and average cost criteria. By studying a Brownian control problem that arises in the conventional heavy traffic regime, we identify (under each cost criterion) an asymptotically optimal policy that prioritizes across different classes according to a workload-dependent dynamic index rule. The proposed policies acknowledge the impact of customer transfers on the optimal allocation of service resources and generate rich managerial insights. One of the technical difficulties in the proof of the asymptotic optimality result is to identify a natural stability condition and formally show that the sequence of diffusion-scaled processes is uniformly stable under that condition.

Discrete-time average-cost mean-field games on Polish spaces

In stochastic dynamic games, when the number of players is sufficiently large and the interactions between agents depend on empirical state distribution, one way to approximate the original game is to introduce infinitepopulation limit of the problem. In the infinite population limit, a generic agent is faced with a so-called mean-field game. In this paper, we study discrete-time mean-field games with average-cost criteria. Using average cost optimality equation and Kakutani's fixed point theorem, we establish the existence of Nash equilibria for mean-field games under drift and minorization conditions on the dynamics of each agent. Then, we show that the equilibrium policy in the mean-field game, when adopted by each agent, is an approximate Nash equilibrium for the corresponding finite-agent game with sufficiently many agents.