Long-run Average Cost Criterion Research Articles

Ergodic Control of Bipartite Matching Queues with Class Change and Matching Failure

Motivated by transplant applications, we study a bipartite matching queue with multiclass customers and multitype resources. Customers may change their classes or abandon the system while waiting in queue, and they may decline the offered resource units which results in matching failure. We are interested in designing efficient instantaneous matching policies that allocate resources upon arrival to waiting customers. Our objective is bicriteria and formulated as a cost functional that linearly combines the long-run average expected reward due to successful matches and the long-run average expected cost from customer waiting and abandonment. We first develop a stability condition on the class change and abandonment rates, which requires at least one customer queue with abandonment and that any queue without abandonment have a class transition path to a queue with abandonment. Under this condition, we construct a simple linear program, referred to as the fluid control problem (FCP), which serves as a lower bound for the original stochastic control problem under any admissible policy. We then propose a randomized matching policy based on the solution of the FCP and show that the proposed policy is asymptotically optimal under both the long-run average and ergodic cost criteria. In addition, we apply our method to study two X matching models with two customer classes and two resource types to provide insights on how the class change and matching failure impact the optimal policies.

Optimal Linear-Quadratic Regulator for a Stochastic System under Mutually Inverse Time Preferences in the Cost

We investigate a long-run behavior of a linear stochastic system. It is assumed that the quadratic cost includes a time-varying function and its multiplicative inverse. Such a specification reflects the fact that time preferences used by agents to assess different types of losses evolve in opposite directions. We consider the case when priority is set for the losses associated with state deviations. The optimal control law is derived with respect to extended long-run average cost criteria. We provide conditions for the existence of an alternative control strategy, which is also optimal and is based on a solution of an algebraic Riccati equation.

A survey of average cost problems in deterministic discrete-time control systems

This paper concerns optimal control problems for infinite-horizon discrete-time deterministic systems with the long-run average cost (AC) criterion. This optimality criterion can be traced back to a paper by Bellman [6] for a class of Markov decision processes (MDPs). We present a survey of some of the main approaches to study the AC problem, namely, the AC optimality (or dynamic programming) equation, the steady state approach, and the vanishing discount approach, emphasizing the difference between the deterministic control problem and the corresponding (stochastic) MDP. Several examples illustrate these approaches and related results. We also state some open problems.

On structural properties of optimal average cost functions in Markov decision processes with Borel spaces and universally measurable policies

We consider Markov decision processes (MDPs) with Borel state and action spaces and universally measurable policies. For several long-run average cost criteria and two classes of MDPs, we prove sufficient conditions for the optimal average cost functions to be constant almost everywhere with respect to certain σ-finite measures. Besides suitable boundedness conditions on the positive parts of the one-stage costs, the key condition here is that each subset of states with positive measure be reachable with probability one under some policy. Our proofs exploit an inequality for the optimal average cost functions and its connection with submartingales, and, in a special case that involves stationary policies, also use the theory of recurrent Markov chains.

On Linear Programming for Constrained and Unconstrained Average-Cost Markov Decision Processes with Countable Action Spaces and Strictly Unbounded Costs

We consider the linear programming approach for constrained and unconstrained Markov decision processes (MDPs) under the long-run average-cost criterion, where the class of MDPs in our study have Borel state spaces and discrete countable action spaces. Under a strict unboundedness condition on the one-stage costs and a recently introduced majorization condition on the state transition stochastic kernel, we study infinite-dimensional linear programs for the average-cost MDPs and prove the absence of a duality gap and other optimality results. Our results do not require a lower-semicontinuous MDP model. Thus, they can be applied to countable action space MDPs where the dynamics and one-stage costs are discontinuous in the state variable. Our proofs make use of the continuity property of Borel measurable functions asserted by Lusin’s theorem.

Average Cost Optimality in Partially Observed Lost-Sales Inventory Systems

We consider a partially observable inventory system in which the inventory level can only be observed when it reaches zero, the unmet demand is lost, and replenishment orders must be decided so as to minimize the long-run average cost. This problem has an infinite-dimensional state space and is therefore difficult to analyze. We prove the existence of an average-cost optimal policy, using the so-called vanishing discount factor approach. As a main methodological contribution, we provide a way to verify the key condition -- the uniform boundedness of the relative discounted value function -- for the partially observed system. To accomplish that we construct -- for the partially observed system -- a valid policy which in a certain sense copies the actions of another policy for the process with a different initial state. To our best knowledge, this paper is the first dealing with inventory models with partially observable inventory levels under the long-run average cost criterion.

Optimal Drift Rate Control and Impulse Control for a Stochastic Inventory/Production System

In this paper, we consider joint drift rate control and impulse control for a stochastic inventory system under a long-run average cost criterion. Assuming the inventory level must be nonnegative, we prove that a $\{(0,q^{\star},Q^{\star},S^{\star}),\{\mu^{\star}(x): x\in[0, S^{\star}]\}\}$ policy is an optimal joint control policy, where the impulse control follows the control band policy $(0,q^{\star},Q^{\star},S^{\star})$, which brings the inventory level up to $q^{\star}$ once it drops to $0$ and brings it down to $Q^{\star}$ once it rises to $S^{\star}$, and the drift rate only depends on the current inventory level and is given by function $\mu^{\star}(x)$ for the inventory level $x\in[0,S^{\star}]$. The optimality of the $\{(0,q^{\star},Q^{\star},S^{\star}),\{\mu^{\star}(x): x\in[0,S^{\star}]\}\}$ policy is proven by using a lower bound approach, in which a critical step is to prove the existence and uniqueness of optimal policy parameters. To prove the existence and uniqueness, we develop a novel analytical method to solve a free boundary problem consisting of an ordinary differential equation and several free boundary conditions. Furthermore, we find that the optimal drift rate $\mu^{\star}(x)$ is first increasing and then decreasing as $x$ increases from 0 to $S^{\star}$ with a turnover point between $Q^{\star}$ and $S^{\star}$.

Local Poisson Equations Associated with Discrete-Time Markov Control Processes

This paper provides a characterization of the optimal average cost function, when the long-run (risk-sensitive) average cost criterion is used. The Markov control model has a denumerable state space with finite set of actions, and the characterization presented is given in terms of a system of local Poisson equations, which gives as a by-product the existence of an optimal stationary policy.

A Characterization of the Optimal Certainty Equivalent of the Average Cost via the Arrow-Pratt Sensitivity Function

This work is concerned with finite-state irreducible Markov decision chains satisfying continuity-compactness requirements. It is supposed that the system is driven by a decision maker with utility function U, which, aside mild conditions, is arbitrary, and the performance of a control policy is measured by the long-run average cost criterion induced by U. The main conclusions about this performance index are as follows: (i) the optimal U-average value function coincides with the optimal V-average index for a certain exponential utility V, and (ii) the average criteria associated with U and V have the same class of optimal stationary policies.

Effective inventory control policies with a minimum order quantity and batch ordering

In this paper, we consider a single-item periodic-review stochastic inventory system with both minimum order quantity (MOQ) and batch ordering requirements. In each time period, the firm can order either none or at least as much as the MOQ. At the same time, if an order is placed, the order quantity is required to be an integral multiple of a given specific batch size. We first adopt a heuristic policy which is specified by two parameters (s,t). Applying a discrete time Markov chain approach, we compute the system cost and optimize this (s,t) policy under the long-run average cost criterion. We also consider a simpler one-parameter policy, the so-called S policy, which is a special case of the (s,t) policy. In an intensive numerical study, we find that (1) both policies perform well in comparison with other policies and (2) the S policy also performs well and is compatible with the (s,t) policy; only in a few cases where demand variation is small, the latter outperforms the former significantly. We also evaluate the effects of some important parameters on system performance.

Flexible Server Allocation and Customer Routing Policies for Two Parallel Queues When Service Rates Are Not Additive

We consider the question of how routing and allocation can be coordinated to meet the challenge of demand variability in a parallel queueing system serving two types of customers. A decision maker decides whether to keep customers at the station at which they arrived or to reroute them to the other station. At the same time, the decision maker has two servers and must decide where to allocate their effort. We analyze this joint decision-making scenario with both routing and station-dependent holding costs, but add an important twist. We allow the combined service rate (when the servers work at the same station) to be superadditive or subadditive. This captures positive or negative externalities that arise during collaboration. We seek an optimal control policy under the discounted or long-run average cost criteria. Our results show that in the superadditive case jobs should never be routed away from the lower-cost queue. When jobs are rerouted from the higher-cost queue to the low-cost queue the optimal control is monotone in the respective queue lengths. Moreover, we show that the optimal allocation is a nonidling priority rule based on the holding costs. In the subadditive case we find that the optimal policy need not exhibit such a simple structure. In fact, the optimal allocation need not prioritize one station (it may split the servers), and the optimal routing need not be monotone in the number of customers in each queue. We characterize the optimal policy for a few canonical cases and discuss why intuitive policies need not be optimal in the general case. An extensive numerical study examines the benefit of dynamically controlling both routing and resource allocation; we discuss when using one of the two levers—dynamic routing and dynamic allocation—is sufficient and when using both controls is warranted.

In This Issue

In This Issue

Performance Evaluation of Modified Echelon <i>(r, Q)</I> Policies for Stochastic Serial Inventory Systems

We consider the classic continuous-review N-stage serial inventory system with a homogeneous Poisson demand arrival process at the most downstream stage (Stage 1). Any shipment to each stage, regardless of its size, incurs a positive fixed setup cost and takes a positive constant leadtime. The optimal policy for this system under the long-run average cost criterion is unknown. Finding a good worst-case performance guarantee for this problem remains an open problem. We tackle this problem by introducing a class of modified echelon (r, Q) policies that do not require Q_{i 1}/Q_i to be a positive integer: Stage i 1 ships to Stage i based on its observation of the echelon inventory position at Stage i; if it is at or below r_i and Stage i 1 has positive on-hand inventory, then a shipment is sent to Stage i to raise its echelon inventory position to r_i Q_i as close as possible. We construct a heuristic policy within this class of policies, which has the following features: First, it has provably primitive-dependent performance bounds. In a two-stage system, the performance of the heuristic policy is guaranteed to be within (1 K_1/K_2) times of the optimal cost, where K_1 is the downstream fixed cost and K_2 is the upstream fixed cost. We also provide an alternative performance bound, which depends on efficiently-computable optimal (r, Q) solutions to N single-stage systems, but tends to be tighter. Second, the heuristic is simple, efficiently computable and numerically performs well, likely to even outperform the optimal integer-ratio echelon (r,Q) policies when K_1 is dominated by K_2. Third, the heuristic is asymptotically optimal, as we take some dominant relationships, between the setup or holding cost primitives at an upstream stage and its immediate downstream stage, to the extreme, e.g., when h_2/h_1\rightarrow0, where h_1 is the downstream holding cost parameter and h_2 is the upstream holding cost parameter.

When to challenge a call in tennis: A Markov decision process approach

In this paper, we develop a Markov decision process (MDP) model to determine when a player should challenge a line call in a game of tennis, if the objective is to maximize the probability of winning the game. The parameters in the model include the relative strength of the players and the fallibility of the officials. The player’s decision depends on the the current score, the number of challenges remaining, the outcome of a successful challenge, and his confidence that the line call is incorrect. The model developed is a multi-chain MDP operating under the long-run average cost criterion. We also performed extensive numerical studies when the player has one challenge remaining, varying the player strengths and the fallibility levels. These studies imply some general intuitive challenge strategies but also exhibit unusual strategic behavior in some game states. For example, in some states it is not optimal for very weak or very strong players to challenge calls. Furthermore, we demonstrate that the challenge decision is not “unimodal” as a function of the player’s strength, i.e., there may be multiple decision thresholds with respect to this parameter.

Modified Echelon (r, Q) Policies with Guaranteed Performance Bounds for Stochastic Serial Inventory Systems

We consider the classic continuous-review Nstage serial inventory system with a homogeneous Poisson demand arrival process at the most downstream stage (Stage 1). Any shipment to each stage, regardless of its size, incurs a positive fixed setup cost and takes a positive constant lead time. The optimal policy for this system under the long-run average cost criterion is unknown. Finding a good worst-case performance guarantee remains an open problem. We tackle this problem by introducing a class of modified echelon (r, Q) policies that do not require Qi + 1/Qi to be a positive integer: Stage i + 1 ships to Stage i based on its observation of the echelon inventory position at Stage i; if it is at or below ri and Stage i + 1 has positive on-hand inventory, then a shipment is sent to Stage i to raise its echelon inventory position to ri + Qi as close as possible. We construct a heuristic policy within this class of policies, which has the following features: First, it has provably primitive-dependent performance bounds. In a two-stage system, the performance of the heuristic policy is guaranteed to be within (1 + K1/K2) times the optimal cost, where K1 is the downstream fixed cost and K2 is the upstream fixed cost. We also provide an alternative performance bound, which depends on efficiently computable optimal (r, Q) solutions to N single-stage systems but tends to be tighter. Second, the heuristic is simple, it is efficiently computable and it performs well numerically; it is even likely to outperform the optimal integer-ratio echelon (r,Q) policies when K1 is dominated by K2. Third, the heuristic is asymptotically optimal when we take some dominant relationships between the setup or holding cost primitives at an upstream stage and its immediate downstream stage to the extreme, for example, when h2/h1 → 0, where h1 is the downstream holding cost parameter and h2 is the upstream holding cost parameter.

Quantifying the risk in age and block replacement policies

Preventive maintenance policies have been studied in the literature without considering the risk due to the cost variability. In this paper, we consider the two most popular preventive replacement policies, namely, age and block replacement policies under long-run average cost and expected unit time cost criteria. To quantify the risk in the preventive maintenance policies, we use the long-run variance of the accumulated cost over a time interval. We numerically derive the Risk-sensitive preventive replacement policies and study the impact of the Risk-sensitive optimality criterion on the managerial decisions. We also examine the performance of the expected unit time cost criterion as an alternative to the traditional long-run average cost criterion.

Average Cost and Stability of Time-Varying Linear Systems

<para xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> In this technical note, the stability for time-varying discrete-time stochastic linear systems, with possibly unbounded trajectories, is associated to the existence of the long-run average cost criteria. Under controllability and observability, the stochastic system is stable in the sense that its state tends asymptotically to the origin in the mean. Further conditions related to a lower bound on the long-run average cost, or with periodic time-varying systems, provides uniform second moment stability. </para>

Solutions of the average cost optimality equation for finite Markov decision chains: risk-sensitive and risk-neutral criteria

This work is concerned with controlled Markov chains with finite state and action spaces. It is assumed that the decision maker has an arbitrary but constant risk sensitivity coefficient, and that the performance of a control policy is measured by the long-run average cost criterion. Within this framework, the existence of solutions of the corresponding risk-sensitive optimality equation for arbitrary cost function is characterized in terms of communication properties of the transition law.

On the existence of stationary optimal policies for partially observed MDPs under the long-run average cost criterion

This paper studies the problem of the existence of stationary optimal policies for finite state controlled Markov chains, with compact action space and imperfect observations, under the long-run average cost criterion. It presents sufficient conditions for existence of solutions to the associated dynamic programming equation, that strengthen past results. There is a detailed discussion comparing the different assumptions commonly found in the literature.

Drift rate control of a Brownian processing system

A system manager dynamically controls a diffusion process Z that lives in a finite interval [0,b]. Control takes the form of a negative drift rate θ that is chosen from a fixed set A of available values. The controlled process evolves according to the differential relationship dZ=dX−θ(Z) dt+dL−dU, where X is a (0,σ) Brownian motion, and L and U are increasing processes that enforce a lower reflecting barrier at Z=0 and an upper reflecting barrier at Z=b, respectively. The cumulative cost process increases according to the differential relationship dξ=c(θ(Z)) dt+p dU, where c(⋅) is a nondecreasing cost of control and p>0 is a penalty rate associated with displacement at the upper boundary. The objective is to minimize long-run average cost. This problem is solved explicitly, which allows one to also solve the following, essentially equivalent formulation: minimize the long-run average cost of control subject to an upper bound constraint on the average rate at which U increases. The two special problem features that allow an explicit solution are the use of a long-run average cost criterion, as opposed to a discounted cost criterion, and the lack of state-related costs other than boundary displacement penalties. The application of this theory to power control in wireless communication is discussed.