Long-run Average Criterion Research Articles

Zero-Sum Non-stationary Stochastic Games with the Long-Run Average Criterion

Zero-Sum Non-stationary Stochastic Games with the Long-Run Average Criterion

Asymptotically optimal routing of a many-server parallel queueing system with long-run average criterion

This paper considers a parallel queueing system with multiple stations, each of which contains many statistically identical servers and has a dedicated queue. Upon each customer arrival, the system manager must decide to which station the customer should be routed, with the objective of minimizing the system’s long-run average delay cost. One feature of this paper is that a customer’s delay cost depends not only on his/her delay, but also on the routed station. Considering this heterogeneity across stations, we propose a routing policy, which can be regarded as an extension of the MED–FSF policy. Under this policy, any arriving customer will be routed to: (i) the station with the minimum value, which depends on the station’s expected delay and the station index when servers in all stations are fully occupied; or otherwise (ii) the station with a fastest idle server. Using asymptotic analysis, we derive diffusion limits of queue-length processes and their stationary distributions under the proposed policy in the Halfin–Whitt regime. Combined with an asymptotic lower bound result for the long-run average delay cost, we show that the proposed routing policy is asymptotically optimal under the considered objective. Finally, we provide numerical experiments to validate the accuracy of our diffusion approximation, and we compare the performance metrics under the proposed policy with those under other commonly used routing policies.

Optimal drift rate control and two-sided impulse control for a Brownian system with the long-run average criterion

Abstract In this paper, we consider a joint drift rate control and two-sided impulse control problem in which the system manager adjusts the drift rate as well as the instantaneous relocation for a Brownian motion, with the objective of minimizing the total average state-related cost and control cost. The system state can be negative. Assuming that instantaneous upward and downward relocations take a different cost structure, which consists of both a setup cost and a variable cost, we prove that the optimal control policy takes an $\left\{ {\!\left( {{s^{\ast}},{q^{\ast}},{Q^{\ast}},{S^{\ast}}} \right),\!\left\{ {{\mu ^{\ast}}(x)\,:\,x \in [ {{s^{\ast}},{S^{\ast}}}]} \right\}} \right\}$ form. Specifically, the optimal impulse control policy is characterized by a quadruple $\left( {{s^{\ast}},{q^{\ast}},{Q^{\ast}},{S^{\ast}}} \right)$ , under which the system state will be immediately relocated upwardly to ${q^{\ast}}$ once it drops to ${s^{\ast}}$ and be immediately relocated downwardly to ${Q^{\ast}}$ once it rises to ${S^{\ast}}$ ; the optimal drift rate control policy will depend solely on the current system state, which is characterized by a function ${\mu ^{\ast}}\!\left( \cdot \right)$ for the system state staying in $[ {{s^{\ast}},{S^{\ast}}}]$ . By analyzing an associated free boundary problem consisting of an ordinary differential equation and several free boundary conditions, we obtain these optimal policy parameters and show the optimality of the proposed policy using a lower-bound approach. Finally, we investigate the effect of the system parameters on the optimal policy parameters as well as the optimal system’s long-run average cost numerically.

Learning the Best Price and Ordering Policy under Fixed Costs and Ambiguous Demand

We study joint inventory-price control in which a firm chooses among a finite number of prices to influence the demand to be realized; also, the firm’s ordering activities incur fixed setup costs. While intending to settle down on an optimal price and figure out an optimal ordering policy all catering to the long-run average criterion, the firm is ambiguous about the stationary distribution of the random demand that it is to face under each price. We propose an adaptive policy in which periods are grouped into intervals, with each being associated with one single price and one single ordering policy. Pricing is based on a learning-while-doing trade-off: a price with the least number of interval visits will be chosen when this number is below a threshold associated with the total number of interval visits under all prices; otherwise, the chosen price will be one with the most promising profit prospect estimated from past experiences. Interval-wise ordering relies on an (s,S) policy most suitable for the empirical distribution learned from past experiences under the chosen price. The power at which the policy’s regret grows in the horizon length T would be below (3 +√29)/10 ≃ 0.839 even when demand patterns are fairly ambiguous. When demand realizations are further confined to a finite support, the bound would be reducible to √2/2 ≃ 0.707.

LP Formulations of Discrete Time Long-Run Average Optimal Control Problems: The NonErgodic Case

We formulate and study the infinite dimensional linear programming (LP) problem associated with the deterministic discrete time long-run average criterion optimal control problem. Along with its du...

Linear programming formulations of deterministic infinite horizon optimal control problems in discrete time

This paper is devoted to a study of infinite horizon optimal control problems with time discounting and time averaging criteria in discrete time. We establish that these problems are related to certain infinite-dimensional linear programming (IDLP) problems. We also establish asymptotic relationships between the optimal values of problems with time discounting and long-run average criteria.

Performance analysis of a reflected fluid production/inventory model

We study the performance of a reflected fluid production/inventory model operating in a stochastic environment that is modulated by a finite state continuous time Markov chain. The process alternates between ON and OFF periods. The ON period is switched to OFF when the content level reaches a predetermined level q and returns to ON when it drops to 0. The ON/OFF periods generate an alternative renewal process. Applying a matrix analytic approach, fluid flow techniques and martingales, we develop methods to obtain explicit formulas for the cost functionals (setup, holding, production and lost demand costs) in the discounted case and under the long-run average criterion. Numerical examples present the trade-off between the holding cost and the loss cost and show that the total cost appears to be a convex function of q.

A Fluid EOQ Model with Markovian Environment

We consider a production-inventory model operating in a stochastic environment that is modulated by a finite state continuous-time Markov chain. When the inventory level reaches zero, an order is placed from an external supplier. The costs (purchasing and holding costs) are modulated by the state at the order epoch time. Applying a matrix analytic approach, fluid flow techniques, and martingales, we develop methods to obtain explicit equations for these cost functionals in the discounted case and under the long-run average criterion. Finally, we extend the model to allow backlogging.

A Fluid EOQ Model with Markovian Environment

We consider a production-inventory model operating in a stochastic environment that is modulated by a finite state continuous-time Markov chain. When the inventory level reaches zero, an order is placed from an external supplier. The costs (purchasing and holding costs) are modulated by the state at the order epoch time. Applying a matrix analytic approach, fluid flow techniques, and martingales, we develop methods to obtain explicit equations for these cost functionals in the discounted case and under the long-run average criterion. Finally, we extend the model to allow backlogging.

A Fluid EOQ Model with Markovian Environment

We consider a production-inventory model operating in a stochastic environment that is modulated by a finite state continuous-time Markov chain. When the inventory level reaches zero, an order is placed from an external supplier. The costs (purchasing and holding costs) are modulated by the state at the order epoch time. Applying a matrix analytic approach, fluid flow techniques, and martingales, we develop methods to obtain explicit equations for these cost functionals in the discounted case and under the long-run average criterion. Finally, we extend the model to allow backlogging.

Group replacement policies for a repairable cold standby system with fixed lead times

Consider a multi-component repairable cold standby system and assume that repaired units are as good as new. The operational times of the units follow phase-type distribution. Downtime cost is occurred when failed components are not repaired or replaced. There are also fixed, unit repair, and replacement costs associated with the maintenance facility, which are carried out after a fixed lead time τ. Closed-form results are derived for three classes of group replacement policies (m-failure, T-age, and (m, T, τ), which is a refinement of the classic (m, T) policy) for the expected discounted case and for the long-run average criteria. Illustrative examples are provided.

Policy Iteration Algorithms for Zero-Sum Stochastic Differential Games with Long-Run Average Payoff Criteria

This paper studies the policy iteration algorithm (PIA) for zero-sum stochastic differential games with the basic long-run average criterion, as well as with its more selective version, the so-called bias criterion. The system is assumed to be a nondegenerate diffusion. We use Lyapunov-like stability conditions that ensure the existence and boundedness of the solution to certain Poisson equation. We also ensure the convergence of a sequence of such solutions, of the corresponding sequence of policies, and, ultimately, of the PIA.

A make-to-stock production/inventory model with MAP arrivals and phase-type demands

We consider a make-to-stock production/inventory model in a random environment with finite storage capacity and restricted backlogging possibility. Our aim is to demonstrate that all cost quantities of interest can be derived in closed form under quite general assumptions on the demand arrival process and on the switches in the production rates. Specifically, the demands arrive according to a Markov additive process governed by a continuous-time Markov chain, and their sizes are independent and have phase-type distributions depending on the type of arrival. The production process switches between predetermined rates which depend on the state of the environment and on the presence or absence of backlogs. Four types of costs are considered: the holding cost for the stock, the cost of lost production due to the finite storage capacity, the shortage cost for the backlogged demand and the cost due to unsatisfied demand. We obtain explicit formulas for these cost functionals in the discounted case and under the long-run average criterion. The derivations are based on optional sampling of a multi-dimensional martingale and on fluid flow techniques.

Average optimal strategies for zero-sum Markov games with poorly known payoff function on one side

We are concerned with two-person zero-sum Markov games with Borel spaces under a long-run average criterion. The payoff function is possibly unbounded and depends on a parameter which is unknown to one of the players. The parameter and the payoff function can be estimated by implementing statistical methods. Thus, our main objective is to combine such estimation procedure with a variant of the so-called vanishing discount approach to construct an average optimal pair of strategies for the game. Our results are applied to a class of zero-sum semi-Markov games.

Illustrated review of convergence conditions of the value iteration algorithm and the rolling horizon procedure for average-cost MDPs

This paper is concerned with the links between the Value Iteration algorithm and the Rolling Horizon procedure, for solving problems of stochastic optimal control under the long-run average criterion, in Markov Decision Processes with finite state and action spaces. We review conditions of the literature which imply the geometric convergence of Value Iteration to the optimal value. Aperiodicity is an essential prerequisite for convergence. We prove that the convergence of Value Iteration generally implies that of Rolling Horizon. We also present a modified Rolling Horizon procedure that can be applied to models without analyzing periodicity, and discuss the impact of this transformation on convergence. We illustrate with numerous examples the different results. Finally, we discuss rules for stopping Value Iteration or finding the length of a Rolling Horizon. We provide an example which demonstrates the difficulty of the question, disproving in particular a conjectured rule proposed by Puterman.

Structural Properties of Optimal Scheduling Policies for Wireless Data Transmission

We analyze a cell with a fixed number of users in a time period network. The base station schedules to serve at most one user in a given time period based on information about the available data rates and other parameter(s) for all the users in the cell. We consider infinitely backlogged queues and model the system as a Markov Decision Process (MDP) and prove the monotonicity of the optimal policy with respect to the starvation age and the available data rate. For this, we consider both the discounted as well as the long-run average criterion. The proofs of the monotonicity properties serve as good illustrations of analyzing MDPs with respect to their optimal solutions.

Semi-Infinite Weighted Markov Decision Processes

In this article, weighted reward Markov decision processes (semi-infinite WMDP) with finite state and countable action spaces are considered. The “weighted reward” refers to an appropriately normalized convex combination of the discounted and long-run average reward criteria. This criterion allows the controller to trade-off short-term rewards versus long-run rewards. We prove that under some conditions, the supremum in the class of general strategies is equivalent to the supremum in the class of relatively simple “ultimately deterministic” strategies. These are strategies that behave just like deterministic stationary strategies, after a certain point in time. We present an iterative algorithm for computing a δ-optimal simple ultimately deterministic strategy. The steps of the algorithm are based on the one developed in Ref. for finite WMDP.

Maximisation of long-run average profit through dynamic control of production and pricing in a manufacturing system

This paper concerns the joint management problem of production control and dynamic pricing to balance the finished goods inventory and market demand in a make-to-stock manufacturing system using a long-run average profit criterion. In the system, the production rate is random, with a controllable mean rate, and the demand is Markov, with a changeable mean rate which depends on the sale price. The management issue is how to dynamically adjust the production rate and the selling price to maximise the long-run average profit. We discover that the optimal policy of dynamic pricing and production control over an infinite horizon is a matter of thresholds. An effective algorithm is suggested.

Comments on: Dynamic priority allocation via restless bandit marginal productivity indices

There is an abundance of real-life situations of the following nature. Several projects, of various types, need to be done by some person, machine, or other device. These require certain amounts of time, while costs (or rewards) are associated with performing such projects, delaying them, or switching between projects. The problem now is: Find a policy that determines, for given numbers of projects of the various types, on which project one should work, so as to minimize (discounted or average) costs. Examples of this problem turn up in administrative settings, in manufacturing, in computer-communications, etc. The problem of dynamically prioritizing projects (traffic classes) may often be formulated as a Markov decision problem, but the state space explosion makes the numerical solution of realistically sized problems usually prohibitive. An important class of heuristic policies, which often provide near-optimal solutions (and sometimes even can be shown to be optimal), is the class of priority index policies. A static index rule employed in minimizing completion costs in single-machine scheduling is the ratio of holding cost rate to processing time: work on the job with highest index. A dynamic index rule has been shown (Gittins and Jones (1974), as referenced in Nino-Mora 2007) to be optimal for the multi-armed bandit problem, viz., sequential allocation of work to a collection of stochastic projects (=bandits), so as to maximize the expected total discounted reward earned over an infinite horizon. Whittle (1988) discusses the multi-armed restless bandit problem, i.e., bandits can change state while being passive, under the long-run average criterion.

Semi-infinite weighted Markov decision processes with perturbation

In this paper, Weighted reward Perturbed Markov Decision Processes with finite state and countable action spaces (semi-infinite WMDP for short) are considered. The ”weighted reward” refers to appropriately normalized convex combination of the discounted and the long-run average reward criteria. This criterion allows the controller to trade-off short-term rewards versus long-run rewards. In every application where both the discounted and the long-run average criteria have been proposed in the past, there is clearly a rationale for considering the weighted criterion. Of course, as with all Markov decision models, the standard weighted criterion model assumes that all the transition probabilities are known precisely. Since, in most applications this would not be the case, we consider the perturbed version of the weighted reward model. In the case of perturbations, we prove that for many models a nearly optimal strategy can be found in the class of relatively “simple ultimately deterministic” strategies. These are strategies which behave just like deterministic stationary strategies, after a certain point of time.