Long-run Average Cost Criterion Research Articles

Drift rate control of a Brownian processing system

A system manager dynamically controls a diffusion process Z that lives in a finite interval [0,b]. Control takes the form of a negative drift rate θ that is chosen from a fixed set A of available values. The controlled process evolves according to the differential relationship dZ=dX−θ(Z) dt+dL−dU, where X is a (0,σ) Brownian motion, and L and U are increasing processes that enforce a lower reflecting barrier at Z=0 and an upper reflecting barrier at Z=b, respectively. The cumulative cost process increases according to the differential relationship dξ=c(θ(Z)) dt+p dU, where c(⋅) is a nondecreasing cost of control and p>0 is a penalty rate associated with displacement at the upper boundary. The objective is to minimize long-run average cost. This problem is solved explicitly, which allows one to also solve the following, essentially equivalent formulation: minimize the long-run average cost of control subject to an upper bound constraint on the average rate at which U increases. The two special problem features that allow an explicit solution are the use of a long-run average cost criterion, as opposed to a discounted cost criterion, and the lack of state-related costs other than boundary displacement penalties. The application of this theory to power control in wireless communication is discussed.

Optimal Predictive Maintenance Policies for a Deteriorating System: The Total Discounted Cost and the Long-Run Average Cost Cases

A predictive maintenance policy for a system subject to degradation is considered. Over time, the system is subject to degradation. The degradation level is observed only at successive inspection times. The system has a nominal life Y, and it fails once the degradation exceeds the nominal life. Failure is detected only by inspection, and is then maintained (corrective maintenance). The system is also maintained when the degradation level exceeds a predetermine level M(preventive maintenance). Each preventive maintenance costs c 1, while the cost of a corrective maintenance is equal to c 1 + d, d ≥ 0. At any inspection, before failure, the cost of inspection is state dependent: if the degradation level at inspection is x, then an inspection cost of c(x) occurs. We determine the optimal maintenance level M using the total discounted cost as well as the long-run average cost criteria. Our results extend those of Grall et al. [Grall, A., Dieulle, L., Berenguer, C., Roussignol, M. (2002). Continuous-time predictive-maintenance scheduling for a deteriorating system. IEEE Trans. Reliability 51:141–149].

Optimal control of dams using Pλ, τM policies and penalty cost

We discuss recent results on optimal control of dams using P λ, τ M policies and penalty cost. Specifically, we assume that the water is released at one of two rates, zero or M, per a unit of time. The release rate is zero until the water reaches level λ, and then the water is released at rate M, until it reaches level τ(τ < λ). Once the water reaches level T again the release rate remains zero until level λ is reached again and the cycle is repeated. At any time, the release rate can be switched from zero to M, with a starting cost of MK 1, or switched from M to zero at a closing cost of MK 2. For each unit of output, a reward A is received. Moreover, there is a penalty cost that accrues at rate g, where g is a bounded measurable function defined on the state space of the content process. We determine the optimal control policy according to the total discounted cost and the long-run average cost criteria.

Optimal maintenance of systems subject to deterioration of the renewal type

We review some of the recent work on optimal maintenance of devices subject to wear of the cumulative renewal process type. The optimality considered using the total discounted cost over the infinite horizon, and the long-run average cost criterion.

Optimal control of a dam using Pλ,τM policies and penalty cost when the input process is a compound Poisson process with positive drift

In this paper we consider the optimal control of an infinite dam using policies assuming that the input process is a compound Poisson process with a non-negative drift term, and using the total discounted cost and long-run average cost criteria. The results of Lee and Ahn (1998) as well as other well-known results are shown to follow from our results.

Optimal control of a dam using P λ,τ M policies and penalty cost when the input process is a compound Poisson process with positive drift

In this paper we consider the optimal control of an infinite dam using policies assuming that the input process is a compound Poisson process with a non-negative drift term, and using the total discounted cost and long-run average cost criteria. The results of Lee and Ahn (1998) as well as other well-known results are shown to follow from our results.

The Stochastic Economic Lot Scheduling Problem: Heavy Traffic Analysis of Dynamic Cyclic Policies

We consider two queueing control problems that are stochastic versions of the economic lot scheduling problem: A single server processes N customer classes, and completed units enter a finished goods inventory that services exogenous customer demand. Unsatisfied demand is backordered, and each class has its own general service time distribution, renewal demand process, and holding and backordering cost rates. In the first problem, a setup cost is incurred when the server switches class, and the setup cost is replaced by a setup time in the second problem. In both problems we employ a long-run average cost criterion and restrict ourselves to a class of dynamic cyclic policies, where idle periods and lot sizes are state-dependent, but the N classes must be served in a fixed sequence. Motivated by existing heavy traffic limit theorems, we make a time scale decomposition assumption that allows us to approximate these scheduling problems by diffusion control problems. Our analysis of the approximating setup cost problem yields a closed-form dynamic lot-sizing policy and a computational procedure for an idling threshold. We derive structural results and an algorithmic procedure for the setup time problem. A computational study compares the proposed policy and several alternative policies to the numerically computed optimal policy.

Constrained markov decision processes with compact state and action spaces: the average case

Constrained Markov decision processes with compact state and action spaces are studied under long-run average reward or cost criteria. By introducing a corresponding Lagrange function, a saddle-point theorem is given, by which the existence of a constrained optimal pair of initial state distribution and policy is shown. Also, under the hypothesis of Doeblin, the functional characterization of a constrained optimal policy is obtained

Dynamic routing and jockeying controls in a two-station queueing system

This paper studies optimal routing and jockeying policies in a two-station parallel queueing system. It is assumed that jobs arrive to the system in a Poisson stream with rate λand are routed to one of two parallel stations. Each station has a single server and a buffer of infinite capacity. The service times are exponential with server-dependent rates, μ 1 and μ 2. Jockeying between stations is permitted. The jockeying cost is cij when a job in station i jockeys to station j, i ≠ j. There is no cost when a new job joins either station. The holding cost in station j is hj, h 1 ≦ h 2, per job per unit time. We characterize the structure of the dynamic routing and jockeying policies that minimize the expected total (holding plus jockeying) cost, for both discounted and long-run average cost criteria. We show that the optimal routing and jockeying controls are described by three monotonically non-decreasing functions. We study the properties of these control functions, their relationships, and their asymptotic behavior. We show that some well-known queueing control models, such as optimal routing to symmetric and asymmetric queues, preemptive or non-preemptive scheduling on homogeneous or heterogeneous servers, are special cases of our system.

Dynamic routing and jockeying controls in a two-station queueing system

This paper studies optimal routing and jockeying policies in a two-station parallel queueing system. It is assumed that jobs arrive to the system in a Poisson stream with rate λand are routed to one of two parallel stations. Each station has a single server and a buffer of infinite capacity. The service times are exponential with server-dependent rates, μ1 and μ2. Jockeying between stations is permitted. The jockeying cost is cij when a job in station i jockeys to station j, i ≠ j. There is no cost when a new job joins either station. The holding cost in station j is hj, h1 ≦ h2, per job per unit time. We characterize the structure of the dynamic routing and jockeying policies that minimize the expected total (holding plus jockeying) cost, for both discounted and long-run average cost criteria. We show that the optimal routing and jockeying controls are described by three monotonically non-decreasing functions. We study the properties of these control functions, their relationships, and their asymptotic behavior. We show that some well-known queueing control models, such as optimal routing to symmetric and asymmetric queues, preemptive or non-preemptive scheduling on homogeneous or heterogeneous servers, are special cases of our system.

Finite state markovian decision processes with weak and strong interactions

This paper is concerned with hierarchical control of a class of finite state continuous time Markovian decision processes having both weak and strong interactions. The state of the process can be divided into several groups such that transitions among the states within each group occur much more frequently than the transitions among the states belonging to different groups. This gives rise to a limiting control problem which is simpler to solve. The limiting problem is obtained by replacing the states in each group by the corresponding average distribution. A nearly optimal solution for the original problem is constructed by using an optimal solution to the limiting problem. Both discounted cost and long-run average cost criteria are considered. Error bounds of constructed controls are obtained for problems with both cost criteria. Finally, computational methods are discussed.

Optimality of a Threshold Policy in theM/M/ queue with repeated vacations

Consider anM/M/1 queueing system with server vacations where the server is turned off as soon as the queue gets empty. We assume that the vacation durations form a sequence of i.i.d. random variables with exponential distribution. At the end of a vacation period, the server may either be turned on if the queue is non empty or take another vacation. The following costs are incurred: a holding cost ofh per unit of time and per customer in the system and a fixed cost of γ each time the server is turned on. We show that there exists a threshold policy that minimizes the long-run average cost criterion. The approach we use was first proposed in Blanc et al. (1990) and enables us to determine explicitly the optimal threshold and the optimal long-run average cost in terms of the model parameters.

Martingale limit theorem and its application to an ergodic controlled Markov chain

Slightly different sufficient conditions for the martingale limit theorem of Loève (1978) are studied for an adapted sequence of random variables { X i , F i , i ⩾ 1}. As an application of this form of the martingale limit theorem, existence of an optimal control for an ergodic controlled Markov chain with long-run average cost criterion is established under a weaker condition than Borkar's.

Average Reward Markov Decision Processes with Multiple Cost Constraints

Abstract We consider constrained Markov decision processes (MDP’s) with compact state and action spaces under long-run average reward or cost criteria, and give the characterization of an optimal pair of initial state distribution and policy, which maximize over all policies the essential infimum of the sample-path average reward subject to multiple average cost constraints. First, applying the idea of occupation measures constrained MDP’s considered here are equivalently transfered to the infinite linear programming and the existence of an optimal pair associated with a randomized stationary policy is shown by compact analysis. Secondly, we introduce the notion of a state-wise mixed stationary policy (s.w.m. policy) to get further results. And for any e>0, the existence of an e-optimal pair associated with a s.w.m. policy is inductively proved by using a Lagrange multiplier. In case of countable state space, the existence of an optimal pair associated with a s.w.m. policy is shown.

Optimal Manufacturing of a Two Product Mix

We consider a machine that works constantly and produces two types of fluid products, one type of product at a time. Whenever there is a positive amount of both types of products, the products are mixed immediately and shipped away. Whenever mixing is infeasible the product that is currently being produced accumulates inventory. The goal is to minimize the long-run average as well as the discounted costs of running such an operation. Inventory incurs a linear holding cost for each type of product. Also, whenever switching from the production of one type of product to the other, a fixed switching cost is incurred. It is found that a stationary periodic policy is optimal. The optimal solution for the long-run average cost criterion is described by closed form formulas. As a special case, we show that the (s, S) policy for the textbook EOQ inventory model is indeed optimal over the class of all policies. Also, a theorem that proves the optimality of cyclic policies within a particular set of policies is introduced in a very general setting.

Optimal Control of the Vacation Scheme in an M/G/1 Queue

In this article the M/G/1 queue with server vacations is considered with the assumption that the decision whether or not to take a new vacation, when the system is empty, depends on the number of vacations already taken through a random outcome. Both descriptive and optimization issues are considered, where the latter is done under the expected long-run average cost criterion with linear holding costs, fixed setup costs and a concave piecewise linear reward function for being on vacation. The optimization problem results in an infinite dimensional fractional program of which the solution yields a (deterministic) policy of the control limit type.

Analysis of an identification algorithm arising in the adaptive estimation of Markov chains

We investigate an algorithm applied to the adaptive estimation of partially observed finite-state Markov chains. The algorithm utilizes the recursive equation characterizing the conditional distribution of the state of the Markov chain, given the past observations. We show that the process “driving” the algorithm has a unique invariant measure for each fixed value of the parameter, and following the ordinary differential equation method for stochastic approximations, establish almost sure convergence of the parameter estimates to the solutions of an associated differential equation. The performance of the adaptive estimation scheme is analyzed by examining the induced controlled Markov process with respect to a long-run average cost criterion.

Ergodic control of multidimensional diffusions, II: Adaptive control

The self-tuning scheme for the adaptive control of a diffusion process is studied with long-run average cost criterion and maximum likelihood estimation of parameters. Asymptotic optimality under a suitable identifiability condition is established under two alternative sets of hypotheses—a Lyapunov-type stability criterion and a condition on cost which penalizes instability.

Shepp statistic for Markov chains application to a long-run average cost criterion

This paper is a generalization to Markov chains of the work of Shepp [6] in the i.i.d case. Shepp studies the limiting values of the averagesTn= (Sn+f(n)–Sn)/f(n) whereSn=X0+X1+ · ·· +Xn,X0= 0,n= 1, 2, ···, is a sum of mutually independent and identically distributed random variables. The functionftakes positive integer values and non-decreasingly tends to infinity. Here we take a class of functionsfin central positionf(n) = [clogn],c&gt; 0,n= 1, 2, ···. There are many refinements of the functionfin the i.i.d case [1], [2]. Here we consider the more general case whereX1,· ··,Xnis an irreducible and recurrent Markov chain. The state space of the chain is either compact or countable.

Shepp statistic for Markov chains application to a long-run average cost criterion

This paper is a generalization to Markov chains of the work of Shepp [6] in the i.i.d case. Shepp studies the limiting values of the averages Tn = (Sn + f(n) – Sn )/f(n) where Sn = X0 + X 1+ · ·· + Xn , X 0 = 0, n = 1, 2, ···, is a sum of mutually independent and identically distributed random variables. The function f takes positive integer values and non-decreasingly tends to infinity. Here we take a class of functions f in central position f(n) = [c log n], c &amp;gt; 0, n = 1, 2, ···. There are many refinements of the function f in the i.i.d case [1], [2]. Here we consider the more general case where X 1 , · ··, Xn is an irreducible and recurrent Markov chain. The state space of the chain is either compact or countable.