Average Cost Criterion Research Articles

Rate of Convergence of Empirical Measures and Costs in Controlled Markov Chains and Transient Optimality

The purpose of this paper is two fold. First, bounds on the rate of convergence of empirical measures in controlled Markov chains are obtained under some recurrence conditions. These include bounds obtained through large deviations and central limit theorem arguments. These results are then applied to optimal control problems. Bounds on the rate of convergence of the empirical measures that are uniform over different sets of policies are derived, resulting in bounds on the rate of convergence of the costs. Finally, new optimal control problems that involve not only average cost criteria but also measures on the transient behavior of the cost, namely the rate of convergence, are introduced and applied to a problem in telecommunications. The solution to these problems rely on the bounds introduced in previous sections.

A note on the Ross\\2-Taylor theorem

In this note the condition used in proving a result due to S.M. Ross and H.M. Taylor are examined. These results pertain to the existence of bounded solutions to the average cost optimality equation for controlled Markov processes with an average cost criterion. In particular, we show how the use of commonly found convexity (concavity) properties of value functions can be employed to verify a seemingly rather restrictive equicontinuity condition. In addition, we remark that several results in the literature can be viewed as special cases of the result by Ross and Taylor, contrary to claims otherwise.

Almost-periodic solutions for Riccati equations of stochastic control

We consider an average quadratic cost criteria for affine stochastic differential equations with almost-periodic coefficients. Under stabilizability and detectability conditions we show that the Riccati equation associated with the quadratic control problem has a unique almost-periodic solution. In the periodic case the corresponding result is proved in [4].

Control policies for two-stage, pull-type production/inventory systems with constrained average cost criterion

Abstract We consider a two-stage, pull-type production/inventory system with a known service mechanism at the first stage. Set-ups and start-ups are involved in the operation of the second stage. We develop a production control policy for the second stage, within the class of (R, r) continuous-review policies, that minimizes the long run average number of set-ups and start-ups subject to a constraint on the number of customers that are lost. We use a semi- Markov decision model to obtain an optimal policy for the operation of the second stage. The optimal policy is in the form of a randomized stationary policy. The structure of the optimal policy suggests the use of a suboptimal look-back policy that delays the set-up at the second stage if the buffer lacks sufficient raw material.

Controlled Markov processes on the infinite planning horizon: Weighted and overtaking cost criteria

Stochastic control problems for controlled Markov processes models with an infinite planning horizon are considered, under some non-standard cost criteria. The classical discounted and average cost criteria can be viewed as complementary, in the sense that the former captures the short-time and the latter the long-time performance of the system. Thus, the authors study a cost criterion obtained as weighted combinations of these criteria, extending to a general state and control space framework several results by Feinberg and Shwartz (1992), and by Krass et al. (1992). In addition, a functional characterization is given for overtaking optimal policies, for problems with countable state spaces and compact control spaces; the authors' approach is based on qualitative properties of the optimality equation for problems with an average cost criterion. >

Perturbation methods in optimal stopping with average cost criterion

By a perturbation method, we prove an existence result on the equation related to the Markovian stopping time problem with average cost, and an optimal stopping time is given

Understanding the lead-time effects in stochastic inventory systems with discounted costs

This paper explores qualitative effects of lead-time uncertainty in a basic continuous-time single-item inventory model with Poisson demand and stochastic lead times. The objective is to minimize the infinite-horizon expected total discounted cost. Order costs are linear, so a base-stock policy is optimal. Traditionally and intuitively, people believe that a (stochastically) longer lead time results in a larger lead-time demand and therefore that the system should have a higher optimal base-stock level. In fact, Song (1994) shows that this is true for models with the average cost criterion. Here, we show that this intuition is not always validated for models with the discounted cost criterion. We identify certain systems in which the lead times do have a monotonic impact on the optimal base-stock levels. Examples include systems with constant, geometric, gamma, or uniform lead-time distributions. In addition, we find that a shorter lead time does not necessarily result in a smaller optimal system cost.

Linear Programming and Average Optimality of Markov Control Processes on Borel Spaces—Unbounded Costs

This paper is concerned with the linear programming formulation of Markov control processes with Borel state and action spaces, and the average cost (AC) criterion. The one-stage cost function may be unbounded. A linear program $EP$ and its dual, $EP^*$, are introduced. Their values, $\inf EP$ and $\sup EP^*$, bound the value (say, $\inf AC$) of the $AC$ problem, i.e., $\sup EP^{*} \leq \inf AC \leq \inf EP$. Conditions are provided for the existence of no duality gap, viz., $\sup EP^{*}=\inf EP$, and also for strong duality, so that both $EP$ and $EP^*$ are solvable and their optimal values satisfy $\max EP^{*} = \min EP$. The latter implies (i) the existence of an $AC$-optimal control policy and that (ii) the $AC$ optimality equation holds almost everywhere. These results are applied to a general vector-valued, additive-noise system with quadratic costs.

Optimal repair and replacement in markovian systems

A deteriorating system is inspected at equally spaced points in time. After each inspection a repair to a better state or a full replacement are possible. We introduce a generalized control limit policy and show that, under reasonable conditions on the system's law of evolution and on the operating, repair and replacement costs, this policy is optimal for the expected total discounted cost, as well as for the (long-run) average cost criteria. Furthermore, we allow for uncertain repair actions (that may even end up in a worse state), and show that the generalized control limit rule is still optimal. The work extends, generalizes and unifies many models in the area of optimal control of repairable systems

Stability and singular perturbations in constrained Markov decision problems

Constrained Markov decision problems (CMDPs) with the average cost criterion and a single ergodic chain, or the discounted cost with a general multichain structure, are considered. Conditions for stability of the optimal value and control to changes of the parameters of the problem, such as immediate costs, transition probabilities, and the discount factor, are established. Singular constrained problems, for which the optimal value and controls exhibit discontinuities, are studied. >

Discrete-Time Controlled Markov Processes with Average Cost Criterion: A Survey

This work is a survey of the average cost control problem for discrete-time Markov processes. The authors have attempted to put together a comprehensive account of the considerable research on this problem over the past three decades. The exposition ranges from finite to Borel state and action spaces and includes a variety of methodologies to find and characterize optimal policies. The authors have included a brief historical perspective of the research efforts in this area and have compiled a substantial yet not exhaustive bibliography. The authors have also identified several important questions that are still open to investigation.

Nearly Optimal Controls for Stochastic Ergodic Problems with Partial Observation

This paper considers the problem of constructing nearly optimal controls for discrete-time, infinite horizon, stochastic control problems under partial observations, with the average cost criterion.

Optimal control of the M/G/1 queue with repeated vacations of the server

An M/G/1 queue where the server may take repeated vacations is considered. Whenever a busy period terminates, the server takes a vacation of random duration. At the end of each vacation, the server may either take a new vacation or resume service; if the queue is found empty, the server always takes a new vacation. The cost structure includes a holding cost per unit of time and per customer in the system and a cost each time the server is turned on. One discounted cost criterion and two average cost criteria are investigated. It is shown that the vacation policy that minimizes the discounted cost criterion over all policies (randomized, history dependent, etc.) converges to a threshold policy as the discount factor goes to zero. This result relies on a nonstandard use of the value iteration algorithm of dynamic programming and is used to prove that both average cost problems are minimized by a threshold policy.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Computing the optimal policy for capacitated inventory models

We consider an infinite-horizon inventory model with finite production capacity and stochastic demands. Federgruen and Zipkin [2][3] showed that a base-stock policy is optimal; however, they did not provide an algorithm for the computation of the optimal policy or the cost. We provide, for the average cost criteria, an algorithm to compute the critical number and the associated cost. Our approach uses results from the theory of storage processes and multi-stage uncapacitated systems

Policy iteration and Newton-Raphson methods for Markov decision processes under average cost criterion

In this paper, we are concerned with Markov decision processes under the average cost criterion. By taking a fractional programming appraoch, we first show that the problem can be reduced to a nonlinear equation whose unknown variable is the optimal average cost to be determined. Next, for this nonlinear equation, we propose a family of solution algoriths parametrized by a natural number (including +∞), whose extreme cases, that is, the algorithms with parameters1 and +∞ correspond to the policy iteration and Newton-Raphson methods, respectively. This result show a relationship between thhe above two methods, and suggests that a suitable choice of the parameter enable us to solve efficiently a given application problem with a specific structure.

Optimal minimal-repair and replacement problem with age dependent cost structure

The optimal minimal-repair and replacement problem of a reliability system under the average cost criterion is formulated as a semi-Markov decision process. It is assumed that the cost structure of the system depends on its age. Under some weak assumptions, it is shown that, among all allowable policies, an optimal policy is a t-policy. That is, failures before age t are minimally repaired, but the system is replaced when a failure after age t occurs.

On distributed dynamic programming

An asynchronous implementation of the value-iteration algorithm of dynamic programming for the expected average cost criterion is presented. The situation where the task of performing each iteration is divided among a number of processors is considered. Each processor iterates on a different set of components of the value function vector using the information it obtains from other processors concerning the remaining components of the vector. The proposed asynchronous algorithm is shown to converge when the processors iterate at different speeds, some performing more iterations than others, when the information received by a processor regarding the other components of the vector may be outdated or may have been received out of order, or when there may be an unpredictable delay in obtaining information from other processors.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

The Hawk–Dove game as an average-cost problem

This paper considers a version of the Hawk–Dove game of Maynard Smith and Price (1973) in which animals compete for a sequence of food items. Actions may depend on an animal's energy reserves. Costs and transition probabilities under a given policy depend on the mean level of aggressiveness, p, of the rest of the population. We find the optimal policy for a single animal under an average cost criterion when ρ is constant over time. We then consider the whole interacting population when individual members follow the same stationary policy. It is shown that the mean aggressiveness, p, asymptotically approaches a limiting value in this population. We then consider the existence of evolutionarily stable strategies for the population. It is shown that such strategies always exist but may not be unique.

The Hawk–Dove game as an average-cost problem

This paper considers a version of the Hawk–Dove game of Maynard Smith and Price (1973) in which animals compete for a sequence of food items. Actions may depend on an animal's energy reserves. Costs and transition probabilities under a given policy depend on the mean level of aggressiveness, p, of the rest of the population. We find the optimal policy for a single animal under an average cost criterion when ρ is constant over time. We then consider the whole interacting population when individual members follow the same stationary policy. It is shown that the mean aggressiveness, p, asymptotically approaches a limiting value in this population. We then consider the existence of evolutionarily stable strategies for the population. It is shown that such strategies always exist but may not be unique.

Singularly perturbed Markov control problem: Limiting average cost

In this paper we consider a singularly perturbed Markov decision process with the limiting average cost criterion. We assume that the underlying process is composed ofn separate irreducible processes, and that the small perturbation is such that it “unites” these processes into a single irreducible process. We formulate the underlying control problem for the singularly perturbed MDP, and call it the “limit Markov control problem” (limit MCP). We prove the validity of the “the limit control principle” which states that an optimal solution to the perturbed MDP can be approximated by an optimal solution of the limit MCP for any sufficiently small perturbation. We also demonstrate that the limit Markov control problem is equivalent to a suitably constructed nonlinear program in the space of long-run state-action frequencies. This approach combines the solutions of the original separated irreducible MDPs with the stationary distribution of a certain “aggregated MDP” and creates a framework for future algorithmic approaches.