Average Cost Case Research Articles

Models and algorithms for skip-free Markov decision processes on trees

We introduce a class of models for multidimensional control problems that we call skip-free Markov decision processes on trees. We describe and analyse an algorithm applicable to Markov decision processes of this type that are skip-free in the negative direction. Starting with the finite average cost case, we show that the algorithm combines the advantages of both value iteration and policy iteration—it is guaranteed to converge to an optimal policy and optimal value function after a finite number of iterations but the computational effort required for each iteration step is comparable with that for value iteration. We show that the algorithm can also be used to solve discounted cost models and continuous-time models, and that a suitably modified algorithm can be used to solve communicating models.

Randomized and Relaxed Strategies in Continuous-Time Markov Decision Processes

One of the goals of this article is to describe a wide class of control strategies, which includes the traditional relaxed strategies, as well as the so called randomized strategies which appeared earlier only in the framework of semi-Markov decision processes. If the objective is the total expected cost up to the accumulation of jumps, then without loss of generality one can consider only Markov relaxed strategies. Under a simple condition, the Markov randomized strategies are also sufficient. An example shows that the mentioned condition is important. Finally, without any conditions, the class of so called Poisson-related strategies is also sufficient in the optimization problems. All the results are applicable to the discounted model, they may be useful also for the case of long-run average cost.

Risk-sensitive control of continuous time Markov chains

We study risk-sensitive control of continuous time Markov chains taking values in discrete state space. We study both finite and infinite horizon problems. In the finite horizon problem we characterize the value function via Hamilton Jacobi Bellman equation and obtain an optimal Markov control. We do the same for infinite horizon discounted cost case. In the infinite horizon average cost case we establish the existence of an optimal stationary control under certain Lyapunov condition. We also develop a policy iteration algorithm for finding an optimal control.

The discounted (R,Q) inventory model—The Shrewd Accountant's Heuristic

The discounted continuous-review (R,Q) inventory model with continuous and stochastic demand is investigated. New optimality conditions are derived, clarifying the difference to the average-cost case, also graphically. Supported by depreciation theory, applied to the value of a setup, the results suggest an insightful and very precise approximation – The Shrewd Accountant's Heuristic – based on a new average-cost model. It deepens and extends the work of Hadley (1964). Three examples are worked out in detail and the model is generalized to Poisson demand and to stochastic lead-times.

Optimal Control of Markov Processes with Age-Dependent Transition Rates

We study optimal control of Markov processes with age-dependent transition rates. The control policy is chosen continuously over time based on the state of the process and its age. We study infinite horizon discounted cost and infinite horizon average cost problems. Our approach is via the construction of an equivalent semi-Markov decision process. We characterise the value function and optimal controls for both discounted and average cost cases.

Technical note: A computationally efficient algorithm for undiscounted Markov decision processes with restricted observations

AbstractWe present a computationally efficient procedure to determine control policies for an infinite horizon Markov Decision process with restricted observations. The optimal policy for the system with restricted observations is a function of the observation process and not the unobservable states of the system. Thus, the policy is stationary with respect to the partitioned state space. The algorithm we propose addresses the undiscounted average cost case. The algorithm combines a local search with a modified version of Howard's (Dynamic programming and Markov processes, MIT Press, Cambridge, MA, 1960) policy iteration method. We demonstrate empirically that the algorithm finds the optimal deterministic policy for over 96% of the problem instances generated. For large scale problem instances, we demonstrate that the average cost associated with the local optimal policy is lower than the average cost associated with an integer rounded policy produced by the algorithm of Serin and Kulkarni Math Methods Oper Res 61 (2005) 311–328. © 2008 Wiley Periodicals, Inc. Naval Research Logistics 2009

Optimization of Joint Replacement Policies for Multipart Systems by a Rollout Framework

Maintaining an asset with life-limited parts, e.g., a jet engine or an electric generator, may be costly. Certain costs, e.g., setup cost, can be shared if some parts of the asset are replaced jointly. Reducing the maintenance cost by good joint replacement policies is difficult in view of complicate asset dynamics, large problem sizes and the irregular optimal policy structures. This paper addresses these difficulties by using a rollout optimization framework. Based on a novel application of time-aggregated Markov decision processes, the ldquoOne-Stage Analysisrdquo method is first developed. The policies obtained from the method are investigated and their effectiveness is demonstrated by examples. This method and the existing threshold method are then improved by the ldquorollout algorithmrdquo for the total cost case and the average cost case. Based on ordinal optimization, it is shown that excessive simulations are not necessary for the rollout algorithm. Numerical testing demonstrates that the policies obtained by the rollout algorithms with either the ldquoOne-Stage Analysisrdquo or the threshold method significantly outperform traditional threshold policies.

Managing the Inventory of an Item with a Replacement Warranty

In this paper, we study a firm that faces demand from two sources: demand for new items and demand to replace failed items under warranty. We model this setting as a multiperiod single-product inventory problem where the demands for new items in different periods are independent and the demands for replacing failed items depend on the number and ages of the items under warranty. We consider backlogging and emergency supply cases, and study both discounted cost and average cost criteria. We prove the optimality of the w-dependent base-stock ordering policy where the base-stock level is a function of w, the vector representing the number of items at different ages currently under warranty. For the special case where the demand for new products is identically distributed, we prove the optimality of a stationary w-dependent base stock policy for the finite-horizon discounted and the infinite-horizon discounted and average cost cases. In our computational study, we find that an optimal w-dependent policy can lead to 69% average improvement in expected costs when compared to a policy that neglects demands from items under warranty.

Structural Properties of Optimal Transmission Policies Over a Randomly Varying Channel

We consider the problem of transmitting packets over a randomly varying point to point channel with the objective of minimizing the expected power consumption subject to a constraint on the average packet delay. By casting it as a constrained Markov decision process in discrete time with time-averaged costs, we prove structural results about the dependence of the optimal policy on buffer occupancy, number of packet arrivals in the previous slot and the channel fading state for both i.i.d. and Markov arrivals and channel fading. The techniques we use to establish such results: convexity, stochastic dominance, decreasing-differences, are among the standard ones for the purpose. Our main contribution, however, is the passage to the average cost case, a notoriously difficult problem for which rather limited results are available. The novel proof techniques used here are likely to have utility in other stochastic control problems well beyond their immediate application considered here.

Impulse Control of Brownian Motion: The Constrained Average Cost Case

When a manufacturer places repeated orders with a supplier to meet changing production requirements, he faces the challenge of finding the right balance between holding costs and the operational costs involved in adjusting the shipment sizes. We consider an inventory whose content fluctuates as a Brownian motion in the absence of control. At any moment, a controller can adjust the inventory level by any positive or negative quantity, but incurs both a fixed cost and a cost proportional to the magnitude of the adjustment. The inventory level must be nonnegative at all times and continuously incurs a linear holding cost. The objective is to minimize long-run average cost. We show that control band policies are optimal for the average cost Brownian control problem and explicitly calculate the parameters of the optimal control band policy. This form of policy is described by three parameters {q,Q,S}, 0 < q ≤ Q < S. When the inventory falls to zero (rises to S), the controller expedites (curtails) shipments to return it to q (Q). Employing apparently new techniques based on methods of Lagrangian relaxation, we show that this type of policy is optimal even with constraints on the size of adjustments and on the maximum inventory level. We also extend these results to the discounted cost problem.

COORDINATED INVENTORY PLANNING FOR NEW AND OLD PRODUCTS UNDER WARRANTY

In this article we study a firm that is facing demand from two sources: demand for new items and demand to replace failed items under warranty. We model this setting as a multiperiod single-product inventory problem where the demands for new items in different periods are independent and the demands for replacing failed items depend on the number of the items under warranty. We consider backlogging and emergency supply cases and study both discounted-cost and average-cost criteria. We prove the optimality of the w-dependent base stock ordering policy, where the base stock level is a function of w, the number of items currently under warranty. For the special case where the demand for new products is stationary, we prove the optimality of a stationary w-dependent base stock policy for the finite-horizon discounted-cost and the infinite-horizon discounted- and average-cost cases. We compare the integrated inventory policy with the one that neglects demands from items under warranty.

Optimal production and backordering policy in failure-prone manufacturing systems

This note considers a failure-prone manufacturing system producing one part-type in discrete mode. The problem is to find the optimal integrated production and backordering policy, in which control decisions include how much the production rate should take when the machine is up and whether an unmet demand should be backorder or rejected. The optimal policy is proved to be a threshold control and can be characterized by three threshold parameters for both discounted cost and long-run average cost cases. The structural relationships between three threshold parameters are established. It is found that the completely backordering policy may be optimal for the discounted cost but will never be optimal for the average cost. In addition, the explicit form of the stationary distribution under such threshold policy is derived, which is then be used to calculate interesting steady-state performance measures and seek the optimal threshold values. A numerical example illustrates the results.

Optimal Predictive Maintenance Policies for a Deteriorating System: The Total Discounted Cost and the Long-Run Average Cost Cases

A predictive maintenance policy for a system subject to degradation is considered. Over time, the system is subject to degradation. The degradation level is observed only at successive inspection times. The system has a nominal life Y, and it fails once the degradation exceeds the nominal life. Failure is detected only by inspection, and is then maintained (corrective maintenance). The system is also maintained when the degradation level exceeds a predetermine level M(preventive maintenance). Each preventive maintenance costs c 1, while the cost of a corrective maintenance is equal to c 1 + d, d ≥ 0. At any inspection, before failure, the cost of inspection is state dependent: if the degradation level at inspection is x, then an inspection cost of c(x) occurs. We determine the optimal maintenance level M using the total discounted cost as well as the long-run average cost criteria. Our results extend those of Grall et al. [Grall, A., Dieulle, L., Berenguer, C., Roussignol, M. (2002). Continuous-time predictive-maintenance scheduling for a deteriorating system. IEEE Trans. Reliability 51:141–149].

A Fast Computational Procedure to Solve the Multi-Item Single Machine Lot Scheduling Optimization Problem: The Average Cost Case

In this paper we present some special procedures for the numerical solution of the optimal scheduling problem of a multi-item single machine. We study the in6nite horizon case when the optimization criterion is the average cost. We establish the solution of the problem in terms of viscosity solutions of the Quasi-Variational Inequality (QVI) system associated to the problem. The existence of solution of the QVI and the uniqueness of the optimal average cost are proved. A method of discretization and a computational procedure are described. They allow us to compute the solution in a short time and with precision of order k. We obtain an estimate for the discretization error and develop an algorithm that converges in a finite number of steps. In our method the nodes of the triangulation mesh are joined by segments of trajectories of the original system. This feature allows us to obtain the k-order precision which, in general, is impossible to obtain by usual methods.

Functional Estimation for a Multicomponent Age Replacement Model

SYNOPTIC ABSTRACTWe consider a multicomponent system with identical components subject to failures, controlled by a replacement rule based on age thresholds. We show how to estimate the expected cost of operating the system as a function of one of the thresholds, in functional form, by simulation. The efficiency of the approach is analyzed for two types of performance measures of the system: (i) The total expected cost over a finite time horizon and (ii) the average cost over an infinite time horizon. For the average cost case, we show under mild conditions that the system is geometrically ergodic, uniformly over the parameter set of interest. Our numerical illustrations suggest that the proposed approach is viable from the practical viewpoint.

Value of Information in Capacitated Supply Chains

We incorporate information flow between a supplier and a retailer in a two-echelon model that captures the capacitated setting of a typical supply chain. We consider three situations: (1) a traditional model where there is no information to the supplier prior to a demand to him except for past data; (2) the supplier knows the (s, S) policy used by the retailer as well as the end-item demand distribution; and (3) the supplier has full information about the state of the retailer. Order up-to policies continue to be optimal for models with information flow for the finite horizon, the infinite horizon discounted and the infinite horizon average cost cases. Study of these three models enables us to understand the relationships between capacity, inventory, and information at the supplier level, as well as how they are affected by the retailer’s (S − s) values and end-item demand distribution. We estimate the savings at the supplier due to information flow and study when information is most beneficial.

Zero-sum stochastic games with unbounded costs: Discounted and average cost cases

Zero-sum stochastic games with countable state space and with finitely many moves available to each player in a given state are treated. As a function of the current state and the moves chosen, player I incurs a nonnegative cost and player II receives this as a reward. For both the discounted and average cost cases, assumptions are given for the game to have a finite value and for the existence of an optimal randomized stationary strategy pair. In the average cost case, the assumptions generalize those given in Sennott (1993) for the case of a Markov decision chain. Theorems of Hoffman and Karp (1966) and Nowak (1992) are obtained as corollaries. Sufficient conditions are given for the assumptions to hold. A flow control example illustrates the results.

Nonzero-sum stochastic games with unbounded costs: Discounted and average cost cases

We treat non-cooperative stochastic games with countable state space and with finitely many players each having finitely many moves available in a given state. As a function of the current state and move vector, each player incurs a nonnegative cost. Assumptions are given for the expected discounted cost game to have a Nash equilibrium randomized stationary strategy. These conditions hold for bounded costs, thereby generalizing Parthasarathy (1973) and Federgruen (1978). Assumptions are given for the long-run average expected cost game to have a Nash equilibrium randomized stationary strategy, under which each player has constant average cost. A flow control example illustrates the results. This paper complements the treatment of the zero-sum case in Sennott (1993a).

Rolling Horizon Procedures in Nonhomogeneous Markov Decision Processes

By far the most common planning procedure found in practice is to approximate the solution to an infinite horizon problem by a series of rolling finite horizon solutions. Although many empirical studies have been done, this so-called rolling horizon procedure has been the subject of few analytic studies. We provide a cost error bound for a general rolling horizon algorithm when applied to infinite horizon nonhomogeneous Markov decision processes, both in the discounted and average cost cases. We show that a Doeblin coefficient of ergodicity acts much like a discount factor to reduce this error. In particular, we show that the error goes to zero for any fixed rolling horizon as this Doeblin measure of control over the future decreases. The theory is illustrated through an application to vehicle deployment.

Critical Number Policies for Inventory Models with Periodic Data

We consider an infinite-horizon in problem with stochastic demands where the data vary periodically. Karlin (Karlin, S. 1960a. Dynamic inventory policy with varying stochastic demands. Management Sci. 6 231–258; Karlin, S. 1960b. Optimal policy for dynamic inventory process with stochastic demands subject to seasonal variations. J. SIAM 8 611–629.) shows that a periodic critical-number policy is optimal and presents an algorithm for computing the critical numbers, assuming discounted costs. Here we develop an alternative, conceptually simpler approach to these problems. The results include a proof of the optimality of such policies for the average-cost case, and a qualitative description of the behavior of the optimal policy as “smoothing” fluctuations in the data.