Average Cost Optimal Policy Research Articles

Average Cost Optimality in Partially Observed Lost-Sales Inventory Systems

We consider a partially observable inventory system in which the inventory level can only be observed when it reaches zero, the unmet demand is lost, and replenishment orders must be decided so as to minimize the long-run average cost. This problem has an infinite-dimensional state space and is therefore difficult to analyze. We prove the existence of an average-cost optimal policy, using the so-called vanishing discount factor approach. As a main methodological contribution, we provide a way to verify the key condition -- the uniform boundedness of the relative discounted value function -- for the partially observed system. To accomplish that we construct -- for the partially observed system -- a valid policy which in a certain sense copies the actions of another policy for the process with a different initial state. To our best knowledge, this paper is the first dealing with inventory models with partially observable inventory levels under the long-run average cost criterion.

Optimal Maintenance of a Production-inventory System With Continuous Repair Times and Idle Periods

In this paper two similar models for the maintenance of a production-inventory system are considered. In both models, an input generating installation supplies a buffer with a raw material and a production unit pulls the raw material from the buffer. The installation in the first model and the production unit in the second model deteriorate stochastically over time and the problem of their optimal preventive maintenance is considered. In the first model, it is assumed that the installation, after the completion of its maintenance, remains idle until the buffer is evacuated, while in the second model, it is assumed that the production unit, after the completion of its maintenance, remains idle until the buffer is filled up. The preventive and corrective repair times of the installation in the first model and the preventive and corrective repair times of the production unit in the second model are continuous random variables with known probability density functions. Under a suitable cost structure, semi-Markov decision processes are considered for both models in order to find a policy that minimizes the long-run expected average cost per unit time. A great number of numerical examples provide strong evidence that, for each fixed buffer content, the average-cost optimal policy is of control-limit type in both models, i.e. it prescribes a preventive maintenance of the installation in the first model and a preventive maintenance of the production unit in the second model if and only if their degree of deterioration is greater than or equal to a critical level. Using the usual regenerative argument, the average cost of the optimal control-limit policy is computed exactly in both models. Four numerical examples are also presented in which the preventive and corrective repair times follow the Exponential, the Weibull, the Gamma and the Log-Normal distribution, respectively.

Cross docking for libraries with a depot

Library organizations in the Netherlands show an increasing interest to employ depots for low-cost storage and demand fulfillment of item requests. Typically, all libraries in an organization have a shared catalog, and, on local unavailability, requests can be shipped from elsewhere in the organization. The depot can be used to consolidate shipment requests by making tours along all libraries, delivering requested items, but also picking up items that have to be stored at the depot, or that have to be shipped from one library to another. Cross docking and delayed shipments are two preferred methods for fulfilling requests that cannot be directly met using on-hand stock at the depot. In this paper, we compare these two methods from an inventory control perspective. We model the library system as a Markov Decision process. For one- and two-location systems, we derive analytical results for the average-cost optimal policy, showing that the decision to store items from the location at the depot satisfies a threshold structure depending on the number of rented items. For larger instances, an effective heuristic is proposed exploiting this threshold structure. In numerical experiments, important managerial insights are obtained by comparing cross docking and delayed shipments in different situations. Cross docking is shown to add most value in systems with low total stock, however, delayed shipments may achieve similar costs as cross docking when stock is high or when tours frequently visit all locations. Furthermore, effective decisions can be based on simple model formulations with memoryless rental time distributions.

Canonical Dynamic Programming Equations subject to Ambiguity

This paper studies the infinite horizon average cost Markov control model subject to ambiguity on the controlled process conditional distribution. The stochastic control problem is formulated as a minimax optimization in which, (i) the existence of optimal policies is established through a pair of canonical dynamic programming equations derived for Borel state and action spaces, and (ii) the controlled process maximizing conditional distribution is characterized through a water-filling solution derived for finite state and action spaces. To obtain average cost optimal policies numerically a policy iteration algorithm is also developed. Finally, as an application of the proposed canonical dynamic programming equations, an example is provided.

OPTIMAL SWITCHING ON AND OFF THE ENTIRE SERVICE CAPACITY OF A PARALLEL QUEUE

This paper studies optimal switching on and off of the entire service capacity of an M/M/∞ queue with holding, running and switching costs. The running costs depend only on whether the system is on or off, and the holding costs are linear. The goal is to minimize average costs per unit time. The main result is that an average-cost optimal policy either always runs the system or is an (M, N)-policy defined by two thresholds M and N, such that the system is switched on upon an arrival epoch when the system size accumulates to N and is switched off upon a departure epoch when the system size decreases to M. It is shown that this optimization problem can be reduced to a problem with a finite number of states and actions, and an average-cost optimal policy can be computed via linear programming. An example, in which the optimal (M, N)-policy outperforms the best (0, N)-policy, is provided. Thus, unlike the case of single-server queues studied in the literature, (0, N)-policies may not be average-cost optimal.

Finite and infinite-horizon single vehicle routing problems with a predefined customer sequence and pickup and delivery

We consider the problem of finding the optimal routing of a single vehicle that starts its route from a depot and picks up from and delivers K different products to N customers that are served according to a predefined customer sequence. The vehicle is allowed during its route to return to the depot to unload returned products and restock with new products. The items of all products are of the same size. For each customer the demands for the products that are delivered by the vehicle and the quantity of the products that is returned to the vehicle are discrete random variables with known joint distribution. Under a suitable cost structure, it is shown that the optimal policy that serves all customers has a specific threshold-type structure. We also study a corresponding infinite-time horizon problem in which the service of the customers is not completed when the last customer has been serviced but it continues indefinitely with the same customer order. For each customer, the joint distribution of the quantities that are delivered and the quantity that is picked up is the same at each cycle. The discounted-cost optimal policy and the average-cost optimal policy have the same structure as the optimal policy in the finite-horizon problem. Numerical results are given that illustrate the structural results.

SINGLE VEHICLE ROUTING PROBLEMS WITH A PREDEFINED CUSTOMER ORDER, UNIFIED LOAD AND STOCHASTIC DISCRETE DEMANDS

We consider the problem of finding the optimal routing of a single vehicle that delivers K different products to N customers that are served according to a particular order. It is assumed that the demands of the customers for each product are discrete random variables, and the total demand of each customer for all products cannot exceed the vehicle capacity. The joint probability mass function of the demands of each customer is known. It is assumed that all products are stored together in the vehicle's single compartment. The policy that serves all customers with the minimum total expected cost is found by implementing a suitable dynamic programming algorithm. We prove that this policy has a specific threshold-type structure. Furthermore, we study a corresponding infinite-time horizon problem in which the service of the customers is not completed when the last customer has been serviced but it continues periodically with the same customer order. The demands of each customer for the products have the same distributions at different periods. The discounted cost optimal policy and the average-cost optimal policy have the same structure as the optimal policy in the finite-horizon problem. Numerical results are given that illustrate the structural results.

Average Cost Markov Decision Processes with Weakly Continuous Transition Probabilities

This paper presents sufficient conditions for the existence of stationary optimal policies for average cost Markov decision processes with Borel state and action sets and weakly continuous transition probabilities. The one-step cost functions may be unbounded, and the action sets may be noncompact. The main contributions of this paper are: (i) general sufficient conditions for the existence of stationary discount optimal and average cost optimal policies and descriptions of properties of value functions and sets of optimal actions, (ii) a sufficient condition for the average cost optimality of a stationary policy in the form of optimality inequalities, and (iii) approximations of average cost optimal actions by discount optimal actions.

Average Cost Optimality in Inventory Models With Dynamic Information Delays

Information delays exist when the most recent inventory information available to the Inventory Manager (IM) is dated. In other words, the IM observes only the inventory level that belongs to an earlier period. Such situations are not uncommon, and they arise when it takes a while to process the demand data and pass the results to the IM. In this paper, we establish that the average cost optimal policy is of state-dependent basestock type with respect to the reference inventory position. We show that the optimal base stock depends on the age and the magnitude of the latest observed delay. We illustrate the results by solving an example with delays of 0 and 1, for which we are able to obtain formulas/bounds for the basestock levels.

Single vehicle routing problems with a predefined customer sequence, compartmentalized load and stochastic demands

We consider the problem of finding the optimal routing of a single vehicle that delivers K different products to N customers according to a particular customer order. The demands of the customers for each product are assumed to be random variables with known distributions. Each product type is stored in its dedicated compartment in the vehicle. Using a suitable dynamic programming algorithm we find the policy that satisfies the demands of the customers with the minimum total expected cost. We also prove that this policy has a specific threshold-type structure. Furthermore, we investigate a corresponding infinite-time horizon problem in which the service of the customers does not stop when the last customer has been serviced but it continues indefinitely with the same customer order. It is assumed that the demands of the customers at different tours have the same distributions. It is shown that the discounted-cost optimal policy and the average-cost optimal policy have the same threshold-type structure as the optimal policy in the original problem. The theoretical results are illustrated by numerical examples.

Optimal maintenance of two stochastically deteriorating machines with an intermediate buffer

We consider a manufacturing system in which an input generating installation transfers a raw material to a subsequent production unit. Both machines deteriorate stochastically with usage and may fail. For each machine the deteriorating process is described by some known transition probabilities between different degrees of deterioration. A buffer has been built between the two machines in order to cope with unexpected failures of the installation. A discrete-time Markov decision model is formulated for the optimal preventive maintenance of both machines. The maintenance times are geometrically distributed and the cost structure includes operating costs, storage costs, maintenance costs and costs due to the lost production. It is proved that for fixed buffer content and for fixed deterioration degree of one machine, the average-cost optimal policy initiates a preventive maintenance of the other machine if and only if its degree of deterioration exceeds some critical level. We study, by means of numerical results, the effect of the variation of some parameters on the optimal policy and on the minimum average cost. For the case in which the maintenance times follow continuous distributions, an approximate discrete-time Markov decision model is proposed.

Optimal maintenance of a production-inventory system with idle periods

In this paper we consider a production-inventory system in which an input generating installation supplies a buffer with a raw material and a production unit pulls the raw material from the buffer with constant rate. The installation deteriorates in time and the problem of its optimal preventive maintenance is considered. It is assumed that the installation after the completion of its maintenance remains idle until the buffer is evacuated. Under a suitable cost structure it is shown that the average-cost optimal policy for fixed buffer content is of control-limit type, i.e. it prescribes a preventive maintenance of the installation if and only if its degree of deterioration is greater than or equal to a critical level. Using the usual regenerative argument, the average cost of a control-limit policy is computed exactly and then, the optimal control-limit policy is determined. Furthermore, the stationary probabilities of the system under the optimal policy are computed.

Average Cost Optimal Policy for a Stochastic Two-Machine Flowshop with Limited Work-in-Process

We consider a production planning problem in a two-machine flowshop subject to breakdown and repair of machines and subject to nonnegativity and upper bound constraints on work-in-process. The objective is to choose machine production rates over time to minimize the long-run average inventory/backlog and production costs. For sufficiently large upper bound on the work-in-process, the problem is formulated as a stochastic dynamic program. We then establish a verification theorem and a partial characterization of the optimal control policy if it exists.

Optimality Inequalities for Average Cost Markov Decision Processes and the Stochastic Cash Balance Problem

For general state and action space Markov decision processes, we present sufficient conditions for the existence of solutions of the average cost optimality inequalities. These conditions also imply the convergence of both the optimal discounted cost value function and policies to the corresponding objects for the average costs per unit time case. Inventory models are natural applications of our results. We describe structural properties of average cost optimal policies for the cash balance problem; an inventory control problem where the demand may be negative and the decision-maker can produce or scrap inventory. We also show the convergence of optimal thresholds in the finite horizon case to those under the expected discounted cost criterion and those under the expected discounted costs to those under the average costs per unit time criterion.

An improved algorithm for the computation of the optimal repair/replacement policy under general repairs

We consider a system which deteriorates with age and may experience a failure at any time instant. On failure, the system may be replaced or repaired. The repair can partially reset the failure intensity of the unit. Under a suitable cost structure it has been proved in the literature that the average-cost optimal policy is of control-limit type, i.e. it conducts a replacement if and only if, on the nth failure, the real age of the system is greater than or equal to a critical value. We develop an efficient special-purpose policy iteration algorithm that generates a sequence of improving control-limit policies. The value determination step of the algorithm is based on the embedding technique. There is strong numerical evidence that the algorithm converges to the optimal policy.

Empirical estimation in average Markov control processes

This work concerns discrete-time Markov control processes with unbounded costs and unknown disturbance distribution θ . Assuming observability of the random disturbance, we estimate θ using its empirical estimator, which, combined with a variant of the vanishing discount factor approach, yields average cost optimal policies.

On the control of a compound immigration process through total catastrophes

We consider a compound immigration process that can be controlled by the introduction of total catastrophes. Under a natural cost structure the average cost optimal policy is found to be of threshold type. Moreover we study the stationary distribution of the population size when the system operates optimally.

Average cost optimal policy for a stochastic two-machine flowshop with limited work-in-process

We consider a production planning problem in a two-machine flowshop subject to breakdown and repair of machines and subject to nonnegativity and upper bound constraints on work-in-process. The objective is to choose machine production rates over time to minimize the long-run average inventory/backlog and production costs. For sufficiently large upper bound on the work-in-process, the problem is formulated as a stochastic dynamic program. We then establish a verification theorem and a partial characterization of the optimal control policy if it exists.

Sequencing and Routing in Multiclass Queueing Networks Part I: Feedback Regulation

This paper establishes new criteria for stability and for instability of multiclass network models under a given stationary policy. It also extends previous results on the approximation of the solution to the average cost optimality equations through an associated fluid model: It is shown that an optimized network possesses a fluid limit model which is itself optimal with respect to a total cost criterion. A general framework for constructing control algorithms for multiclass queueing networks is proposed based on these general results. Network sequencing and routing problems are considered as special cases. The following aspects of the resulting feedback regulation policies are developed in the paper: The policies are stabilizing and are, in fact, geometrically ergodic for a Markovian model. Numerical examples are given. In each case it is shown that the feedback regulation policy closely resembles the average-cost optimal policy. A method is proposed for reducing variance in simulation for a network controlled using a feedback regulation policy.

Stochastic Dynamic Programming and the Control of Queueing Systems

Optimization Criteria. Finite Horizon Optimization. Infinite Horizon Discounted Cost Optimization. An Inventory Model. Average Cost Optimization for Finite State Spaces. Average Cost Optimization Theory for Countable State Spaces. Computation of Average Cost Optimal Policies for Infinite State Spaces. Optimization Under Actions at Selected Epochs. Average Cost Optimization of Continuous Time Processes. Appendices. Bibliography. Index.