Average Cost Criterion Research Articles

Continuous-time Markov decision processes under the risk-sensitive average cost criterion

This paper studies continuous-time Markov decision processes under the risk-sensitive average cost criterion. The state space is a finite set, the action space is a Borel space, the cost and transition rates are bounded, and the risk-sensitivity coefficient can take any positive real number. Under the mild conditions, we develop a new approach to establish the existence of a solution to the risk-sensitive average cost optimality equation and obtain the existence of an optimal deterministic stationary policy.

Experimental Results Indicating Lattice‐Dependent Policies May Be Optimal for General Assemble‐To‐Order Systems

We consider an assemble‐to‐order (ATO) system with multiple products, multiple components which may be demanded in different quantities by different products, possible batch ordering of components, random lead times, and lost sales. We model the system as an infinite‐horizon Markov decision process under the average cost criterion. A control policy specifies when a batch of components should be produced, and whether an arriving demand for each product should be satisfied. Previous work has shown that a lattice‐dependent base‐stock and lattice‐dependent rationing (LBLR) policy is an optimal stationary policy for a special case of the ATO model presented here (the generalized M‐system). In this study, we conduct numerical experiments to evaluate the use of an LBLR policy for our general ATO model as a heuristic, comparing it to two other heuristics from the literature: a state‐dependent base‐stock and state‐dependent rationing (SBSR) policy, and a fixed base‐stock and fixed rationing (FBFR) policy. Remarkably, LBLR yields the globally optimal cost in each of more than 22,500 instances of the general problem, outperforming SBSR and FBFR with respect to both objective value (by up to 2.6% and 4.8%, respectively) and computation time (by up to three orders and one order of magnitude, respectively) in 350 of these instances (those on which we compare the heuristics). LBLR and SBSR perform significantly better than FBFR when replenishment batch sizes imperfectly match the component requirements of the most valuable or most highly demanded product. In addition, LBLR substantially outperforms SBSR if it is crucial to hold a significant amount of inventory that must be rationed.

Controlled Semi-Markov Chains with Risk-Sensitive Average Cost Criterion

This work concerns with semi-Markov decision chains on a finite state space. Assuming that the controller has a constant and positive risk-sensitive coefficient, an optimality equation for the corresponding (long-run) risk-sensitive average cost index is formulated and, under suitable continuity-compactness conditions, it is shown that a solution of such an equation determines the optimal average cost, as well as an optimal stationary policy. Additionally, if the underlying Markov chain is communicating, then it is proved that the optimality equation has a solution.

A Multiobjective Optimization Framework for Online Stochastic Optimal Control in Hybrid Electric Vehicles

The increasing urgency to extract additional efficiency from hybrid propulsion systems has led to the development of advanced power management control algorithms. In this paper, we address the problem of online optimization of the supervisory power management control in parallel hybrid electric vehicles (HEVs). We model HEV operation as a controlled Markov chain and show that the control policy yielding the Pareto optimal solution minimizes online the long-run expected average cost per unit time criterion. The effectiveness of the proposed solution is validated through simulation and compared with the solution derived with dynamic programming using the average cost criterion. Both solutions achieved the same cumulative fuel consumption demonstrating that the online Pareto control policy is an optimal control policy.

Interactive effects of seasonal-demand characteristics on manufacturing systems

In this paper, we identify different characteristics of seasonal demands and analyse their impact on manufacturing systems. In particular, we consider a single-stage production facility that has flexibility to adjust its production rates or inventory target levels. Using Markov decision process formulation, we analyse a model that minimises the total cost under infinite horizon discounted and average cost criteria. Through extensive numerical studies, we reveal several insights related to the individual and joint impact of seasonal demand characteristics that are of practical interests to manufacturers.

Actor-Critic Algorithms with Online Feature Adaptation

We develop two new online actor-critic control algorithms with adaptive feature tuning for Markov Decision Processes (MDPs). One of our algorithms is proposed for the long-run average cost objective, while the other works for discounted cost MDPs. Our actor-critic architecture incorporates parameterization both in the policy and the value function. A gradient search in the policy parameters is performed to improve the performance of the actor. The computation of the aforementioned gradient, however, requires an estimate of the value function of the policy corresponding to the current actor parameter. The value function, on the other hand, is approximated using linear function approximation and obtained from the critic. The error in approximation of the value function, however, results in suboptimal policies. In our article, we also update the features by performing a gradient descent on the Grassmannian of features to minimize a mean square Bellman error objective in order to find the best features. The aim is to obtain a good approximation of the value function and thereby ensure convergence of the actor to locally optimal policies. In order to estimate the gradient of the objective in the case of the average cost criterion, we utilize the policy gradient theorem, while in the case of the discounted cost objective, we utilize the simultaneous perturbation stochastic approximation (SPSA) scheme. We prove that our actor-critic algorithms converge to locally optimal policies. Experiments on two different settings show performance improvements resulting from our feature adaptation scheme.

A unified approach to time-aggregated Markov decision processes

This paper presents a unified approach to time-aggregated Markov decision processes (MDPs) with an average cost criterion. The approach is based on a framework in which a time-aggregated MDP constitutes a semi-Markov decision process (SMDP). By analyzing the performance sensitivity formulas of this SMDP, a number of optimization algorithms for time aggregated MDPs, including those previously reported in the literature, can be developed in a simple and intuitive way.

A Characterization of the Optimal Certainty Equivalent of the Average Cost via the Arrow-Pratt Sensitivity Function

This work is concerned with finite-state irreducible Markov decision chains satisfying continuity-compactness requirements. It is supposed that the system is driven by a decision maker with utility function U, which, aside mild conditions, is arbitrary, and the performance of a control policy is measured by the long-run average cost criterion induced by U. The main conclusions about this performance index are as follows: (i) the optimal U-average value function coincides with the optimal V-average index for a certain exponential utility V, and (ii) the average criteria associated with U and V have the same class of optimal stationary policies.

A poisson equation for the risk-sensitive average cost in semi-markov chains

This work concerns semi-Markov chains evolving on a finite state space. The system development generates a cost when a transition is announced, as well as a holding cost which is incurred continuously during each sojourn time. It is assumed that these costs are paid by an observer with positive and constant risk-sensitivity, and the overall performance of the system is measured by the corresponding (long-run) risk-sensitive average cost criterion. In this framework, conditions are provided under which the average index does not depend on the initial state and is characterized in terms of a single Poisson equation.

Potential-Based Least-Squares Policy Iteration for a Parameterized Feedback Control System

In the paper, a potential-based policy iteration method is proposed for optimal control of a stochastic dynamic system with an average cost criterion and a parameterized control law. In this method, the potential function and the optimal control parameters are obtained via a least-squares-based approach. The potential estimation algorithm is derived from a temporal difference learning method, which can be viewed as a continuous version of the least-squares policy evaluation algorithm. The policy iteration algorithm is validated by solving a linear quadratic gaussian problem in the simulation.

Special Issue: ACM SIGMETRICS 2014

The five papers in this special issue are based on a selection of top theoretical papers from the ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems, which took place at Austin, Texas, in June 2014. As the premier conference on the measurement and modeling of computer systems, the technical program featured papers that covered both theory and applications from a wide variety of areas, including algorithms, communication networks, dynamics and control, optimization, performance analysis, resource allocation and scheduling, and stochastic modeling, among others. The program consisted of 40 papers selected from more than 237 international submissions by a technical program committee of well-established researchers from all over the world. We seek with this special issue to highlight some of the recent theoretical work on the mathematical analysis and modeling of computer systems. The paper of Larranaga, Ayesta, and Verloop deals with a resource allocation problem in a multi-class server with convex holding costs and user impatience under the average cost criterion. Although the optimal policy has a complex dependency on all the input parameters and state information, the authors derive simple index policies obtained by solving a relaxed version of the optimal stochastic control problem and combining results from restless multi-armed bandits and queueing theory.

Effective inventory control policies with a minimum order quantity and batch ordering

In this paper, we consider a single-item periodic-review stochastic inventory system with both minimum order quantity (MOQ) and batch ordering requirements. In each time period, the firm can order either none or at least as much as the MOQ. At the same time, if an order is placed, the order quantity is required to be an integral multiple of a given specific batch size. We first adopt a heuristic policy which is specified by two parameters (s,t). Applying a discrete time Markov chain approach, we compute the system cost and optimize this (s,t) policy under the long-run average cost criterion. We also consider a simpler one-parameter policy, the so-called S policy, which is a special case of the (s,t) policy. In an intensive numerical study, we find that (1) both policies perform well in comparison with other policies and (2) the S policy also performs well and is compatible with the (s,t) policy; only in a few cases where demand variation is small, the latter outperforms the former significantly. We also evaluate the effects of some important parameters on system performance.

Optimal pricing barriers in a regulated market using reflected diffusion processes

We consider a class of one-dimensional (1D) reflected stochastic differential equations (SDEs). Such reflected SDE models arise as the key approximating processes in a regulated financial market system, and our main goal is to determine the set of optimal pricing barriers. We consider the running cost associated with the deviation of the process from the desired target level, and also the control cost from the interventions in an effort to keep the process inside the boundaries. Both a long-time average (ergodic) cost criterion and an infinite horizon discount cost criterion, where the discount factor is allowed to vary from one period to another, are studied, with numerical examples illustrating our main results.

Asymptotically optimal index policies for an abandonment queue with convex holding cost

We investigate a resource allocation problem in a multi-class server with convex holding costs and user impatience under the average cost criterion. In general, the optimal policy has a complex dependency on all the input parameters and state information. Our main contribution is to derive index policies that can serve as heuristics and are shown to give good performance. Our index policy attributes to each class an index, which depends on the number of customers currently present in that class. The index values are obtained by solving a relaxed version of the optimal stochastic control problem and combining results from restless multi-armed bandits and queueing theory. They can be expressed as a function of the steady-state distribution probabilities of a one-dimensional birth-and-death process. For linear holding cost, the index can be calculated in closed-form and turns out to be independent of the arrival rates and the number of customers present. In the case of no abandonments and linear holding cost, our index coincides with the $$c\mu $$cμ-rule, which is known to be optimal in this simple setting. For general convex holding cost, we derive properties of the index value in limiting regimes: we consider the behavior of the index (i) as the number of customers in a class grows large, which allows us to derive the asymptotic structure of the index policies, (ii) as the abandonment rate vanishes, which allows us to retrieve an index policy proposed for the multi-class M/M/1 queue with convex holding cost and no abandonments, and (iii) as the arrival rate goes to either 0 or $$\infty $$?, representing light-traffic and heavy-traffic regimes, respectively. We show that Whittle's index policy is asymptotically optimal in both light-traffic and heavy-traffic regimes. To obtain further insights into the index policy, we consider the fluid version of the relaxed problem and derive a closed-form expression for the fluid index. The latter is shown to coincide with the index values for the stochastic model in asymptotic regimes. For arbitrary convex holding cost the fluid index can be seen as the $$Gc\mu /\theta $$Gcμ/?-rule; that is, including abandonments into the generalized $$c\mu $$cμ-rule ($$Gc\mu $$Gcμ-rule). Numerical experiments for a wide range of parameters have shown that the Whittle index policy and the fluid index policy perform very well for a broad range of parameters.

A Characterization of Stationary Nash Equilibria of Single Controller Constrained Stochastic Games

We consider a two player finite state-action general sum single controller constrained stochastic game with both discounted and average cost criteria. We consider the situation where player 1 has subscription-based constraints and player 2, who controls the transition probabilities, has realization-based constraints which can also depend on the strategies of player 1. It is known that a stationary Nash equilibrium for discounted case exists under strong Slater condition, while, for the average case, stationary Nash equilibrium exists if additionally the Markov chain is unichain. For each case we show that the set of stationary Nash equilibria of this game has one to one correspondence with the set of global minimizers of a certain nonconvex mathematical program. If the constraints of player 2 do not depend on the strategies of player 1, then the mathematical program reduces to a quadratic program. The known linear programs for zero sum games of this class can be obtained as a special case of above quadratic programs.

Asymptotic Optimality and Rates of Convergence of Quantized Stationary Policies in Stochastic Control

We consider the discrete approximation of stationary policies for a discrete-time Markov decision process with Polish state and action spaces under total, discounted, and average cost criteria. Deterministic stationary quantizer policies are introduced and shown to be able to approximate optimal deterministic stationary policies with arbitrary precision under mild technical conditions, thus demonstrating that one can search for $\varepsilon$ -optimal policies within the class of quantized control policies. We also derive explicit bounds on the approximation error in terms of the quantization rate.

Strong average optimality criterion for continuous-time Markov decision processes

This paper deals with continuous-time Markov decision processes with the unbounded transition rates under the strong average cost criterion. The state and action spaces are Borel spaces, and the costs are allowed to be unbounded from above and from below. Under mild conditions, we first prove that the finite-horizon optimal value function is a solution to the optimality equation for the case of uncountable state spaces and unbounded transition rates, and that there exists an optimal deterministic Markov policy. Then, using the two average optimality inequalities, we show that the set of all strong average optimal policies coincides with the set of all average optimal policies, and thus obtain the existence of strong average optimal policies. Furthermore, employing the technique of the skeleton chains of controlled continuous-time Markov chains and Chapman-Kolmogorov equation, we give a new set of sufficient conditions imposed on the primitive data of the model for the verification of the uniform exponential ergodicity of continuous-time Markov chains governed by stationary policies. Finally, we illustrate our main results with an example.

Approximate dynamic programming for stochastic linear control problems on compact state spaces

This paper addresses Markov Decision Processes over compact state and action spaces. We investigate the special case of linear dynamics and piecewise-linear and convex immediate costs for the average cost criterion. This model is very general and covers many interesting examples, for instance in inventory management. Due to the curse of dimensionality, the problem is intractable and optimal policies usually cannot be computed, not even for instances of moderate size.We show the existence of optimal policies and of convex and bounded relative value functions that solve the average cost optimality equation under reasonable and easy-to-check assumptions. Based on these insights, we propose an approximate relative value iteration algorithm based on piecewise-linear convex relative value function approximations. Besides computing good policies, the algorithm also provides lower bounds to the optimal average cost, which allow us to bound the optimality gap of any given policy for a given instance.The algorithm is applied to the well-studied Multiple Sourcing Problem as known from inventory management. Multiple sourcing is known to be a hard problem and usually tackled by parametric heuristics. We analyze several MSP instances with two and more suppliers and compare our results to state-of-the-art heuristics. For the considered scenarios, our policies are always at least as good as the best known heuristic, and strictly better in most cases. Moreover, by using the computed lower bounds we show for all instances that the optimality gap has never exceeded 5%, and that it has been much smaller for most of them.

Index policies for a multi-class queue with convex holding cost and abandonments

We investigate a resource allocation problem in a multi-class server with convex holding costs and user impatience under the average cost criterion. In general, the optimal policy has a complex dependency on all the input parameters and state information. Our main contribution is to derive index policies that can serve as heuristics and are shown to give good performance. Our index policy attributes to each class an index, which depends on the number of customers currently present in that class. The index values are obtained by solving a relaxed version of the optimal stochastic control problem and combining results from restless multi-armed bandits and queueing theory. They can be expressed as a function of the steady-state distribution probabilities of a one-dimensional birth-and-death process. For linear holding cost, the index can be calculated in closed-form and turns out to be independent of the arrival rates and the number of customers present. In the case of no abandonments and linear holding cost, our index coincides with the c μ-rule, which is known to be optimal in this simple setting. For general convex holding cost we derive properties of the index value in limiting regimes: we consider the behavior of the index (i) as the number of customers in a class grows large, which allows us to derive the asymptotic structure of the index policies, and (ii) as the abandonment rate vanishes, which allows us to retrieve an index policy proposed for the multi-class M/M/1 queue with convex holding cost and no abandonments. In fact, in a multi-server environment it follows from recent advances that the index policy is asymptotically optimal for linear holding cost. To obtain further insights into the index policy, we consider the fluid version of the relaxed problem and derive a closed-form expression for the fluid index. The latter coincides with the stochastic model in case of linear holding costs. For arbitrary convex holding cost the fluid index can be seen as the Gc μθ-rule, that is, including abandonments into the generalized c μ-rule ( Gc μ-rule). Numerical experiments show that our index policies become optimal as the load in the system increases.

Operations sequencing for a multi‐stage production inventory system

AbstractThis article studies operations sequencing for a multi‐stage production inventory system with lead times under predictable (deterministic) yield losses and random demand. We consider various cases with either full or partial release of work‐in‐process inventories, for either pre‐operation or post‐operation cost structures, and under either the total discounted or average cost criteria. We derive necessary and sufficient criteria for the optimal sequence of operations in all cases. While the criteria differ in their specific forms, they all lead to the same principal: those operations with (1) lower yields, (2) lower processing costs, (3) longer lead times, and (4) lower inventory holding costs should be placed higher upstream in the system.Copyright © 2014 Wiley Periodicals, Inc. Naval Research Logistics 61: 144–154, 2014