Average Cost Criterion Research Articles

Optimal server assignment in a two-stage tandem queueing system

We study Markovian queueing systems consisting of two stations in tandem. There is a dedicated server in each station and an additional server that can be assigned to any station. Assuming that linear holding costs are incurred by jobs in the system and two servers can collaborate to work on the same job, we determine structural properties of optimal server assignment policies under the discounted and the average cost criteria.

OPTIMAL CONTROL POLICIES FOR AN M/M/1 QUEUE WITH A REMOVABLE SERVER AND DYNAMIC SERVICE RATES

AbstractWe consider an M/M/1 queue with a removable server that dynamically chooses its service rate from a set of finitely many rates. If the server is off, the system must warm up for a random, exponentially distributed amount of time, before it can begin processing jobs. We show under the average cost criterion, that work conserving policies are optimal. We then demonstrate the optimal policy can be characterized by a threshold for turning on the server and the optimal service rate increases monotonically with the number in system. Finally, we present some numerical experiments to provide insights into the practicality of having both a removable server and service rate control.

Finite-State Approximations to Discounted and Average Cost Constrained Markov Decision Processes

In this paper, we consider the finite-state approximation of a discrete-time constrained Markov decision process (MDP) under the discounted and average cost criteria. Using the linear programming formulation of the constrained discounted cost problem, we prove the asymptotic convergence of the optimal value of the finite-state model to the optimal value of the original model. With further continuity condition on the transition probability, we also establish a method to compute approximately optimal policies. For the average cost, instead of using the finite-state linear programming approximation method, we use the original problem definition to establish the finite-state asymptotic approximation of the constrained problem and compute approximately optimal policies. Under Lipschitz-type regularity conditions on the components of the MDP, we also obtain explicit rate of convergence bounds quantifying how the approximation improves as the size of the approximating finite-state space increases.

Solving traffic queues at controlled-signalized intersections in continuous-time Markov games

The traffic signal control plays a fundamental role to improve the efficiency and efficacy of traffic flows in traffic networks. This paper is the first work in which we consider a mathematically rigorous study of the continuous-time, discrete state, multi-traffic signal control problem using a non-cooperative game theory approach. The solution of the problem is circumscribed to an ergodic, controllable, discrete state, continuous-time Markov game computed under the expected average cost criterion. This paper provides several main contributions. First, we present a general continuous-time queue model, which is employed as the fundamental scheme of a computationally tractable game theory approach for the signal control continuous-time Markov game. This model is transformed into a discrete state Poisson process where the vehicles leave the queue in the order they arrive. Second, in this problem, each signal controller (player) aims at finding green time that minimizes its signal and queuing delay. Then, a conflict appears when each signal controller tries to minimize its queue. We study the problem of computing a Nash equilibrium for this game. Our third contribution employs a proximal/gradient method for computing the Nash equilibrium point of the game. By introducing new restrictions over the signal controller and adding a restriction for continuous-time Markov chains, we obtain the set of average optimal policies, which is one of the main results of this paper. Hence, our final contribution shows, in simulation, the usefulness of the proposed method with an application example.

A weak convergence approach to inventory control using a long-term average criterion

Abstract In this paper we continue the examination of inventory control in which the inventory is modeled by a diffusion process and a long-term average cost criterion is used to make decisions. The class of such models under consideration has general drift and diffusion coefficients, and boundary points that are consistent with the notion that demand should tend to reduce the inventory level. The conditions on the cost functions are greatly relaxed from those in Helmes et al. (2017). Characterization of the cost of a general (s, S) policy as a function of two variables naturally leads to a nonlinear optimization problem over the ordering levels s and S. Existence of an optimizing pair (s*, S*) is established for these models under very weak conditions; nonexistence of an optimizing pair is also discussed. Using average expected occupation and ordering measures and weak convergence arguments, weak conditions are given for the optimality of the (s*, S*) ordering policy in the general class of admissible policies. The analysis involves an auxiliary function that is globally C2 and which, together with the infimal cost, solves a particular system of linear equations and inequalities related to but different from the long-term average Hamilton‒Jacobi‒Bellman equation. This approach provides an analytical solution to the problem rather than a solution involving intricate analysis of the stochastic processes. The range of applicability of these results is illustrated on a drifted Brownian motion inventory model, both unconstrained and reflected, and on a geometric Brownian motion inventory model under two different cost structures.

Risk-sensitive average continuous-time Markov decision processes with unbounded rates

In this paper we study the risk-sensitive average cost criterion for continuous-time Markov decision processes in the class of all randomized Markov policies. The state space is a denumerable set, and the cost and transition rates are allowed to be unbounded. Under the suitable conditions, we establish the optimality equation of the auxiliary risk-sensitive first passage optimization problem and obtain the properties of the corresponding optimal value function. Then by a technique of constructing the appropriate approximating sequences of the cost and transition rates and employing the results on the auxiliary optimization problem, we show the existence of a solution to the risk-sensitive average optimality inequality and develop a new approach called the risk-sensitive average optimality inequality approach to prove the existence of an optimal deterministic stationary policy. Furthermore, we give some sufficient conditions for the verification of the simultaneous Doeblin condition, use a controlled birth and death system to illustrate our conditions and provide an example for which the risk-sensitive average optimality strict inequality occurs.

Characterization of the Optimal Risk-Sensitive Average Cost in Denumerable Markov Decision Chains

This work is concerned with Markov decision chains on a denumerable state space. The controller has a positive risk-sensitivity coefficient, and the performance of a control policy is measured by a risk-sensitive average cost criterion. Besides standard continuity-compactness conditions, it is assumed that the state process is communicating under any stationary policy, and that the simultaneous Doeblin condition holds. In this context, it is shown that if the cost function is bounded from below, and the superior limit average index is finite at some point, then (i) the optimal superior and inferior limit average value functions coincide and are constant, (ii) the optimal average cost is characterized via an extended version of the Collatz-Wielandt formula in the theory of positive matrices, and (iii) an optimality inequality is established, from which a stationary optimal policy is obtained. Moreover, an explicit example is given to show that, even if the cost function is bounded, the strict inequality may occur in the optimality relation.

Optimal Control of Customers to the Service Facility with Two Types of Customers

In this article, we considered a discrete-time service facility system, viewed as a Markov Decision Process (MDP). Decisions are taken at discrete time epochs to control admissions to the system. Here the queue before the server is divided into eligible queue and potential queue. Potential queue has two types of customers (Priority and nonpriority). It is assumed that demands arrived throughout the period but they are satisfied only at the end of the period. The MDP based on average cost criteria is used to find the optimal policy to be implemented for the system. Numerical example is provided to illustrate the problem.

Optimal Drift Rate Control and Impulse Control for a Stochastic Inventory/Production System

In this paper, we consider joint drift rate control and impulse control for a stochastic inventory system under a long-run average cost criterion. Assuming the inventory level must be nonnegative, we prove that a $\{(0,q^{\star},Q^{\star},S^{\star}),\{\mu^{\star}(x): x\in[0, S^{\star}]\}\}$ policy is an optimal joint control policy, where the impulse control follows the control band policy $(0,q^{\star},Q^{\star},S^{\star})$, which brings the inventory level up to $q^{\star}$ once it drops to $0$ and brings it down to $Q^{\star}$ once it rises to $S^{\star}$, and the drift rate only depends on the current inventory level and is given by function $\mu^{\star}(x)$ for the inventory level $x\in[0,S^{\star}]$. The optimality of the $\{(0,q^{\star},Q^{\star},S^{\star}),\{\mu^{\star}(x): x\in[0,S^{\star}]\}\}$ policy is proven by using a lower bound approach, in which a critical step is to prove the existence and uniqueness of optimal policy parameters. To prove the existence and uniqueness, we develop a novel analytical method to solve a free boundary problem consisting of an ordinary differential equation and several free boundary conditions. Furthermore, we find that the optimal drift rate $\mu^{\star}(x)$ is first increasing and then decreasing as $x$ increases from 0 to $S^{\star}$ with a turnover point between $Q^{\star}$ and $S^{\star}$.

Convergence of Finite Element Methods for Singular Stochastic Control

A numerical method is proposed for a class of stochastic control problems including singular behavior. This method solves an infinite-dimensional linear program equivalent to the stochastic control problem using a finite element type approximation, which results in a solvable finite-dimensional program. The discretization scheme as well as the necessary assumptions are discussed, and a detailed convergence analysis for the discretization scheme is given. Its performance is illustrated by two examples featuring a long-term average cost criterion.

On Stochastic Sensor Network Scheduling for Multiple Processes

We consider the problem of multiple sensor scheduling for remote state estimation of multiple process over a shared link. In this problem, a set of sensors monitor mutually independent dynamical systems in parallel but only one sensor can access the shared channel at each time to transmit the data packet to the estimator. We propose a stochastic event-based sensor scheduling in which each sensor makes transmission decisions based on both channel accessibility and distributed event-triggering conditions. The corresponding minimum mean squared error (MMSE) estimator is explicitly given. Considering information patterns accessed by sensor schedulers, time-based ones can be treated as a special case of the proposed one. By ultilizing realtime information, the proposed schedule outperforms the time-based ones in terms of the estimation quality. Resorting to solving an Markov decision process (MDP) problem with average cost criterion, we can find optimal parameters for the proposed schedule. As for practical use, a greedy algorithm is devised for parameter design, which has rather low computational complexity. We also provide a method to quantify the performance gap between the schedule optimized via MDP and any other schedules.

CONTINUOUS REVIEW INVENTORY MODELS FOR PERISHABLE ITEMS WITH LEADTIMES

We consider a continuous review (s, S) model of perishable items with lost sales. Once items are perished the entire inventory drops instantaneously to zero. The total cost includes the cost of: ordering, unsatisfied demand, units destroyed, holding, and fixed cost of perishability. Both the time to perishability and the lead times are assumed to be exponentially distributed while two cases of demand distribution are considered: Poisson and compound Poisson with general demand sizes. We study the average cost criterion and provide computational results on the problem of finding the optimal re-order level,s, and order up-to level,S. None of the known work on the subject is as general as the model presented here. Our analysis leads to several insights on the optimal (s, S) policies for perishable items in the presence of lead times. For example, we demonstrate that the effectiveness of a heuristic that ignores perishability (and is also analyzed here) decreases with the demand variability and that the cost may either increase or decrease with this variability.

Continuous inventory models of diffusion type: Long-term average cost criterion

This paper establishes conditions for optimality of an $(s,S)$ ordering policy for the minimization of the long-term average cost of one-dimensional diffusion inventory models. The class of such models under consideration have general drift and diffusion coefficients and boundary points that are consistent with the notion that demand should tend to reduce the inventory level. Characterization of the cost of a general $(s,S)$ policy as a function $F$ of two variables naturally leads to a nonlinear optimization problem over the ordering levels $s$ and $S$. Existence of an optimizing pair $(s_{*},S_{*})$ is established for these models. Using the minimal value $F_{*}$ of $F$, along with $(s_{*},S_{*})$, a function $G$ is identified which is proven to be a solution of a quasi-variational inequality provided a simple condition holds. At this level of generality, optimality of the $(s_{*},S_{*})$ ordering policy is established within a large class of ordering policies for which local martingale and transversality conditions involving $G$ hold. For specific models, optimality of an $(s,S)$ policy in the general class of admissible policies can be established using comparison results. This most general optimality result is shown for the classical drifted Brownian motion inventory model with holding and fixed plus proportional ordering costs. Optimality of an $(s,S)$ ordering policy is also extended to the general class of admissible policies for a geometric Brownian motion inventory model with fixed plus level-dependent ordering costs. However, for a drifted Brownian motion process with reflection at $\{0\}$, a new class of non-Markovian policies is introduced which have lower costs than the $(s,S)$ policies. In addition, interpreting reflection at $\{0\}$ as “just-in-time” ordering, a necessary and sufficient condition is given that determines when just-in-time ordering is better than traditional $(s,S)$ policies.

Average cost criterion induced by the regular utility function for continuous-time Markov decision processes

In this paper we study the average cost criterion induced by the regular utility function (U-average cost criterion) for continuous-time Markov decision processes. This criterion is a generalization of the risk-sensitive average cost and expected average cost criteria. We first introduce an auxiliary risk-sensitive first passage optimization problem and obtain the properties of the corresponding optimal value function under the slight conditions. Then we show that the pair of the optimal value functions of the risk-sensitive average cost criterion and the risk-sensitive first passage criterion is a solution to the optimality equation of the risk-sensitive average cost criterion allowing the risk-sensitivity parameter to take any nonzero value. Moreover, we have that the optimal value function of the risk-sensitive average cost criterion is continuous with respect to the risk-sensitivity parameter. Finally, we give the connections between the U-average cost criterion and the average cost criteria induced by the identity function and the exponential utility function, and prove the existence of a U-average optimal deterministic stationary policy in the class of all randomized Markov policies.

Local Poisson Equations Associated with Discrete-Time Markov Control Processes

This paper provides a characterization of the optimal average cost function, when the long-run (risk-sensitive) average cost criterion is used. The Markov control model has a denumerable state space with finite set of actions, and the characterization presented is given in terms of a system of local Poisson equations, which gives as a by-product the existence of an optimal stationary policy.

Reformulation of the linear program for completely ergodic MDPs with average cost criteria

In this paper, we study discrete infinite-time horizon undiscounted finite-state and finite-action \(\textit{MDP}\)s with average cost criteria. Based on a combinatorial approach, we show that completely ergodic \(\textit{MDP}\)s can be reduced to a generalized minimum cost circulation problem with equality constraints. We show that although the reduced problem has the same complexity as the original one, it enables a linear programming problem with an underlying network structure for which strongly polynomial algorithms can be developed.

Adaptive Multi-Resource Allocation for Cloudlet-Based Mobile Cloud Computing System

Mobile cloud computing utilizing cloudlet is an emerging technology to improve the quality of mobile services. In this paper, to better overcome the main bottlenecks of the computation capability of cloudlet and the wireless bandwidth between mobile devices and cloudlet, we consider the multi-resource allocation problem for the cloudlet environment with resource-intensive and latency-sensitive mobile applications. The proposed multi-resource allocation strategy enhances the quality of mobile cloud service, in terms of the system throughput (the number of admitted mobile applications) and the service latency. We formulate the resource allocation model as a semi-Markov decision process under the average cost criterion, and solve the optimization problem using linear programming technology. Through maximizing the long-term reward while meeting the system requirements of the request blocking probability and service time latency, an optimal resource allocation policy is calculated. From simulation result, it is indicated that the system adaptively adjusts the allocation policy about how much resource to allocate and whether to utilize the distant cloud according to the traffic of mobile service requests and the availability of the resource in the system. Our algorithm outperforms greedy admission control over a broad range of environments.

Ergodic diffusion control of multiclass multi-pool networks in the Halfin–Whitt regime

We consider Markovian multiclass multi-pool networks with heterogeneous server pools, each consisting of many statistically identical parallel servers, where the bipartite graph of customer classes and server pools forms a tree. Customers form their own queue and are served in the first-come first-served discipline, and can abandon while waiting in queue. Service rates are both class and pool dependent. The objective is to study the limiting diffusion control problems under the long run average (ergodic) cost criteria in the Halfin–Whitt regime. Two formulations of ergodic diffusion control problems are considered: (i) both queueing and idleness costs are minimized, and (ii) only the queueing cost is minimized while a constraint is imposed upon the idleness of all server pools. We develop a recursive leaf elimination algorithm that enables us to obtain an explicit representation of the drift for the controlled diffusions. Consequently, we show that for the limiting controlled diffusions, there always exists a stationary Markov control under which the diffusion process is geometrically ergodic. The framework developed in [Ann. Appl. Probab. 25 (2015) 3511–3570] is extended to address a broad class of ergodic diffusion control problems with constraints. We show that the unconstrained and constrained problems are well posed, and we characterize the optimal stationary Markov controls via HJB equations.

A Duality Framework for Stochastic Optimal Control of Complex Systems

We address the problem of minimizing the long-run expected average cost of a complex system consisting of interactive subsystems. We formulate a multiobjective optimization problem of the one-stage expected costs of the subsystems and provide a duality framework to prove that the control policy yielding the Pareto optimal solution minimizes the average cost criterion of the system. We provide the conditions of existence and a geometric interpretation of the solution. For practical situations with constraints consistent to those studied here, our results imply that the Pareto control policy may be of value when we seek to derive online the optimal control policy in complex systems.

Inventory Rationing for Multiple Class Demand under Continuous Review

We consider how a firm should ration inventory to multiple classes in a stochastic demand environment with partial, class‐dependent backlogging where the firm incurs a fixed setup cost when ordering from its supplier. We present an infinite‐horizon, average cost criterion Markov decision problem formulation for the case with zero lead times. We provide an algorithm that determines the optimal rationing policy, and show how to find the optimal base‐stock reorder policy. Numerical studies indicate that the optimal policy is similar to that given by the equivalent deterministic problem and relies on tracking both the current inventory and the rate that backorder costs are accumulating. Our study of the case of non‐zero lead time shows that a heuristic combining the optimal, zero lead time policy with an allocation policy based on a single‐period profit management problem is effective.