Discounted Cost Criterion Research Articles

Discrete-time stopping games with risk-sensitive discounted cost criterion

Discrete-time stopping games with risk-sensitive discounted cost criterion

Optimal cash management using impulse control

We consider the impulse control of Lévy processes under the infinite horizon, discounted cost criterion. Our motivating example is the cash management problem in which a controller is charged a fixed plus proportional cost for adding to or withdrawing from his/her reserve, plus an opportunity cost for keeping any cash on hand. Our main result is to provide a verification theorem for the optimality of control band policies in this scenario. We also analyze the transient and steady-state behavior of the controlled process under control band policies and explicitly solve for an optimal policy in the case in which the Lévy process to be controlled is the sum of a Brownian motion with drift and a compound Poisson process with exponentially distributed jump sizes.

Continuous-Time Markov Decision Processes Under the Risk-Sensitive First Passage Discounted Cost Criterion

Continuous-Time Markov Decision Processes Under the Risk-Sensitive First Passage Discounted Cost Criterion

Inventory dispensation and restocking in multiclass inventory systems: A decoupling approach

We consider an inventory system with Poisson demand arrivals arising from multiple classes. Restocking epochs are specified by a given sequence. At each epoch, we can place an order for any number of items for immediate delivery. We have to decide the inventory dispensation policy, which determines whether we should satisfy a demand, or reject it, or backlog it until the next restocking epoch. We allow class‐dependent rejection and backlogging costs. We also incorporate holding costs and procurement costs. We obtain the optimal restocking and inventory dispensation policies under the total discounted cost criterion as well as the long‐run average cost criterion. In particular, we develop efficient algorithms to compute the optimal policies (OPs) by decoupling the inventory dispensation policy from the restocking policy. We show that the OP operates as follows: At the restocking epochs, we order up to a fixed base‐stock level. We satisfy the demands from a class if the inventory is above a critical level that depends on the class and time to the next restocking epoch; else we reject or backlog it depending on whether the next restocking epoch is later or earlier than a class‐dependent critical number. The numerical experiments for the stochastic and deterministic interstocking periods are conducted to compare the OPs and costs.

Risk-sensitive first passage stochastic games with unbounded costs

ABSTRACT This paper studies discrete-time nonzero-sum stochastic games under the risk-sensitive first passage discounted cost criterion. The state space is a countable set and the costs are allowed to be unbounded. Under the suitable optimality conditions, we prove that the risk-sensitive first passage discounted optimal value function of each player is a unique solution to the risk-sensitive first passage optimality equation via an approximation method. Moreover, by the risk-sensitive first passage discounted optimality equation, we show the existence of a randomized Markov Nash equilibrium. Finally, three examples are given to illustrate the results.

Constrained Discounted Stochastic Games

In this paper, we consider a large class of constrained non-cooperative stochastic Markov games with countable state spaces and discounted cost criteria. In one-player case, i.e., constrained discounted Markov decision models, it is possible to formulate a static optimisation problem whose solution determines a stationary optimal strategy (alias control or policy) in the dynamical infinite horizon model. This solution lies in the compact convex set of all occupation measures induced by strategies, defined on the set of state–action pairs. In case of n-person discounted games the occupation measures are induced by strategies of all players. Therefore, it is difficult to generalise the approach for constrained discounted Markov decision processes directly. It is not clear how to define the domain for the best-response correspondence whose fixed point induces a stationary equilibrium in the Markov game. This domain should be the Cartesian product of compact convex sets in a locally convex topological vector space. One of our main results shows how to overcome this difficulty and define a constrained non-cooperative static game whose Nash equilibrium induces a stationary Nash equilibrium in the Markov game. This is done for games with bounded cost functions and positive initial state distribution. An extension to a class of Markov games with unbounded costs and arbitrary initial state distribution relies on an approximation of the unbounded game by bounded ones with positive initial state distributions. In the unbounded case, we assume the uniform integrability of the discounted costs with respect to all probability measures induced by strategies of the players, defined on the space of plays (histories) of the game. Our assumptions are weaker than those applied in earlier works on discounted dynamic programming or stochastic games using the so-called weighted norm approach.

Risk-sensitive discounted cost criterion for continuous-time Markov decision processes on a general state space

In this paper, we consider risk-sensitive discounted control problem for continuous-time jump Markov processes taking values in general state space. The transition rates of underlying continuous-time jump Markov processes and the cost rates are allowed to be unbounded. Under certain Lyapunov condition, we establish the existence and uniqueness of the solution to the Hamilton–Jacobi–Bellman equation. Also, we prove the existence of optimal risk-sensitive control in the class of Markov control and completely characterized the optimal control.

Zero-sum games for pure jump processes with risk-sensitive discounted cost criteria

<p style='text-indent:20px;'>In this paper we study zero-sum stochastic games for pure jump processes on a general state space with risk sensitive discounted criteria. We establish a saddle point equilibrium in Markov strategies for bounded cost function. We achieve our results by studying relevant Hamilton-Jacobi-Isaacs equations.</p>

Continuous-time zero-sum games for markov decision processes with discounted risk-sensitive cost criterion on a general state space

We consider zero-sum stochastic games for controlled continuous time Markov processes on a general state space with risk-sensitive discounted cost criteria. The transition and cost rates are possibly unbounded. Under a stability assumption, we prove the existence of a saddle-point equilibrium in the class of Markov strategies and give a characterization in terms of the corresponding Hamilton-Jacobi-Isaacs (HJI) equation. Also, we illustrate our results and assumptions by an example.

Impulse Control with Discontinuous Setup Costs: Discounted Cost Criterion

This paper studies a continuous-review backlogged inventory model considered by [K. L. Helmes, R. H. Stockbridge, and C. Zhu, SIAM J. Control. Optim., 53 (2015), pp. 2100--2140] but with discontinuous quantity-dependent setup cost for each order. In particular, the setup cost is characterized by a two-step function and a higher cost would be charged once the order quantity exceeds a threshold $Q$. Unlike the optimality of the $(s,S)$-type policy obtained by Helmes, Stockbridge, and Zhu for continuous setup cost with the discounted cost criterion, we find that, in our model, although some $(s,S)$-type policy is indeed optimal in some cases, the $(s,S)$-type policy cannot always be optimal. In particular, we show that there exist cases in which an $(s,S)$ policy is optimal for some initial levels but it is strictly worse than a generalized $(s,\{S(x):x\leq s\})$ policy for the other initial levels. Under $(s,\{S(x):x\leq s\})$ policy, it orders nothing for $x>s$ and orders up to level $S(x)$ for $x\leq s$, where $S(x)$ is a nonconstant function of $x$. We further prove the optimality of such $(s,\{S(x):x\leq s\})$ policy in a large subset of admissible policies for those initial levels. Moreover, the optimality is obtained through establishing a more general lower bound theorem which will also be applicable in solving some other optimization problems by the lower bound approach.

Point-Based Value Iteration and Approximately Optimal Dynamic Sensor Selection for Linear-Gaussian Processes

The problem of synthesizing an optimal sensor selection policy is pertinent to a variety of engineering applications ranging from event detection to autonomous navigation. We consider such a synthesis problem in the context of linear-Gaussian systems over an infinite time horizon with a discounted cost criterion. We formulate this problem in terms of a value iteration over the continuous space of covariance matrices. To obtain a computationally tractable solution, we subsequently formulate an approximate sensor selection problem, which is solvable through a point-based value iteration over a finite “mesh” of covariance matrices with a user-defined bounded trace. We provide theoretical guarantees bounding the suboptimality of the sensor selection policies synthesized through this method and provide numerical examples comparing them to known results.

Nonzero-sum risk-sensitive stochastic differential games with discounted costs

We study nonzero-sum stochastic differential games with risk-sensitive discounted cost criteria. Under fairly general conditions on drift term and diffusion coefficients, we establish a Nash equilibrium in Markov strategies for the discounted cost criterion. We achieve our results by studying relevant systems of coupled HJB equations.

Asymptotic Optimality of Finite Model Approximations for Partially Observed Markov Decision Processes With Discounted Cost

We consider finite model approximations of discrete-time partially observed Markov decision processes (POMDPs) under the discounted cost criterion. After converting the original partially observed stochastic control problem to a fully observed one on the belief space, the finite models are obtained through the uniform quantization of the state and action spaces of the belief space Markov decision process (MDP). Under mild assumptions on the components of the original model, it is established that the policies obtained from these finite models are nearly optimal for the belief space MDP, and so, for the original partially observed problem. The assumptions essentially require that the belief space MDP satisfies a mild weak continuity condition. We provide an example and introduce explicit approximation procedures for the quantization of the set of probability measures on the state space of POMDP (i.e., belief space).

Stability Estimation for Markov Control Processes with Discounted Cost

This article explores controllable Borel spaces, stationary, homogeneous Markov processes, discrete time with infinite horizon, with bounded cost functions and using the expected total discounted cost criterion. The problem of the estimation of stability for this type of process is set. The central objective is to obtain a bounded stability index expressed in terms of the Levy-Prokhorov metric; likewise, sufficient conditions are provided for the existence of such inequalities.

Approximate Nash Equilibria in Partially Observed Stochastic Games with Mean-Field Interactions

Establishing the existence of Nash equilibria for partially observed stochastic dynamic games is known to be quite challenging, with the difficulties stemming from the noisy nature of the measurements available to individual players (agents) and the decentralized nature of this information. When the number of players is sufficiently large and the interactions among agents is of the mean-field type, one way to overcome this challenge is to investigate the infinite-population limit of the problem, which leads to a mean-field game. In this paper, we consider discrete-time partially observed mean-field games with infinite-horizon discounted-cost criteria. Using the technique of converting the original partially observed stochastic control problem to a fully observed one on the belief space and the dynamic programming principle, we establish the existence of Nash equilibria for these game models under very mild technical conditions. Then, we show that the mean-field equilibrium policy, when adopted by each agent, forms an approximate Nash equilibrium for games with sufficiently many agents.

The Benefits of State Aggregation with Extreme-Point Weighting for Assemble-to-Order Systems

We provide a new method for solving a very general model of an assemble-to-order system: multiple products, multiple components that may be demanded in different quantities by different products, batch production, random lead times, and lost sales, modeled as a Markov decision process under the discounted cost criterion. A control policy specifies when a batch of components should be produced and whether an arriving demand for each product should be satisfied. As optimal solutions for our model are computationally intractable for even moderately sized systems, we approximate the optimal cost function by reformulating it on an aggregate state space and restricting each aggregate state to be represented by its extreme original states. Our aggregation drastically reduces the value iteration computational burden. We derive an upper bound on the distance between aggregate and optimal solutions. This guarantees that the value iteration algorithm for the original problem initialized with the aggregate solution converges to the optimal solution. We also establish the optimality of a lattice-dependent base-stock and rationing policy in the aggregate problem when certain product and component characteristics are incorporated into the aggregation/disaggregation schemes. This enables us to further alleviate the value iteration computational burden in the aggregate problem by eliminating suboptimal actions. Leveraging all of our results, we can solve the aggregate problem for systems of up to 22 components, with an average distance of 11.09% from the optimal cost in systems of up to 4 components (for which we could solve the original problem to optimality). The e-companion is available at https://doi.org/10.1287/opre.2017.1710 .

Discrete-Time Hybrid Control in Borel Spaces

A discrete-time hybrid control model with Borel state and action spaces is introduced. In this type of models, the dynamic of the system is composed by two sub-dynamics affecting the evolution of the state; one is of a standard-type that runs almost every time and another is of a special-type that is active under special circumstances. The controller is able to use two different type of actions, each of them is applied to each of the two sub-dynamics, and the activations of these sub-dynamics are possible according to an activation rule that can be handled by the controller. The aim for the controller is to find a control policy, containing a mix of actions (of either standard- or special-type), with the purpose of minimizing an infinite-horizon discounted cost criterion whose discount factor is dependent on the state-action history and may be equal to one at some stages. Two different sets of conditions are proposed to guarantee (i) the finiteness of the cost criterion, (ii) the characterization of the optimal value function and (iii) the existence of optimal control policies; to do so, we employ the dynamic programming approach. A useful characterization that signalizes the accurate times between changes of sub-dynamics in terms of the so-named contact set is also provided. Finally, we introduce two examples that illustrate our results and also show that control models such as discrete-time impulse control models and discrete-time switching control models become special cases of our present hybrid model.

Customizing exponential semi-Markov decision processes under the discounted cost criterion

The uniformization technique is a widely used method for establishing the existence of optimal policies with certain monotonicity properties. This technique converts a semi-Markov decision process with exponential sojourn times (ESMDP) into an equivalent discrete-time Markov decision process by defining some fictitious jumps. This study proposes a new device, called customization, which can convert a given ESMDP into another equivalent ESMDP whose formulation possibly simplifies mathematical analysis. The customization technique uses the fictitious jump idea to establish the equivalence under deterministic stationary policies just like the uniformization technique. However, it allows the transition rates of the new ESMDP to be different. Moreover, it can be applied even when the transition rates of the initial ESMDP are unbounded. This flexibility can be very useful in analyzing the problems where the uniformization is not applicable or not so helpful. We analyze a complex optimal replacement problem and an infinite server queueing problem with unbounded transition rates to demonstrate the applicability and advantages of customization.

Continuous-Time Constrained Stochastic Games under the Discounted Cost Criteria

In this paper, we consider the continuous-time nonzero-sum constrained stochastic games with the discounted cost criteria. The state space is denumerable and the action space of each player is a general Polish space, while the transition rates and cost functions are allowed to be unbounded from below and from above. The strategies for each player may be history-dependent and randomized. Models with these features seemingly have not been handled in the previous literature. By constructing a sequence of continuous-time finite-state game models to approximate the original denumerable-state game model, we prove the existence of constrained Nash equilibria for the constrained games with denumerable states.

Rationally Inattentive Control of Markov Processes

The article poses a general model for optimal control subject to information constraints, motivated in part by recent work of Sims and others on information-constrained decision making by economic agents. In the average-cost optimal control framework, the general model introduced in this paper reduces to a variant of the linear-programming representation of the average-cost optimal control problem, subject to an additional mutual information constraint on the randomized stationary policy. The resulting optimization problem is convex and admits a decomposition based on the Bellman error, which is the object of study in approximate dynamic programming. The theory is illustrated through the example of information-constrained linear quadratic Gaussian control problem. Some results on the infinite-horizon discounted-cost criterion are also presented.