Markov Policies Research Articles

A state-dependent perishability (s, S) inventory model with random batch demands

We consider continuous-review perishable inventory models with random lead times and state-dependent Poisson demand. The paper revises an earlier work of Barron and Baron (IISE Trans 1–52, 2019). While the former studies unit Poisson demands, this paper deals with demand uncertainty and allows for random batch demands. We conduct a comprehensive analysis of two main models that have different lead times and perish times under backorders or lost sales. Thus, our models can be applied to many industries, in situations where the system is subject to random perishability, random lead time, and demand uncertainty. With a probabilistic approach, we derive a long-run average cost function under the (S, s) replenishment policy. Numerical examples are used to demonstrate the impact of changing batch size and other system parameters on the optimal policy. Our numerical study indicates that, although the Markovian policy can be used as a good approximation of the average total cost, it performs better for a general perish time. We further show that the optimal cost may differ for a different average batch size, while the batch variability seems to provide some robustness.

When to preempt? Age of information minimization under link capacity constraint

In this paper, we consider a scenario where a source continuously monitors an object and sends time-stamped status updates to a destination through a rate-limited link. We assume updates arrive randomly at the source according to a Bernoulli process. Due to the link capacity constraint, it takes multiple time slots for the source to complete the transmission of an update. Therefore, when a new update arrives at the source during the transmission of another update, the source needs to decide whether to skip the new arrival or to switch to it, in order to minimize the expected average age of information (AoI) at the destination. We start with the setting where all updates are of the same size, and prove that within a broadly defined class of online policies, the optimal policy should be a renewal policy, and has a sequential switching property. We then show that the optimal decision of the source in any time slot has threshold structures, and only depends on the age of the update being transmitted and the AoI at the destination. We then consider the setting where updates are of different sizes, and show that the optimal Markovian policy also has a multiple-threshold structure. For each of the settings, we explicitly identify the thresholds by formulating the problem as a Markov Decision Process (MDP), and solve it through value iteration. Special structural properties of the corresponding optimal policy are utilized to reduce the computational complexity of the value iteration algorithm.

Markov decision processes with sequential sensor measurements

Markov decision processes (MDPs) have been used to formulate many decision-making problems in science and engineering. The objective is to synthesize the best decision (action selection) policies to maximize expected rewards (or minimize costs) for a stochastic dynamical system. In this paper, we introduce a new type of sensor measurement to the MDP model that provides additional information about the stochastic process, and hence that information can be incorporated in the decision policy to increase the performance. The new model is tailored for environments with high uncertainty. With the additional measurements, more refined information on the possible state transition is provided in real-time before taking an action. This new MDP model with sequential measurements is referred to as sequentially-observed MDP (SO-MDP). We show that the SO-MDP shares some similar properties with a standard MDP; among randomized history dependent policies, deterministic Markovian policies are still optimal. Optimal SO-MDP policies have the advantage of producing better total rewards than standard MDP policies due to the additional measurements, however computing these policies is more complex. We present two algorithms for solving the finite-horizon SO-MDP problem: the first algorithm is based on linear-programming, and the second algorithm is based on dynamic programming. We show that the complexity of computing optimal policies of the SO-MDP model with perfect sensors is the same as standard MDP. Simulations demonstrate that the SO-MDP model outperforms the standard MDP model in the presence of high environmental uncertainty.

On the Adjoint Markov Policies in Stochastic Differential Games

We consider time-homogeneous uniformly nondegenerate stochastic differential games in domains and propose constructing $\varepsilon$-optimal strategies and policies by using adjoint Markov strategies and adjoint Markov policies which are actually time-homogeneous Markov, however, relative not to the original process but to a couple of processes governed by a system consisting of the main original equation and of an adjoint stochastic equations of the same type as the main one. We show how to find $\varepsilon$-optimal strategies and policies in these classes by using the solvability in Sobolev spaces of not the original Isaacs equation but of its appropriate modification. We also give an example of a uniformly nondegenerate game where our assumptions are not satisfied and where we conjecture that there are no not only optimal Markov but even $\varepsilon$-optimal adjoint (time-homogeneous) Markov strategies for one of the players.

Risk-sensitive average continuous-time Markov decision processes with unbounded rates

In this paper we study the risk-sensitive average cost criterion for continuous-time Markov decision processes in the class of all randomized Markov policies. The state space is a denumerable set, and the cost and transition rates are allowed to be unbounded. Under the suitable conditions, we establish the optimality equation of the auxiliary risk-sensitive first passage optimization problem and obtain the properties of the corresponding optimal value function. Then by a technique of constructing the appropriate approximating sequences of the cost and transition rates and employing the results on the auxiliary optimization problem, we show the existence of a solution to the risk-sensitive average optimality inequality and develop a new approach called the risk-sensitive average optimality inequality approach to prove the existence of an optimal deterministic stationary policy. Furthermore, we give some sufficient conditions for the verification of the simultaneous Doeblin condition, use a controlled birth and death system to illustrate our conditions and provide an example for which the risk-sensitive average optimality strict inequality occurs.

A dynamic game approach to distributionally robust safety specifications for stochastic systems

This paper presents a new safety specification method that is robust against errors in the probability distribution of disturbances. Our proposed distributionally robust safe policy maximizes the probability of a system remaining in a desired set for all times, subject to the worst possible disturbance distribution in an ambiguity set. We propose a dynamic game formulation of constructing such policies and identify conditions under which a non-randomized Markov policy is optimal. Based on this existence result, we develop a practical design approach to safety-oriented stochastic controllers with limited information about disturbance distributions. However, an associated Bellman equation involves infinite-dimensional minimax optimization problems since the disturbance distribution may have a continuous density. To alleviate computational issues, we propose a duality-based reformulation method that converts the infinite-dimensional minimax problem into a semi-infinite program that can be solved using existing convergent algorithms. We prove that there is no duality gap, and that this approach thus preserves optimality. The results of numerical tests confirm that the proposed method is robust against distributional errors in disturbances, while a standard stochastic safety verification tool is not.

MARKOV DECISION POLICY BASED DATABASE SECURITY

An advanced database security system using Markov based policy is proposed. In large databases in this work, along with it’s a difficult task to keep the data up to date where huge data of dynamic nature occurs. In e-commerce sites, large data lands up and the database might not get updated. This causes staleness in data and also, if the database undergoes updation, it might not be available for access during that time. Sometimes, updation is not possible without human intervention, hence Markov based policy is used which would perform fixed interval updates effectively and lead to cost savings. In addition to updation, this policy can be used for the security of the system;the proposed system has a middleware that will restore the database using the Markov updation policy in case of attack. The middleware will process the changes in the data and display the details of the attacker and also the changes done by the attacker. Thus, this database system can also be secured using the Markov policy and this three layered security framework.

On uniqueness of time-consistent Markov policies for quasi-hyperbolic consumers under uncertainty

We give a set of sufficient conditions for uniqueness of a time-consistent stationary Markov consumption policy for a quasi-hyperbolic household under uncertainty. To the best of our knowledge, this uniqueness result is the first presented in the literature for general settings, i.e. under standard assumptions on preferences, as well as some new condition on a transition probability. This paper advocates a “generalized Bellman equation” method to overcome some predicaments of the known methods and also extends our recent existence result. Our method also works for returns unbounded from above. We provide a few natural extensions of optimal policy uniqueness: convergent and accurate computational algorithm, monotone comparative statics result and generalized Euler equation.

Business cycles and emission trading with banking

We propose an emission cap adjustment policy for an emission trading system (ETS) that allows banking of allowances for later use and faces business cycle uncertainty. The forward looking banking decisions cause a commitment problem with respect to regulator’s next period allowance supply decision. We tackle the commitment problem by focusing on the Markov equilibrium of the policy. The equilibrium policy has a surprisingly simple structure, combining features from both price and quantity instruments. The regulator’s ability to choose a new cap for each period does not render banking obsolete, but on the contrary, makes banking an integral part of the equilibrium policy. Using data from the EU ETS to calibrate the model, we solve the Markov equilibrium policy for the case of carbon emissions regulation and compare Markov policies with and without the possibility to bank allowances. A counterfactual analysis of the EU ETS, during the period of 2009–2013, shows that the optimal Markov policy would have led to a steadier allowance price level and a stricter emission regulation than what was actually observed in the EU ETS.

A Convex Optimization Approach to Distributionally Robust Markov Decision Processes With Wasserstein Distance

We consider the problem of constructing control policies that are robust against distribution errors in the model parameters of Markov decision processes. The Wasserstein metric is used to model the ambiguity set of admissible distributions. We prove the existence and optimality of Markov policies and develop convex optimization-based tools to compute and analyze the policies. Our methods, which are based on the Kantorovich convex relaxation and duality principle, have the following advantages. First, the proposed dual formulation of an associated Bellman equation resolves the infinite dimensionality issue that is inherent in its original formulation when the nominal distribution has a finite support. Second, our duality analysis identifies the structure of a worst-case distribution and provides a simple decentralized method for its construction. Third, a sensitivity analysis tool is developed to quantify the effect of ambiguity set parameters on the performance of distributionally robust policies. The effectiveness of our proposed tools is demonstrated through a human-centered air conditioning problem.

Average cost criterion induced by the regular utility function for continuous-time Markov decision processes

In this paper we study the average cost criterion induced by the regular utility function (U-average cost criterion) for continuous-time Markov decision processes. This criterion is a generalization of the risk-sensitive average cost and expected average cost criteria. We first introduce an auxiliary risk-sensitive first passage optimization problem and obtain the properties of the corresponding optimal value function under the slight conditions. Then we show that the pair of the optimal value functions of the risk-sensitive average cost criterion and the risk-sensitive first passage criterion is a solution to the optimality equation of the risk-sensitive average cost criterion allowing the risk-sensitivity parameter to take any nonzero value. Moreover, we have that the optimal value function of the risk-sensitive average cost criterion is continuous with respect to the risk-sensitivity parameter. Finally, we give the connections between the U-average cost criterion and the average cost criteria induced by the identity function and the exponential utility function, and prove the existence of a U-average optimal deterministic stationary policy in the class of all randomized Markov policies.

Continuous-time Markov decision processes with risk-sensitive finite-horizon cost criterion

This paper studies continuous-time Markov decision processes with a denumerable state space, a Borel action space, bounded cost rates and possibly unbounded transition rates under the risk-sensitive finite-horizon cost criterion. We give the suitable optimality conditions and establish the Feynman–Kac formula, via which the existence and uniqueness of the solution to the optimality equation and the existence of an optimal deterministic Markov policy are obtained. Moreover, employing a technique of the finite approximation and the optimality equation, we present an iteration method to compute approximately the optimal value and an optimal policy, and also give the corresponding error estimations. Finally, a controlled birth and death system is used to illustrate the main results.

Constrained Continuous-Time Markov Decision Processes on the Finite Horizon

This paper studies the constrained (nonhomogeneous) continuous-time Markov decision processes on the finite horizon. The performance criterion to be optimized is the expected total reward on the finite horizon, while N constraints are imposed on similar expected costs. Introducing the appropriate notion of the occupation measures for the concerned optimal control problem, we establish the following under some suitable conditions: (a) the class of Markov policies is sufficient; (b) every extreme point of the space of performance vectors is generated by a deterministic Markov policy; and (c) there exists an optimal Markov policy, which is a mixture of no more than $$N+1$$N+1 deterministic Markov policies.

Finite-horizon optimality for continuous-time Markov decision processes with unbounded transition rates

In this paper we focus on the finite-horizon optimality for denumerable continuous-time Markov decision processes, in which the transition and reward/cost rates are allowed to be unbounded, and the optimality is over the class of all randomized history-dependent policies. Under mild reasonable conditions, we first establish the existence of a solution to the finite-horizon optimality equation by designing a technique of approximations from the bounded transition rates to unbounded ones. Then we prove the existence of ε (≥ 0)-optimal Markov policies and verify that the value function is the unique solution to the optimality equation by establishing the analog of the Itô-Dynkin formula. Finally, we provide an example in which the transition rates and the value function are all unbounded and, thus, obtain solutions to some of the unsolved problems by Yushkevich (1978).

Finite-horizon optimality for continuous-time Markov decision processes with unbounded transition rates

In this paper we focus on the finite-horizon optimality for denumerable continuous-time Markov decision processes, in which the transition and reward/cost rates are allowed to be unbounded, and the optimality is over the class of all randomized history-dependent policies. Under mild reasonable conditions, we first establish the existence of a solution to the finite-horizon optimality equation by designing a technique of approximations from the bounded transition rates to unbounded ones. Then we prove the existence of ε (≥ 0)-optimal Markov policies and verify that the value function is the unique solution to the optimality equation by establishing the analog of the Itô-Dynkin formula. Finally, we provide an example in which the transition rates and the value function are all unbounded and, thus, obtain solutions to some of the unsolved problems by Yushkevich (1978).

Ergodic control of multi-class $M/M/N+M$ queues in the Halfin–Whitt regime

We study a dynamic scheduling problem for a multi-class queueing network with a large pool of statistically identical servers. The arrival processes are Poisson, and service times and patience times are assumed to be exponentially distributed and class dependent. The optimization criterion is the expected long time average (ergodic) of a general (nonlinear) running cost function of the queue lengths. We consider this control problem in the Halfin–Whitt (QED) regime, that is, the number of servers $n$ and the total offered load $\mathbf{r}$ scale like $n\approx\mathbf{r}+\hat{\rho}\sqrt{\mathbf{r}}$ for some constant $\hat{\rho}$. This problem was proposed in [Ann. Appl. Probab. 14 (2004) 1084–1134, Section 5.2]. The optimal solution of this control problem can be approximated by that of the corresponding ergodic diffusion control problem in the limit. We introduce a broad class of ergodic control problems for controlled diffusions, which includes a large class of queueing models in the diffusion approximation, and establish a complete characterization of optimality via the study of the associated HJB equation. We also prove the asymptotic convergence of the values for the multi-class queueing control problem to the value of the associated ergodic diffusion control problem. The proof relies on an approximation method by spatial truncation for the ergodic control of diffusion processes, where the Markov policies follow a fixed priority policy outside a fixed compact set.

Partial hedging of American options in discrete time and complete markets: convex duality and optimal Markov policies

In this paper we study the partial hedging of American options in discrete time and complete markets. This problem leads to a stochastic game maximizing over stopping times and minimizing over stochastic processes satisfying a budget constraint. In the first part of the paper we introduce a dual problem maximizing over two-dimensional vectors of randomized stopping times and parametrized by Lagrange multipliers. We show that the value function of the hedging problem is conjugate to the dual problem. The solution to the dual problem for a fixed Lagrange multiplier is given as the solution of a two-dimensional coupled singular control problem under which the American option is decomposed in terms of risk and cost. In the second part of the paper we show, under a Markov assumption, the existence of an optimal deterministic Markov policy with respect to an enlarged state process.

Markov decision evolutionary game theoretic learning for cooperative sensing of unmanned aerial vehicles

As one of the major contributions of biology to competitive decision making, evolutionary game theory provides a useful tool for studying the evolution of cooperation. To achieve the optimal solution for unmanned aerial vehicles (UAVs) that are carrying out a sensing task, this paper presents a Markov decision evolutionary game (MDEG) based learning algorithm. Each individual in the algorithm follows a Markov decision strategy to maximize its payoff against the well known Tit-for-Tat strategy. Simulation results demonstrate that the MDEG theory based approach effectively improves the collective payoff of the team. The proposed algorithm can not only obtain the best action sequence but also a sub-optimal Markov policy that is independent of the game duration. Furthermore, the paper also studies the emergence of cooperation in the evolution of self-regarded UAVs. The results show that it is the adaptive ability of the MDEG based approach as well as the perfect balance between revenge and forgiveness of the Tit-for-Tat strategy that the emergence of cooperation should be attributed to.

Strong average optimality criterion for continuous-time Markov decision processes

This paper deals with continuous-time Markov decision processes with the unbounded transition rates under the strong average cost criterion. The state and action spaces are Borel spaces, and the costs are allowed to be unbounded from above and from below. Under mild conditions, we first prove that the finite-horizon optimal value function is a solution to the optimality equation for the case of uncountable state spaces and unbounded transition rates, and that there exists an optimal deterministic Markov policy. Then, using the two average optimality inequalities, we show that the set of all strong average optimal policies coincides with the set of all average optimal policies, and thus obtain the existence of strong average optimal policies. Furthermore, employing the technique of the skeleton chains of controlled continuous-time Markov chains and Chapman-Kolmogorov equation, we give a new set of sufficient conditions imposed on the primitive data of the model for the verification of the uniform exponential ergodicity of continuous-time Markov chains governed by stationary policies. Finally, we illustrate our main results with an example.

Stochastic Control of Relay Channels With Cooperative and Strategic Users

This paper studies node cooperation in a wireless network from the MAC layer perspective. A simple relay channel with a source, a relay, and a destination node is considered where the source can transmit a packet directly to the destination or transmit through the relay. The tradeoff between average energy and delay is studied by posing the problem as a stochastic dynamical optimization problem. The following two cases are considered: 1) nodes are cooperative and information is decentralized, and 2) nodes are strategic and information is centralized. With decentralized information and cooperative nodes, a structural result is proven that the optimal policy is the solution of a Bellman-type fixed-point equation over a time invariant state space. For specific cost functions reflecting transmission energy consumption and average delay, numerical results are presented showing that a policy found by solving this fixed-point equation outperforms conventionally used time-division multiple access (TDMA) and random access (RA) policies. When nodes are strategic and information is common knowledge, it is shown that cooperation can be induced by exchange of payments between the nodes, imposed by the network designer such that the socially optimal Markov policy corresponding to the centralized solution is the unique subgame perfect equilibrium of the resulting dynamic game.