Stationary Policy Research Articles

Rank-1 transition uncertainties in constrained Markov decision processes

We consider an infinite-horizon discounted constrained Markov decision process (CMDP) with uncertain transition probabilities. We assume that the uncertainty in transition probabilities has a rank-1 matrix structure and the underlying uncertain parameters belong to a polytope. We formulate the uncertain CMDP problem using a robust optimization framework. To derive reformulation of the robust CMDP problem, we restrict to the class of stationary policies and show that it is equivalent to a bilinear programming problem. We provide a simple example where a Markov policy performs better than the optimal policy in the class of stationary policies, implying that, unlike in classical CMDP problem, an optimal policy of the robust CMDP problem need not be present in the class of stationary policies. For the case of a single uncertain parameter, we propose sufficient conditions under which an optimal policy of the restricted robust CMDP problem is unaffected by uncertainty. The numerical experiments are performed on randomly generated instances of a machine replacement problem and a well-known class of problems called Garnets.

Optimal transmission strategy for multiple Markovian fading channels: Existence, structure, and approximation

This paper investigates the optimal transmission strategy for remote state estimation over multiple Markovian fading channels. A smart sensor is used to obtain a local state estimate of a system, and transmits it to a remote estimator. A new transmission strategy is proposed by co-designing the channel allocation and the transmission power control. The co-designing problem is modeled as a constrained Markov decision process (CMDP) to minimize the expected average estimation error covariance subject to the energy constraint over an infinite horizon. The CMDP is then relaxed as an unconstrained Markov decision process (UMDP) using the Lagrange multiplier method. Sufficient conditions for the existence of the optimal stationary policy for the UMDP are established to obtain the optimal transmission strategy. The structure of the optimal transmission power control policy for the UMDP with discounted cost is also elucidated. Taking account of the discrete-continuous hybrid action space, a parameterized deep Q-network (P-DQN) algorithm is employed to obtain an approximate optimal policy for the UMDP. Finally, a moving vehicle example is introduced to illustrate the effectiveness of the developed methods.

Operational policy development for dynamic restoration of lakes in a changing climate; application of innovative hedging rules in a system dynamics platform

Growing agricultural and domestic demand exacerbates water shortages globally, creating substantial environmental challenges, especially in lakes and wetlands ecosystems. This paper presents a stationary policy to restore and sustain the water level in natural lakes using a new version of hedging rule that accounts for time-dependent environmental demand and the water allocation to the agricultural and municipal sectors under different climate change projections. The proposed framework is demonstrated via operational policy derived to rehabilitate Lake Urmia in Iran. To simplify the modeling process, all reservoirs in the basin are replaced with an equivalent reservoir (ER) to allocate the available water between potential users. The ER is then operated using the set of hedging rules derived for staged restoration and sustainability of the lake level while meeting other stakeholder objectives. A system dynamics-based model is used to simulate the multi-sectoral system of the basin while using a built-in optimization algorithm to develop the most desirable multi-period operational policy. The lake-level condition is investigated by producing lake-level duration curves, while the reservoir performance indices (reliability, resiliency, and vulnerability) are used for assessing water supply in the basin. The results indicate that the proposed framework is highly effective in restoring lake level while meeting agricultural and municipal water demand in the basin. The proposed model provides a stationary policy for the lake restoration accounting for the dynamic variation of the lake level and fluctuations of the reservoirs inflow due to climate variability and change.

Markov decision processes with burstiness constraints

We consider a Markov Decision Process (MDP), over a finite or infinite horizon, augmented by so-called (σ,ρ)-burstiness constraints. Such constraints, which had been introduced within the framework of network calculus, are meant to limit some additive quantity to a given rate over any time interval, plus a term which allows for occasional and limited bursts. We introduce this class of constraints for MDP models, and formulate the corresponding constrained optimization problems. Due to the burstiness constraints, constrained optimal policies are generally history-dependent. We use a recursive form of the constraints to define an augmented-state model, for which sufficiency of Markov or stationary policies is recovered and the standard theory may be applied, albeit over a larger state space. The analysis is mainly devoted to a characterization of feasible policies, followed by application to the constrained MDP optimization problem. A simple queuing example serves to illustrate some of the concepts and calculations involved.

Lord Curzon's Tribal Policy: An Appraisal of its Aims of Securing Pacification and Contentment of Frontier Tribesmen

The British Empire, which encapsulated one-fourth of humanity, had few difficult frontiers, and the Northwest Frontier figured prominently among these. The Indian subcontinent was the most prized jewel of its coveted crown, and its preservation was considered vital given the influence and power of other European rivals, i.e., French and Russians in the region. Initially, the Inactive or Stationary Policy was pursued, and later in the 1870s, aggressive and penetrative Policy was adopted in the Tribal Belt. The aggressive Forward Policy alienated the tribespeople, which eventually resulted in the uprising of 1897. The British finally realized that only the Policy of contentment and pacification would ensure long-lasting peace and gain the loyalty of the tribe members. Lord Curzon, the expert on frontier affairs, was the main exponent of this Policy of conciliation to win the confidence of the inhabitants of the most sensitive Frontier of the empire.

Adaptive discounted control for piecewise deterministic Markov processes

The main goal of this paper is to study the adaptive infinite-horizon discounted continuous-time optimal control problem of piecewise deterministic Markov processes (PDMPs) with the control acting continuously on the jump intensity λ and on the transition measure Q of the process. It is assumed that jump parameters (λ and Q), as well the continuous and boundary costs (Cg and Ci respectively), depend on an unknown parameter β⁎. It is shown that the principle of estimation and control holds, that is, the strategy consisting of choosing, at each stage n, an action according to an optimal stationary policy, where the true but unknown parameter β⁎ is replaced by its estimated value βˆn, is asymptotically discount optimal, provided that the sequence of estimators {βˆn} of β⁎ is strongly consistent, that is, βˆn converge to β⁎ almost surely. In the framework of PDMPs, the so-called discrepancy function depends on the derivative along the flow of the value function as well as on some boundary conditions, which brings new challenges in the analysis of this problem.

Autonomous Platoon Control With Integrated Deep Reinforcement Learning and Dynamic Programming

Deep Reinforcement Learning (DRL) is regarded as a potential method for car-following control and has been mostly studied to support a single following vehicle. However, it is more challenging to learn a stable and efficient car-following policy when there are multiple following vehicles in a platoon, especially with unpredictable leading vehicle behavior. In this context, we adopt an integrated DRL and Dynamic Programming (DP) approach to learn autonomous platoon control policies, which embeds the Deep Deterministic Policy Gradient (DDPG) algorithm into a finite-horizon value iteration framework. Although the DP framework can improve the stability and performance of DDPG, it has the limitations of lower sampling and training efficiency. In this paper, we propose an algorithm, namely Finite-Horizon-DDPG with Sweeping through reduced state space using Stationary approximation (FH-DDPG-SS), which uses three key ideas to overcome the above limitations, i.e., transferring network weights backward in time, stationary policy approximation for earlier time steps, and sweeping through reduced state space. In order to verify the effectiveness of FH-DDPG-SS, simulation using real driving data is performed, where the performance of FH-DDPG-SS is compared with those of the benchmark algorithms. Finally, platoon safety and string stability for FH-DDPG-SS are demonstrated.

Word-of-Mouth on Action: Analysis of Optimal Shipment Policy When Customers are Resentful

Word-of-Mouth (WoM) communication via online reviews plays a vital role in customers’ purchasing decisions. As such, retailers must consider the impact of WoM to manage customer perceptions and future demand. This paper considers an online shopping system with premium and regular customers. Building on the behavioral and operations management literature, we model customer preferences based on the perceived service quality indicated by WoM and integrate this into the retailer’s operational problem to determine a shipment policy regarding the timing of consolidated shipments and the treatment of regular demand. First, we study the e-tailer’s problem when they have no knowledge of WoM and only react to the changes in demand. We analyze the long-term behavior of customer demand and show that potential market size and customer sensitivity are the key parameters determining this behavior. Then, we build a model to integrate the knowledge of WoM into operational decision-making and partially characterize the optimal solution. We show that (i) underpromise-and-overdeliver can be a hurtful strategy since it creates a false sense of fast delivery for the regular service, (ii) relaxations in operational constraints may hurt profitability due to the associated difficulties of managing perceptions, and (iii) seeking a stationary policy can lead to suboptimal solutions; therefore, cyclic policies should also be considered when appropriate.

Learning Optimal Stochastic Sensor Scheduling for Remote Estimation With Channel Capacity Constraint

Scheduling for multiple sensors to observe multiple systems is investigated. Only one sensor can transmit a measurement to the remote estimator over a Markovian fading channel at each time instant. A stochastic scheduling protocol is proposed, which first chooses the system to be observed via a probability distribution, and then chooses the sensor to transmit the measurement via another distribution. The stochastic sensor scheduling is modeled as a Markov decision process (MDP). A sufficient condition is derived to ensure the stability of remote estimation error covariance by a contraction mapping operator. In addition, the existence of an optimal deterministic and stationary policy is proved. To overcome the curse of dimensionality, the deep deterministic policy gradient, a recent deep reinforcement learning algorithm, is utilized to obtain an optimal policy for the MDP. Finally, a practical example is given to demonstrate that the developed scheduling algorithm significantly outperforms other policies.

Energy scheduling for DoS attack over multi-hop networks: Deep reinforcement learning approach

This paper studies the energy scheduling for Denial-of-Service (DoS) attack against remote state estimation over multi-hop networks. A smart sensor observes a dynamic system, and transmits its local state estimate to a remote estimator. Due to the limited communication range of the sensor, some relay nodes are employed to deliver data packets from the sensor to the remote estimator, which constitutes a multi-hop network. To maximize the estimation error covariance with energy constraint, a DoS attacker needs to determine the energy level implemented on each channel. This problem is formulated as an associated Markov decision process (MDP), and the existence of an optimal deterministic and stationary policy (DSP) is proved for the attacker. Besides, a simple threshold structure of the optimal policy is obtained, which significantly reduces the computational complexity. Furthermore, an up-to-date deep reinforcement learning (DRL) algorithm, dueling double Q-network (D3QN), is introduced to approximate the optimal policy. Finally, a simulation example illustrates the developed results and verifies the effectiveness of D3QN for optimal DoS attack energy scheduling.

A note on the existence of optimal stationary policies for average Markov decision processes with countable states

In many practical stochastic dynamic optimization problems with countable states, the optimal policy possesses certain structural properties. For example, the (s,S) policy in inventory control, the well-known cμ-rule and the recently discovered c/μ-rule (Xia et al. (2022)) in scheduling of queues. A presumption of such results is that an optimal stationary policy exists. There are many research works regarding to the existence of optimal stationary policies of Markov decision processes with countable state spaces (see, e.g., Bertsekas (2012); Hernández-Lerma and Lasserre (1996); Puterman (1994); Sennott (1999)). However, these conditions are usually not easy to verify in such optimization problems. In this paper, we study the optimization of long-run average of continuous-time Markov decision processes with countable state spaces. We provide an intuitive approach to prove the existence of an optimal stationary policy. The approach is simply based on compactness of the policy space, with a special designed metric, and the continuity of the long-run average in the space. Our method is capable to handle cost functions unbounded from both above and below, which makes a complementary contribution to the literature work where the cost function is unbounded from only one side. Examples are provided to illustrate the application of our main results.

Risk-Sensitive Average Optimality for Discrete-Time Markov Decision Processes

In this paper we study the risk-sensitive average optimality for discrete-time Markov decision processes with denumerable states and unbounded costs. We derive the multiplicative Poisson equation under the suitable ergodicity conditions via an approximation method. Moreover, we prove the existence of a unique solution to the risk-sensitive average cost optimality equation and give an equivalent characterization of the set of all optimal stationary policies. Finally, we present the policy iteration algorithm and show its convergence.

An axiomatic approach to Markov decision processes

This paper presents an axiomatic approach to finite Markov decision processes where the discount rate is zero. One of the principal difficulties in the no discounting case is that, even if attention is restricted to stationary policies, a strong overtaking optimal policy need not exists. We provide preference foundations for two criteria that do admit optimal policies: 0-discount optimality and average overtaking optimality. As a corollary of our results, we obtain conditions on a decision maker’s preferences which ensure that an optimal policy exists. These results have implications for disciplines where dynamic programming problems arise, including automatic control, dynamic games, and economic development.

Contextual Bandits with Cross-Learning

In the classic contextual bandits problem, in each round t, a learner observes some context c, chooses some action i to perform, and receives some reward [Formula: see text]. We consider the variant of this problem in which in addition to receiving the reward [Formula: see text], the learner also learns the values of [Formula: see text] for some other contexts [Formula: see text] in set [Formula: see text], that is, the rewards that would be achieved by performing that action under different contexts [Formula: see text]. This variant arises in several strategic settings, such as learning how to bid in nontruthful repeated auctions, which has gained a lot of attention lately as many platforms have switched to running first price auctions. We call this problem the contextual bandits problem with cross-learning. The best algorithms for the classic contextual bandits problem achieve [Formula: see text] regret against all stationary policies, in which C is the number of contexts, K the number of actions, and T the number of rounds. We design and analyze new algorithms for the contextual bandits problem with cross-learning and show that their regret has better dependence on the number of contexts. Under complete cross-learning in which the rewards for all contexts are learned when choosing an action, that is, set [Formula: see text] contains all contexts, we show that our algorithms achieve regret [Formula: see text], removing the dependence on C. For any other cases, that is, under partial cross-learning in which [Formula: see text] for some context–action pair of (i, c), the regret bounds depend on how the sets [Formula: see text] impact the degree to which cross-learning between contexts is possible. We simulate our algorithms on real auction data from an ad exchange running first price auctions and show that they outperform traditional contextual bandit algorithms. Supplemental Material: The online appendix is available at https://doi.org/10.1287/moor.2022.1313 .

Sensor scheduling design for complex networks under a distributed state estimation framework

This paper investigates sensor scheduling for state estimation of complex networks over shared transmission channels. For a complex network of dynamical systems, referred to as nodes, a sensor network is adopted to measure and estimate the system states in a distributed way, where a sensor is used to measure a node. The estimates are transmitted from sensors to the associated nodes, in the presence of one-step time delay and subject to packet loss. Due to limited transmission capability, only a portion of sensors are allowed to send information at each time step. The goal of this paper is to seek an optimal sensor scheduling policy minimizing the overall estimation errors. Under a distributed state estimation framework, this problem is reformulated as a Markov decision process, where the one-stage reward for each node is strongly coupled. The feasibility of the problem reformulation is ensured. In addition, an easy-to-check condition is established to guarantee the existence of an optimal deterministic and stationary policy. Moreover, it is found that the optimal policies have a threshold, which can be used to reduce the computational complexity in obtaining these policies. Finally, the effectiveness of the theoretical results is illustrated by several simulation examples.

Convex analytic method revisited: Further optimality results and performance of deterministic policies in average cost stochastic control

The convex analytic method has proved to be a very versatile method for the study of infinite horizon average cost optimal stochastic control problems. In this paper, we revisit the convex analytic method and make three primary contributions: (i) We present an existence result for controlled Markov models that lack weak continuity of the transition kernel but are strongly continuous in the action variable for every fixed state variable. (ii) For average cost stochastic control problems in standard Borel spaces, while existing results establish the optimality of stationary (possibly randomized) policies, few results are available on the optimality of deterministic policies. We review existing results and present further conditions under which an average cost optimal stochastic control problem admits optimal solutions that are deterministic stationary. (iii) We establish conditions under which the performance under stationary deterministic (and also quantized) policies is dense in the set of performance values under randomized stationary policies.

Theoretical Guarantees of Fictitious Discount Algorithms for Episodic Reinforcement Learning and Global Convergence of Policy Gradient Methods

When designing algorithms for finite-time-horizon episodic reinforcement learning problems, a common approach is to introduce a fictitious discount factor and use stationary policies for approximations. Empirically, it has been shown that the fictitious discount factor helps reduce variance, and stationary policies serve to save the per-iteration computational cost. Theoretically, however, there is no existing work on convergence analysis for algorithms with this fictitious discount recipe. This paper takes the first step towards analyzing these algorithms. It focuses on two vanilla policy gradient (VPG) variants: the first being a widely used variant with discounted advantage estimations (DAE), the second with an additional fictitious discount factor in the score functions of the policy gradient estimators. Non-asymptotic convergence guarantees are established for both algorithms, and the additional discount factor is shown to reduce the bias introduced in DAE and thus improve the algorithm convergence asymptotically. A key ingredient of our analysis is to connect three settings of Markov decision processes (MDPs): the finite-time-horizon, the average reward and the discounted settings. To our best knowledge, this is the first theoretical guarantee on fictitious discount algorithms for the episodic reinforcement learning of finite-time-horizon MDPs, which also leads to the (first) global convergence of policy gradient methods for finite-time-horizon episodic reinforcement learning.

Regime switching optimal growth model with risk sensitive preferences

We consider a risk-sensitive optimization of consumption-utility on an infinite time horizon where the one-period investment gain depends on an underlying economic state whose evolution over time is assumed to be described by a discrete-time, finite-state, Markov chain. We suppose that the production function also depends on a sequence of independent and identically distributed (i.i.d.) random shocks. For the sake of generality, the utility and the production functions are allowed to be unbounded from above. Under the Markov regime-switching model, it is shown that the value function of optimization problem satisfies an optimality equation and that the optimality equation has a unique solution in a particular class of functions. Furthermore, we show that an optimal policy exists in the class of stationary policies. We also derive the Euler equation of optimal consumption. Furthermore, the existence of a joint stationary distribution of the optimal growth process and the underlying regime process is examined. Finally, we present a numerical solution by considering a power utility and some hypothetical values of parameters in a regime switching extension of the Cobb–Douglas production rate function.

Sleep, Sense or Transmit: Energy-Age Tradeoff for Status Update With Two-Threshold Optimal Policy

Age-of-Information (AoI), or simply age, which measures the data freshness, is essential for real-time Internet-of-Things (IoT) applications. On the other hand, energy saving is urgently required by many energy-constrained IoT devices. This paper studies the energy-age tradeoff for status update from a sensor to a monitor over an error-prone channel. The sensor can sleep, sense and transmit a new update, or retransmit by considering both sensing energy and transmit energy. An infinite-horizon average cost problem is formulated as a Markov decision process (MDP) with the objective of minimizing the weighted sum of average AoI and average energy consumption. By solving the associated discounted cost problem and analyzing the Markov chain under the optimal policy, we prove that there exists a threshold optimal stationary policy with only two thresholds, i.e., one threshold on the AoI at the transmitter (AoIT) and the other on the AoI at the receiver (AoIR). Moreover, the two thresholds can be efficiently found by a line search. Numerical results show the performance of the optimal policies and the tradeoff curves with different parameters. Comparisons with the conventional policies show that considering sensing energy is of significant impact on the policy design, and introducing sleep mode greatly expands the tradeoff range.

A framework for the analysis of self-confirming policies

This paper provides a general framework for analyzing self-confirming policies. We study self-confirming equilibria in recurrent decision problems with incomplete information about the true stochastic model. We characterize stationary monetary policies in a linear-quadratic setting.