Class Of Markov Decision Processes Research Articles

A Maximum Divergence Approach to Optimal Policy in Deep Reinforcement Learning.

Model-free reinforcement learning algorithms based on entropy regularized have achieved good performance in control tasks. Those algorithms consider using the entropy-regularized term for the policy to learn a stochastic policy. This work provides a new perspective that aims to explicitly learn a representation of intrinsic information in state transition to obtain a multimodal stochastic policy, for dealing with the tradeoff between exploration and exploitation. We study a class of Markov decision processes (MDPs) with divergence maximization, called divergence MDPs. The goal of the divergence MDPs is to find an optimal stochastic policy that maximizes the sum of both the expected discounted total rewards and a divergence term, where the divergence function learns the implicit information of state transition. Thus, it can provide better-off stochastic policies to improve both in robustness and performance in a high-dimension continuous setting. Under this framework, the optimality equations can be obtained, and then a divergence actor-critic algorithm is developed based on the divergence policy iteration method to address large-scale continuous problems. The experimental results, compared to other methods, show that our approach achieved better performance and robustness in the complex environment particularly. The code of DivAC can be found in https://github.com/yzyvl/DivAC.

A survey of average cost problems in deterministic discrete-time control systems

This paper concerns optimal control problems for infinite-horizon discrete-time deterministic systems with the long-run average cost (AC) criterion. This optimality criterion can be traced back to a paper by Bellman [6] for a class of Markov decision processes (MDPs). We present a survey of some of the main approaches to study the AC problem, namely, the AC optimality (or dynamic programming) equation, the steady state approach, and the vanishing discount approach, emphasizing the difference between the deterministic control problem and the corresponding (stochastic) MDP. Several examples illustrate these approaches and related results. We also state some open problems.

Transmission schedule for jointly optimizing remote state estimation and wireless sensor network lifetime

This paper studies scheduling multiple sensors for jointly optimizing remote state estimation and wireless sensor network lifetime. Each sensor in the network sends its packet to the remote center for further estimation over wireless communication channels. Sensors in the network will inevitably run out of battery power due to transmission consumption and then reach the end of sensor network lifetime. This article focuses on the problem how to schedule multiple sensors minimizing the total remote estimation errors and simultaneously prolonging the sensor network lifetime. The problem is formulated as a special class of Markov decision process (MDP), stochastic shortest path (SSP) problem. Under this framework, we first derive a set of conditions to guarantee the optimality of structural policies in SSP. Subsequently, threshold structure of the optimal scheduling scheme is obtained by verifying the conditions. Moreover, two Value-of-Information-based multi-sensor scheduling algorithms and a linear-architecture-based learning algorithm are designed to tackle the curse of dimensionality and unknown channel statistics in large-scale sensor networks. Numerical simulations are given to validate effectiveness of our results.

Markov Decision Processes with Incomplete Information and Semiuniform Feller Transition Probabilities

This paper deals with control of partially observable discrete-time stochastic systems. It introduces and studies the class of Markov Decision Processes with Incomplete information and with semi-uniform Feller transition probabilities. The important feature of this class of models is that the classic reduction of such a model with incomplete observation to the completely observable Markov Decision Process with belief states preserves semi-uniform Feller continuity of transition probabilities. Under mild assumptions on cost functions, optimal policies exist, optimality equations hold, and value iterations converge to optimal values for this class of models. In particular, for Partially Observable Markov Decision Processes the results of this paper imply new and generalize several known sufficient conditions on transition and observation probabilities for the existence of optimal policies, validity of optimality equations, and convergence of value iterations.

Exploiting Partial Observability for Optimal Deception

Deception is a useful tool in situations where an agent operates in the presence of its adversaries. We consider a setting where a supervisor provides a reference policy to an agent, expects the agent to operate in an environment by following the reference policy, and partially observes the agent's behavior. The agent instead follows a different, deceptive policy to achieve a different task. We model the environment with a Markov decision process and study the synthesis of optimal deceptive policies under partial observability. We formalize the notion of deception as a hypothesis testing problem and show that the synthesis of optimal deceptive policies is NP-hard. As an approximation, we consider the class of mixture policies, which provides a convex optimization formulation of the deception problem. We give an algorithm that converges to the optimal mixture policy. We also consider a special class of Markov decision processes where the transition and observation functions are deterministic. For this case, we give a randomized algorithm for path planning that generates a path for the agent in polynomial time and achieves the optimal value for the considered objective function.

On Linear Programming for Constrained and Unconstrained Average-Cost Markov Decision Processes with Countable Action Spaces and Strictly Unbounded Costs

We consider the linear programming approach for constrained and unconstrained Markov decision processes (MDPs) under the long-run average-cost criterion, where the class of MDPs in our study have Borel state spaces and discrete countable action spaces. Under a strict unboundedness condition on the one-stage costs and a recently introduced majorization condition on the state transition stochastic kernel, we study infinite-dimensional linear programs for the average-cost MDPs and prove the absence of a duality gap and other optimality results. Our results do not require a lower-semicontinuous MDP model. Thus, they can be applied to countable action space MDPs where the dynamics and one-stage costs are discontinuous in the state variable. Our proofs make use of the continuity property of Borel measurable functions asserted by Lusin’s theorem.

L^*-based learning of Markov decision processes (extended version)

Automata learning techniques automatically generate systemmodels fromtest observations. Typically, these techniques fall into two categories: passive and active. On the one hand, passive learning assumes no interaction with the system under learning and uses a predetermined training set, e.g., system logs. On the other hand, active learning techniques collect training data by actively querying the system under learning, allowing one to steer the discovery ofmeaningful information about the systemunder learning leading to effective learning strategies. A notable example of active learning technique for regular languages is Angluin’s L^*-algorithm. The L^*-algorithm describes the strategy of a student who learns the minimal deterministic finite automaton of an unknown regular language L by asking a succinct number of queries to a teacher who knows L.In this work, we study L^* -based learning of deterministic Markov decision processes, a class of Markov decision processes where an observation following an action uniquely determines a successor state. For this purpose, we first assume an ideal setting with a teacher who provides perfect information to the student. Then, we relax this assumption and present a novel learning algorithm that collects information by sampling execution traces of the system via testing.Experiments performed on an implementation of our sampling-based algorithm suggest that our method achieves better accuracy than state-of-the-art passive learning techniques using the same amount of test obser vations. In contrast to existing learning algorithms which assume a predefined number of states, our algorithm learns the complete model structure including the state space.

Markov decision processes with quasi-hyperbolic discounting

We study Markov decision processes with Borel state spaces under quasi-hyperbolic discounting. This type of discounting nicely models human behaviour, which is time-inconsistent in the long run. The decision maker has preferences changing in time. Therefore, the standard approach based on the Bellman optimality principle fails. Within a dynamic game-theoretic framework, we prove the existence of randomised stationary Markov perfect equilibria for a large class of Markov decision processes with transitions having a density function. We also show that randomisation can be restricted to two actions in every state of the process. Moreover, we prove that under some conditions, this equilibrium can be replaced by a deterministic one. For models with countable state spaces, we establish the existence of deterministic Markov perfect equilibria. Many examples are given to illustrate our results, including a portfolio selection model with quasi-hyperbolic discounting.

On the convergence of projective-simulation\u2013based reinforcement learning in Markov decision processes

In recent years, the interest in leveraging quantum effects for enhancing machine learning tasks has significantly increased. Many algorithms speeding up supervised and unsupervised learning were established. The first framework in which ways to exploit quantum resources specifically for the broader context of reinforcement learning were found is projective simulation. Projective simulation presents an agent-based reinforcement learning approach designed in a manner which may support quantum walk-based speedups. Although classical variants of projective simulation have been benchmarked against common reinforcement learning algorithms, very few formal theoretical analyses have been provided for its performance in standard learning scenarios. In this paper, we provide a detailed formal discussion of the properties of this model. Specifically, we prove that one version of the projective simulation model, understood as a reinforcement learning approach, converges to optimal behavior in a large class of Markov decision processes. This proof shows that a physically inspired approach to reinforcement learning can guarantee to converge.

Recursive stochastic games with positive rewards

We study the complexity of a class of Markov decision processes and, more generally, stochastic games, called 1-exit Recursive Markov Decision Processes (1-RMDPs) and 1-exit Recursive Simple Stochastic Games (1-RSSGs), with strictly positive rewards. These are a class of finitely presented countable-state zero-sum turn-based stochastic games that subsume standard finite-state MDPs and Condon's simple stochastic games. They correspond to optimization and game versions of several classic stochastic models, with rewards. In particular, they correspond to the MDP and game versions of multi-type branching processes and stochastic context-free grammars with strictly positive rewards. The goal of the two players in the game is to maximize/minimize the total expected reward generated by a play of the game. Such stochastic models arise naturally as models of probabilistic procedural programs with recursion, and the problems we address are motivated by the goal of analyzing the optimal/pessimal expected running time in such a setting.We first show that in such games both players have optimal deterministic “stackless and memoryless” optimal strategies. We then provide polynomial-time algorithms for computing the exact optimal expected reward (which may be infinite, but is otherwise rational), and optimal strategies, for both the maximizing and minimizing single-player versions of the game, i.e., for (1-exit) Recursive Markov Decision Processes (1-RMDPs). It follows that the quantitative decision problem for positive reward 1-RSSGs is in NP ∩ coNP. We show that Condon's well-known quantitative termination problem for finite-state simple stochastic games (SSGs) which she showed to be in NP ∩ coNP reduces to a special case of the reward problem for 1-RSSGs, namely, deciding whether the value is ∞. By contrast, for finite-state SSGs with strictly positive rewards, deciding if this expected reward value is ∞ is solvable in P-time. We also show that there is a simultaneous strategy improvement algorithm that converges in a finite number of steps to the value and optimal strategies of a 1-RSSG with positive rewards.

The Linear Programming Approach to Reach-Avoid Problems for Markov Decision Processes

One of the most fundamental problems in Markov decision processes is analysis and control synthesis for safety and reachability specifications. We consider the stochastic reach-avoid problem, in which the objective is to synthesize a control policy to maximize the probability of reaching a target set at a given time, while staying in a safe set at all prior times. We characterize the solution to this problem through an infinite dimensional linear program. We then develop a tractable approximation to the infinite dimensional linear program through finite dimensional approximations of the decision space and constraints. For a large class of Markov decision processes modeled by Gaussian mixtures kernels we show that through a proper selection of the finite dimensional space, one can further reduce the computational complexity of the resulting linear program. We validate the proposed method and analyze its potential with numerical case studies.

Effects of System Parameters on the Optimal Cost and Policy in a Class of Multidimensional Queueing Control Problems

We consider a class of Markov Decision Processes frequently employed to model queueing and inventory control problems. For these problems, we explore how changes in different system input parameters (transition rates, costs, discount rates etc.) affect the optimal cost and the optimal policy when the state space of the problem is multidimensional. To address a large class of problems, we introduce two generic dynamic programming operators to model different types of controlled events. For these operators, we derive sufficient conditions to propagate monotonicity and supermodularity properties of the value function. These properties allow to predict how changes in system input parameters affect the optimal cost and policy. Finally, we explore the case when several parameters are changed at the same time. The online appendix is available at https://doi.org/10.1287/opre.2017.1600 .

Near optimality of quantized policies in stochastic control under weak continuity conditions

This paper studies the approximation of optimal control policies by quantized (discretized) policies for a very general class of Markov decision processes (MDPs). The problem is motivated by applications in networked control systems, computational methods for MDPs, and learning algorithms for MDPs. We consider the finite-action approximation of stationary policies for a discrete-time Markov decision process with discounted and average costs under a weak continuity assumption on the transition probability, which is a significant relaxation of conditions required in earlier literature. The discretization is constructive, and quantized policies are shown to approximate optimal deterministic stationary policies with arbitrary precision. The results are applied to the fully observed reduction of a partially observed Markov decision process, where weak continuity is a much more reasonable assumption than more stringent conditions such as strong continuity or continuity in total variation.

LP-Based Approaches to Stationary-Constrained Markov Decision Problems

We study a class of Markov Decision Processes (MDPs) under stationary constraints (particularly but not exclusively, chance constraints). Our model focuses on problems where the state space is finite but the control at each state may take real values. It is straightforward to formulate the problem as a constrained nonlinear optimization problem, but solving it requires either projection, penalty or gradient-based methods that may show slow convergence, particularly because this type of problems may not be strictly convex.In this paper we extend results that are known for the discrete control model and show that the solution of a binary randomized control problem is optimal under the assumption that the dependency on the control variable is linear. If the dependency is piecewise linear with several breakpoints, then the randomized problem is not equivalent to the original problem; however we show that a particular solution always exists that is also the solution to the original continuous problem. This special solution takes the form of a two-action control policy for each state. Solving each two-action randomized problem can be done in polynomial time by linear programming (LP). This leads to an efficient solution when the number of states is small, but for a large state space the complexity can be prohibitive. In a restricted yet interesting special case of piecewise linear, multiple actions setting where the actions are linearly ordered, we show that the problem can again be reduced to a single LP.To overcome the computational complexity in the general case, we propose an alternative approach that involves solving a sequence of linear programs modeling the stationary measure and the optimal solution concurrently. We conclude the paper by discussing future extensions of this framework to application settings where the state space is continuous as well.

Markov Decision Processes with Threshold Based Piecewise Linear Optimal Policies

This letter investigates the structure of the optimal policy for a class of Markov decision processes (MDPs), having convex and piecewise linear cost function. The optimal policy is proved to have a piecewise linear structure that alternates flat and constant-slope pieces, resembling a staircase with tilted rises and as many steps (thresholds) as the breakpoints of the cost function. This particular structure makes it possible to express the policy in a very compact manner, particularly suitable to be stored in low-end devices. More importantly, the threshold-based form of the optimal policy can be exploited to reduce the computational complexity of the iterative dynamic programming (DP) algorithm used to solve the problem. These results apply to a rather wide set of optimization problems, typically involving the management of a resource buffer such as the energy stored in a battery, or the packets queued in a wireless node.

Monotone optimal control for a class of Markov decision processes

This paper provides a unified framework to study monotone optimal control for a class of Markov decision processes through D-multimodularity. We demonstrate that each system in this class can be classified as either a substitution-type or a complement-type system according to the possible transition set, which can be used as a classification mechanism that integrates a variety of models in the literature. We develop a generic proof of the structural properties of both types of system. In particular, we show that D-multimodularity is a generally sufficient condition for monotone optimal control of different types of system in this class. With this unified theory, there is no need to pursue each problem ad hoc and the structural properties of this class of MDPs follow with ease.

Minimum Distance Estimation for a Class of Markov Decision Processes

We develop a two-step estimator for a class of Markov decision processes with continuous control that is intuitive and simple to implement. Making use of the monotonicity assumption we estimate the expected continuation value functions nonparametrically in the first stage. In the second stage our estimator minimizes a minimum distance criterion that measures the divergence between the nonparametric conditional distribution function and a model implied simulated semiparametric counterpart. We show that our minimum distance estimator is asymptotically normal and converges at the parametric rate under some regularity conditions. We estimate the expected value function by kernel smoothing and derive its pointwise distribution theory. We illustrate how our estimation methodology forms a basis for the estimation of dynamic models with different class of control variable(s) as well as a class of Markovian games.

Partially Observed Stochastic Shortest Path Problems With Approximate Solution by Neurodynamic Programming

We analyze a class of Markov decision processes with imperfect state information that evolve on an infinite time horizon and have a total cost criterion. In particular, we are interested in problems with stochastic shortest path structure, assuming the following: 1) the existence of a policy that guarantees termination with probability one and 2) the property that any policy that fails to guarantee termination has infinite expected cost from some initial state. We also assume that termination is perfectly recognized. In this paper, we clarify and expand upon arguments (given in an earlier paper) for establishing the existence, uniqueness, and characterization of stationary optimal policies, and the convergence of value and policy iteration. We also present an illustrative example, involving the search for a partially observed target that moves randomly on a grid, and we develop a simulation-based algorithm (based on neurodynamic programming techniques) for computing policies that approximately minimize the expected number of stages to complete the search.

Uniqueness and Stability of Optimal Policies of Finite State Markov Decision Processes

In this paper we consider infinite horizon discrete-time optimal control of Markov decision processes (MDPs) with finite state spaces and compact action sets. We restrict attention to unicost MDPs, which form a class that contains all the weakly communicating MDPs. The unicost MDPs are characterized as those MDPs for which there exists a solution to the single optimality equation. We address the problem of uniqueness and stability of minimizing Markov actions. Our main result asserts that when we endow the set of unicost MDPs with a certain natural metric, under which it is complete, then the class of MDPs with essentially unique and stable minimizing Markov actions contains the intersection of countably many open dense sets (hence is itself dense). Thus, the property of having essentially unique and stable minimizing Markov actions is generic for unicost MDPs.

Solution to the risk-sensitive average cost optimality equation in a class of Markov decision processes with finite state space

This work concerns discrete-time Markov decision processes with finite state space and bounded costs per stage. The decision maker ranks random costs via the expectation of the utility function associated to a constant risk sensitivity coefficient, and the performance of a control policy is measured by the corresponding (long-run) risk-sensitive average cost criterion. The main structural restriction on the system is the following communication assumption: For every pair of states x and y, there exists a policy π, possibly depending on x and y, such that when the system evolves under π starting at x, the probability of reaching y is positive. Within this framework, the paper establishes the existence of solutions to the optimality equation whenever the constant risk sensitivity coefficient does not exceed certain positive value.