Space Of Occupation Measures Research Articles

Further remarks on absorbing Markov decision processes

In this note, based on the recent remarkable results of Dufour and Prieto-Rumeau, we deduce that for an absorbing Markov decision process with a given initial state, under a standard compactness-continuity condition, the space of occupation measures has the same convergent sequences, when it is endowed with the weak topology and with the weak-strong topology. We provided two examples demonstrating that imposed condition cannot be replaced with its popular alternative, and the above assertion does not hold for the space of marginals of occupation measures on the state space. Moreover, the examples also clarify some results in the previous literature.

Constrained Markov Decision Processes with Non-constant Discount Factor

This paper studies constrained Markov decision processes under the total expected discounted cost optimality criterion, with a state-action dependent discount factor that may take any value between zero and one. Both the state and the action space are assumed to be Borel spaces. By using the linear programming approach, consisting in stating the control problem as a linear problem on a set of occupation measures, we show the existence of an optimal stationary Markov policy. Our results are based on the study of both weak-strong topologies in the space of occupation measures and Young measures in the space of Markov policies.

Some advances on constrained Markov decision processes in Borel spaces with random state-dependent discount factors

ABSTRACT This paper addresses a class of discrete-time Markov decision processes in Borel spaces with a finite number of cost constraints. The constrained control model considers costs of discounted type with state-dependent discount factors which are subject to external disturbances. Our objective is to prove the existence of optimal control policies and characterize them according to certain optimality criteria. Specifically, by rewriting appropriately our original constrained problem as a new one on a space of occupation measures, we apply the direct method to show solvability. Next, the problem is defined as a convex program, and we prove that the existence of a saddle point of the associated Lagrangian operator is equivalent to the existence of an optimal control policy for the constrained problem. Finally, we turn our attention to multi-objective optimization problems, where the existence of Pareto optimal policies can be obtained from the existence of saddle-points of the aforementioned Lagrangian or equivalently from the existence of optimal control policies of constrained problems.

Spectral Inequalities for Nonnegative Tensors and Their Tropical Analogues

We extend some characterizations and inequalities for the eigenvalues of nonnegative matrices, such as Donsker–Varadhan, Friedland–Karlin, Karlin–Ost inequalities, to nonnegative tensors. Our approach involves a correspondence between nonnegative tensors, ergodic control and entropy maximization: we show in particular that the logarithm of the spectral radius of a tensor is given by en entropy maximization problem over a space of occupation measures. We study in particular the tropical analogue of the spectral radius, that we characterize as a limit of the classical spectral radius, and we give an explicit combinatorial formula for this tropical spectral radius.

Convex analytic approach to constrained discounted Markov decision processes with non-constant discount factors

In this paper we develop the convex analytic approach to a discounted discrete-time Markov decision process (DTMDP) in Borel state and action spaces with N constraints. Unlike the classic discounted models, we allow a non-constant discount factor. After defining and characterizing the corresponding occupation measures, the original constrained DTMDP is written as a convex program in the space of occupation measures, whose compactness and convexity we show. In particular, we prove that every extreme point of the space of occupation measures can be generated by a deterministic stationary policy for the DTMDP. For the resulting convex program, we prove that it admits a solution that can be expressed as a convex combination of N+1 extreme points of the space of occupation measures. One of its consequences is the existence of a randomized stationary optimal policy for the original constrained DTMDP.

A separation principle for partially observed control of singular stochastic processes

The analysis of partially observed stochastic control problems often replaces the unknown state process with its conditional distribution given the observations. This technique rewrites the dynamics in terms of knowable processes whose costs coincide with the original processes. This paper considers stochastic processes having singular behavior and presents an approach which separates the determination of the optimal control from the task of estimating the conditional distribution of the unknown process. It involves using a martingale problem characterization for the dynamics followed by a further characterization using occupation measures. This final characterization forms the basis for an equivalent linear programming formulation of the problem over the space of occupation measures.