Borel Spaces Research Articles

On Average Optimality for Non-Stationary Markov Decision Processes in Borel Spaces

This paper studies Borel models of nonstationary Markov decision processes (MDPs) with average criteria. We establish a suitable fixed point theorem, which is used to show the existence of solutions to the average optimality equation (AOE), without using contractions. The existence of optimal policies follows from the obtained solutions to the AOE. Furthermore, we show that versions of the rolling horizon algorithm can be used to produce an optimal policy or an ϵ-optimal policy. Finally, we compare the optimality conditions imposed in this paper with the existing ones in the literature, and demonstrate that they can be satisfied while the previous ones are not. Funding: This research was supported by National Key Research and Development Program of China [2022YFA1004600], National Natural Science Foundation of China [No. 12301170], Guangdong Basic and Applied Research Foundation [2023A1515012644], and the Fundamental Research Funds for the Central Universities, Sun Yat-Sen University [No. 23qnpy39].

Orbit pseudometrics and a universality property of the Gromov-Hausdorff distance

We consider the notion of Borel reducibility between pseudometrics on standard Borel spaces introduced and studied recently by Cúth, Doucha and Kurka, as well as the notion of an orbit pseudometric, a continuous version of the notion of an orbit equivalence relation. It is well known that the relation of isometry of Polish metric spaces is bireducible with a universal orbit equivalence relation. We prove a version of this result for pseudometrics, showing that the Gromov-Hausdorff distance of Polish metric spaces is bireducible with a universal element in a certain class of orbit pseudometrics.

Stationary Almost Markov ε-Equilibria for Discounted Stochastic Games with Borel Spaces and Unbounded Payoffs

Stationary Almost Markov ε-Equilibria for Discounted Stochastic Games with Borel Spaces and Unbounded Payoffs

Constrained Markov Decision Processes with Non-constant Discount Factor

This paper studies constrained Markov decision processes under the total expected discounted cost optimality criterion, with a state-action dependent discount factor that may take any value between zero and one. Both the state and the action space are assumed to be Borel spaces. By using the linear programming approach, consisting in stating the control problem as a linear problem on a set of occupation measures, we show the existence of an optimal stationary Markov policy. Our results are based on the study of both weak-strong topologies in the space of occupation measures and Young measures in the space of Markov policies.

Risk-sensitive discounted Markov decision processes with unbounded reward functions and Borel spaces

This paper attempts to study the risk-sensitive discounted discrete-time Markov decision processes in Borel spaces, in which the reward functions are allowed to be unbounded from above and from below. We find mild conditions imposed on the primitive data of the decision processes, which not only ensure the existence of a solution to the optimality equation (OE in short), but also are the generalization of the bounded reward case. Furthermore, using the OE and a novel technique, we prove the existence of an optimal policy out of the class of randomized history-dependent policies. Finally, we illustrate our results with an inventory system.

Extension of monotone operators and Lipschitz maps invariant for a group of isometries

Abstract We study monotone operators in reflexive Banach spaces that are invariant with respect to a group of suitable isometric isomorphisms, and we show that they always admit a maximal extension which preserves the same invariance. A similar result applies to Lipschitz maps in Hilbert spaces, thus providing an invariant version of Kirszbraun–Valentine extension theorem. We then provide a relevant application to the case of monotone operators in $L^{p}$ -spaces of random variables which are invariant with respect to measure-preserving isomorphisms, proving that they always admit maximal dissipative extensions which are still invariant by measure-preserving isomorphisms. We also show that such operators are law invariant, a much stronger property which is also inherited by their resolvents, the Moreau–Yosida approximations, and the associated semigroup of contractions. These results combine explicit representation formulae for the maximal extension of a monotone operator based on self-dual Lagrangians and a refined study of measure-preserving maps in standard Borel spaces endowed with a nonatomic measure, with applications to the approximation of arbitrary couplings between measures by sequences of maps.

A categorical characterization of relative entropy on standard Borel spaces

We give a categorical treatment, in the spirit of Baez and Fritz, of relative entropy for probability distributions defined on standard Borel spaces. We define a category suitable for reasoning about statistical inference on standard Borel spaces. We define relative entropy as a functor into Lawvere's category and we show convexity, lower semicontinuity and uniqueness.

Markov decision processes approximation with coupled dynamics via Markov deterministic control systems

Abstract This article presents an approximation of discrete Markov decision processes with small noise on Borel spaces with an infinite horizon and an expected total discounted cost by the corresponding deterministic Markov process. In both cases, the dynamics evolve through a system consisting of two coupled difference equations. It is assumed that the difference equations of the system are perturbed by a small noise. Under our assumptions, a bound for the stability index is given, and the optimal cost convergence rate is estimated using a small perturbation parameter. Moreover, the convergence of the optimal policy on compact subsets is verified. Finally, two examples are presented to illustrate the developed theory.

Generalized Polish spaces at regular uncountable cardinals

AbstractIn the context of generalized descriptive set theory, we systematically compare and analyze various notions of Polish‐like spaces and standard ‐Borel spaces for an uncountable (regular) cardinal satisfying . As a result, we obtain a solid framework where one can develop the theory in full generality. We also provide natural characterizations of the generalized Cantor and Baire spaces. Some of the results obtained considerably extend previous work from Coskey and Schlicht (Fund. Math. 232 (2016) 227–248), Galeotti (Ph.D. thesis, Universität Hamburg, 2019), and Lücke and Schlicht (Israel J. Math. 209 (2015) 421–461), and answer some questions contained therein.

Zero-sum games involving teams against teams: Existence of equilibria, and comparison and regularity in information

Many emerging problems involve teams of agents taking part in a game. Such problems require a stochastic analysis with regard to the correlation structures among the agents belonging to a given team. In the context of standard Borel state, measurement and action spaces (thus, including real vector spaces and finite spaces), this paper makes the following contributions for two teams of finitely many agents taking part in a zero-sum game: (i) An existence result will be presented for saddle-point equilibria in zero-sum games involving teams against teams when common randomness is assumed to be available in each team with an analysis on conditions for compactness of strategic team measures to be presented. (ii) Blackwell’s ordering of information structures is generalized to n-player teams with standard Borel spaces, where correlated garbling of information structures is introduced as a key attribute; (iii) building on this result Blackwell’s ordering of information structures is established for team-against-team zero-sum game problems. (iv) Finally, continuity of the equilibrium value of team-against-team zero-sum game problems in the space of information structures under total variation is established.

A UNIVERSAL CHARACTERIZATION OF STANDARD BOREL SPACES

AbstractWe prove that the category $\mathsf {SBor}$ of standard Borel spaces is the (bi-)initial object in the 2-category of countably complete Boolean (countably) extensive categories. This means that $\mathsf {SBor}$ is the universal category admitting some familiar algebraic operations of countable arity (e.g., countable products and unions) obeying some simple compatibility conditions (e.g., products distribute over disjoint unions). More generally, for any infinite regular cardinal $\kappa $ , the dual of the category $\kappa \mathsf {Bool}_{\kappa }$ of $\kappa $ -presented $\kappa $ -complete Boolean algebras is (bi-)initial in the 2-category of $\kappa $ -complete Boolean ( $\kappa $ -)extensive categories.

Some advances on constrained Markov decision processes in Borel spaces with random state-dependent discount factors

ABSTRACT This paper addresses a class of discrete-time Markov decision processes in Borel spaces with a finite number of cost constraints. The constrained control model considers costs of discounted type with state-dependent discount factors which are subject to external disturbances. Our objective is to prove the existence of optimal control policies and characterize them according to certain optimality criteria. Specifically, by rewriting appropriately our original constrained problem as a new one on a space of occupation measures, we apply the direct method to show solvability. Next, the problem is defined as a convex program, and we prove that the existence of a saddle point of the associated Lagrangian operator is equivalent to the existence of an optimal control policy for the constrained problem. Finally, we turn our attention to multi-objective optimization problems, where the existence of Pareto optimal policies can be obtained from the existence of saddle-points of the aforementioned Lagrangian or equivalently from the existence of optimal control policies of constrained problems.

Convex analytic method revisited: Further optimality results and performance of deterministic policies in average cost stochastic control

The convex analytic method has proved to be a very versatile method for the study of infinite horizon average cost optimal stochastic control problems. In this paper, we revisit the convex analytic method and make three primary contributions: (i) We present an existence result for controlled Markov models that lack weak continuity of the transition kernel but are strongly continuous in the action variable for every fixed state variable. (ii) For average cost stochastic control problems in standard Borel spaces, while existing results establish the optimality of stationary (possibly randomized) policies, few results are available on the optimality of deterministic policies. We review existing results and present further conditions under which an average cost optimal stochastic control problem admits optimal solutions that are deterministic stationary. (iii) We establish conditions under which the performance under stationary deterministic (and also quantized) policies is dense in the set of performance values under randomized stationary policies.

Monads for Measurable Queries in Probabilistic Databases

We consider a bag (multiset) monad on the category of standard Borel spaces, and show that it gives a free measurable commutative monoid. Firstly, we show that a recent measurability result for probabilistic database queries (Grohe and Lindner, ICDT 2020) follows quickly from the fact that queries can be expressed in monad-based terms. We also extend this measurability result to a fuller query language. Secondly, we discuss a distributive law between probability and bag monads, and we illustrate that this is useful for generating probabilistic databases.

On structural properties of optimal average cost functions in Markov decision processes with Borel spaces and universally measurable policies

We consider Markov decision processes (MDPs) with Borel state and action spaces and universally measurable policies. For several long-run average cost criteria and two classes of MDPs, we prove sufficient conditions for the optimal average cost functions to be constant almost everywhere with respect to certain σ-finite measures. Besides suitable boundedness conditions on the positive parts of the one-stage costs, the key condition here is that each subset of states with positive measure be reachable with probability one under some policy. Our proofs exploit an inequality for the optimal average cost functions and its connection with submartingales, and, in a special case that involves stationary policies, also use the theory of recurrent Markov chains.

De Finetti’s Theorem in Categorical Probability

We present a novel proof of de Finetti's Theorem characterizing permutation-invariant probability measures of infinite sequences of variables, so-called exchangeable measures. The proof is phrased in the language of Markov categories, which provide an abstract categorical framework for probability and information flow. The diagrammatic and abstract nature of the arguments makes the proof intuitive and easy to follow. We also show how the usual measure-theoretic version of de Finetti's Theorem for standard Borel spaces is an instance of this result.

Constrained optimality problem of Markov decision processes with Borel spaces and varying discount factors

Constrained optimality problem of Markov decision processes with Borel spaces and varying discount factors

Nonzero-sum Risk-Sensitive Average Stochastic Games: The Case of Unbounded Costs

In this paper, we study discrete-time nonzero-sum stochastic games under the risk-sensitive average cost criterion. The state space is a denumerable set, the action spaces of players are Borel spaces, and the cost functions are unbounded. Under suitable conditions, we first introduce the risk-sensitive first passage payoff functions and obtain their properties. Then, we establish the existence of a solution to the risk-sensitive average cost optimality equation of each player for the case of unbounded cost functions and show the existence of a randomized stationary Nash equilibrium in the class of randomized history-dependent strategies. Finally, we use a controlled population system to illustrate the main results.

Comparison of Information Structures for Zero-Sum Games and a Partial Converse to Blackwell Ordering in Standard Borel Spaces

In statistical decision theory involving a single decision maker, an information structure is said to be better than another one if for any cost function involving a hidden state variable and an ac...

Minimal definable graphs of definable chromatic number at least three

Abstract We show that there is a Borel graph on a standard Borel space of Borel chromatic number three that admits a Borel homomorphism to every analytic graph on a standard Borel space of Borel chromatic number at least three. Moreover, we characterize the Borel graphs on standard Borel spaces of vertex-degree at most two with this property and show that the analogous result for digraphs fails.