Continuous Transition Probabilities Research Articles

On Approximate and Weak Correlated Equilibria in Constrained Discounted Stochastic Games

In this paper, we consider constrained discounted stochastic games with a countably generated state space and norm continuous transition probability having a density function. We prove existence of approximate stationary equilibria and stationary weak correlated equilibria. Our results imply the existence of stationary Nash equilibrium in ARAT stochastic games.

Zero-Sum Average Cost Semi-Markov Games with Weakly Continuous Transition Probabilities and a Minimax Semi-Markov Inventory Problem

Zero-Sum Average Cost Semi-Markov Games with Weakly Continuous Transition Probabilities and a Minimax Semi-Markov Inventory Problem

MDPs with setwise continuous transition probabilities

This paper describes the structure of optimal policies for infinite-state Markov Decision Processes with setwise continuous transition probabilities. The action sets may be noncompact. The objective criteria are either the expected total discounted and undiscounted costs or average costs per unit time. The analysis of optimality equations and inequalities is based on the optimal selection theorem for inf-compact functions introduced in this paper.

Calibration of Transition Intensities for a Multistate Model: Application to Long-Term Care

We consider a non-homogeneous continuous time Markov chain model for Long-Term Care with five states: the autonomous state, three dependent states of light, moderate and severe dependence levels and the death state. For a general approach, we allow for non null intensities for all the returns from higher dependence levels to all lesser dependencies in the multi-state model. Using data from the 2015 Portuguese National Network of Continuous Care database, as the main research contribution of this paper, we propose a method to calibrate transition intensities with the one step transition probabilities estimated from data. This allows us to use non-homogeneous continuous time Markov chains for modeling Long-Term Care. We solve numerically the Kolmogorov forward differential equations in order to obtain continuous time transition probabilities. We assess the quality of the calibration using the Portuguese life expectancies. Based on reasonable monthly costs for each dependence state we compute, by Monte Carlo simulation, trajectories of the Markov chain process and derive relevant information for model validation and premium calculation.

Reduction of total-cost and average-cost MDPs with weakly continuous transition probabilities to discounted MDPs

This note describes sufficient conditions under which total-cost and average-cost Markov decision processes (MDPs) with general state and action spaces, and with weakly continuous transition probabilities, can be reduced to discounted MDPs. For undiscounted problems, these reductions imply the validity of optimality equations and the existence of stationary optimal policies. The reductions also provide methods for computing optimal policies. The results are applied to a capacitated inventory control problem with fixed costs and lost sales.

On the convergence of optimal actions for Markov decision processes and the optimality of (s, S) inventory policies

AbstractThis article studies convergence properties of optimal values and actions for discounted and average‐cost Markov decision processes (MDPs) with weakly continuous transition probabilities and applies these properties to the stochastic periodic‐review inventory control problem with backorders, positive setup costs, and convex holding/backordering costs. The following results are established for MDPs with possibly non‐compact action sets and unbounded cost functions: (i) convergence of value iterations to optimal values for discounted problems with possibly non‐zero terminal costs, (ii) convergence of optimal finite‐horizon actions to optimal infinite‐horizon actions for total discounted costs, as the time horizon tends to infinity, and (iii) convergence of optimal discount‐cost actions to optimal average‐cost actions for infinite‐horizon problems, as the discount factor tends to 1. Being applied to the setup‐cost inventory control problem, the general results on MDPs imply the optimality of (s, S) policies and convergence properties of optimal thresholds. In particular this article analyzes the setup‐cost inventory control problem without two assumptions often used in the literature: (a) the demand is either discrete or continuous or (b) the backordering cost is higher than the cost of backordered inventory if the amount of backordered inventory is large.© 2017 Wiley Periodicals, Inc. Naval Research Logistics 65: 619–637, 2018

A class of iterated function systems with adapted piecewise constant transition probabilities: Asymptotic stability and Hausdorff dimension of the invariant measure

A class of iterated function systems (IFS) with non-overlapping or just-touching contractions on closed real intervals and adapted piecewise constant transition probabilities are studied. We give criteria for the existence and the uniqueness of an invariant probability measure for the IFSs and for the asymptotic stability of the system in terms of bounds of transition probabilities. The proofs are mainly based on the symbolic system associated with the contractions, an extended alphabet and usual theorems for Markov chains. Additionally, in case there exists a unique invariant measure, we obtain its Hausdorff dimension as the ratio of the entropy over the Lyapunov exponent. This result extends the formula, established in the literature for continuous transition probabilities, to the case considered here of piecewise constant probabilities. The main idea of this proof consists in finding good lower and upper bounds for the invariant measure.

On the optimality equation for average cost Markov decision processes and its validity for inventory control

As is well known, average-cost optimality inequalities imply the existence of stationary optimal policies for Markov decision processes with average costs per unit time, and these inequalities hold under broad natural conditions. This paper provides sufficient conditions for the validity of the average-cost optimality equation for an infinite state problem with weakly continuous transition probabilities and with possibly unbounded one-step costs and noncompact action sets. These conditions also imply the convergence of sequences of discounted relative value functions to average-cost relative value functions and the continuity of average-cost relative value functions. As shown in this paper, the classic periodic-review setup-cost inventory control problem with backorders and convex holding/backlog costs satisfies these conditions. Therefore, the optimality inequality holds in the form of an equality with a continuous average-cost relative value function for this problem. In addition, the K-convexity of discounted relative value functions and their convergence to average-cost relative value functions, when the discount factor increases to 1, imply the K-convexity of average-cost relative value functions. This implies that average-cost optimal (s, S) policies for the inventory control problem can be derived from the average-cost optimality equation.

Partially Observable Total-Cost Markov Decision Processes with Weakly Continuous Transition Probabilities

This paper describes sufficient conditions for the existence of optimal policies for partially observable Markov decision processes (POMDPs) with Borel state, observation, and action sets, when the goal is to minimize the expected total costs over finite or infinite horizons. For infinite-horizon problems, one-step costs are either discounted or assumed to be nonnegative. Action sets may be noncompact and one-step cost functions may be unbounded. The introduced conditions are also sufficient for the validity of optimality equations, semicontinuity of value functions, and convergence of value iterations to optimal values. Since POMDPs can be reduced to completely observable Markov decision processes (COMDPs), whose states are posterior state distributions, this paper focuses on the validity of the above-mentioned optimality properties for COMDPs. The central question is whether the transition probabilities for the COMDP are weakly continuous. We introduce sufficient conditions for this and show that the transition probabilities for a COMDP are weakly continuous, if transition probabilities of the underlying Markov decision process are weakly continuous and observation probabilities for the POMDP are continuous in total variation. Moreover, the continuity in total variation of the observation probabilities cannot be weakened to setwise continuity. The results are illustrated with counterexamples and examples.

Continuity properties of a factor of Markov chains

Abstract Starting from a Markov chain with a finite or a countable infinite alphabet, we consider the chain obtained when all but one symbol are indistinguishable for the practitioner. We study conditions on the transition matrix of the Markov chain ensuring that the image chain has continuous or discontinuous transition probabilities with respect to the past.

Stochastic Processes With Random Contexts: A Characterization and Adaptive Estimators for the Transition Probabilities

This paper introduces the concept of random context representations for the transition probabilities of a finite-alphabet stochastic process. Processes with these representations generalize context tree processes (a.k.a. variable length Markov chains), and are proven to coincide with processes whose transition probabilities are almost surely continuous functions of the (infinite) past. This is similar to a classical result by Kalikow about continuous transition probabilities. Existence and uniqueness of a minimal random context representation are proven, and an estimator of the transition probabilities based on this representation is shown to have very good "pastwise adaptativity" properties. In particular, it achieves minimax performance, up to logarithmic factors, for binary renewal processes with bounded $2+\gamma$ moments.

On the vanishing discount factor approach for Markov decision processes with weakly continuous transition probabilities

This note deals with average cost Markov decision processes with Borel state and control spaces, possibly unbounded costs and non-compact action subsets under the assumption of weak continuity of the transition law. It provides an elementary proof of the existence of average cost optimal stationary policies using the vanishing discount factor approach.

H∞ static output feedback controller design for continuous Markov jump systems with incomplete transition probabilities

The [Formula: see text] static output feedback control of continuous Markov jump systems with incomplete transition probabilities is investigated in this paper. Employing the property of continuous transition probabilities and a matrix transformation technique, relaxed sufficient conditions for the existence of a solution to the [Formula: see text] static output feedback control problem are established in terms of linear matrix inequalities. In addition, the obtained results are extended to Markov jump nonlinear systems. Numerical examples are provided to demonstrate the effectiveness of the derived results.

Average Cost Markov Decision Processes with Weakly Continuous Transition Probabilities

This paper presents sufficient conditions for the existence of stationary optimal policies for average cost Markov decision processes with Borel state and action sets and weakly continuous transition probabilities. The one-step cost functions may be unbounded, and the action sets may be noncompact. The main contributions of this paper are: (i) general sufficient conditions for the existence of stationary discount optimal and average cost optimal policies and descriptions of properties of value functions and sets of optimal actions, (ii) a sufficient condition for the average cost optimality of a stationary policy in the form of optimality inequalities, and (iii) approximations of average cost optimal actions by discount optimal actions.

New analysis and synthesis conditions for continuous Markov jump linear systems with partly known transition probabilities

In this study, the stability analysis and synthesis problems for continuous Markov jump linear systems with partly known transition probabilities are investigated. The partly known transition probabilities cover the cases that some elements are known, some are unknown with known lower and upper bounds and some are completely unknown. By making full use of the continuous transition probability matrix property, that is, the sum of transition probabilities is 0 for each row, a new method for the analysis and synthesis is presented in terms of solvability of a set of linear matrix inequalities. Compared to the existing results in the literature, it is shown that the proposed method is more effective to deal with the considered transition probabilities. Numerical examples are given to show the validity of the proposed method.

Zero-Sum Ergodic Semi-Markov Games with Weakly Continuous Transition Probabilities

Zero-sum ergodic semi-Markov games with weakly continuous transition probabilities and lower semicontinuous, possibly unbounded, payoff functions are studied. Two payoff criteria are considered: the ratio average and the time average. The main result concerns the existence of a lower semicontinuous solution to the optimality equation and its proof is based on a fixed-point argument. Moreover, it is shown that the ratio average as well as the time average payoff stochastic games have the same value. In addition, one player possesses an e-optimal stationary strategy (e>0), whereas the other has an optimal stationary strategy.

Computing the Distributions of Economic Models via Simulation

This paper studies the convergence properties of a Monte Carlo algorithm for computing distributions of state variables when the underlying model is a Markov chain with absolutely continuous transition probabilities. We show that the L1 error of the estimator always converges to zero with probability one. In addition, rates of convergence are established for L1 and integral mean squared errors. The algorithm is shown to have many applications in economics.

A single-chain-based multidimensional Markov chain model for subsurface characterization

Multidimensional Markov chain models in geosciences were often built on multiple chains, one in each direction, and assumed these 1-D chains to be independent of each other. Thus, unwanted transitions (i.e., transitions of multiple chains to the same location with unequal states) inevitably occur and have to be excluded in estimating the states at unobserved locations. This consequently may result in unreliable estimates, such as underestimation of small classes (i.e., classes with smaller than average areas) in simulated realizations. This paper presents a single-chain-based multidimensional Markov chain model for estimation (i.e., prediction and conditional stochastic simulation) of spatial distribution of subsurface formations with borehole data. The model assumes that a single Markov chain moves in a lattice space, interacting with its nearest known neighbors through different transition probability rules in different cardinal directions. The conditional probability distribution of the Markov chain at the location to be estimated is formulated in an explicit form by following the Bayes’ Theorem and the conditional independence of sparse data in cardinal directions. Since no unwanted transitions are involved, the model can estimate all classes fairly. Transiogram models (i.e., 1-D continuous Markov transition probability diagrams) are used to provide transition probability input with needed lags to generalize the model. Therefore, conditional simulation can be conducted directly and efficiently. The model provides an alternative for heterogeneity characterization of subsurface formations.

A Fixed Point Approach to Solve the Average Cost Optimality Equation for Semi-Markov Decision Processes with Feller Transition Probabilities

Semi-Markov control processes with Feller (also known as weakly continuous) transition probabilities and unbounded cost functions are considered. We give sufficient conditions for the existence of a lower semicontinuous and continuous solutions to the average cost optimality equation. The proof is based on a fixed point argument.

On Markov Games with Average Reward Criterion and Weakly Continuous Transition Probabilities

In this paper we consider two‐person zero‐sum stochastic games with the average reward criterion and Borel state and action spaces. A geometric drift condition is assumed. We show that the optimality (Shapley) equation has a unique solution if the transition probability function is weakly continuous, the stage reward is lower semicontinuous, and the set‐valued mappings of admissible actions satisfy some semicontinuity assumptions. Furthermore, the minimizing player has an optimal stationary strategy and the maximizing player has an ε‐optimal stationary strategy for every $\varepsilon > 0 $.