Discounted Continuous-Time Markov Decision Processes with Unbounded Rates: The Convex Analytic Approach

Abstract

This paper deals with constrained discounted continuous-time Markov decision processes, also known as controlled jump Markov processes, with Borel state and action spaces. Under some conditions imposed on the primitives, allowing unbounded transition rates and unbounded (from both above and below) cost rates, first, we study the space of occupation measures. Then we reformulate the original problem as a linear program over the space of those measures and undertake the duality analysis. Finally, under some compactness-continuity conditions, we show the existence of a stationary optimal policy out of the class of randomized history-dependent policies.