Ergodic Control of Continuous-Time Markov Chains with Pathwise Constraints

Abstract

This paper deals with unichain ergodic continuous-time controlled Markov chains (CMCs) with a denumerable state space and possibly unbounded reward rates as well as unbounded transition rates. The problem we are concerned with is to find control policies that maximize a sample-path average reward over the family of admissible policies for which a certain pathwise average cost is below a given value with probability one. To study this problem, first, we give conditions for the existence of sample-path average optimal policies. Then we analyze constrained average reward CMCs with “expected” constraints, and, finally, we apply these results to our original control problem with pathwise constraints. Examples on the control of a queueing system and an epidemic process illustrate the feasibility of our approach.