On the First Passage $g$-Mean-Variance Optimality for Discounted Continuous-Time Markov Decision Processes

Abstract

This paper considers the discounted continuous-time Markov decision processes (MDPs) in Borel spaces and with unbounded transition rates. The discount factors are allowed to depend on states and actions. Main attention is concentrated on the set $F_g$ of stationary policies attaining a given mean performance $g$ up to the first passage of the continuous-time MDP to an arbitrarily fixed target set. Under suitable conditions, we prove the existence of a $g$-mean-variance optimal policy that minimizes the first passage variance over the set $F_g$ using a transformation technique, and also give the value iteration and policy iteration algorithms for computing the $g$-variance value function and a $g$-mean-variance optimal policy, respectively. Two examples are analytically solved to demonstrate the application of our results.