Approximate Optimal Cost and Policies of First Passage Markov Decision Processes with Countable-State Space and Discount Factors

Abstract

This paper is concerned with the approximate optimal cost and policies of discrete-time Markov decision processes (DTMDPs) with constraints and state-dependent discount factors. For any countable-state first passage DTMDPs, we construct a sequence of truncated control models with finite states, transform their constrained optimality problems into the equivalent linear program problems by the convex analytic approach, and obtain their optimal costs and policies. Then, we prove that the optimal cost and policies of the original first passage DTMDPs can be approximated by those of the truncated models, respectively. Finally, we illustrate such approximation by a controlled queueing system.