Finite approximation of the first passage models for discrete-time Markov decision processes with varying discount factors

Abstract

This paper deals with the finite approximation of the first passage models for discrete-time Markov decision processes with varying discount factors. For a given control model ź$\mathcal {M}$ with denumerable states and compact Borel action sets, and possibly unbounded reward functions, under reasonable conditions we prove that there exists a sequence of control models źn$\mathcal {M}_{n}$ such that the first passage optimal rewards and policies of źn$\mathcal {M}_{n}$ converge to those of ź$\mathcal {M}$, respectively. Based on the convergence theorems, we propose a finite-state and finite-action truncation method for the given control model ź$\mathcal {M}$, and show that the first passage optimal reward and policies of ź$\mathcal {M}$ can be approximated by those of the solvable truncated finite control models. Finally, we give the corresponding value and policy iteration algorithms to solve the finite approximation models.