Analysis of averages over distributions of Markov processes

Abstract

In problems of optimal control of Markov decision processes and optimal design of experiments, the occupation measure of a Markov process is designed in order to maximize a specific reward function. When the memory of such a process is too long, or the process is non-Markovian but mixing, it makes sense to approximate it by that of a shorter memory Markov process. This note provides a specific bound for the approximation error introduced in these schemes. The derived bound is then applied to the proposed solution of a recently introduced approach to optimal input design for nonlinear systems.