Near Optimality of Finite Memory Policies for POMPDs with Continuous Spaces

Abstract

We study an approximation method for partially observed Markov decision processes (POMDPs) with continuous spaces. Belief MDP reduction has been the standard approach to study POMDPs, which, due to its uncountable state space and strict regularity properties however, requires rigorous approximation methods for practical applications. In this work, we focus on an approximation procedure via discretizing the observation space and constructing a fully observed finite MDP model using a finite length history of the discrete observations and control actions. We show that the resulting policy is nearly optimal under some regularity assumptions on the channel, and under certain controlled filter stability requirements for the hidden state process. We also provide a Q learning algorithm that uses a finite memory of discretized information variables, and prove its convergence to the optimality equation of the finite fully observed MDP constructed using the approximation method.