Control Theory Meets POMDPs: A Hybrid Systems Approach

Abstract

Partially observable Markov decision processes (POMDPs) provide a modeling framework for a variety of sequential decision making under uncertainty scenarios in artificial intelligence (AI). Since the states are not directly observable in a POMDP, decision making has to be performed based on the output of a Bayesian filter (continuous beliefs); hence, making POMDPs intractable to solve and analyze. To overcome the complexity challenge of POMDPs, we apply techniques from the control theory. Our contributions are fourfold. 1) We begin by casting the problem of analyzing a POMDP into analyzing the behavior of a discrete-time switched system. 2) Then, in order to estimate the reachable belief space of a POMDP, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.,</i> the set of all possible evolutions given an initial belief distribution over the states and a set of actions and observations, we find overapproximations in terms of sublevel sets of Lyapunov-like functions. 3) Furthermore, in order to verify safety and performance requirements of a given POMDP, we formulate a barrier certificate theorem, wherein we show that if there exists a barrier certificate satisfying a set of inequalities along the solutions to the belief update equation of the POMDP, the safety and performance properties are guaranteed to hold. In both cases 2) and 3), the calculations can be decomposed and solved in parallel. 4) Finally, we show that the conditions we formulate can be computationally implemented as a set of sum-of-squares programs. We illustrate the applicability of our method by addressing two problems in active ad scheduling and machine teaching.