Partially observed Markov decision processes (POMDPs)

Abstract

A POMDP is a controlled HMM. Recall from §2.4 that an HMM consists of an X -state Markov chain { x k } observed via a noisy observation process { y k }. Figure 7.1 displays the schematic setup of a POMDP where the action u k affects the state and/or observation (sensing) process of the HMM. The HMM filter (discussed extensively in Chapter 3) computes the posterior distribution π k of the state. The posterior π k is called the belief state . In a POMDP, the stochastic controller depicted in Figure 7.1 uses the belief state to choose the next action. This chapter is organized as follows. §7.1 describes the POMDP model. Then §7.2 gives the belief state formulation and the Bellman's dynamic programming equation for the optimal policy of a POMDP. It is shown that a POMDP is equivalent to a continuous-state MDP where the states are belief states (posteriors). Bellman's equation for continuous-state MDP was discussed in §6.3. §7.3 gives a toy example of a POMDP. Despite being a continuous-state MDP, §7.4 shows that for finite horizon POMDPs, Bellman's equation has a finite dimensional characterization. §7.5 discusses several algorithms that exploit this finite dimensional characterization to compute the optimal policy. §7.6 considers discounted cost infinite horizon POMDPs. As an example of a POMDP, optimal search of a moving target is discussed in §7.7. Finite horizon POMDP A POMDP model with finite horizon N is a 7-tuple ( X , U , Y , P ( u ), B ( u ), c ( u ), c N ). Partially observed Markov decision process (POMDP) schematic setup. The Markov system together with noisy sensor constitute a hidden Markov model (HMM). The HMM filter computes the posterior (belief state) π k of the state of the Markov chain. The controller (decision-maker) then chooses the action u k at time k based on π k . 1. X = {1, 2, …, X } denotes the state space and x k ∈ X denotes the state of a controlled Markov chain at time k = 0, 1, …, N . 2. U = {1, 2, …, U } denotes the action space with u k ∈ U denoting the action chosen at time k by the controller.