14 - Hidden Markov Models

Abstract

In the Markov decision process, the states are visible in the sense that the state sequence of the processes is known. Thus, we can refer to this model as a visible Markov decision model. In the partially observable Markov decision process (POMDP), the underlying process is a Markov chain whose internal states are hidden from the observer. In this chapter, we consider a variation of the POMDP called the hidden Markov model (HMM), in which the state sequence that the process passes through is not known but can only be guessed through a sequence of observations of the dynamics of the process. The difference between HMM and POMDP is that in POMDP we have control over the state transitions through the actions we take, but not so in HMM. Thus, in HMM, the states are not observable and we have no control over state transitions.