Partially Observable Markov Decision Process Model Research Articles

Adaptive Multimodality Sensing of Landmines

The problem of adaptive multimodality sensing of landmines is considered based on electromagnetic induction (EMI) and ground-penetrating radar (GPR) sensors. Two formulations are considered based on a partially observable Markov decision process (POMDP) framework. In the first formulation, it is assumed that sufficient training data are available, and a POMDP model is designed based on physics-based features, with model selection performed via a variational Bayes analysis of several possible models. In the second approach, the training data are assumed absent or insufficient, and a lifelong-learning approach is considered, in which exploration and exploitation are integrated. We provide a detailed description of both formulations, with example results presented using measured EMI and GPR data, for buried mines and clutter

Apprentissage actif dans les processus décisionnels de Markov partiellement observables L'algorithme MEDUSA

We study a problem inspired from robotics in which we want to find an optimal policy to learn a Partially Observable Markov Decision Process (POMDP) when the agent only has an imperfect model of its environment. To help the agent in its task we assume the availability of an external operator (an oracle), that can provide information about the underlying state. We present the algorithm MEDUSA, which improves an initial POMDP model using experimentation through the environment and a minimum number of queries. We also show how MEDUSA handles non-stationary environments and how it can withstand noise in the query answer.

Finding Approximate POMDP solutions Through Belief Compression

Standard value function approaches to finding policies for Partially Observable Markov Decision Processes (POMDPs) are generally considered to be intractable for large models. The intractability of these algorithms is to a large extent a consequence of computing an exact, optimal policy over the entire belief space. However, in real-world POMDP problems, computing the optimal policy for the full belief space is often unnecessary for good control even for problems with complicated policy classes. The beliefs experienced by the controller often lie near a structured, low-dimensional subspace embedded in the high-dimensional belief space. Finding a good approximation to the optimal value function for only this subspace can be much easier than computing the full value function. We introduce a new method for solving large-scale POMDPs by reducing the dimensionality of the belief space. We use Exponential family Principal Components Analysis (Collins, Dasgupta & Schapire, 2002) to represent sparse, high-dimensional belief spaces using small sets of learned features of the belief state. We then plan only in terms of the low-dimensional belief features. By planning in this low-dimensional space, we can find policies for POMDP models that are orders of magnitude larger than models that can be handled by conventional techniques. We demonstrate the use of this algorithm on a synthetic problem and on mobile robot navigation tasks.