Partially Observable Markov Decision Process Model Research Articles

Dialogue POMDP components (Part II): learning the reward function

The partially observable Markov decision process (POMDP) framework has been applied in dialogue systems as a formal framework to represent uncertainty explicitly while being robust to noise. In this context, estimating the dialogue POMDP model components (states, observations, and reward) is a significant challenge as they have a direct impact on the optimized dialogue POMDP policy. Learning states and observations sustaining a POMDP have been both covered in the first part (Part I), whereas this part (Part II) covers learning the reward function, that is required by the POMDP. To this end, we propose two specific algorithms based on inverse reinforcement learning (IRL). The first is called POMDP-IRL-BT (BT for belief transition) and it approximates a belief transition model, similar to the Markov decision process transition models. The second is a point-based POMDP-IRL algorithm, denoted by PB-POMDP-IRL (PB for point-based), that approximates the value of the new beliefs, which occurs in the computation of the policy values, using a linear approximation of expert beliefs. Ultimately, we apply the two algorithms on healthcare dialogue management in order to learn a dialogue POMDP from dialogues collected by SmartWheeler (an intelligent wheelchair).

Partially Observable Markov Decision Processes (POMDPs) and Wireless Body Area Networks (WBAN): A Survey

Wireless body area network (WBAN) is a promising candidate for future health monitoring system. Nevertheless, the path to mature solutions is still facing a lot of challenges that need to be overcome. Energy efficient scheduling is one of these challenges given the scarcity of available energy of biosensors and the lack of portability. Therefore, researchers from academia, industry and health sectors are working together to realize practical solutions for these challenges. The main difficulty in WBAN is the uncertainty in the state of the monitored system. Intelligent learning approaches such as a Markov Decision Process (MDP) were proposed to tackle this issue. A Markov Decision Process (MDP) is a form of Markov Chain in which the transition matrix depends on the action taken by the decision maker (agent) at each time step. The agent receives a reward, which depends on the action and the state. The goal is to find a function, called a policy, which specifies which action to take in each state, so as to maximize some utility functions (e.g., the mean or expected discounted sum) of the sequence of rewards. A partially Observable Markov Decision Processes (POMDP) is a generalization of Markov decision processes that allows for the incomplete information regarding the state of the system. In this case, the state is not visible to the agent. This has many applications in operations research and artificial intelligence. Due to incomplete knowledge of the system, this uncertainty makes formulating and solving POMDP models mathematically complex and computationally expensive. Limited progress has been made in terms of applying POMPD to real applications. In this paper, we surveyed the existing methods and algorithms for solving POMDP in the general domain and in particular in Wireless body area network (WBAN). In addition, the papers discussed recent real implementation of POMDP on practical problems of WBAN. We believe that this work will provide valuable insights for the newcomers who would like to pursue related research in the domain of WBAN.

Planning for Human–Robot Interaction in Socially Situated Tasks

This article presents the results of a study on the effects of representing time and intention in models of socially situated tasks on the quality of policies for robot behavior. The ability to reason about how others’ observable actions relate to their unobservable intentions is an important part of interpreting and responding to social behavior. It is also often necessary to observe the timing of actions in order to disambiguate others’ intentions. Therefore, our proposed approach is to model these interactions as time-indexed partially observable Markov decision processes (POMDPs). The intentions of humans are represented as hidden state in the POMDP models, and the time-dependence of actions by both humans and the robot are explicitly modelled. Our hypothesis is that planning for these interactions with a model that represents time dependent action outcomes and uncertainty about others’ intentions will achieve better results than simpler models that make fixed assumptions about people’s intentionality or abstract away time-dependent effects. A driving interaction governed by social conventions and involving ambiguity in the other driver’s intent was used as the scenario with which to test this hypothesis. A robot car controlled by policies from time-dependent POMDP models or by policies from two less expressive model variants performed this interaction in a driving simulator with human drivers. The time-dependent POMDP policies achieved better results than those of the models without explicit time representation or human intention as hidden state, both according to the reward obtained and to people’s subjective impressions of how socially appropriate and natural the robot’s behavior was. These results demonstrate both the relative superiority of these representation choices and the effectiveness of this approach to planning for socially situated tasks.

Relational approach to knowledge engineering for POMDP-based assistance systems as a translation of a psychological model

Assistive systems for persons with cognitive disabilities (e.g., dementia) are difficult to build due to the wide range of different approaches people can take to accomplishing the same task, and the significant uncertainties that arise from both the unpredictability of client’s behaviours and from noise in sensor readings. Partially observable Markov decision process (POMDP) models have been used successfully as the reasoning engine behind such assistive systems for small multi-step tasks such as hand washing. POMDP models are a powerful, yet flexible framework for modelling assistance that can deal with uncertainty and utility. Unfortunately, POMDPs usually require a very labour intensive, manual procedure for their definition and construction. Our previous work has described a knowledge driven method for automatically generating POMDP activity recognition and context sensitive prompting systems for complex tasks. We call the resulting POMDP a SNAP (SyNdetic Assistance Process). The spreadsheet-like result of the analysis does not correspond to the POMDP model directly and the translation to a formal POMDP representation is required. To date, this translation had to be performed manually by a trained POMDP expert. In this paper, we formalise and automate this translation process using a probabilistic relational model (PRM) encoded in a relational database. The database encodes the relational skeleton of the PRM, and includes the goals, action preconditions, environment states, cognitive model, client and system actions (i.e., the outcome of the SNAP analysis), as well as relevant sensor models. The database is easy to approach for someone who is not an expert in POMDPs, allowing them to fill in the necessary details of a task using a simple and intuitive procedure. The database, when filled, implicitly defines a ground instance of the relational skeleton, which we extract using an automated procedure, thus generating a POMDP model of the assistance task. A strength of the database is that it allows constraints to be specified, such that we can verify the POMDP model is, indeed, valid for the task given the analysis. We demonstrate the method by eliciting three assistance tasks from non-experts: handwashing, and toothbrushing for elderly persons with dementia, and on a factory assembly task for persons with a cognitive disability. We validate the resulting POMDP models using case-based simulations to show that they are reasonable for the domains. We also show a complete case study of a designer specifying one database, including an evaluation in a real-life experiment with a human actor.

Modelling traditional Chinese medicine therapy planning with POMDP

During the traditional Chinese medicine (TCM) treatment procedure, the manifestations of patients could be observed but the health state and TCM diagnosis of patient are uncertain. Thus, the real–world TCM therapy planning is a typical kind of dynamic decision making under uncertainty. Partially observable Markov decision process (POMDP) constitutes a powerful mathematical model for planning and is suitable for TCM therapy planning. In this paper, we apply POMDP to solve TCM therapy planning problem with all the dynamics inferred from TCM clinical data for type 2 diabetes treatment. This POMDP model contains 55 health states, 67 observation variables and 414 actions, it could order prescriptions for patients with type 2 diabetes. The results demonstrate that the POMDP model for TCM therapy planning is reasonable and helpful in clinical practice.

Particle Guidance: Applying POMDPs to the Optimization of Mid-Course Guidance Laws for Long-Range Missiles

During the mid-course phase of an air-to-air missile, choosing the optimal Guidance Point (GP) so as to maximize lock-on success and minimize intercept time is critical. Given low computational resources available on board and a very constrained maneuvering time frame, GP-based algorithms must be efficient. We suggest an innovative approach using Reinforcement Learning (RL) to produce finite state controllers that can be executed efficiently – using table lookup – to meet the strict time limits of a target engagement. Instead of hand-crafting a GP-picking algorithm for every combination of sensor and aircraft configuration, one promising alternative models a missile-target engagement as a Partially Observable Markov Decision Process (POMDP) and automatically generates a controller for picking the best GP by solving the POMDP model. Using a recently developed offline algorithm called Monte Carlo Value Iteration (MCVI) we constructed continuous-state POMDP models and solved them directly, without discretizing the entire state space.Invited session “Missile Guidance Navigation & Control” (pm846)

People, sensors, decisions

The ratio of healthcare professionals to care recipients is dropping at an alarming rate, particularly for the older population. It is estimated that the number of persons with Alzheimer's disease, for example, will top 100 million worldwide by the year 2050 [Alzheimer's Disease International 2009]. It will become harder and harder to provide needed health services to this population of older adults. Further, patients are becoming more aware and involved in their own healthcare decisions. This is creating a void in which technology has an increasingly important role to play as a tool to connect providers with recipients. Examples of interactive technologies range from telecare for remote regions to computer games promoting fitness in the home. Currently, such technologies are developed for specific applications and are difficult to modify to suit individual user needs. The future potential economic and social impact of technology in the healthcare field therefore lies in our ability to make intelligent devices that are customizable by healthcare professionals and their clients, that are adaptive to users over time, and that generalize across tasks and environments. A wide application area for technology in healthcare is for assistance and monitoring in the home. As the population ages, it becomes increasingly dependent on chronic healthcare, such as assistance for tasks of everyday life (washing, cooking, dressing), medication taking, nutrition, and fitness. This article will present a summary of work over the past decade on the development of intelligent systems that provide assistance to persons with cognitive disabilities. These systems are unique in that they are all built using a common framework, a decision-theoretic model for general-purpose assistance in the home. In this article, we will show how this type of general model can be applied to a range of assistance tasks, including prompting for activities of daily living, assistance for art therapists, and stroke rehabilitation. This model is a Partially Observable Markov Decision Process (POMDP) that can be customized by end-users, that can integrate complex sensor information, and that can adapt over time. These three characteristics of the POMDP model will allow for increasing uptake and long-term efficiency and robustness of technology for assistance.

OR Forum—A POMDP Approach to Personalize Mammography Screening Decisions

Breast cancer is the most common nonskin cancer and the second leading cause of cancer death in U.S. women. Although mammography is the most effective modality for breast cancer screening, it has several potential risks, including high false-positive rates. Therefore, the balance of benefits and risks, which depend on personal characteristics, is critical in designing a mammography screening schedule. In contrast to prior research and existing guidelines that consider population-based screening recommendations, we propose a personalized mammography screening policy based on the prior screening history and personal risk characteristics of women. We formulate a finite-horizon, partially observable Markov decision process (POMDP) model for this problem. Our POMDP model incorporates two methods of detection (self or screen), age-specific unobservable disease progression, and age-specific mammography test characteristics. We solve this POMDP optimally after setting transition probabilities to values estimated from a validated microsimulation model. Additional published data is used to specify other model inputs such as sensitivity and specificity of test results. Our results show that our proposed personalized screening schedules outperform the existing guidelines with respect to the total expected quality-adjusted life years, while significantly decreasing the number of mammograms and false-positives. We also report the lifetime risk of developing undetected invasive cancer associated with each screening scenario.

Optimization of Prostate Biopsy Referral Decisions

Prostate cancer is the most common solid tumor in American men and is screened for using prostate-specific antigen (PSA) tests. We report on a nonstationary partially observable Markov decision process (POMDP) for prostate biopsy referral decisions. The core states are the patients' prostate cancer related health states, and PSA test results are the observations. Transition probabilities and rewards are inferred from the Mayo Clinic Radical Prostatectomy Registry and the medical literature. The objective of our model is to maximize expected quality-adjusted life years. We solve the POMDP model to obtain an age and belief (probability of having prostate cancer) dependent optimal biopsy referral policy. We also prove a number of structural properties including the existence of a control-limit type policy for the biopsy referral decision. Our empirical results demonstrate a nondecreasing belief threshold in age, and we provide sufficient conditions under which PSA screening should be discontinued for older patients. Finally, the benefits of screening under the optimal biopsy referral policy are estimated, and sensitivity analysis is used to prioritize the model parameters that would benefit from additional data collection.

Recognizing and learning models of social exchange strategies for the regulation of social interactions in open agent societies

Abstract Regulation of social exchanges refers to controlling social exchanges between agents so that the balance of exchange values involved in the exchanges are continuously kept—as far as possible—near to equilibrium. Previous work modeled the social exchange regulation problem as a POMDP (Partially Observable Markov Decision Process), and defined the policyToBDIplans algorithm to extract BDI (Beliefs, Desires, Intentions) plans from POMDP models, so that the derived BDI plans can be applied to keep in equilibrium social exchanges performed by BDI agents. The aim of the present paper is to extend that BDI-POMDP agent model for self-regulation of social exchanges with a module, based on HMM (Hidden Markov Model), for recognizing and learning partner agents’ social exchange strategies, thus extending its applicability to open societies, where new partner agents can freely appear at any time. For the recognition problem, patterns of refusals of exchange proposals are analyzed, as such refusals are produced by the partner agents. For the learning problem, HMMs are used to capture probabilistic state transition and observation functions that model the social exchange strategy of the partner agent, in order to translate them into POMDP’s action-based state transition and observation functions. The paper formally addresses the problem of translating HMMs into POMDP models and vice versa, introducing the translation algorithms and some examples. A discussion on the results of simulations of strategy-based social exchanges is presented, together with an analysis about related work on social exchanges in multiagent systems.

A Modified Memory-Based Reinforcement Learning Method for Solving POMDP Problems

Partially observable Markov decision processes (POMDP) provide a mathematical framework for agent planning under stochastic and partially observable environments. The classic Bayesian optimal solution can be obtained by transforming the problem into Markov decision process (MDP) using belief states. However, because the belief state space is continuous and multi-dimensional, the problem is highly intractable. Many practical heuristic based methods are proposed, but most of them require a complete POMDP model of the environment, which is not always practical. This article introduces a modified memory-based reinforcement learning algorithm called modified U-Tree that is capable of learning from raw sensor experiences with minimum prior knowledge. This article describes an enhancement of the original U-Tree's state generation process to make the generated model more compact, and also proposes a modification of the statistical test for reward estimation, which allows the algorithm to be benchmarked against some traditional model-based algorithms with a set of well known POMDP problems.

Rapid specification and automated generation of prompting systems to assist people with dementia

Activity recognition in intelligent environments could play a key role for supporting people in their activities of daily life. Partially observable Markov decision process (POMDP) models have been used successfully, for example, to assist people with dementia when carrying out small multistep tasks such as hand washing. POMDP models are a powerful, yet flexible framework for modeling assistance that can deal with uncertainty and utility in a theoretically well-justified manner. Unfortunately, POMDPs usually require a very labor-intensive, manual set-up procedure. This paper describes a knowledge-driven method for automatically generating POMDP activity recognition and context-sensitive prompting systems for complex tasks. It starts with a psychologically justified description of the task and the particular environment in which it is to be carried out that can be generated from empirical data. This is then combined with a specification of the available sensors and effectors to build a working prompting system. The method is illustrated by building a system that prompts through the task of making a cup of tea in a real-world kitchen. The case is made that, with further development and tool support, the method could feasibly be used in a clinical or industrial setting.

Optimally managing under imperfect detection: a method for plant invasions

Summary1. Failing to account for uncertainty in the detection of invasive plants may lead to inefficient management strategies and wasted resources. Smart strategies to manage plant invasions requires consideration of the economic costs and benefits, and plant life‐history characteristics as well as imperfect detection.2. We develop a partially observable Markov decision process (POMDP) to provide optimal management actions when we are uncertain about the presence of invasive plants. The optimal strategy depends on the probability of being in a particular state. We ask the question, ‘When is it preferable to use a less efficient, less costly action to a more efficient, more costly action?’ We apply the POMDP to branched broomrape Orobanche ramosa, a parasitic plant species at the centre of a national eradication campaign in South Australia.3. The optimal strategy depends on the ability to detect the invasive species and the location of the infested site. For high detection rates, if the site is a satellite infestation, management should employ the more efficient, more costly action (i.e. soil fumigation) the year the weed is detected followed by monitoring. When the detection probability is low, then it is optimal to employ the less efficient, low cost action (i.e. host denial) in the years the species is not detected. For sites in the centre of the infestation, management should employ the less costly, less efficient action. While the optimal strategy is insensitive to colonization, the likelihood of local eradication diminishes as colonization probability increases, highlighting the importance of limiting colonization if eradication is to be achieved.4. Synthesis and applications. Providing decision support for managing ecological systems is a key role of applied research. Formulating this support within a decision theory context provides a framework for good decision‐making. The POMDP model is a novel decision support tool for optimal sequential decision making when invasive plants are difficult to detect. The model can determine the best management action to employ based on the location of the infestation and can inform when to switch to alternative management actions that buffer against imperfect detection.

Bayesian update of dialogue state: A POMDP framework for spoken dialogue systems

This paper describes a statistically motivated framework for performing real-time dialogue state updates and policy learning in a spoken dialogue system. The framework is based on the partially observable Markov decision process (POMDP), which provides a well-founded, statistical model of spoken dialogue management. However, exact belief state updates in a POMDP model are computationally intractable so approximate methods must be used. This paper presents a tractable method based on the loopy belief propagation algorithm. Various simplifications are made, which improve the efficiency significantly compared to the original algorithm as well as compared to other POMDP-based dialogue state updating approaches. A second contribution of this paper is a method for learning in spoken dialogue systems which uses a component-based policy with the episodic Natural Actor Critic algorithm. The framework proposed in this paper was tested on both simulations and in a user trial. Both indicated that using Bayesian updates of the dialogue state significantly outperforms traditional definitions of the dialogue state. Policy learning worked effectively and the learned policy outperformed all others on simulations. In user trials the learned policy was also competitive, although its optimality was less conclusive. Overall, the Bayesian update of dialogue state framework was shown to be a feasible and effective approach to building real-world POMDP-based dialogue systems.

A tutorial on partially observable Markov decision processes

The partially observable Markov decision process (POMDP) model of environments was first explored in the engineering and operations research communities 40 years ago. More recently, the model has been embraced by researchers in artificial intelligence and machine learning, leading to a flurry of solution algorithms that can identify optimal or near-optimal behavior in many environments represented as POMDPs. The purpose of this article is to introduce the POMDP model to behavioral scientists who may wish to apply the framework to the problem of understanding normative behavior in experimental settings. The article includes concrete examples using a publicly-available POMDP solution package.

An online multi-agent co-operative learning algorithm in POMDPs

Solving partially observable Markov decision processes (POMDPs) is a complex task that is often intractable. This paper examines the problem of finding an optimal policy for POMDPs. While a lot of effort has been made to develop algorithms to solve POMDPs, the question of automatically finding good low-dimensional spaces in multi-agent co-operative learning domains has not been explored thoroughly. To identify this question, an online algorithm CMEAS is presented to improve the POMDP model. This algorithm is based on a look-ahead search to find the best action to execute at each cycle. Thus the overwhelming complexity of computing a policy for each possible situation is avoided. A series of simulations demonstrate this good strategy and performance of the proposed algorithm when multiple agents co-operate to find an optimal policy for POMDPs.

Spoken language interaction with model uncertainty: an adaptive human–robot interaction system

Spoken language is one of the most intuitive forms of interaction between humans and agents. Unfortunately, agents that interact with people using natural language often experience communication errors and do not correctly understand the user's intentions. Recent systems have successfully used probabilistic models of speech, language and user behaviour to generate robust dialogue performance in the presence of noisy speech recognition and ambiguous language choices, but decisions made using these probabilistic models are still prone to errors owing to the complexity of acquiring and maintaining a complete model of human language and behaviour. In this paper, a decision-theoretic model for human–robot interaction using natural language is described. The algorithm is based on the Partially Observable Markov Decision Process (POMDP), which allows agents to choose actions that are robust not only to uncertainty from noisy or ambiguous speech recognition but also unknown user models. Like most dialogue systems, a POMDP is defined by a large number of parameters that may be difficult to specify a priori from domain knowledge, and learning these parameters from the user may require an unacceptably long training period. An extension to the POMDP model is described that allows the agent to acquire a linguistic model of the user online, including new vocabulary and word choice preferences. The approach not only avoids a training period of constant questioning as the agent learns, but also allows the agent actively to query for additional information when its uncertainty suggests a high risk of mistakes. The approach is demonstrated both in simulation and on a natural language interaction system for a robotic wheelchair application.

Adaptive Multimodality Sensing of Landmines

The problem of adaptive multimodality sensing of landmines is considered based on electromagnetic induction (EMI) and ground-penetrating radar (GPR) sensors. Two formulations are considered based on a partially observable Markov decision process (POMDP) framework. In the first formulation, it is assumed that sufficient training data are available, and a POMDP model is designed based on physics-based features, with model selection performed via a variational Bayes analysis of several possible models. In the second approach, the training data are assumed absent or insufficient, and a lifelong-learning approach is considered, in which exploration and exploitation are integrated. We provide a detailed description of both formulations, with example results presented using measured EMI and GPR data, for buried mines and clutter

Apprentissage actif dans les processus décisionnels de Markov partiellement observables L'algorithme MEDUSA

We study a problem inspired from robotics in which we want to find an optimal policy to learn a Partially Observable Markov Decision Process (POMDP) when the agent only has an imperfect model of its environment. To help the agent in its task we assume the availability of an external operator (an oracle), that can provide information about the underlying state. We present the algorithm MEDUSA, which improves an initial POMDP model using experimentation through the environment and a minimum number of queries. We also show how MEDUSA handles non-stationary environments and how it can withstand noise in the query answer.

Finding Approximate POMDP solutions Through Belief Compression

Standard value function approaches to finding policies for Partially Observable Markov Decision Processes (POMDPs) are generally considered to be intractable for large models. The intractability of these algorithms is to a large extent a consequence of computing an exact, optimal policy over the entire belief space. However, in real-world POMDP problems, computing the optimal policy for the full belief space is often unnecessary for good control even for problems with complicated policy classes. The beliefs experienced by the controller often lie near a structured, low-dimensional subspace embedded in the high-dimensional belief space. Finding a good approximation to the optimal value function for only this subspace can be much easier than computing the full value function. We introduce a new method for solving large-scale POMDPs by reducing the dimensionality of the belief space. We use Exponential family Principal Components Analysis (Collins, Dasgupta & Schapire, 2002) to represent sparse, high-dimensional belief spaces using small sets of learned features of the belief state. We then plan only in terms of the low-dimensional belief features. By planning in this low-dimensional space, we can find policies for POMDP models that are orders of magnitude larger than models that can be handled by conventional techniques. We demonstrate the use of this algorithm on a synthetic problem and on mobile robot navigation tasks.