Partially Observable Markov Decision Processes Framework Research Articles

Value of information analysis in non-stationary stochastic decision environments: A reliability-assisted POMDP approach

Optimal management of systems over their service life as they face a multitude of uncertainties remains a significant challenge. While additional information can reduce uncertainties, collecting new information incurs cost and may include observation error. Value of Information (VoI) analysis facilitates quantitative assessment of the expected net benefits of collecting new information. Moreover, partially observable Markov decision processes (POMDPs) can be integrated within VoI analysis to efficiently capture the sequential decision-making environments for systems. The assumption of stationary environment in existing POMDP frameworks for VoI analysis may not be valid, however, in many applications such as deterioration processes which are often non-stationary. To address this gap, this paper presents a new approach called VoI-R-POMDP. A new POMDP framework is proposed to accurately describe non-stationary processes using multiple integrated transition models. New strategies based on reliability concepts are developed to accurately and efficiently determine the parameters of the proposed POMDP model based on prior information. A new formulation of the observation function based on Bayes’ theorem is also derived. The proposed framework is applied to a corroding beam example. Results indicate that VoI-R-POMDP can accurately and efficiently describe the deterioration process and thus provide accurate VoI estimates for non-stationary systems.

A Partially Observable Markov-Decision-Process-Based Blackboard Architecture for Cognitive Agents in Partially Observable Environments

Partial observability, or the inability of an agent to fully observe the state of its environment, exists in many real-world problem domains. However, most cognitive architectures do not have a theoretical foundation that allows for a systematic approach to handling partial observability. To address this issue, in this article, we propose a novel cognitive architecture based on the theory of partially observable Markov decision processes (POMDPs). We focus on one of the most fundamental cognitive architectures, the blackboard architecture, and reformulate it using the POMDP framework so that the agent can automatically decide how to use its information processing modules in partially observable environments. We propose a two-layer POMDP model for the blackboard architecture in which one layer models the dynamics of the environment and the other layer models the dynamics of the blackboard and knowledge sources within the agent. We also provide a planning method to find the optimal actions for the two-layer POMDP model based on the partially observable upper confidence bounds applied to trees (PO-UCT) algorithm. We validate the proposed architecture in both numerical and real-robot experiments.

Decision Making Under Uncertainty: Theory and Application [Bookshelf

The unmanned air vehicles and self-driving cars of the future will require a high degree of autonomy. Not only will they need feedback loops that are conducive to a wide variety of environmental conditions, but they will also require higher levels of reasoning and planning that can efficiently handle unexpected conditions and scenarios. Since the dynamics of such systems will have inherent uncertainty (and the available environmental sensors like radar, lidar, and computer vision have high levels of noise and associated uncertainty), solutions will require statistical modeling and computation. To this end, this text discusses a framework for formulating these types of problems as partially observable Markov decision processes (POMDPs). The book proposes solving POMDPs by using approximate dynamic programming. The content of the book is divided into two main parts. Part I outlines the basic theory, and Part II contains several chapters that detail nontrivial applications of the theory to solve several problems that are currently of interest. Kochenderfer is the primary author of Part I, but each chapter in Part II is written by a different set of authors from different subject domains, providing a compelling argument that POMDPs are a suitable modeling and solution technique for a variety of different applications. This book is a timely and well-written introduction to modeling complex decision and control problems using the POMDP framework. The author has done an admirable job of bringing together various approximation techniques that have been developed to make the solution of this class of problems tractable. Since the techniques introduced in the book can be applied to a wide variety of problems that include self-driving and self-flying vehicles, the book is a welcome addition to the literature.

Optimum inspection and maintenance policies for corroded structures using partially observable Markov decision processes and stochastic, physically based models

Stochastic control methods have a history of implementation in risk management and life-cycle cost procedures for civil engineering structures. The synergy of stochastic control methods and Bayesian principles can result in Partially Observable Markov Decision Processes (POMDPs) that allow consideration of uncertainty within the entire domain of the model and expand available policy options compared to other state-of-the art methods. The superior attributes of POMDPs enable optimum decisions which are based on the belief space or otherwise only on the best knowledge that a decision-maker can have at each time. In this work the effort is mostly based in modeling and solving the problem of finding optimal policies for the maintenance and management of aging structures through a POMDP framework with large state spaces that can adequately and sufficiently describe real-life problems. In order to form the POMDP framework, stochastic, physically based models can be used and their connection to the control process is explained in detail. Specific examples of a corroded existing structure are presented, based on non-stationary POMDPs, for both infinite and finite horizon cases with 332 and 14,009 states respectively. Results from both cases are compared and discussed and the capabilities of the method become apparent.

Addressing structural and observational uncertainty in resource management

Most natural resource management and conservation problems are plagued with high levels of uncertainties, which make good decision making difficult. Although some kinds of uncertainties are easily incorporated into decision making, two types of uncertainty present more formidable difficulties. The first, structural uncertainty, represents our imperfect knowledge about how a managed system behaves. The second, observational uncertainty, arises because the state of the system must be inferred from imperfect monitoring systems. The former type of uncertainty has been addressed in ecology using Adaptive Management (AM) and the latter using the Partially Observable Markov Decision Processes (POMDP) framework. Here we present a unifying framework that extends standard POMDPs and encompasses both standard POMDPs and AM. The approach allows any system variable to be observed or not observed and uses any relevant observed variable to update beliefs about unknown variables and parameters. This extends standard AM, which only uses realizations of the state variable to update beliefs and extends standard POMDP by allowing more general stochastic dependence among the observable variables and the state variables. This framework enables both structural and observational uncertainty to be simultaneously modeled. We illustrate the features of the extended POMDP framework with an example.

UAV Path Planning in a Dynamic Environment via Partially Observable Markov Decision Process

A path-planning algorithm to guide unmanned aerial vehicles (UAVs) for tracking multiple ground targets based on the theory of partially observable Markov decision processes (POMDPs) is presented. A variety of features of interest are shown to be easy to incorporate into the framework by plugging in the appropriate models, which demonstrates the power and flexibility of the POMDP framework. Specifically, it is shown how to incorporate the following features by appropriately formulating the POMDP action space, transition law, and objective function: 1) control UAVs with both forward acceleration and bank angle subject to constraints; 2) account for the effect of wind disturbance on UAVs; 3) avoid collisions between UAVs and obstacles and among UAVs; 4) track targets while evading threats; 5) track evasive targets; and 6) mitigate track swaps.

A POMDP Framework for Coordinated Guidance of Autonomous UAVs for Multitarget Tracking

This paper discusses the application of the theory of partially observable Markov decision processes (POMDPs) to the design of guidance algorithms for controlling the motion of unmanned aerial vehicles (UAVs) with onboard sensors to improve tracking of multiple ground targets. While POMDP problems are intractable to solve exactly, principled approximation methods can be devised based on the theory that characterizes optimal solutions. A new approximation method called nominal belief-state optimization (NBO), combined with other application-specific approximations and techniques within the POMDP framework, produces a practical design that coordinates the UAVs to achieve good long-term mean-squared-error tracking performance in the presence of occlusions and dynamic constraints. The flexibility of the design is demonstrated by extending the objective to reduce the probability of a track swap in ambiguous situations.

Planning treatment of ischemic heart disease with partially observable Markov decision processes

Diagnosis of a disease and its treatment are not separate, one-shot activities. Instead, they are very often dependent and interleaved over time. This is mostly due to uncertainty about the underlying disease, uncertainty associated with the response of a patient to the treatment and varying cost of different diagnostic (investigative) and treatment procedures. The framework of partially observable Markov decision processes (POMDPs) developed and used in the operations research, control theory and artificial intelligence communities is particularly suitable for modeling such a complex decision process. In this paper, we show how the POMDP framework can be used to model and solve the problem of the management of patients with ischemic heart disease (IHD), and demonstrate the modeling advantages of the framework over standard decision formalisms.