Partially Observable Markov Decision Process Framework Research Articles

Dynamic Optimization of Random Access in Deadline-Constrained Broadcasting

This paper considers dynamic optimization of random access in deadline-constrained broadcasting with frame-synchronized traffic. Under the non-retransmission setting, we define a dynamic control scheme that allows each active node to determine the transmission probability based on the local knowledge of current delivery urgency and contention intensity (i.e., the number of active nodes). For an idealized environment where the contention intensity is completely known, we develop a Markov Decision Process (MDP) framework, by which an optimal scheme for maximizing the timely delivery ratio (TDR) can be explicitly obtained. For a realistic environment where the contention intensity is incompletely known, we develop a Partially Observable MDP (POMDP) framework, by which an optimal scheme can only in theory be found. To overcome the infeasibility in obtaining an optimal or near-optimal scheme from the POMDP framework, we investigate the behaviors of the optimal scheme for extreme cases in the MDP framework, and leverage intuition gained from these behaviors together with an approximation on the contention intensity knowledge to propose a heuristic scheme for the realistic environment with TDR close to the maximum TDR in the idealized environment. We further generalize the heuristic scheme to support retransmissions. Numerical results are provided to validate our study.

A POMDP-Based Approach to Joint Antenna Selection and User Scheduling for Multi-User Massive MIMO Communication

We devise a partially observable Markov decision process (POMDP) based joint antenna selection and user scheduling (JASUS) policy for a massive MIMO base station, equipped with only a small number of RF chains, that serves a large number of users. The users are served at different time slots within a frame. Relying on partial CSI obtained from training between the selected antennas and the users, at the beginning of each frame, the BS assigns each user to a time slot in the frame and selects a subset of antennas to serve the users scheduled in each time slot. Assuming that the channels evolve according to a Markov process and relying on zero-forcing beamforming, we formulate our JASUS problem using a POMDP framework to devise a real-time decision-making policy that maximizes the expected long-term sum-rate. We rigorously prove that for positively correlated two-state channel models, the myopic policy provides the optimal solution to our POMDP-based JASUS problem for any number of RF chains and for any number of users. Based on this, we model the Rayleigh fading channels as first-order Gauss-Markov processes and devise a low-complexity myopic policy-based JASUS algorithm for massive MU-MIMO systems that only relies on partial CSI.

Planning for Automated Vehicles with Human Trust

Recent work has considered personalized route planning based on user profiles, but none of it accounts for human trust. We argue that human trust is an important factor to consider when planning routes for automated vehicles. This article presents a trust-based route-planning approach for automated vehicles. We formalize the human-vehicle interaction as a partially observable Markov decision process (POMDP) and model trust as a partially observable state variable of the POMDP, representing the human’s hidden mental state. We build data-driven models of human trust dynamics and takeover decisions, which are incorporated in the POMDP framework, using data collected from an online user study with 100 participants on the Amazon Mechanical Turk platform. We compute optimal routes for automated vehicles by solving optimal policies in the POMDP planning and evaluate the resulting routes via human subject experiments with 22 participants on a driving simulator. The experimental results show that participants taking the trust-based route generally reported more positive responses in the after-driving survey than those taking the baseline (trust-free) route. In addition, we analyze the trade-offs between multiple planning objectives (e.g., trust, distance, energy consumption) via multi-objective optimization of the POMDP. We also identify a set of open issues and implications for real-world deployment of the proposed approach in automated vehicles.

A POMDP-Based Antenna Selection for Massive MIMO Communication

We use a partially observable Markov decision process (POMDP) framework to design an optimal antenna selection policy for downlink transmit beamforming at a multi-antenna base station (BS) equipped with only a limited number of RF chains. Assuming that the channel state evolves according to a finite-state Markov process and that only the channel coefficients which correspond to previously selected antennas, are available at the BS, we use the POMDP framework for antenna selection with the aim to maximize the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">long-term expected downlink data rate</i> . To avoid the high computational complexity of the value iteration algorithm, we focus on the myopic policy and prove that <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">in the case of positively correlated two-state Markov model for the channel over each antenna, the myopic policy is optimal for antenna selection for any number of RF chains</i> . Based on this finding, for general fading channels, we propose to quantize each channel into two levels and apply the myopic policy for antenna selection. Our simulation results show that using this two-state coarse channel quantization for antenna selection results in only a small loss in performance, as compared to the antenna selection technique which uses full channel state information without quantization.

PALO bounds for reinforcement learning in partially observable stochastic games

A partially observable stochastic game (POSG) is a general model for multiagent decision making under uncertainty. Perkins’ Monte Carlo exploring starts for partially observable Markov decision process (POMDP) (MCES-P) integrates Monte Carlo exploring starts (MCES) into a local search of the policy space to offer an elegant template for model-free reinforcement learning in POSGs. However, multiagent reinforcement learning in POSGs is tremendously more complex than in single agent settings due to the heterogeneity of agents and discrepancy of their goals. In this article, we generalize reinforcement learning under partial observability to self-interested and cooperative multiagent settings under the POSG umbrella. We present three new templates for multiagent reinforcement learning in POSGs. MCES for interactive POMDP (MCESIP) extends MCES-P by maintaining predictions of the other agent’s actions based on dynamic beliefs over models. MCES for multiagent POMDP (MCES-MP) generalizes MCES-P to the canonical multiagent POMDP framework, with a single policy mapping joint observations of all agents to joint actions. Finally, MCES for factored-reward multiagent POMDP (MCES-FMP) has each agent individually mapping joint observations to their own action. We use probabilistic approximate locally optimal (PALO) bounds to analyze sample complexity, thereby instantiating these templates to PALO learning. We promote sample efficiency by including a policy space pruning technique and evaluate the approaches on six benchmark domains as well as compare with the state-of-the-art techniques, which demonstrates that MCES-IP and MCES-FMP yield improved policies with fewer samples compared to the previous baselines.

Modeling of UAV path planning based on IMM under POMDP framework

In order to enhance the capability of tracking targets autonomously of unmanned aerial vehicle (UAV), the partially observable Markov decision process (POMDP) model for UAV path planning is established based on the POMDP framework. The elements of the POMDP model are analyzed and described. The state transfer law in the model can be described by the method of interactive multiple model (IMM) due to the diversity of the target motion law, which is used to switch the motion model to accommodate target maneuvers, and hence improving the tracking accuracy. The simulation results show that the model can achieve efficient planning for the UAV route, and effective tracking for the target. Furthermore, the path planned by this model is more reasonable and efficient than that by using the single state transition law.

A POMDP Framework to Find Optimal Inspection and Maintenance Policies via Availability and Profit Maximization for Manufacturing Systems

Maintenance can be the factor of either increasing or decreasing system's availability, so it is valuable work to evaluate a maintenance policy from cost and availability point of view, simultaneously and according to decision maker's priorities. This study proposes a Partially Observable Markov Decision Process (POMDP) framework for a partially observable and stochastically deteriorating system in which inspection and maintenance optimal policies of Condition Based Maintenance (CBM) must be determined to maximize the average profit and availability of the system simultaneously. A recent exact method named Accelerated Vector Pruning method (AVP) and some other popular estimating and exact methods are applied and compared in solving such problems.

A POMDP framework to find Optimal Policy in Sustainable Maintenance

The increasing importance of these two subjects, maintenance and cleaner environment, beside the relations between them, encourages us to investigate a mathematical Markovian model for Condition Based Maintenance problem while considering environmental effects. In this paper, the problem of proposing maintenance optimal policy for a partially observable, stochastically deteriorating system is studied, in order to maximize the average profit of the system with consideration of sustainability aspects. The modeling of this Condition Based Sustainable Maintenance (CBSM) problem is done using mathematical methods such as Partially Observable Markov Decision Process (POMDP) and Bayesian theory. A new exact method named Accelerated Vector Pruning method and other popular estimating and exact methods are applied and compared in solving the presented CBSM model and several managerial conclusions were obtained.

A POMDP framework for integrated scheduling of infrastructure maintenance and inspection

This work presents an optimization scheme for maintenance and inspection scheduling of the infrastructure system whose states are nearly impossible or prohibitively expensive to estimate or measure online. The suggested framework describes state transition under the observation uncertainty as Partially Observable Markov Decision Process (POMDP) and can integrate heterogeneous scheduling jobs including maintenance, inspection, and sensor installation within a single model. The proposed approach performs survival analysis to obtain time-variant transition probabilities. A POMDP problem is then formulated via state augmentation. The resulting large-scale POMDP is solved by an approximate point-based solver. We exploit the idea of receding horizon control to the POMDP framework as a feedback rule for the online evaluation. Water distribution pipeline is analyzed as an illustrative example, and the results indicate that the proposed POMDP framework can improve the overall cost for maintenance tasks and thus the system’s sustainability.

Optimal Radio Frequency Energy Harvesting With Limited Energy Arrival Knowledge

We develop optimal sleeping and harvesting policies for radio frequency (RF) energy harvesting devices, formalizing the following intuition: when the ambient RF energy is low, devices consume more energy being awake than what can be harvested and should enter sleep mode; when the ambient RF energy is high, on the other hand, it is essential to wake up and harvest. Toward this end, we consider a scenario with intermittent energy arrivals described by a two-state Gilbert–Elliott Markov chain model. The challenge is that the state of the Markov chain can only be observed during the harvesting action, and not while in sleep mode. Two scenarios are studied under this model. In the first scenario, we assume that the transition probabilities of the Markov chain are known and formulate the problem as a partially observable Markov decision process (POMDP). We prove that the optimal policy has a threshold structure and derive the optimal decision parameters. In the practical scenario where the ratio between the reward and the penalty is neither too large nor too small, the POMDP framework and the threshold-based optimal policies are very useful for finding non-trivial optimal sleeping times. In the second scenario, we assume that the Markov chain parameters are unknown and formulate the problem as a Bayesian adaptive POMDP and propose a heuristic posterior sampling algorithm to reduce the computational complexity. The performance of our approaches is demonstrated via numerical examples.

A POMDP framework for forwarding mechanism in named data networking

As a novel networking paradigm adapted to the shift of Internet usage, the named data networking (NDN) introduces both opportunities and challenges. The pervasiveness of NDN’s built-in caching plays an important role in reducing redundant content transmission. However, the volatile availability of cached items calls for an efficient forwarding strategy to ensure data delivery performance. In this paper, the partially observable Markov decision process (POMDP) is exploited to the design of NDN request forwarding mechanism based upon the key concept of event. Since the exact POMDP solution is intractable, a simulation-based approximation algorithm with reduced complexity yet comparable performance is developed as a solution tool for our POMDP abstract. By adopting the policy resulted from our POMDP-based approach for target interface selection, minimal network cost can be achieved. Simulation results demonstrate that the proposed scheme outperforms the existing forwarding strategies in NDN.

Training Beam Sequence Design for Millimeter-Wave MIMO Systems: A POMDP Framework

In this paper, adaptive training beam sequence design for efficient channel estimation in large millimeter-wave(mmWave) multiple-input multiple-output (MIMO) channels is considered. By exploiting the sparsity in large mmWave MIMO channels and imposing a Markovian random walk assumption on the movement of the receiver and reflection clusters, the adaptive training beam sequence design and channel estimation problem is formulated as a partially observableMarkov decision process (POMDP) problem that finds non-zero bins in a two-dimensional grid. Under the proposed POMDP framework, optimal and suboptimal adaptive training beam sequence design policies are derived. Furthermore, a very fast suboptimal greedy algorithm is developed based on a newly proposed reduced sufficient statistic to make the computational complexity of the proposed algorithm low to a level for practical implementation. Numerical results are provided to evaluate the performance of the proposed training beam design method. Numerical results show that the proposed training beam sequence design algorithms yield good performance.

Joint spectrum sensing and access for stable dynamic spectrum aggregation

Spectrum aggregation is an emerging technology to satisfy the data rate requirement of broadband services for next-generation wireless communication systems. In dynamic spectrum environment, in which the spectrum availability is time-varying, it is quite challenging to maintain the stability of spectrum aggregation. In this paper, we investigate the spectrum sensing and access schemes to minimize the times of channel switching for achieving stable dynamic spectrum aggregation, taking into consideration the hardware limitations of spectrum sensing and aggregation capability. We develop an analytical framework for the joint spectrum sensing and access problem based on partially observable Markov decision process (POMDP). Especially, we derive the reward function by estimation of the stability of different spectrum sensing and access strategies. Based on the POMDP framework, we propose a rollout-based suboptimal spectrum sensing and access scheme which approximates the value function of POMDP, and propose a differential training method to improve its robustness. It is proved that the rollout policy achieves performance improvement over the basis heuristics. The simulation results show that the proposed POMDP-based spectrum sensing and access scheme improves the system stability significantly and achieves near-optimal performance with a much lower complexity.

Conversational system for information navigation based on POMDP with user focus tracking

We address a spoken dialogue system which conducts information navigation in a style of small talk. The system uses Web news articles as an information source, and the user can receive information about the news of the day through interaction. The goal and procedure of this kind of dialogue are not well defined. An empirical approach based on a partially observable Markov decision process (POMDP) has recently been widely used for dialogue management, but it assumes a definite task goal and information slots, which does not hold in our application system. In this work, we formulate the problem of dialogue management as a selection of modules and optimize it with POMDP by tracking the dialogue state and focus of attention. The POMDP-based dialogue manager receives a user intention that is classified by a spoken language understanding (SLU) component based on logistic regression (LR). The manager also receives a user focus that is detected by the SLU component based on conditional random fields (CRFs). These dialogue states are used for selecting appropriate modules by policy function, which is optimized by reinforcement learning. The reward function is defined by the quality of interaction to encourage long interaction of information navigation with users. The module which responds to user queries is based on a similarity of predicate-argument (P-A) structures that are automatically defined from a domain corpus. It allows for flexible response generation even if the system cannot find exact matching information to the user query. The system also proactively presents information by following the user focus and retrieving a news article based on the similarity measure even if the user does not make any utterance. Experimental evaluations with real dialogue sessions demonstrate that the proposed system outperformed the conventional rule-based system in terms of dialogue state tracking and action selection. Effect of focus detection in the POMDP framework is also confirmed.

Monitoring as a partially observable decision problem

Monitoring is an important and costly activity in resource management problems such as containing invasive species, protecting endangered species, preventing soil erosion, and regulating contracts for environmental services. Recent studies have viewed optimal monitoring as a Partially Observable Markov Decision Process (POMDP), which provides a framework for sequential decision making under stochastic resource dynamics and uncertainty about the resource state. We present an overview of the POMDP framework and its applications to resource monitoring. We discuss the concept of the information content provided by monitoring systems and illustrate how information content affects optimal monitoring strategies. Finally, we demonstrate how the timing of monitoring in relation to resource treatment and transition can have substantial effects on optimal monitoring strategies.

Bayesian update of dialogue state: A POMDP framework for spoken dialogue systems

This paper describes a statistically motivated framework for performing real-time dialogue state updates and policy learning in a spoken dialogue system. The framework is based on the partially observable Markov decision process (POMDP), which provides a well-founded, statistical model of spoken dialogue management. However, exact belief state updates in a POMDP model are computationally intractable so approximate methods must be used. This paper presents a tractable method based on the loopy belief propagation algorithm. Various simplifications are made, which improve the efficiency significantly compared to the original algorithm as well as compared to other POMDP-based dialogue state updating approaches. A second contribution of this paper is a method for learning in spoken dialogue systems which uses a component-based policy with the episodic Natural Actor Critic algorithm. The framework proposed in this paper was tested on both simulations and in a user trial. Both indicated that using Bayesian updates of the dialogue state significantly outperforms traditional definitions of the dialogue state. Policy learning worked effectively and the learned policy outperformed all others on simulations. In user trials the learned policy was also competitive, although its optimality was less conclusive. Overall, the Bayesian update of dialogue state framework was shown to be a feasible and effective approach to building real-world POMDP-based dialogue systems.

Multivariate Bayesian process control for a finite production run

Most industrial products and processes are characterized by several, typically correlated measurable variables, which jointly describe the product or process quality. Various control charts such as Hotelling’s T2, EWMA and CUSUM charts have been developed for multivariate quality control, where the values of the chart parameters, namely the sample size, sampling interval and the control limits are determined to satisfy given economic and/or statistical requirements. It is well known that this traditional non-Bayesian approach to a control chart design is not optimal, but very few results regarding the form of the optimal Bayesian control policy have appeared in the literature, all limited to a univariate chart design. In this paper, we consider a multivariate Bayesian process mean control problem for a finite production run under the assumption that the observations are values of independent, normally distributed vectors of random variables. The problem is formulated in the POMDP (partially observable Markov decision process) framework and the objective is to determine a control policy minimizing the total expected cost. It is proved that under standard operating and cost assumptions the control limit policy is optimal. Cost comparisons with the benchmark chi-squared chart and the MEWMA chart show that the Bayesian chart is highly cost effective, the savings are larger for smaller values of the critical Mahalanobis distance between the in-control and out-of-control process mean.

Prioritizing Point-Based POMDP Solvers

Recent scaling up of partially observable Markov decision process (POMDP) solvers toward realistic applications is largely due to point-based methods that quickly converge to an approximate solution for medium-sized domains. These algorithms compute a value function for a finite reachable set of belief points, using backup operations. Point-based algorithms differ on the selection of the set of belief points and on the order by which backup operations are executed on the selected belief points. We first show how current algorithms execute a large number of backups that can be removed without reducing the quality of the value function. We demonstrate that the ordering of backup operations on a predefined set of belief points is important. In the simpler domain of MDP solvers, prioritizing the order of equivalent backup operations on states is known to speed up convergence. We generalize the notion of prioritized backups to the POMDP framework, showing how existing algorithms can be improved by prioritizing backups. We also present a new algorithm, which is the prioritized value iteration, and show empirically that it outperforms current point-based algorithms. Finally, a new empirical evaluation measure (in addition to the standard runtime comparison), which is based on the number of atomic operations and the number of belief points, is proposed in order to provide more accurate benchmark comparisons.

Decentralized cognitive MAC for opportunistic spectrum access in ad hoc networks: A POMDP framework

We propose decentralized cognitive MAC protocols that allow secondary users to independently search for spectrum opportunities without a central coordinator or a dedicated communication channel. Recognizing hardware and energy constraints, we assume that a secondary user may not be able to perform full-spectrum sensing or may not be willing to monitor the spectrum when it has no data to transmit. We develop an analytical framework for opportunistic spectrum access based on the theory of partially observable Markov decision process (POMDP). This decision-theoretic approach integrates the design of spectrum access protocols at the MAC layer with spectrum sensing at the physical layer and traffic statistics determined by the application layer of the primary network. It also allows easy incorporation of spectrum sensing error and constraint on the probability of colliding with the primary users. Under this POMDP framework, we propose cognitive MAC protocols that optimize the performance of secondary users while limiting the interference perceived by primary users. A suboptimal strategy with reduced complexity yet comparable performance is developed. Without additional control message exchange between the secondary transmitter and receiver, the proposed decentralized protocols ensure synchronous hopping in the spectrum between the transmitter and the receiver in the presence of collisions and spectrum sensing errors