Partially Observable Markov Decision Process Research Articles

A UAV Navigation Approach Based on Deep Reinforcement Learning in Large Cluttered 3D Environments

This paper is concerned with the problem of using deep reinforcement learning methods to enable safe navigation of UAVs in unknown large-scale environments containing a large number of obstacles. The ability of UAV to navigate autonomously is a precondition for performing disaster rescue and logistics deliveries. In this paper, the perception-constrained navigation of UAV is modeled as a partially observable Markov decision process (POMDP), and a fast recurrent stochastic valued gradient algorithm based on the actot-critic framework is designed for online solution. Compared with traditional SLAM-based and reactive obstacle avoidance-based approaches, the proposed navigation method can map the raw sensing information into navigation signals, and the learned policy deployed to the UAV can avoid the memory occupation and computational consumption required for real-time map building. Through experiments in a newly designed simulation environment, it is demonstrated that the proposed algorithm can enable the UAV to navigate safely in unknown cluttered large-scale environments and outperform the state-of-the-art DRL algorithm in terms of performance.

A POMDP-Based Approach to Joint Antenna Selection and User Scheduling for Multi-User Massive MIMO Communication

We devise a partially observable Markov decision process (POMDP) based joint antenna selection and user scheduling (JASUS) policy for a massive MIMO base station, equipped with only a small number of RF chains, that serves a large number of users. The users are served at different time slots within a frame. Relying on partial CSI obtained from training between the selected antennas and the users, at the beginning of each frame, the BS assigns each user to a time slot in the frame and selects a subset of antennas to serve the users scheduled in each time slot. Assuming that the channels evolve according to a Markov process and relying on zero-forcing beamforming, we formulate our JASUS problem using a POMDP framework to devise a real-time decision-making policy that maximizes the expected long-term sum-rate. We rigorously prove that for positively correlated two-state channel models, the myopic policy provides the optimal solution to our POMDP-based JASUS problem for any number of RF chains and for any number of users. Based on this, we model the Rayleigh fading channels as first-order Gauss-Markov processes and devise a low-complexity myopic policy-based JASUS algorithm for massive MU-MIMO systems that only relies on partial CSI.

Data Analytics of an Information System Based on a Markov Decision Process and a Partially Observable Markov Decision Process

Data analytics of an information system is conducted based on a Markov decision process (MDP) and a partially observable Markov decision process (POMDP) in this paper. Data analytics over a finite planning horizon and an infinite planning horizon for a discounted MDP is performed, respectively. Value iteration (VI), policy iteration (PI), and Q-learning are utilized in the data analytics for a discounted MDP over an infinite planning horizon to evaluate the validity of the MDP model. The optimal policy to minimize the total expected cost of states of the information system is obtained based on the MDP. In the analytics for a discounted POMDP over an infinite planning horizon of the information system, the effects of various parameters on the total expected cost of the information system are studied.

Predictive trajectory planning for autonomous vehicles at intersections using reinforcement learning

In this work we put forward a predictive trajectory planning framework to help autonomous vehicles plan future trajectories. We develop a partially observable Markov decision process (POMDP) to model this sequential decision making problem, and a deep reinforcement learning solution methodology to learn high-quality policies. The POMDP model utilizes driving scenarios, condensed into graphs, as inputs. More specifically, an input graph contains information on the history trajectory of the subject vehicle, predicted trajectories of other agents in the scene (e.g., other vehicles, pedestrians, and cyclists), as well as predicted risk levels posed by surrounding vehicles to devise safe, comfortable, and energy-efficient trajectories for the subject vehicle to follow. In order to obtain sufficient driving scenarios to use as training data, we propose a simulation framework to generate socially acceptable driving scenarios using a real world autonomous vehicle dataset. The simulation framework utilizes Bayesian Gaussian mixture models to learn trajectory patterns of different agent types, and Gibbs sampling to ensure that the distribution of simulated scenarios matches that of the real-world dataset collected by an autonomous fleet. We evaluate the proposed work in two complex urban driving environments: a non-signalized T-junction and a non-signalized lane merge intersection. Both environments provide vastly more complex driving scenarios compared to a highway driving environment, which has been mostly the focus of previous studies. The framework demonstrates promising performance for planning horizons as long as five seconds. We compare safety, comfort, and energy efficiency of the planned trajectories against human-driven trajectories in both experimental driving environments, and demonstrate that it outperforms human-driven trajectories in a statistically significant fashion in all aspects.

A Markovian model for the spread of the SARS-CoV-2 virus

We propose a Markovian stochastic approach to model the spread of a SARS-CoV-2-like infection within a closed group of humans. The model takes the form of a Partially Observable Markov Decision Process (POMDP), whose states are given by the number of subjects in different health conditions. The model also exposes the different parameters that have an impact on the spread of the disease and the various decision variables that can be used to control it (e.g, social distancing, number of tests administered to single out infected subjects). The model describes the stochastic phenomena that underlie the spread of the epidemic and captures, in the form of deterministic parameters, some fundamental limitations in the availability of resources (hospital beds and test swabs). The model lends itself to different uses. For a given control policy, it is possible to verify if it satisfies an analytical property on the stochastic evolution of the state (e.g., to compute probability that the hospital beds will reach a fill level, or that a specified percentage of the population will die). If the control policy is not given, it is possible to apply POMDP techniques to identify an optimal control policy that fulfils some specified probabilistic goals. Whilst the paper primarily aims at the model description, we show with numeric examples some of its potential applications.

Condition-based pavement management systems accounting for model uncertainty and facility heterogeneity with belief updates

Due to the tighter budget for pavement management, schedules of inspection activities should be jointly optimized with the maintenance and reconstruction (M&R) plans for pavement systems. Conducting inspections every year is unnecessary and will decrease the budget for M&R activities, while infrequent inspections may lead to suboptimal M&R planning due to the lack of accurate information. This paper presents a methodology for jointly optimizing the inspection scheduling and M&R planning for pavement systems, considering model uncertainty and facility-specific heterogeneity. The problem is defined as a Partially Observable Markov Decision Process (POMDP) model, accounting for the tradeoff between the information value and inspection costs. Moreover, a statistical learning method is used to update the prediction of pavement conditions using the collected inspection data and, eventually, improve the condition-based decisions. This “belief update” process can gradually reduce the model uncertainty as the dataset size increases. We demonstrate the proposed stochastic optimization framework through a numerical example with a system of fifty heterogenous pavement facilities under a combined budget for inspection and M&R activities. Several managerial insights and implications are discussed. For example, the optimal inspection frequencies are less sensitive to the budget; and the agency should perform fewer reconstructions and more rehabilitations when the budget is limited.

Improving anti-jamming decision-making strategies for cognitive radar via multi-agent deep reinforcement learning

Most of the existing anti-jamming decision-making methods overly rely on the subjective experience of radar operators. However, due to the rapid development of cognitive radar and modern electronic warfare, conventional anti-jamming decision-making methods can no longer adapt to the complex and changing electromagnetic environment. The advent of deep reinforcement learning (DRL) provides a new attractive solution for this issue. In this paper, an adversarial anti-jamming decision-making network for cognitive radar via multi-agent deep reinforcement learning (MDRL) is proposed, which has good self-learning ability and can meet the requirements of intelligent, dynamic and real-time in modern electronic warfare. Since competitive decision-makers are considered and these two confrontational sides are not able to obtain the completely accurate information of each other, the environment model is specifically constructed as a partially observable Markov decision process (POMDP). Then, a decision-making network is designed based on deterministic deep deterministic policy gradient (DDPG) algorithm to explore the competition between cognitive radar and smart jammer. In order to overcome the environment non-stationarity, the decision-making network is trained and tested in a special MDRL framework. The experimental results demonstrate that the proposed method is effective in anti-jamming decision-making system of cognitive radar. Furthermore, the two confrontational sides show high decision-making ability and perform well in the adversarial scenario by comparing with other training policies, which demonstrate that confrontational training with powerful opponents can improve the intelligence level of all agents.

Delay-Oriented Scheduling in 5G Downlink Wireless Networks Based on Reinforcement Learning With Partial Observations

5G wireless networks are expected to satisfy different delay requirements of various traffics by network resource scheduling. Existing scheduling methods perform poorly in practice due to their unrealistic assumption on the access to the full channel state information (CSI) or the explicit mathematical expression of network delay. In this paper, we consider the delay-oriented packet scheduling problem in multi-cell 5G downlink networks with multiple users and traffic types (e.g., FTP, VoIP and video streaming), and formulate it as a partially observable Markov decision process (POMDP). We design a delay-oriented downlink scheduling framework based on deep reinforcement learning (DRL) to autonomously schedule the active traffic flows without the full channel information. Furthermore, a recurrent proximal policy optimization (RPPO) algorithm is proposed to perceive the underlying state and accelerate learning under different time granularities, with the policy gradient theorem under POMDP strictly proved. By incorporating the future traffic information provided by a proposed spatial-temporal prediction algorithm, RPPO can balance the load and achieve lower delay in real-time multi-cell multi-user scenarios. Results of extensive experiments on a realistic 5G simulator demonstrate that our framework significantly outperforms existing approaches in terms of both tail delay and average delay for up to 48% and 41.7%, respectively.

POMDP Planning Under Object Composition Uncertainty: Application to Robotic Manipulation

Manipulating unknown objects in a cluttered environment is difficult because segmentation of the scene into objects, that is, object composition, is uncertain. Due to the uncertainty, prior work has either identified the “best” object composition and decided on manipulation actions accordingly or tried to greedily gather information about the “best” object composition. We instead, first, use different possible object compositions in planning, second, utilize object composition information provided by robot actions, third, consider the effect of competing object hypotheses on the desired task. We cast the manipulation planning problem as a partially observable Markov decision process (POMDP) that plans over possible object composition hypotheses. The POMDP chooses the action that maximizes long-term expected task-specific utility, and while doing so, considers informative actions and the effect of different object hypotheses on succeeding in the task. In simulation and physical robotic experiments, a probabilistic approach outperforms using the most likely object composition, and long term planning outperforms greedy decision making.

Deep reinforcement learning for optimal well control in subsurface systems with uncertain geology

A general control policy framework based on deep reinforcement learning (DRL) is introduced for closed-loop decision making in subsurface flow settings. Traditional closed-loop modeling workflows in this context involve the repeated application of data assimilation/history matching and robust optimization steps. Data assimilation can be particularly challenging in cases where both the geological style (scenario) and individual model realizations are uncertain. The closed-loop reservoir management (CLRM) problem is formulated here as a partially observable Markov decision process (POMDP), with the associated optimization problem solved using a proximal policy optimization algorithm. This provides a control policy that instantaneously maps flow data observed at wells (as are available in practice) to optimal well pressure settings. The policy is represented by a temporal convolution and gated transformer blocks. Training is performed in a preprocessing step with an ensemble of prior geological models, which can be drawn from multiple geological scenarios. Example cases involving the production of oil via water injection, with both 2D and 3D geological models, are presented. The DRL-based methodology is shown to result in an increase in NPV of 15% (for the 2D cases) and 33% (3D cases) relative to robust optimization over prior models, and to an improvement of 2 - 7% in NPV relative to traditional CLRM. The solutions from the control policy are found to be comparable to those from deterministic optimization, in which the geological model is assumed to be known, even when multiple geological scenarios are considered. The control policy approach results in a 76% decrease in computational cost relative to traditional CLRM with the algorithms and parameter settings considered in this work.

A Discrete Partially Observable Markov Decision Process Model for the Maintenance Optimization of Oil and Gas Pipelines

Corrosion is one of the major causes of failure in pipelines for transporting oil and gas products. To mitigate the impact of this problem, organizations perform different maintenance operations, including detecting corrosion, determining corrosion growth, and implementing optimal maintenance policies. This paper proposes a partially observable Markov decision process (POMDP) model for optimizing maintenance based on the corrosion progress, which is monitored by an inline inspection to assess the extent of pipeline corrosion. The states are defined by dividing the deterioration range equally, whereas the actions are determined based on the specific states and pipeline attributes. Monte Carlo simulation and a pure birth Markov process method are used for computing the transition matrix. The cost of maintenance and failure are considered when calculating the rewards. The inline inspection methods and tool measurement errors may cause reading distortion, which is used to formulate the observations and the observation function. The model is demonstrated with two numerical examples constructed based on problems and parameters in the literature. The result shows that the proposed model performs well with the added advantage of integrating measurement errors and recommending actions for multiple-state situations. Overall, this discrete model can serve the maintenance decision-making process by better representing the stochastic features.

The Intelligent Prospector v1.0: geoscientific model development and prediction by sequential data acquisition planning with application to mineral exploration

Abstract. Geoscientific models are based on geoscientific data; hence, building better models, in the sense of attaining better predictions, often means acquiring additional data. In decision theory, questions of what additional data are expected to best improve predictions and decisions is within the realm of value of information and Bayesian optimal survey design. However, these approaches often evaluate the optimality of one additional data acquisition campaign at a time. In many real settings, certainly in those related to the exploration of Earth resources, a large sequence of data acquisition campaigns possibly needs to be planned. Geoscientific data acquisition can be expensive and time-consuming, requiring effective measurement campaign planning to optimally allocate resources. Each measurement in a data acquisition sequence has the potential to inform where best to take the following measurements; however, directly optimizing a closed-loop measurement sequence requires solving an intractable combinatoric search problem. In this work, we formulate the sequential geoscientific data acquisition problem as a partially observable Markov decision process (POMDP). We then present methodologies to solve the sequential problem using Monte Carlo planning methods. We demonstrate the effectiveness of the proposed approach on a simple 2D synthetic exploration problem. Tests show that the proposed sequential approach is significantly more effective at reducing uncertainty than conventional methods. Although our approach is discussed in the context of mineral resource exploration, it likely has bearing on other types of geoscientific model questions.

Task-guided IRL in POMDPs that scales

In inverse reinforcement learning (IRL), a learning agent infers a reward function encoding the underlying task using demonstrations from experts. However, many existing IRL techniques make the often unrealistic assumption that the agent has access to full information about the environment. We remove this assumption by developing an algorithm for IRL in partially observable Markov decision processes (POMDPs). We address two limitations of existing IRL techniques. First, they require an excessive amount of data due to the information asymmetry between the expert and the learner. Second, most of these IRL techniques require solving the computationally intractable forward problem—computing an optimal policy given a reward function—in POMDPs. The developed algorithm reduces the information asymmetry while increasing the data efficiency by incorporating task specifications expressed in temporal logic into IRL. Such specifications may be interpreted as side information available to the learner a priori in addition to the demonstrations. Further, the algorithm avoids a common source of algorithmic complexity by building on causal entropy as the measure of the likelihood of the demonstrations as opposed to entropy. Nevertheless, the resulting problem is nonconvex due to the so-called forward problem. We solve the intrinsic nonconvexity of the forward problem in a scalable manner through a sequential linear programming scheme that guarantees to converge to a locally optimal policy. In a series of examples, including experiments in a high-fidelity Unity simulator, we demonstrate that even with a limited amount of data and POMDPs with tens of thousands of states, our algorithm learns reward functions and policies that satisfy the task while inducing similar behavior to the expert by leveraging the provided side information.

Predictive Modelling of a Honeypot System Based on a Markov Decision Process and a Partially Observable Markov Decision Process

A honeypot is used to attract and monitor attacker activities and capture valuable information that can be used to help practice good cybersecurity. Predictive modelling of a honeypot system based on a Markov decision process (MDP) and a partially observable Markov decision process (POMDP) is performed in this paper. Analyses over a finite planning horizon and an infinite planning horizon for a discounted MDP are respectively conducted. Four methods, including value iteration (VI), policy iteration (PI), linear programming (LP), and Q-learning, are used in the analyses over an infinite planning horizon for the discounted MDP. The results of the various methods are compared to evaluate the validity of the created MDP model and the parameters in the model. The optimal policy to maximise the total expected reward of the states of the honeypot system is achieved, based on the MDP model employed. In the modelling over an infinite planning horizon for the discounted POMDP of the honeypot system, the effects of the observation probability of receiving commands, the probability of attacking the honeypot, the probability of the honeypot being disclosed, and transition rewards on the total expected reward of the honeypot system are studied.

Welfare Maximization Algorithm for Solving Budget-Constrained Multi-Component POMDPs

Partially Observable Markov Decision Processes (POMDPs) provide an efficient way to model real-world sequential decision making processes. Motivated by the problem of maintenance and inspection of a group of infrastructure components with independent dynamics, this letter presents an algorithm to find the optimal policy for a multi-component budget-constrained POMDP. We first introduce a budgeted-POMDP model (b-POMDP) which enables us to find the optimal policy for a POMDP while adhering to budget constraints. Next, we prove that the value function or maximal collected reward for a special class of b-POMDPs is a concave function of the budget for the finite horizon case. Our second contribution is an algorithm to calculate the optimal policy for a multi-component budget-constrained POMDP by finding the optimal budget split among the individual component POMDPs. The optimal budget split is posed as a welfare maximization problem and the solution is computed by exploiting the concavity of the value function. We illustrate the effectiveness of the proposed algorithm by proposing a maintenance and inspection policy for a group of real-world infrastructure components with different deterioration dynamics, inspection and maintenance costs. We show that the proposed algorithm vastly outperforms the policies currently used in practice.

Deep Reinforcement Learning-Based Air-to-Air Combat Maneuver Generation in a Realistic Environment

Artificial intelligence is becoming increasingly important in the air combat domain. Most air combat research now assumes that all aircraft information is known. In practical applications, however, some aircraft information, such as their position, attitude, velocity, etc., can be incorrect or impossible to obtain due to realistic limitations and sensor errors. In this paper, we propose a deep reinforcement learning-based framework for developing a model capable of performing within visual range (WVR) air-to-air combat under the conditions of a partially observable Markov decision process (POMDP) with insufficient information. To deal robustly with such a situation, we use recurrent neural networks and apply a soft actor-critic (SAC) algorithm to cope effectively with realistic limitations and sensor errors. Additionally, to raise the efficiency and effectiveness of learning, we apply the curriculum learning technique to restrict the scope of exploration in state space. Finally, simulations and experiments show that the proposed techniques can deal with practical problems caused by sensor limitations and errors in a noisy environment while also being efficient and effective in reducing the training time for learning.

Personalized and Energy-Efficient Health Monitoring: A Reinforcement Learning Approach

We consider a network of controlled sensors that monitor the unknown health state of a patient. We assume that the health state process is a Markov chain with a transition matrix that is unknown to the controller. At each timestep, the controller chooses a subset of sensors to activate, which incurs an energy (i.e., battery) cost. Activating more sensors improves the estimation of the unknown state, which introduces an energy-accuracy tradeoff. Our goal is to minimize the combined energy and state misclassification costs over time. Activating sensors now also provides measurements that can be used to learn the model, improving future decisions. Therefore, the learning aspect is intertwined with the energy-accuracy tradeoff. While Reinforcement Learning (RL) is often used when the model is unknown, it cannot be directly applied in health monitoring since the controller does not know the (health) state. Therefore, the monitoring problem is a partially observable Markov decision process (POMDP) where the cost feedback is also only partially available since the misclassification cost is unknown. To overcome this difficulty, we propose a monitoring algorithm that combines RL for POMDPs and online estimation of the expected misclassification cost based on a Hidden Markov Model (HMM). We show empirically that our algorithm achieves comparable performance with a monitoring system that assumes a known transition matrix and quantizes the belief state. It also outperforms the model-based approach where the estimated transition matrix is used for value iteration. Thus, our algorithm can be useful in designing energy-efficient and personalized health monitoring systems.

Investigations on Speed Planning Algorithm and Trajectory Tracking Control of Intersection Scenarios Without Traffic Signs

Efficient intersection planning is one of the most challenging tasks for an autonomous vehicle at present. The degree of response to other traffic participants and whether the path tracking effect is good are the key aspects that determine the performance of planning algorithm for an autonomous vehicle. In order to deal with these problems, in this paper, the frame structure of POMDP (Partially Observable Markov Decision Processes) and the time distance coordinate system are combined to carry out the speed planning for the self-propelled vehicle. The speed planning designs a novel POMDP model to ensure effective prediction of behavioral intentions and decision-making at intersections with no traffic signs. Specifically, the method is considered both the driving position and the time range of arrival of the vehicle at the intersection in real-world situations. At the same time, the algorithm is projected the state information into the time distance coordinate system in the form of probability, and used the value iteration method to solve the optimal solution, so as to improve the adaptability of the algorithm to real-world scenarios. Furthermore, MPC control is added on this basis. Rational use of dynamic MPC and kinematic MPC can effectively reduce the tracking error, improve the stability of tracking, and carry out finer control when the speed frequently fluctuates.

D3GF: A Study on Optimal Defense Performance Evaluation of Drone-Type Moving Target Defense Through Game Theory

Based on the paradigm shift in modern warfare, ground forces conduct pilot operations of wireless unmanned maneuvering systems, such as tactical drones, in the form of manned and unmanned cooperative tactics after deploying the relevant systems in the battlefield. However, security considerations for relevant systems are limited to the scope of using only the existing end-to-end encryption and public key authentication modules, and no defense strategy to actively respond to specialized cyber-electronic warfare threats has been officially established. To drastically reduce both the potential attack surface and security vulnerabilities of drones employed in network-centric-warfare, a proactive defense technology that expires the effectiveness of attacks by avoiding invasive action at the target is expected to be essential. Accordingly, this paper proposes the concept of active moving-target-defense (MTD), an element of cyber deception that minimizes the rate of success of cyber-attacks while conversely maximizing both defense predominance and attack complexity asymmetrically, exclusive according to partially observable Markov decision process (POMDP)-based threat modeling that considers both the internal and external operation sequences of target drones. To optimally design the proposed drone-type MTD based on the Pareto frontier, we additionally advanced and simulated a drone-based defensive deception game framework (D3GF), which represents a general-sum combat framework reflecting decision logics such as the perfect Bayesian Nash equilibrium, stochastic Stackelberg, and partial signal game. This study was conducted to compare and calculate the efficiencies of the drone-type MTD’s deceptive defense, which had not been considered in prior studies, by unique environmental features inside and outside the drone. Furthermore, we conducted a detailed performance evaluation considering game metrics based on sensitivity analysis. Hereafter, the drone-type MTD will be extended into an actual active drone protection technology combining cyber flare-type avoidance strategies and cyber camouflage-type disarrangement strategies by expanding its optimization domain as a hypergame, while integrating it with drone decoy elements.

Blockchain Mining With Multiple Selfish Miners

This paper studies a fundamental problem regarding the security of blockchain PoW consensus on how the existence of multiple misbehaving miners influences the profitability of selfish mining. Each selfish miner maintains a private chain and makes it public opportunistically for acquiring more rewards incommensurate to his Hash power. We first establish a general Markov chain model to characterize the state transition of public and private chains for Basic Selfish Mining ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">BSM</i> ), and derive the stationary <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">profitable threshold</i> of Hash power in closed form. It reduces from 25% for a single attacker to below 21.48% for two symmetric attackers theoretically, and further reduces to around 10% with eight symmetric attackers experimentally. We next explore the profitable threshold when one of the attackers performs strategic mining based on Partially Observable Markov Decision Process (POMDP) that only half of the attributes pertinent to a mining state are observable to him. An online algorithm is presented to compute the nearly optimal policy efficiently despite the large state space and high dimensional belief space. The profitable threshold is much lower for the strategic attacker. Last, we formulate a simple model of absolute mining revenue that yields an interesting observation: selfish mining is never profitable at the first difficulty adjustment period, but relies on the reimbursement of stationary selfish mining gains in future periods. The delay till being profitable of an attacker increases with the decrease of his Hash power, making blockchain miners more cautious about performing selfish mining.