Partially Observable Markov Decision Process Research Articles

Partially observable Markov decision process-based optimal maintenance planning with time-dependent observations

The growing technological capability for real-time condition monitoring (CM) of industrial equipment has spurred significant interest in methods for optimal maintenance planning using CM signals. Existing approaches for maintenance policy development consider degradation to be either fully or partially observable. For the more general case of partial observability, it is usually assumed that the relationship between the underlying degradation process and the observed condition is time-invariant. In this paper, we address this major shortcoming by modeling observed CM signals through an underlying failure process wherein the linkage is time-dependent piecewise linear with jumps, and then utilizing a Partially Observed Markov Decision Process (POMDP) to determine the optimal maintenance strategy. We investigate the structure of the policy and show that, under certain conditions, a control-limit policy exists, i.e., a belief threshold exists beyond which the optimal action is to preventively maintain the unit. Finally, we present a case study based on battery resistance data and demonstrate that our modeling procedure offers a maintenance policy that is superior to those from other competing models.

SHM-informed life-cycle intelligent maintenance of fatigue-sensitive detail using Bayesian forecasting and Markov decision process

Civil and maritime engineering systems must be efficiently managed to control the failure risk at an acceptable level as their performance is gradually degraded throughout the operational life, caused by fatigue and corrosion. Structural health monitoring develops a timely capability to assess the structural condition and performance metrics. However, using actual long-term monitoring data to guide the life-cycle management under stochastic environments has not been sufficiently studied. To realize an optimal maintenance strategy within the service life, an integrated monitoring-based optimal management framework is developed on the basis of the partially observable Markov decision processes (POMDPs) and Bayesian forecasting. In the proposed framework, the stochastic fatigue processes are quantified by the state transition matrix. The Bayesian dynamic linear model is embedded in POMDPs as a continuous observation part to forecast the cycling impacts and estimate the deterioration rate using long-term dynamic strain responses. In addition, making use of the special features of the problem considered in this paper, an adaptive discretization strategy is proposed to alleviate the complexity of large discrete observed spaces in the POMDP. The applicability and feasibility of the framework are evaluated by intelligent maintenance of fatigue-sensitive components with real-world monitoring data. After solving the POMDP by an efficient offline solver, the results obtained in this paper demonstrate that structural interventions are uneconomical to extend the life when a welded detail is approaching its end of life due to the normal service. Furthermore, if multiple interventions are available, the framework can find optimal maintenance actions based on the trade-off between long-term utility and the corresponding cost. This framework as the prototype could also be adjusted to aid life-cycle intelligent maintenance of other types of components under different deterioration scenarios.

Research on Wargame Decision-Making Method Based on Multi-Agent Deep Deterministic Policy Gradient

Wargames are essential simulators for various war scenarios. However, the increasing pace of warfare has rendered traditional wargame decision-making methods inadequate. To address this challenge, wargame-assisted decision-making methods that leverage artificial intelligence techniques, notably reinforcement learning, have emerged as a promising solution. The current wargame environment is beset by a large decision space and sparse rewards, presenting obstacles to optimizing decision-making methods. To overcome these hurdles, a Multi-Agent Deep Deterministic Policy Gradient (MADDPG) based wargame decision-making method is presented. The Partially Observable Markov Decision Process (POMDP), joint action-value function, and the Gumbel-Softmax estimator are applied to optimize MADDPG in order to adapt to the wargame environment. Furthermore, a wargame decision-making method based on the improved MADDPG algorithm is proposed. Using supervised learning in the proposed approach, the training efficiency is improved and the space for manipulation before the reinforcement learning phase is reduced. In addition, a policy gradient estimator is incorporated to reduce the action space and to obtain the global optimal solution. Furthermore, an additional reward function is designed to address the sparse reward problem. The experimental results demonstrate that our proposed wargame decision-making method outperforms the pre-optimization algorithm and other algorithms based on the AC framework in the wargame environment. Our approach offers a promising solution to the challenging problem of decision-making in wargame scenarios, particularly given the increasing speed and complexity of modern warfare.

Deep learning-based selective spectrum sensing and allocation in cognitive vehicular radio networks

The main challenge with Vehicular Ad-Hoc Networks (VANETs) for assisting Intelligent Transportation Services (ITSs) is ensuring effective data delivery under various network circumstances despite the scarcity of radio frequency spectrum channels. Meanwhile, Dynamic Spectrum Access (DSA) utilizing Cognitive Radio (CR) technology shows its potential to solve the channel shortage issue. Reliable Spectrum Sensing (SS) and Opportunistic Spectrum Allocation (OSA) are the two most important and challenging aspects of deciding the success of the deployment of CR-enabled VANETs (CR-VANETs). In this paper, we propose three relevant issues of CR-VANETs in a single framework, i.e., reliable Cooperative Spectrum Sensing (CSS), channel indexing for selective SS, and best channel allocation to the CR users. In CSS, we use the local SS decision with more critical attributes like the geographical position of sensing signal acquisition and timestamp to obtain the global CSS session using the Deep Reinforcement Learning (DRL) technique. Selective channel-based spectrum sensing is essential to minimize the sensing overload by CR users. The present work uses the time series analysis through the deep learning-based Long short-term memory (LSTM) model for indexing the primary user channels for selective SS. Finally, for the channel allocation to the CR-VANETs, we model the complex environment as a Partial Observable Markov Decision Process (POMDP) framework and solve using a value iteration-based algorithm. The simulation results show the proposed work's efficacy over the existing works.

Reinforcement Learning-Based Lane Change Decision for CAVs in Mixed Traffic Flow under Low Visibility Conditions

As an important stage in the development of autonomous driving, mixed traffic conditions, consisting of connected autonomous vehicles (CAVs) and human-driven vehicles (HDVs), have attracted more and more attention. In fact, the randomness of human-driven vehicles (HDV) is the largest challenge for connected autonomous vehicles (CAV) to make reasonable decisions, especially in lane change scenarios. In this paper, we propose the problem of lane change decisions for CAV in low visibility and mixed traffic conditions for the first time. First, we consider the randomness of HDV in this environment and construct a finite state machine (FSM) model. Then, this study develops a partially observed Markov decision process (POMDP) for describing the problem of lane change. In addition, we use the modified deep deterministic policy gradient (DDPG) to solve the problem and get the optimal lane change decision in this environment. The reward designing takes the comfort, safety and efficiency of the vehicle into account, and the introduction of transfer learning accelerates the adaptation of CAV to the randomness of HDV. Finally, numerical experiments are conducted. The results show that, compared with the original DDPG, the modified DDPG has a faster convergence velocity. The strategy learned by the modified DDPG can complete the lane change in most of the scenarios. The comparison between the modified DDPG and the rule-based decisions indicates that the modified DDPG has a stronger adaptability to this special environment and can grasp more lane change opportunities.

Partially observable collaborative model for optimizing personalized treatment selection

Precision medicine that enables personalized treatment decision support has become an increasingly important research topic in chronic disease care. The main challenges in designing a treatment algorithm include modeling individual disease progression dynamics and designing adaptive treatment selection strategy. This study aims to develop an adaptive treatment selection framework tailored to an individual patient’s disease progression pattern and treatment response. We propose a Partially Observable Collaborative Model (POCM) to capture the individual variations in a heterogeneous population and optimize treatment outcomes in three stages. The POCM first infers the disease progression models by subgroup patterns using population data in stage one and then fine-tunes the models for individual patients with a small number of treatment trials in stage two. In stage three, we show how the treatment policies based on the Partially Observable Markov Decision Process (POMDP) can be tailored to individual patients by utilizing the disease models learned from the POCM. Using a simulated population of chronic depression patients, we show that the POCM can more accurately estimate the personal disease progression than the traditional method of solving a hidden Markov model. We also compare the POMDP treatment policies with other heuristic policies and demonstrate that the POCM-based policies give the highest net monetary benefits in majority of parameter settings. To conclude, the POCM method is a promising approach to model the chronic disease progression process and recommend a personalized treatment plan for individual patients in a heterogeneous population.

Future memories are not needed for large classes of POMDPs

Optimal policies for partially observed Markov decision processes (POMDPs) are history-dependent: Decisions are made based on the entire history of observations. Memoryless policies, which take decisions based on the last observation only, are generally considered useless in the literature because we can construct POMDP instances for which optimal memoryless policies are arbitrarily worse than history-dependent ones. Our purpose is to challenge this belief. We show that optimal memoryless policies can be computed efficiently using mixed integer linear programming (MILP), and perform reasonably well on a wide range of instances from the literature. When strengthened with valid inequalities, the linear relaxation of this MILP provides high quality upper-bounds on the value of an optimal history dependent policy. Furthermore, when used with a finite horizon POMDP problem with memoryless policies as rolling optimization problem, a model predictive control approach leads to an efficient history-dependent policy, which we call the short memory in the future (SMF) policy. Basically, the SMF policy leverages these memoryless policies to build an approximation of the Bellman value function. Numerical experiments show the efficiency of our approach on benchmark instances from the literature.

Ontology-based hybrid commonsense reasoning framework for handling context abnormalities in uncertain and partially observable environments

Ambient intelligence (AmI) systems aim to provide users with context-aware assistance services intended to improve the quality of their lives in terms of autonomy, safety, and well-being. Taking the uncertainty and partial observability of these environments into account is of major importance for context recognition and, more specifically, to detect and solve context abnormalities such as those related to the user's behavior or those related to context attribute prediction. In this paper, an ontology-based framework integrating machine learning and probabilistic planning within commonsense reasoning is proposed to recognize the user's context and abnormalities associated with it. The reasoning is performed using event calculus in answer set programming (ECASP); ECASP allows for abductive and temporal reasoning, which results in an eXplainable AI (XAI) approach. A context ontology is proposed to axiomatize the reasoning and introduce the notion of probabilistic fluents into the EC formalism in order to perform probabilistic reasoning. The reasoning incorporates probabilistic planning based on a partially observable Markov decision process (POMDP) to solve knowledge incompleteness. To evaluate the proposed framework, real-life scenarios, based on the Orange4Home and SIMADL public datasets are implemented and discussed.

Compositional Construction of Safety Controllers for Networks of Continuous-Space POMDPs

In this article, we propose a compositional framework for the synthesis of safety controllers for networks of partially observable discrete-time stochastic control systems (also known as continuous-space partially observable Markov decision processes (POMDPs)). Given an estimator, we utilize a discretization-free approach to synthesize controllers ensuring safety specifications over finite-time horizons. The proposed framework is based on a notion of so-called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">local control barrier functions</i> computed for subsystems in two different ways. In the first scheme, no prior knowledge of estimation accuracy is needed. The second framework utilizes a probability bound on the estimation accuracy using a notion of so-called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">stochastic simulation functions</i> . In both proposed schemes, we derive sufficient small-gain-type conditions in order to compositionally construct control barrier functions for interconnected POMDPs using local barrier functions computed for subsystems. The constructed control barrier functions for the overall networks enable us to compute lower bounds on the probabilities that the interconnected POMDPs avoid certain unsafe regions in finite-time horizons. We demonstrate the effectiveness of our proposed approaches by applying them to an adaptive cruise control problem.

Recorded recurrent deep reinforcement learning guidance laws for intercepting endoatmospheric maneuvering missiles

This work proposes a recorded recurrent twin delayed deep deterministic (RRTD3) policy gradient algorithm to solve the challenge of constructing guidance laws for intercepting endoatmospheric maneuvering missiles with uncertainties and observation noise. The attack-defense engagement scenario is modeled as a partially observable Markov decision process (POMDP). Given the benefits of recurrent neural networks (RNNs) in processing sequence information, an RNN layer is incorporated into the agent’s policy network to alleviate the bottleneck of traditional deep reinforcement learning methods while dealing with POMDPs. The measurements from the interceptor’s seeker during each guidance cycle are combined into one sequence as the input to the policy network since the detection frequency of an interceptor is usually higher than its guidance frequency. During training, the hidden states of the RNN layer in the policy network are recorded to overcome the partially observable problem that this RNN layer causes inside the agent. The training curves show that the proposed RRTD3 successfully enhances data efficiency, training speed, and training stability. The test results confirm the advantages of the RRTD3-based guidance laws over some conventional guidance laws.

Ensure: Towards Reliable Control of Cyber-Physical Systems Under Uncertainty

Cyber-physical systems (CPSs) are complex ensembles of physical and cyber components that cooperate to offer dynamic and adaptive functionalities. Uncertainty can arise from a plethora of sources in the entangled components, ranging from the unreliable perception, the nondeterministic action effects, to even the changes in the environment. Existing controlling approaches, such as those using Markov decision process, have limited ability in handling uncertainty. To address the challenge, in this article, we novelly propose using partially observable Markov decision processes (POMDPs) to model CPS under uncertainty and show that common types of uncertainties can be modeled by partial observations and nondeterministic actions over probabilistic distributions. With POMDPs, strategies that can optimally control CPS are synthesized. We further propose a strategywise verification method, which resolves the difficult problem of verifying the entire POMDP, to offer reliable controlling strategies. Experiments on two representative cases of CPS show promising results compared with existing approaches.

Deep reinforcement learning for the olfactory search POMDP: a quantitative benchmark.

The olfactory search POMDP (partially observable Markov decision process) is a sequential decision-making problem designed to mimic the task faced by insects searching for a source of odor in turbulence, and its solutions have applications to sniffer robots. As exact solutions are out of reach, the challenge consists in finding the best possible approximate solutions while keeping the computational cost reasonable. We provide a quantitative benchmarking of a solver based on deep reinforcement learning against traditional POMDP approximate solvers. We show that deep reinforcement learning is a competitive alternative to standard methods, in particular to generate lightweight policies suitable for robots.

A UAV Navigation Approach Based on Deep Reinforcement Learning in Large Cluttered 3D Environments

This paper is concerned with the problem of using deep reinforcement learning methods to enable safe navigation of UAVs in unknown large-scale environments containing a large number of obstacles. The ability of UAV to navigate autonomously is a precondition for performing disaster rescue and logistics deliveries. In this paper, the perception-constrained navigation of UAV is modeled as a partially observable Markov decision process (POMDP), and a fast recurrent stochastic valued gradient algorithm based on the actot-critic framework is designed for online solution. Compared with traditional SLAM-based and reactive obstacle avoidance-based approaches, the proposed navigation method can map the raw sensing information into navigation signals, and the learned policy deployed to the UAV can avoid the memory occupation and computational consumption required for real-time map building. Through experiments in a newly designed simulation environment, it is demonstrated that the proposed algorithm can enable the UAV to navigate safely in unknown cluttered large-scale environments and outperform the state-of-the-art DRL algorithm in terms of performance.

A POMDP-Based Approach to Joint Antenna Selection and User Scheduling for Multi-User Massive MIMO Communication

We devise a partially observable Markov decision process (POMDP) based joint antenna selection and user scheduling (JASUS) policy for a massive MIMO base station, equipped with only a small number of RF chains, that serves a large number of users. The users are served at different time slots within a frame. Relying on partial CSI obtained from training between the selected antennas and the users, at the beginning of each frame, the BS assigns each user to a time slot in the frame and selects a subset of antennas to serve the users scheduled in each time slot. Assuming that the channels evolve according to a Markov process and relying on zero-forcing beamforming, we formulate our JASUS problem using a POMDP framework to devise a real-time decision-making policy that maximizes the expected long-term sum-rate. We rigorously prove that for positively correlated two-state channel models, the myopic policy provides the optimal solution to our POMDP-based JASUS problem for any number of RF chains and for any number of users. Based on this, we model the Rayleigh fading channels as first-order Gauss-Markov processes and devise a low-complexity myopic policy-based JASUS algorithm for massive MU-MIMO systems that only relies on partial CSI.

Data Analytics of an Information System Based on a Markov Decision Process and a Partially Observable Markov Decision Process

Data analytics of an information system is conducted based on a Markov decision process (MDP) and a partially observable Markov decision process (POMDP) in this paper. Data analytics over a finite planning horizon and an infinite planning horizon for a discounted MDP is performed, respectively. Value iteration (VI), policy iteration (PI), and Q-learning are utilized in the data analytics for a discounted MDP over an infinite planning horizon to evaluate the validity of the MDP model. The optimal policy to minimize the total expected cost of states of the information system is obtained based on the MDP. In the analytics for a discounted POMDP over an infinite planning horizon of the information system, the effects of various parameters on the total expected cost of the information system are studied.

Predictive trajectory planning for autonomous vehicles at intersections using reinforcement learning

In this work we put forward a predictive trajectory planning framework to help autonomous vehicles plan future trajectories. We develop a partially observable Markov decision process (POMDP) to model this sequential decision making problem, and a deep reinforcement learning solution methodology to learn high-quality policies. The POMDP model utilizes driving scenarios, condensed into graphs, as inputs. More specifically, an input graph contains information on the history trajectory of the subject vehicle, predicted trajectories of other agents in the scene (e.g., other vehicles, pedestrians, and cyclists), as well as predicted risk levels posed by surrounding vehicles to devise safe, comfortable, and energy-efficient trajectories for the subject vehicle to follow. In order to obtain sufficient driving scenarios to use as training data, we propose a simulation framework to generate socially acceptable driving scenarios using a real world autonomous vehicle dataset. The simulation framework utilizes Bayesian Gaussian mixture models to learn trajectory patterns of different agent types, and Gibbs sampling to ensure that the distribution of simulated scenarios matches that of the real-world dataset collected by an autonomous fleet. We evaluate the proposed work in two complex urban driving environments: a non-signalized T-junction and a non-signalized lane merge intersection. Both environments provide vastly more complex driving scenarios compared to a highway driving environment, which has been mostly the focus of previous studies. The framework demonstrates promising performance for planning horizons as long as five seconds. We compare safety, comfort, and energy efficiency of the planned trajectories against human-driven trajectories in both experimental driving environments, and demonstrate that it outperforms human-driven trajectories in a statistically significant fashion in all aspects.

A Markovian model for the spread of the SARS-CoV-2 virus

We propose a Markovian stochastic approach to model the spread of a SARS-CoV-2-like infection within a closed group of humans. The model takes the form of a Partially Observable Markov Decision Process (POMDP), whose states are given by the number of subjects in different health conditions. The model also exposes the different parameters that have an impact on the spread of the disease and the various decision variables that can be used to control it (e.g, social distancing, number of tests administered to single out infected subjects). The model describes the stochastic phenomena that underlie the spread of the epidemic and captures, in the form of deterministic parameters, some fundamental limitations in the availability of resources (hospital beds and test swabs). The model lends itself to different uses. For a given control policy, it is possible to verify if it satisfies an analytical property on the stochastic evolution of the state (e.g., to compute probability that the hospital beds will reach a fill level, or that a specified percentage of the population will die). If the control policy is not given, it is possible to apply POMDP techniques to identify an optimal control policy that fulfils some specified probabilistic goals. Whilst the paper primarily aims at the model description, we show with numeric examples some of its potential applications.

Improving anti-jamming decision-making strategies for cognitive radar via multi-agent deep reinforcement learning

Most of the existing anti-jamming decision-making methods overly rely on the subjective experience of radar operators. However, due to the rapid development of cognitive radar and modern electronic warfare, conventional anti-jamming decision-making methods can no longer adapt to the complex and changing electromagnetic environment. The advent of deep reinforcement learning (DRL) provides a new attractive solution for this issue. In this paper, an adversarial anti-jamming decision-making network for cognitive radar via multi-agent deep reinforcement learning (MDRL) is proposed, which has good self-learning ability and can meet the requirements of intelligent, dynamic and real-time in modern electronic warfare. Since competitive decision-makers are considered and these two confrontational sides are not able to obtain the completely accurate information of each other, the environment model is specifically constructed as a partially observable Markov decision process (POMDP). Then, a decision-making network is designed based on deterministic deep deterministic policy gradient (DDPG) algorithm to explore the competition between cognitive radar and smart jammer. In order to overcome the environment non-stationarity, the decision-making network is trained and tested in a special MDRL framework. The experimental results demonstrate that the proposed method is effective in anti-jamming decision-making system of cognitive radar. Furthermore, the two confrontational sides show high decision-making ability and perform well in the adversarial scenario by comparing with other training policies, which demonstrate that confrontational training with powerful opponents can improve the intelligence level of all agents.

Condition-based pavement management systems accounting for model uncertainty and facility heterogeneity with belief updates

Due to the tighter budget for pavement management, schedules of inspection activities should be jointly optimized with the maintenance and reconstruction (M&R) plans for pavement systems. Conducting inspections every year is unnecessary and will decrease the budget for M&R activities, while infrequent inspections may lead to suboptimal M&R planning due to the lack of accurate information. This paper presents a methodology for jointly optimizing the inspection scheduling and M&R planning for pavement systems, considering model uncertainty and facility-specific heterogeneity. The problem is defined as a Partially Observable Markov Decision Process (POMDP) model, accounting for the tradeoff between the information value and inspection costs. Moreover, a statistical learning method is used to update the prediction of pavement conditions using the collected inspection data and, eventually, improve the condition-based decisions. This “belief update” process can gradually reduce the model uncertainty as the dataset size increases. We demonstrate the proposed stochastic optimization framework through a numerical example with a system of fifty heterogenous pavement facilities under a combined budget for inspection and M&R activities. Several managerial insights and implications are discussed. For example, the optimal inspection frequencies are less sensitive to the budget; and the agency should perform fewer reconstructions and more rehabilitations when the budget is limited.

Delay-Oriented Scheduling in 5G Downlink Wireless Networks Based on Reinforcement Learning With Partial Observations

5G wireless networks are expected to satisfy different delay requirements of various traffics by network resource scheduling. Existing scheduling methods perform poorly in practice due to their unrealistic assumption on the access to the full channel state information (CSI) or the explicit mathematical expression of network delay. In this paper, we consider the delay-oriented packet scheduling problem in multi-cell 5G downlink networks with multiple users and traffic types (e.g., FTP, VoIP and video streaming), and formulate it as a partially observable Markov decision process (POMDP). We design a delay-oriented downlink scheduling framework based on deep reinforcement learning (DRL) to autonomously schedule the active traffic flows without the full channel information. Furthermore, a recurrent proximal policy optimization (RPPO) algorithm is proposed to perceive the underlying state and accelerate learning under different time granularities, with the policy gradient theorem under POMDP strictly proved. By incorporating the future traffic information provided by a proposed spatial-temporal prediction algorithm, RPPO can balance the load and achieve lower delay in real-time multi-cell multi-user scenarios. Results of extensive experiments on a realistic 5G simulator demonstrate that our framework significantly outperforms existing approaches in terms of both tail delay and average delay for up to 48% and 41.7%, respectively.