Observable Markov Decision Process Model Research Articles

Optimal Joint Defense and Monitoring for Networks Security under Uncertainty: A POMDP-Based Approach

The increasing interconnectivity in our infrastructure poses a significant security challenge, with external threats having the potential to penetrate and propagate throughout the network. Bayesian attack graphs have proven to be effective in capturing the propagation of attacks in complex interconnected networks. However, most existing security approaches fail to systematically account for the limitation of resources and uncertainty arising from the complexity of attacks and possible undetected compromises. To address these challenges, this paper proposes a partially observable Markov decision process (POMDP) model for network security under uncertainty. The POMDP model accounts for uncertainty in monitoring and defense processes, as well as the probabilistic attack propagation. This paper develops two security policies based on the optimal stationary defense policy for the underlying POMDP state process (i.e., a network with known compromises): the estimation-based policy that performs the defense actions corresponding to the optimal minimum mean square error state estimation and the distribution-based policy that utilizes the posterior distribution of network compromises to make defense decisions. Optimal monitoring policies are designed to specifically support each of the defense policies, allowing dynamic allocation of monitoring resources to capture network vulnerabilities/compromises. The performance of the proposed policies is examined in terms of robustness, accuracy, and uncertainty using various numerical experiments.

Cancer Screening Benefits Maximization Using Markov Decision Process Models: A Systematic Review

Context: Due to the chronic nature of cancer, screening programs were a set of sequential decisions taken over time. Markov decision process (MDP) and partially observable Markov decision process (POMDP) models were the mathematical tools applied in studies, including sequential decision-making such as screening protocols for medical decision-making. Objectives: The main goal of this study was to investigate optimal policy for cancer screening using MDP and POMDP models. Methods: We performed a review of articles published within July 2000 to November 2022 in PubMed, Web of Science, and Scopus databases. The stopping age, the type of optimal strategy, the benefits of the optimal policy, and the relationship between age and risk threshold were extracted. Studies that did not use MDPs and POMDPs as the mathematical maximization models in cancer screening, review articles, editorials or commentaries, non-English articles, and those that did not focus on optimization were excluded. Results: From 532 articles, 6 studies met the study criteria. All studies suggested that the optimal policy was control-limit, and the cancer risk threshold was a non-decreasing function of age. Three studies specified a stopping age for cancer screening. In five studies, the optimal policies outperformed the guidelines or no screening strategy. Conclusions: Two essential factors in screening decisions were cancer risk and age, which were individual variables. The control-limit policy included these factors in decision-making for cancer screening. These policies highlighted personalized screening and showed that this type of screening could outperform cancer screening guidelines regarding economic and clinical benefits.

A Reinforcement Learning Based Coordinated but Differentiated Load Frequency Control Method With Heterogeneous Frequency Regulation Resources

As the diversification of frequency regulation resources, it is increasingly important to find a load frequency control (LFC) scheme that can effectively coordinate different resources. This paper proposes a coordinated but differentiated LFC control method that utilizes heterogeneous frequency regulation resources with their characteristics fully considered. The optimal LFC models for the single area and multi-area power systems are formulated as partially observable Markov decision process model (POMDP) and partially observable Markov game (POMG) considering the accessibility of system information. A deep reinforcement learning method called recurrent proximal policy optimization is leveraged to solve the POMDP and POMG models. Simulations based on historical data demonstrate that the proposed control scheme can better coordinate different resources and achieve superior control performance.

Dynamic joint sensor selection and maintenance optimization in partially observable deteriorating systems

We consider a degrading system with costly but silent failures. The system can be partially observed using a set of heterogeneous noisy sensors at a given cost, where each sensor noise level is a function of the system state. The goal is to devise a dynamic joint sensor selection and maintenance optimization model to minimize the total expected discounted cost of the system. We develop an infinite-horizon discrete-time partially observable Markov decision process model to dynamically prescribe the sensor subsets for system monitoring and the timing of maintenance activities. We perform numerical experiments for a system with up to ten sensors and compare the proposed optimal policy with four heuristic policies. We observe that the proposed approach consistently outperforms all heuristic policies considered. Through a comprehensive analysis of numerical experiments, our study shows the efficacy of the proposed model’s optimal policy in strategically selecting sensors to achieve precise system state estimation while managing costs. Through these experiments, we find that it is optimal to use lower-cost, higher-variance sensors and often fuse them for more accurate estimations. However, the choice depends on the system state and sensor quality distribution. Importantly, our model excels in determining the best action by considering these factors, underscoring its practical applicability.

Demand response‐based autonomous voltage control for transmission and active distribution networks using modified partially observed Markov decision process model

AbstractTo fully utilize the voltage regulation capacity and interaction characteristics of the Transmission and Distribution (T&D) system, a novel Modified Partially Observed Markov Decision Process (MPOMDP)‐based Reinforcement Learning (RL) scheme for Autonomous Voltage Control is proposed, which incorporates Demand Response (DR) and cooperation with the Transmission Network . The proposed scheme consists of two vital components: an MPOMDP block and a Modified Asynchronous Advantage Actor‐Critic‐based RL block. The MPOMDP block innovatively exploits the confidence interval of the observed state to make a better perception of the precise system state by introducing two new probability vectors. Then the MPOMDP block is fed into the underlying architecture of the RL block for asynchronously capturing features and optimal decision‐making, where the solving framework additionally brings in a public data buffer to realize boundary information sharing. Case studies are conducted on a modified T&D system considering N‐1 contingencies, with a training dataset from a district in Suzhou, China. Simulation results demonstrate that the proposed scheme can achieve significant voltage optimization while ensuring fast convergence speed.

Optimal Scheduling in IoT-Driven Smart Isolated Microgrids Based on Deep Reinforcement Learning

In this paper, we investigate the scheduling issue of diesel generators (DGs) in an Internet of Things (IoT)-Driven isolated microgrid (MG) by deep reinforcement learning (DRL). The renewable energy is fully exploited under the uncertainty of renewable generation and load demand. The DRL agent learns an optimal policy from history renewable and load data of previous days, where the policy can generate real-time decisions based on observations of past renewable and load data of previous hours collected by connected sensors. The goal is to reduce operating cost on the premise of ensuring supply-demand balance. In specific, a novel finite-horizon partial observable Markov decision process (POMDP) model is conceived considering the spinning reserve. In order to overcome the challenge of discrete-continuous hybrid action space due to the binary DG switching decision and continuous energy dispatch (ED) decision, a DRL algorithm, namely the hybrid action finite-horizon RDPG (HAFH-RDPG), is proposed. HAFH-RDPG seamlessly integrates two classical DRL algorithms, i.e., deep Q-network (DQN) and recurrent deterministic policy gradient (RDPG), based on a finite-horizon dynamic programming (DP) framework. Extensive experiments are performed with real-world data in an IoT-driven MG to evaluate the capability of the proposed algorithm in handling the uncertainty due to inter-hour and inter-day power fluctuation and to compare its performance with those of the benchmark algorithms.

A Corrosion Maintenance Model Using Continuous State Partially Observable Markov Decision Process for Oil and Gas Pipelines

This paper proposes a continuous state partially observable Markov decision process (POMDP) model for the corrosion maintenance of oil and gas pipelines. The maintenance operations include complex and extensive activities to detect the corrosion type, determine its severity, predict the deterioration rate, and plan future inspection (monitoring) schemes and maintenance policy. A POMDP model is formulated as a decision-making support tool to effectively handle partially observed corrosion defect levels. It formulates states as the pipeline’s degradation level using a probability distribution. Inline inspection (ILI) methods estimate the latest state of the pipeline, which also defines the initial state of the optimization process. The set of actions comprises corrosion mitigation operations. The errors associated with the ILI method are used to construct the observation function for the model. The sum of inspection, maintenance operations, and failure costs for a given state and action are formulated as rewards. Numerical experiments are made based on data collected from the literature. The results showed that different policies, whether derived from solvers (theoretical) or determined from practical experience, can be compared to identify the best maintenance alternative using the model. It was also observed that the choice of the solvers is important since they affect the discounted rewards and the run time to obtain them. The model approximates the parameters and uncertainty associated with the propagation of corrosion, proficiency of inspection methods, and implementation of maintenance policies. Overall, it can be applied to improve the maintenance decision-making process for the oil and gas pipeline as it incorporates the stochastic features of the operation.

A Supply Chain Inventory Management Method for Civil Aircraft Manufacturing Based on Multi-Agent Reinforcement Learning

Effective supply chain inventory management is crucial for large-scale manufacturing industries such as civil aircraft and automobile manufacturing to ensure efficient manufacturing. Generally, the main manufacturer makes the annual inventory management plan, and contacts with suppliers when some material is approaching critical inventory level according to the actual production schedule, which increases the difficulty of inventory management. In recent years, many researchers have focused on using reinforcement learning method to study inventory management problems. Current approaches were mainly designed for the supply chain with single-node multi-material or multi-node single-material mode, which are not suitable to the civil aircraft manufacturing supply chain with multi-node multi-material mode. To deal with this problem, we formulated the problem as a partially observable Markov decision process (POMDP) model and proposed a multi-agent reinforcement learning method for supply chain inventory management, in which the dual-policy and information transmission mechanism was designed to help the supply chain participant improve the global information utilization efficiency of the supply chain and the coordination efficiency with other participants. The experiment results show that our method has about 45% performance improvement on efficiency compared with current reinforcement learning-based methods.

A parallel game model‐based intrusion response system for cross‐layer security in industrial internet of things

SummaryWith the rise of industrialization, the importance of the industrial Internet of Things (IIoT) has increased significantly, and with it comes a variety of security threats. Therefore, the security of these networks is critical. Industrial Response Systems (IRSs), as the last line of security, plays an important role in the security system of the Industrial Internet of Things. In this paper, a new IRS model based on the non‐cooperative game is proposed. First, by combining the Partially Observable Markov Decision Process (POMDP) model with the stochastic game model based on the expanded attack tree, our model could effectively perceive the changes at each node. Second, our model incorporates the alarms of intrusion detection system (IDS) and the physical quantities of sensors in Industrial Cyber‐Physical System (ICPS) into the quantization system so that the model can respond to intruders more accurately and comprehensively. Finally, we develop this model based on multiprocessors to speed up the solution process, and adopt an approximation algorithm to reduce the number of iterations of the POMDP

FABRIC: A Framework for the Design and Evaluation of Collaborative Robots with Extended Human Adaptation

A limitation for collaborative robots (cobots) is their lack of ability to adapt to human partners, who typically exhibit an immense diversity of behaviors. We present an autonomous framework as a cobot’s real-time decision-making mechanism to anticipate a variety of human characteristics and behaviors, including human errors, toward a personalized collaboration. Our framework handles such behaviors in two levels: (1) short-term human behaviors are adapted through our novel Anticipatory Partially Observable Markov Decision Process (A-POMDP) models, covering a human’s changing intent (motivation), availability, and capability; (2) long-term changing human characteristics are adapted by our novel Adaptive Bayesian Policy Selection (ABPS) mechanism that selects a short-term decision model, e.g., an A-POMDP, according to an estimate of a human’s workplace characteristics, such as her expertise and collaboration preferences. To design and evaluate our framework over a diversity of human behaviors, we propose a pipeline where we first train and rigorously test the framework in simulation over novel human models. Then, we deploy and evaluate it on our novel physical experiment setup that induces cognitive load on humans to observe their dynamic behaviors, including their mistakes, and their changing characteristics such as their expertise. We conduct user studies and show that our framework effectively collaborates non-stop for hours and adapts to various changing human behaviors and characteristics in real-time. That increases the efficiency and naturalness of the collaboration with a higher perceived collaboration, positive teammate traits, and human trust. We believe that such an extended human-adaptation is a key to the long-term use of cobots.

A multi-objective constrained partially observable Markov decision process model for breast cancer screening

A multi-objective constrained partially observable Markov decision process model for breast cancer screening

Condition-based pavement management systems accounting for model uncertainty and facility heterogeneity with belief updates

Due to the tighter budget for pavement management, schedules of inspection activities should be jointly optimized with the maintenance and reconstruction (M&R) plans for pavement systems. Conducting inspections every year is unnecessary and will decrease the budget for M&R activities, while infrequent inspections may lead to suboptimal M&R planning due to the lack of accurate information. This paper presents a methodology for jointly optimizing the inspection scheduling and M&R planning for pavement systems, considering model uncertainty and facility-specific heterogeneity. The problem is defined as a Partially Observable Markov Decision Process (POMDP) model, accounting for the tradeoff between the information value and inspection costs. Moreover, a statistical learning method is used to update the prediction of pavement conditions using the collected inspection data and, eventually, improve the condition-based decisions. This “belief update” process can gradually reduce the model uncertainty as the dataset size increases. We demonstrate the proposed stochastic optimization framework through a numerical example with a system of fifty heterogenous pavement facilities under a combined budget for inspection and M&R activities. Several managerial insights and implications are discussed. For example, the optimal inspection frequencies are less sensitive to the budget; and the agency should perform fewer reconstructions and more rehabilitations when the budget is limited.

A Discrete Partially Observable Markov Decision Process Model for the Maintenance Optimization of Oil and Gas Pipelines

Corrosion is one of the major causes of failure in pipelines for transporting oil and gas products. To mitigate the impact of this problem, organizations perform different maintenance operations, including detecting corrosion, determining corrosion growth, and implementing optimal maintenance policies. This paper proposes a partially observable Markov decision process (POMDP) model for optimizing maintenance based on the corrosion progress, which is monitored by an inline inspection to assess the extent of pipeline corrosion. The states are defined by dividing the deterioration range equally, whereas the actions are determined based on the specific states and pipeline attributes. Monte Carlo simulation and a pure birth Markov process method are used for computing the transition matrix. The cost of maintenance and failure are considered when calculating the rewards. The inline inspection methods and tool measurement errors may cause reading distortion, which is used to formulate the observations and the observation function. The model is demonstrated with two numerical examples constructed based on problems and parameters in the literature. The result shows that the proposed model performs well with the added advantage of integrating measurement errors and recommending actions for multiple-state situations. Overall, this discrete model can serve the maintenance decision-making process by better representing the stochastic features.

A Dynamic Deep Reinforcement Learning-Bayesian Framework for Anomaly Detection

To assure the successful operation of connected and automated vehicles, it is critical to detect and isolate anomalous and/or faulty information in a timely manner. To do so, anomaly detection techniques should be implemented in real-time where if the probability of anomalous information exceeds a certain threshold, the information is dealt with accordingly. Traditionally, the threshold for judging whether the data is anomalous is fixed and determined a priori. However, not only does this approach fail to account for the feedback obtained during a trip on the performance of the algorithms, but it also fails to respond to potential changes in rates of anomalies. Hence, it is important to develop an approach that can dynamically alter this threshold in response to exogenous factors to assure reliable and robust system operation. We develop a mathematical framework which utilizes a dynamic threshold for an anomaly classification algorithm in order to maximize the safety of a trip. Specifically, we develop and pair an anomaly classification algorithm based on convolutional neural networks (CNN), with a partially observable Markov decision process (POMDP) model. We solve the resulting POMDP model using the asynchronous advantage actor critic (A3C) deep reinforcement learning algorithm. The prescribed policy determines the anomaly classification threshold in real-time that maximizes the performance. Our numerical experiments show that the POMDP model outperforms state-of-the-art benchmarks, especially under more difficult to detect anomaly profiles.

Optimization of active surveillance strategies for heterogeneous patients with prostate cancer

Prostate cancer (PCa) is common in American men with long latent periods, during which the disease is asymptomatic. Active surveillance is a monitoring strategy commonly used for patients diagnosed with low‐risk PCa who may harbor latent high‐risk PCa. The optimal monitoring strategy attempts to minimize the disutility of testing while ensuring that the patient is detected at the earliest time when the disease progresses. Unfortunately, guidelines for the active surveillance of PCa are often one‐size‐fits‐all strategies that ignore the heterogeneity among multiple patient types. In contrast, personalized strategies based on partially observable Markov decision process (POMDP) models are challenging to implement in practice given the large number of possible strategies that can be used. This article presents a two‐stage stochastic programming approach that selects a set of strategies for predefined cardinality based on patients' disease risks. The first‐stage decision variables include binary variables for the selection of periods at which to test patients in each strategy and the assignment of multiple patient types to strategies. The objective is to maximize a weighted reward function that considers the need for cancer detection, missed detection, and cost of monitoring patients. We discuss the structure and complexity of the model and reformulate a logic‐based Bender's decomposition formulation that can solve realistic instances to optimality. We present a case study for the active surveillance of PCa and show that our model results in strategies that vary in intensity according to patient disease risk. Finally, we show that our model can generate a small number of strategies that can significantly improve the existing “one‐size‐fits‐all” guideline strategies used in practice.

Screening for preclinical Alzheimer's disease: Deriving optimal policies using a partially observable Markov model.

Alzheimer's Disease (AD) is believed to be the most common type of dementia. Even though screening for AD has been discussed widely, there is no screening program implemented as part of a policy in any country. Current medical research motivates focusing on the preclinical stages of the disease in a modeling initiative. We develop a partially observable Markov decision process model to determine optimal screening programs. The model contains disease free and preclinical AD partially observable states and the screening decision is taken while an individual is in one of those states. An observable diagnosed preclinical AD state is integrated along with observable mild cognitive impairment, AD and death states. Transition probabilities among states are estimated using data from Knight Alzheimer's Disease Research Center (KADRC) and relevant literature. With an objective of maximizing expected total quality-adjusted life years (QALYs), the output of the model is an optimal screening program that specifies at what points in time an individual over 50 years of age with a given risk of AD will be directed to undergo screening. The screening test used to diagnose preclinical AD has a positive disutility, is imperfect and its sensitivity and specificity are estimated using the KADRC data set. We study the impact of a potential intervention with a parameterized effectiveness and disutility on model outcomes for three different risk profiles (low, medium and high). When intervention effectiveness and disutility are at their best, the optimal screening policy is to screen every year between ages 50 and 95, with an overall QALY gain of 0.94, 1.9 and 2.9 for low, medium and high risk profiles, respectively. As intervention effectiveness diminishes and/or its disutility increases, the optimal policy changes to sporadic screening and then to never screening. Under several scenarios, some screening within the time horizon is optimal from a QALY perspective. Moreover, an in-depth analysis of costs reveals that implementing these policies are either cost-saving or cost-effective.

Data pooling for multiple single-component systems under population heterogeneity

We consider multiple newly designed single-component systems with a known finite lifespan. The component in each system can be replaced preventively to avoid a costly failure. This component is also in use for the first time, and therefore, there is no historical data on the lifetime of the component. In this case, the probability distribution of the component lifetime can be estimated based on expert opinions. However, there can be different opinions on the lifetime distribution. There are two populations where components can come from: a weak and a strong population. We assume that the components always come from the same population. However, the true type of the population is unknown. We build a discrete-time partially observable Markov decision process model to find the optimal replacement policy which minimizes the expected total cost throughout the lifespan. To resolve the uncertainty regarding population heterogeneity, we update the belief by using the data collected from all systems, allowing us to investigate the effect of so-called data pooling. First, we generate insights about the structure of the optimal policy. We then compare the cost per system under the optimal policy with the cost per system under two benchmark heuristics that follow the single-system optimal policy with and without data pooling, respectively. In our numerical experiments, we show that the cost reduction relative to the worst benchmark heuristic can be up to 5.6% for data pooling with two systems, and this increases up to 14% for 20 systems. Additionally, the effect of various input parameters on the costs is analyzed.

Data Integrity Attack in Dynamic State Estimation of Smart Grid: Attack Model and Countermeasures

A smart grid integrates advanced sensors, efficient measurement methods, progressive control technologies, and other techniques and devices to achieve safe, efficient and economical operation of the grid system. However, the diversified and open environment of a smart grid makes energy and information of the smart grid vulnerable to malicious attacks. As a representative cyber-physical attack, the data integrity attack has an extremely severe impact on the grid operation for it can bypass the traditional detection mechanisms by adjusting the attack vector. In this paper, we first present the attack strategy against dynamic state estimation of power grid in the perspective of adversary and formulate the data integrity attack detection problem that has the characteristic of sequential decision making as a partially observable Markov decision process. Then, a deep reinforcement learning-based approach is proposed to detect against data integrity attacks, which utilizes the Long Short-Term Memory layer to extract the state features of previous time steps in determining whether the system is currently under attack. Moreover, the noisy networks are employed to ensure effective agent exploration, which prevents the agent from sticking to the non-optimal policy. The principle of a multi-step learning is adopted to increase the estimation accuracy of Q value. To address the sparse rewards problem, the prioritized experience replay is proposed to increase training efficiency. Simulation results demonstrated that the proposed detection approach surpasses the benchmarks in the comparison metrics: delay error rate and false rate. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —In this paper, we present a deep reinforcement learning-based algorithm to defend against the data integrity attacks of smart grid. Most of the previous works discretized the system states and utilized the current state information to identify whether the system is under attack. For this reason, the detection policy may totally ignored the continuously changing characteristics of the grid states, which will lead to poor detection performance. Moreover, the attacked system states only accounts for a small part of the entire grid operation states, the probability of sampling the experience containing the attack state is extremely small, which limits the learning efficiency of previous RL-based detection approaches. In order to increase the accuracy of detection, we first present the attack strategy against power grid’s dynamic state estimation in the perspective of adversary and formulate the partially observable Markov decision process model of attack detection problem. Moreover, we propose a deep reinforcement learning-based detection approach combining the LSTM network to extract the system state features of the previous time steps to determine whether the system is currently being attacked. To address the sparse rewards problem, the prioritized experience replay is used to increase learning efficiency. The experiments demonstrate the effectiveness of proposed detection scheme compared with benchmarks in terms of detection delay as well as accuracy. In conclusion, the proposed detection scheme is helpful in defending against the data integrity attacks without obtaining the opponent’s strategy in advance and can be conveniently applied to the real-world security management system of smart grid.

The neurocomputational bases of explore-exploit decision-making

Flexible decision-making requires animals to forego immediate rewards (exploitation) and try novel choice options (exploration) to discover if they are preferable to familiar alternatives.Using the same task and a partially observable Markov decision process (POMDP) model to quantify the value of choices, we first determined that the computational basis for managing explore-exploit tradeoffs is conserved across monkeys and humans. We then used fMRI to identify where in the human brain the immediate value of exploitative choices and relative uncertainty about the value of exploratory choices were encoded. Consistent with prior neurophysiological evidence in monkeys, we observed divergent encoding of reward value and uncertainty in prefrontal and parietal regions, including frontopolar cortex, and parallel encoding of these computations in motivational regions including the amygdala, ventral striatum, and orbitofrontal cortex. These results clarify the interplay between prefrontal and motivational circuits that supports adaptive explore-exploit decisions in humans and nonhuman primates.

A Machine Learning–Enabled Partially Observable Markov Decision Process Framework for Early Sepsis Prediction

Sepsis is a life-threatening condition, caused by the body’s extreme response to an infection. In the United States, 1.7 million cases of sepsis occur annually, resulting in 265,000 deaths. Delayed diagnosis and treatment are associated with higher mortality rates. An exponential rise in the availability of medical data has allowed for the development of sophisticated machine learning algorithms to predict sepsis earlier than the onset. However, these models often underperform, as the training data are retrospective and do not fully capture the uncertain future. In this study, we develop a novel framework, which we refer to as MLePOMDP, to leverage and combine the underlying, high-level knowledge about sepsis progression and machine learning (ML) for classification. Specifically, we use a hidden Markov model to describe sepsis development at a high level, where the ML model makes the higher-order “observations” from temporal data. Consequently, a partially observable Markov decision process (POMDP) model is developed to make classification decisions. We analytically establish that the optimal policy is of threshold-type, which we exploit to efficiently optimize MLePOMDP. MLePOMDP is calibrated and tested using high-frequency physiological data collected from bedside monitors. Different from past POMDP-based frameworks, MLePOMDP is developed for a prediction task using a very small state definition, produces highly interpretable results, and accounts for a novel and clinically meaningful action space. Our results show that MLePOMDP outperforms machine learning–based benchmarks by up to 8% in precision. Importantly, MLePOMDP is able to reduce false alarms by up to 28%. An additional experiment is conducted to show the generalizability of MLePOMDP to different patient cohorts. Summary of Contribution: This study develops a novel real-time decision support framework for early sepsis prediction by integrating well-known machine learning models (random forest and neural networks) with a well-established sequential decision-making model, namely, a partially observable Markov decision process (POMDP). The structural properties of the optimal policy are further explored and a threshold-type structure is established, which is then leveraged to develop a customized algorithm to solve the problem more efficiently. The resulting framework demonstrates the benefit of applying POMDPs to augment machine learning outputs. Specifically, the framework results in the reduction of false alarms in sepsis predictions where decisions are made in real time, hence improving the overall prediction precision.