Observable Markov Decision Process Model Research Articles

Deep Q-Network-Enhanced Self-Tuning Control of Particle Swarm Optimization

Particle Swarm Optimization (PSO) is a widespread evolutionary technique that has successfully solved diverse optimization problems across various application fields. However, when dealing with more complex optimization problems, PSO can suffer from premature convergence and may become stuck in local optima. The primary goal is accelerating convergence and preventing solutions from falling into these local optima. This paper introduces a new approach to address these shortcomings and improve overall performance: utilizing a reinforcement deep learning method to carry out online adjustments of parameters in a homogeneous Particle Swarm Optimization, where all particles exhibit identical search behaviors inspired by models of social influence among uniform individuals. The present method utilizes an online parameter control to analyze and adjust each primary PSO parameter, particularly the acceleration factors and the inertia weight. Initially, a partially observed Markov decision process model at the PSO level is used to model the online parameter adaptation. Subsequently, a Hidden Markov Model classification, combined with a Deep Q-Network, is implemented to create a novel Particle Swarm Optimization named DPQ-PSO, and its parameters are adjusted according to deep reinforcement learning. Experiments on different benchmark unimodal and multimodal functions demonstrate superior results over most state-of-the-art methods regarding solution accuracy and convergence speed.

Transdiagnostic Anxiety-Related Increases in Information Sampling are Associated With Altered Valuation.

Excessive information sampling in psychiatric patients characterized by high trait anxiety has been inconsistently linked with alterations in inferential and valuation processes. Methodological limitations could account in part for these inconsistencies. To address this, computational models of inference and valuation were applied to data collected from a transdiagnostic sample of adults with and without an anxiety or compulsive disorder using a version of the beads task with enhanced experimental controls. Participants diagnosed with an anxiety or compulsive disorder (n = 35) and healthy controls (n = 23) completed the beads task with three majority-to-minority ratios of blue versus green beads (60:40, 75:25, 90:10). First, a Bayesian belief-updating model was fit to quantify the iterative process by which new information (bead color) and prior beliefs were integrated to influence current beliefs about jar identity. Next, a parameterized partially observable Markov decision process model was used to parse the contribution of value-based decisions to sampling behavior and included a relative subjective cost parameter, Csub , for each bead-ratio condition. Higher trait anxiety was associated with more draws-to-decision, most robustly in the 90:10 bead-ratio condition. Only relative subjective cost of sampling decisions, and not inferential differences in weighting of new or old information, satisfactorily accounted for this relation. Specifically, lower Csub(0.9) was associated with more trait anxiety and more draws-to-decision. In a condition with high objective evidence strength, transdiagnostic trait-anxiety-related increases in information sampling were explained by a cost-benefit analysis where relatively higher subjective cost was assigned to an incorrect guess, highlighting valuation as a potential treatment target for future research.

Optimal Joint Defense and Monitoring for Networks Security under Uncertainty: A POMDP-Based Approach

The increasing interconnectivity in our infrastructure poses a significant security challenge, with external threats having the potential to penetrate and propagate throughout the network. Bayesian attack graphs have proven to be effective in capturing the propagation of attacks in complex interconnected networks. However, most existing security approaches fail to systematically account for the limitation of resources and uncertainty arising from the complexity of attacks and possible undetected compromises. To address these challenges, this paper proposes a partially observable Markov decision process (POMDP) model for network security under uncertainty. The POMDP model accounts for uncertainty in monitoring and defense processes, as well as the probabilistic attack propagation. This paper develops two security policies based on the optimal stationary defense policy for the underlying POMDP state process (i.e., a network with known compromises): the estimation-based policy that performs the defense actions corresponding to the optimal minimum mean square error state estimation and the distribution-based policy that utilizes the posterior distribution of network compromises to make defense decisions. Optimal monitoring policies are designed to specifically support each of the defense policies, allowing dynamic allocation of monitoring resources to capture network vulnerabilities/compromises. The performance of the proposed policies is examined in terms of robustness, accuracy, and uncertainty using various numerical experiments.

Cancer Screening Benefits Maximization Using Markov Decision Process Models: A Systematic Review

Context: Due to the chronic nature of cancer, screening programs were a set of sequential decisions taken over time. Markov decision process (MDP) and partially observable Markov decision process (POMDP) models were the mathematical tools applied in studies, including sequential decision-making such as screening protocols for medical decision-making. Objectives: The main goal of this study was to investigate optimal policy for cancer screening using MDP and POMDP models. Methods: We performed a review of articles published within July 2000 to November 2022 in PubMed, Web of Science, and Scopus databases. The stopping age, the type of optimal strategy, the benefits of the optimal policy, and the relationship between age and risk threshold were extracted. Studies that did not use MDPs and POMDPs as the mathematical maximization models in cancer screening, review articles, editorials or commentaries, non-English articles, and those that did not focus on optimization were excluded. Results: From 532 articles, 6 studies met the study criteria. All studies suggested that the optimal policy was control-limit, and the cancer risk threshold was a non-decreasing function of age. Three studies specified a stopping age for cancer screening. In five studies, the optimal policies outperformed the guidelines or no screening strategy. Conclusions: Two essential factors in screening decisions were cancer risk and age, which were individual variables. The control-limit policy included these factors in decision-making for cancer screening. These policies highlighted personalized screening and showed that this type of screening could outperform cancer screening guidelines regarding economic and clinical benefits.

A Reinforcement Learning Based Coordinated but Differentiated Load Frequency Control Method With Heterogeneous Frequency Regulation Resources

As the diversification of frequency regulation resources, it is increasingly important to find a load frequency control (LFC) scheme that can effectively coordinate different resources. This paper proposes a coordinated but differentiated LFC control method that utilizes heterogeneous frequency regulation resources with their characteristics fully considered. The optimal LFC models for the single area and multi-area power systems are formulated as partially observable Markov decision process model (POMDP) and partially observable Markov game (POMG) considering the accessibility of system information. A deep reinforcement learning method called recurrent proximal policy optimization is leveraged to solve the POMDP and POMG models. Simulations based on historical data demonstrate that the proposed control scheme can better coordinate different resources and achieve superior control performance.

A POMDP model-based online risk mitigation method for autonomous inland vessels

Autonomous surface vessels (ASVs) increasingly gain appeal in the maritime industry for their high efficiency and improved navigational capabilities. However, risks originating from various internal and external factors such as faults, traffic, harsh weather conditions, etc., can affect their guidance and control capabilities and impact nominal vessel operations. The existing risk mitigation methods mainly focus on the vessel's guidance system and do not consider unsafe actions due to the control system. In this paper, we propose a new method based on a partially observable Markov decision process (POMDP) model for the online risk mitigation of autonomous inland vessels. The POMDP model-based method utilizes information about situational awareness to assist the vessel's planning and control system in real-time decision-making during hazardous situations, thereby ensuring that the vessel remains in a minimum-risk condition. Based on the identified risk-influencing factors (RIFs), the transition probabilities are updated by a Bayesian belief network (BBN). A case study of an autonomous inland vessel navigating in a confined waterway is presented to demonstrate the capability of the proposed method.

Dynamic joint sensor selection and maintenance optimization in partially observable deteriorating systems

We consider a degrading system with costly but silent failures. The system can be partially observed using a set of heterogeneous noisy sensors at a given cost, where each sensor noise level is a function of the system state. The goal is to devise a dynamic joint sensor selection and maintenance optimization model to minimize the total expected discounted cost of the system. We develop an infinite-horizon discrete-time partially observable Markov decision process model to dynamically prescribe the sensor subsets for system monitoring and the timing of maintenance activities. We perform numerical experiments for a system with up to ten sensors and compare the proposed optimal policy with four heuristic policies. We observe that the proposed approach consistently outperforms all heuristic policies considered. Through a comprehensive analysis of numerical experiments, our study shows the efficacy of the proposed model’s optimal policy in strategically selecting sensors to achieve precise system state estimation while managing costs. Through these experiments, we find that it is optimal to use lower-cost, higher-variance sensors and often fuse them for more accurate estimations. However, the choice depends on the system state and sensor quality distribution. Importantly, our model excels in determining the best action by considering these factors, underscoring its practical applicability.

Demand response‐based autonomous voltage control for transmission and active distribution networks using modified partially observed Markov decision process model

AbstractTo fully utilize the voltage regulation capacity and interaction characteristics of the Transmission and Distribution (T&D) system, a novel Modified Partially Observed Markov Decision Process (MPOMDP)‐based Reinforcement Learning (RL) scheme for Autonomous Voltage Control is proposed, which incorporates Demand Response (DR) and cooperation with the Transmission Network . The proposed scheme consists of two vital components: an MPOMDP block and a Modified Asynchronous Advantage Actor‐Critic‐based RL block. The MPOMDP block innovatively exploits the confidence interval of the observed state to make a better perception of the precise system state by introducing two new probability vectors. Then the MPOMDP block is fed into the underlying architecture of the RL block for asynchronously capturing features and optimal decision‐making, where the solving framework additionally brings in a public data buffer to realize boundary information sharing. Case studies are conducted on a modified T&D system considering N‐1 contingencies, with a training dataset from a district in Suzhou, China. Simulation results demonstrate that the proposed scheme can achieve significant voltage optimization while ensuring fast convergence speed.

Optimal Scheduling in IoT-Driven Smart Isolated Microgrids Based on Deep Reinforcement Learning

In this paper, we investigate the scheduling issue of diesel generators (DGs) in an Internet of Things (IoT)-Driven isolated microgrid (MG) by deep reinforcement learning (DRL). The renewable energy is fully exploited under the uncertainty of renewable generation and load demand. The DRL agent learns an optimal policy from history renewable and load data of previous days, where the policy can generate real-time decisions based on observations of past renewable and load data of previous hours collected by connected sensors. The goal is to reduce operating cost on the premise of ensuring supply-demand balance. In specific, a novel finite-horizon partial observable Markov decision process (POMDP) model is conceived considering the spinning reserve. In order to overcome the challenge of discrete-continuous hybrid action space due to the binary DG switching decision and continuous energy dispatch (ED) decision, a DRL algorithm, namely the hybrid action finite-horizon RDPG (HAFH-RDPG), is proposed. HAFH-RDPG seamlessly integrates two classical DRL algorithms, i.e., deep Q-network (DQN) and recurrent deterministic policy gradient (RDPG), based on a finite-horizon dynamic programming (DP) framework. Extensive experiments are performed with real-world data in an IoT-driven MG to evaluate the capability of the proposed algorithm in handling the uncertainty due to inter-hour and inter-day power fluctuation and to compare its performance with those of the benchmark algorithms.

A Corrosion Maintenance Model Using Continuous State Partially Observable Markov Decision Process for Oil and Gas Pipelines

This paper proposes a continuous state partially observable Markov decision process (POMDP) model for the corrosion maintenance of oil and gas pipelines. The maintenance operations include complex and extensive activities to detect the corrosion type, determine its severity, predict the deterioration rate, and plan future inspection (monitoring) schemes and maintenance policy. A POMDP model is formulated as a decision-making support tool to effectively handle partially observed corrosion defect levels. It formulates states as the pipeline’s degradation level using a probability distribution. Inline inspection (ILI) methods estimate the latest state of the pipeline, which also defines the initial state of the optimization process. The set of actions comprises corrosion mitigation operations. The errors associated with the ILI method are used to construct the observation function for the model. The sum of inspection, maintenance operations, and failure costs for a given state and action are formulated as rewards. Numerical experiments are made based on data collected from the literature. The results showed that different policies, whether derived from solvers (theoretical) or determined from practical experience, can be compared to identify the best maintenance alternative using the model. It was also observed that the choice of the solvers is important since they affect the discounted rewards and the run time to obtain them. The model approximates the parameters and uncertainty associated with the propagation of corrosion, proficiency of inspection methods, and implementation of maintenance policies. Overall, it can be applied to improve the maintenance decision-making process for the oil and gas pipeline as it incorporates the stochastic features of the operation.

A Supply Chain Inventory Management Method for Civil Aircraft Manufacturing Based on Multi-Agent Reinforcement Learning

Effective supply chain inventory management is crucial for large-scale manufacturing industries such as civil aircraft and automobile manufacturing to ensure efficient manufacturing. Generally, the main manufacturer makes the annual inventory management plan, and contacts with suppliers when some material is approaching critical inventory level according to the actual production schedule, which increases the difficulty of inventory management. In recent years, many researchers have focused on using reinforcement learning method to study inventory management problems. Current approaches were mainly designed for the supply chain with single-node multi-material or multi-node single-material mode, which are not suitable to the civil aircraft manufacturing supply chain with multi-node multi-material mode. To deal with this problem, we formulated the problem as a partially observable Markov decision process (POMDP) model and proposed a multi-agent reinforcement learning method for supply chain inventory management, in which the dual-policy and information transmission mechanism was designed to help the supply chain participant improve the global information utilization efficiency of the supply chain and the coordination efficiency with other participants. The experiment results show that our method has about 45% performance improvement on efficiency compared with current reinforcement learning-based methods.

A parallel game model‐based intrusion response system for cross‐layer security in industrial internet of things

SummaryWith the rise of industrialization, the importance of the industrial Internet of Things (IIoT) has increased significantly, and with it comes a variety of security threats. Therefore, the security of these networks is critical. Industrial Response Systems (IRSs), as the last line of security, plays an important role in the security system of the Industrial Internet of Things. In this paper, a new IRS model based on the non‐cooperative game is proposed. First, by combining the Partially Observable Markov Decision Process (POMDP) model with the stochastic game model based on the expanded attack tree, our model could effectively perceive the changes at each node. Second, our model incorporates the alarms of intrusion detection system (IDS) and the physical quantities of sensors in Industrial Cyber‐Physical System (ICPS) into the quantization system so that the model can respond to intruders more accurately and comprehensively. Finally, we develop this model based on multiprocessors to speed up the solution process, and adopt an approximation algorithm to reduce the number of iterations of the POMDP

FABRIC: A Framework for the Design and Evaluation of Collaborative Robots with Extended Human Adaptation

A limitation for collaborative robots (cobots) is their lack of ability to adapt to human partners, who typically exhibit an immense diversity of behaviors. We present an autonomous framework as a cobot’s real-time decision-making mechanism to anticipate a variety of human characteristics and behaviors, including human errors, toward a personalized collaboration. Our framework handles such behaviors in two levels: (1) short-term human behaviors are adapted through our novel Anticipatory Partially Observable Markov Decision Process (A-POMDP) models, covering a human’s changing intent (motivation), availability, and capability; (2) long-term changing human characteristics are adapted by our novel Adaptive Bayesian Policy Selection (ABPS) mechanism that selects a short-term decision model, e.g., an A-POMDP, according to an estimate of a human’s workplace characteristics, such as her expertise and collaboration preferences. To design and evaluate our framework over a diversity of human behaviors, we propose a pipeline where we first train and rigorously test the framework in simulation over novel human models. Then, we deploy and evaluate it on our novel physical experiment setup that induces cognitive load on humans to observe their dynamic behaviors, including their mistakes, and their changing characteristics such as their expertise. We conduct user studies and show that our framework effectively collaborates non-stop for hours and adapts to various changing human behaviors and characteristics in real-time. That increases the efficiency and naturalness of the collaboration with a higher perceived collaboration, positive teammate traits, and human trust. We believe that such an extended human-adaptation is a key to the long-term use of cobots.

A multi-objective constrained partially observable Markov decision process model for breast cancer screening

A multi-objective constrained partially observable Markov decision process model for breast cancer screening

Condition-based pavement management systems accounting for model uncertainty and facility heterogeneity with belief updates

Due to the tighter budget for pavement management, schedules of inspection activities should be jointly optimized with the maintenance and reconstruction (M&R) plans for pavement systems. Conducting inspections every year is unnecessary and will decrease the budget for M&R activities, while infrequent inspections may lead to suboptimal M&R planning due to the lack of accurate information. This paper presents a methodology for jointly optimizing the inspection scheduling and M&R planning for pavement systems, considering model uncertainty and facility-specific heterogeneity. The problem is defined as a Partially Observable Markov Decision Process (POMDP) model, accounting for the tradeoff between the information value and inspection costs. Moreover, a statistical learning method is used to update the prediction of pavement conditions using the collected inspection data and, eventually, improve the condition-based decisions. This “belief update” process can gradually reduce the model uncertainty as the dataset size increases. We demonstrate the proposed stochastic optimization framework through a numerical example with a system of fifty heterogenous pavement facilities under a combined budget for inspection and M&R activities. Several managerial insights and implications are discussed. For example, the optimal inspection frequencies are less sensitive to the budget; and the agency should perform fewer reconstructions and more rehabilitations when the budget is limited.

A Discrete Partially Observable Markov Decision Process Model for the Maintenance Optimization of Oil and Gas Pipelines

Corrosion is one of the major causes of failure in pipelines for transporting oil and gas products. To mitigate the impact of this problem, organizations perform different maintenance operations, including detecting corrosion, determining corrosion growth, and implementing optimal maintenance policies. This paper proposes a partially observable Markov decision process (POMDP) model for optimizing maintenance based on the corrosion progress, which is monitored by an inline inspection to assess the extent of pipeline corrosion. The states are defined by dividing the deterioration range equally, whereas the actions are determined based on the specific states and pipeline attributes. Monte Carlo simulation and a pure birth Markov process method are used for computing the transition matrix. The cost of maintenance and failure are considered when calculating the rewards. The inline inspection methods and tool measurement errors may cause reading distortion, which is used to formulate the observations and the observation function. The model is demonstrated with two numerical examples constructed based on problems and parameters in the literature. The result shows that the proposed model performs well with the added advantage of integrating measurement errors and recommending actions for multiple-state situations. Overall, this discrete model can serve the maintenance decision-making process by better representing the stochastic features.

A Dynamic Deep Reinforcement Learning-Bayesian Framework for Anomaly Detection

To assure the successful operation of connected and automated vehicles, it is critical to detect and isolate anomalous and/or faulty information in a timely manner. To do so, anomaly detection techniques should be implemented in real-time where if the probability of anomalous information exceeds a certain threshold, the information is dealt with accordingly. Traditionally, the threshold for judging whether the data is anomalous is fixed and determined a priori. However, not only does this approach fail to account for the feedback obtained during a trip on the performance of the algorithms, but it also fails to respond to potential changes in rates of anomalies. Hence, it is important to develop an approach that can dynamically alter this threshold in response to exogenous factors to assure reliable and robust system operation. We develop a mathematical framework which utilizes a dynamic threshold for an anomaly classification algorithm in order to maximize the safety of a trip. Specifically, we develop and pair an anomaly classification algorithm based on convolutional neural networks (CNN), with a partially observable Markov decision process (POMDP) model. We solve the resulting POMDP model using the asynchronous advantage actor critic (A3C) deep reinforcement learning algorithm. The prescribed policy determines the anomaly classification threshold in real-time that maximizes the performance. Our numerical experiments show that the POMDP model outperforms state-of-the-art benchmarks, especially under more difficult to detect anomaly profiles.

Optimization of active surveillance strategies for heterogeneous patients with prostate cancer

Prostate cancer (PCa) is common in American men with long latent periods, during which the disease is asymptomatic. Active surveillance is a monitoring strategy commonly used for patients diagnosed with low‐risk PCa who may harbor latent high‐risk PCa. The optimal monitoring strategy attempts to minimize the disutility of testing while ensuring that the patient is detected at the earliest time when the disease progresses. Unfortunately, guidelines for the active surveillance of PCa are often one‐size‐fits‐all strategies that ignore the heterogeneity among multiple patient types. In contrast, personalized strategies based on partially observable Markov decision process (POMDP) models are challenging to implement in practice given the large number of possible strategies that can be used. This article presents a two‐stage stochastic programming approach that selects a set of strategies for predefined cardinality based on patients' disease risks. The first‐stage decision variables include binary variables for the selection of periods at which to test patients in each strategy and the assignment of multiple patient types to strategies. The objective is to maximize a weighted reward function that considers the need for cancer detection, missed detection, and cost of monitoring patients. We discuss the structure and complexity of the model and reformulate a logic‐based Bender's decomposition formulation that can solve realistic instances to optimality. We present a case study for the active surveillance of PCa and show that our model results in strategies that vary in intensity according to patient disease risk. Finally, we show that our model can generate a small number of strategies that can significantly improve the existing “one‐size‐fits‐all” guideline strategies used in practice.

Screening for preclinical Alzheimer's disease: Deriving optimal policies using a partially observable Markov model.

Alzheimer's Disease (AD) is believed to be the most common type of dementia. Even though screening for AD has been discussed widely, there is no screening program implemented as part of a policy in any country. Current medical research motivates focusing on the preclinical stages of the disease in a modeling initiative. We develop a partially observable Markov decision process model to determine optimal screening programs. The model contains disease free and preclinical AD partially observable states and the screening decision is taken while an individual is in one of those states. An observable diagnosed preclinical AD state is integrated along with observable mild cognitive impairment, AD and death states. Transition probabilities among states are estimated using data from Knight Alzheimer's Disease Research Center (KADRC) and relevant literature. With an objective of maximizing expected total quality-adjusted life years (QALYs), the output of the model is an optimal screening program that specifies at what points in time an individual over 50 years of age with a given risk of AD will be directed to undergo screening. The screening test used to diagnose preclinical AD has a positive disutility, is imperfect and its sensitivity and specificity are estimated using the KADRC data set. We study the impact of a potential intervention with a parameterized effectiveness and disutility on model outcomes for three different risk profiles (low, medium and high). When intervention effectiveness and disutility are at their best, the optimal screening policy is to screen every year between ages 50 and 95, with an overall QALY gain of 0.94, 1.9 and 2.9 for low, medium and high risk profiles, respectively. As intervention effectiveness diminishes and/or its disutility increases, the optimal policy changes to sporadic screening and then to never screening. Under several scenarios, some screening within the time horizon is optimal from a QALY perspective. Moreover, an in-depth analysis of costs reveals that implementing these policies are either cost-saving or cost-effective.

Data pooling for multiple single-component systems under population heterogeneity

We consider multiple newly designed single-component systems with a known finite lifespan. The component in each system can be replaced preventively to avoid a costly failure. This component is also in use for the first time, and therefore, there is no historical data on the lifetime of the component. In this case, the probability distribution of the component lifetime can be estimated based on expert opinions. However, there can be different opinions on the lifetime distribution. There are two populations where components can come from: a weak and a strong population. We assume that the components always come from the same population. However, the true type of the population is unknown. We build a discrete-time partially observable Markov decision process model to find the optimal replacement policy which minimizes the expected total cost throughout the lifespan. To resolve the uncertainty regarding population heterogeneity, we update the belief by using the data collected from all systems, allowing us to investigate the effect of so-called data pooling. First, we generate insights about the structure of the optimal policy. We then compare the cost per system under the optimal policy with the cost per system under two benchmark heuristics that follow the single-system optimal policy with and without data pooling, respectively. In our numerical experiments, we show that the cost reduction relative to the worst benchmark heuristic can be up to 5.6% for data pooling with two systems, and this increases up to 14% for 20 systems. Additionally, the effect of various input parameters on the costs is analyzed.