Markov Decision Process Model Research Articles

Markov-Decision-Process-Based Reinforcement Learning for Emergency Vehicles of Dispatch and Reallocation Strategies through the Lens of Efficiency and Fairness: Evidence from Chinese Emergency Medical Services System

Efficient proactive emergency vehicle planning is crucial for effective emergency response management systems. This paper develops a flexible deep reinforcement learning framework to address uncertainties in dynamic emergency requests and complex traffic conditions. The Markov decision process model is proposed to optimize ambulance dispatch and reallocation, focusing on minimizing average response time, reducing delays from traffic congestion and patient severity misclassification, and enhancing general fairness. To tackle the challenges of the unbounded MDP state space, we employ a least-squares-based approximate policy iteration model for the upper bound and a linear programming model for the lower bound. We use Chinese emergency medical services system data for computational experiments and select average response time, fraction of delay, risk level, and Gini coefficient to evaluate the performance of our model. The numerical results demonstrate our optimal flexible policy outperforms the first-in-first-served rule, improving both efficiency and equity in medical decision-making.

Optimal acceptance of incompatible kidneys

Incompatibility between patient and donor is a major barrier in kidney transplantation (KT). The increasing shortage of kidney donors has driven the development of desensitization techniques to overcome this immunological challenge. Compared with compatible KT, patients undergoing incompatible KTs are more likely to experience rejection, infection, malignancy, and graft loss. We study the optimal acceptance of possibly incompatible kidneys for individual end-stage kidney disease patients. To capture the effects of incompatibility, we propose a Markov Decision Process (MDP) model that explicitly includes compatibility as a state variable. The resulting higher-dimensional model makes it more challenging to analyze, but under suitable conditions, we derive structural properties including control limit-type optimal policies that are easy to compute and implement. Numerical examples illustrate the behavior of the optimal policy under different mismatch levels and highlight the importance of explicitly incorporating the incompatibility level into the acceptance decision when desensitization therapy is an option.

On the Concept of Opportunity Cost in Integrated Demand Management and Vehicle Routing

Integrated demand management and vehicle routing problems are characterized by a stream of customers arriving dynamically over a booking horizon and requesting logistical services, fulfilled by a given fleet of vehicles during a service horizon. Prominent examples are attended home delivery and same-day delivery problems, where customers commonly have heterogeneous preferences regarding service fulfillment and requests differ in profitability. Thus, demand management methods are applied to steer the booking process to maximize total profit considering the cost of the routing decisions for the resulting orders. To measure the requests’ profitability for any demand management method, it is common to estimate their opportunity cost. In the context of integrated demand management and vehicle routing problems, this estimation differs substantially from the estimation in the well-examined demand management problems of traditional revenue management applications as, for example, found in the airline or car rental industry. This is because of the unique interrelation of demand control decisions and vehicle routing decisions as it inhibits a clear quantification and attribution of cost, and of displaced revenue, to certain customer requests. In this paper, we extend the theoretical foundation of opportunity cost in integrated demand management and vehicle routing problems. By defining and analyzing a generic Markov decision process model, we formally derive a definition of opportunity cost and prove opportunity cost properties on a general level. Hence, our findings are valid for a wide range of specific problems. Further, based on these theoretical findings, we propose approximation approaches that have not yet been applied in the existing literature, and evaluate their potential in a computational study. Thereby, we provide evidence that the theoretical results can be practically exploited in the development of solution algorithms. Funding: This work was supported by the University of the Bundeswehr Munich. Supplemental Material: The online appendices are available at https://doi.org/10.1287/trsc.2024.0644 .

A study on inventory control system for a supply chain using Markov decision processes

This paper examines the application of Markov Decision Processes (MDPs) for controlling supply chain inventories. MDPs effectively simulate decision-making problems related to uncertainty, facilitating the determination of optimal inventory policies. The MDP framework addresses various inventory management challenges, including demand fluctuations, lead times, and holding costs. The study investigates the modeling of inventory management as a Markov decision process, detailing the states, actions, and transitions within the MDP model, along with their respective advantages and disadvantages. The research employs policy and value iteration techniques to evaluate and optimize inventory management policies. The paper assesses the proposed MDP-based inventory control system through simulations that utilize supply chain data, aiming to identify optimal policies using the MDP model. A comparative analysis of the MDP approach against conventional inventory management methods is conducted to demonstrate its efficacy in reducing costs and enhancing service levels. Additionally, the paper proposes the incorporation of multiple commodities, multi-echelon supply chains, and perishability considerations into the MDP model. The findings indicate that MDPs facilitate improved optimization of inventory policies, cost reductions, and enhanced customer service, thereby emphasizing the significance of computational complexity and the necessity for accurate data. This research provides a comprehensive investigation into the role of MDPs within inventory control systems, contributing valuable insights to the field of supply chain management. Ultimately, this study lays the groundwork for advancements in MDP-based inventory control methodologies.

Deep Q-Network-Enhanced Self-Tuning Control of Particle Swarm Optimization

Particle Swarm Optimization (PSO) is a widespread evolutionary technique that has successfully solved diverse optimization problems across various application fields. However, when dealing with more complex optimization problems, PSO can suffer from premature convergence and may become stuck in local optima. The primary goal is accelerating convergence and preventing solutions from falling into these local optima. This paper introduces a new approach to address these shortcomings and improve overall performance: utilizing a reinforcement deep learning method to carry out online adjustments of parameters in a homogeneous Particle Swarm Optimization, where all particles exhibit identical search behaviors inspired by models of social influence among uniform individuals. The present method utilizes an online parameter control to analyze and adjust each primary PSO parameter, particularly the acceleration factors and the inertia weight. Initially, a partially observed Markov decision process model at the PSO level is used to model the online parameter adaptation. Subsequently, a Hidden Markov Model classification, combined with a Deep Q-Network, is implemented to create a novel Particle Swarm Optimization named DPQ-PSO, and its parameters are adjusted according to deep reinforcement learning. Experiments on different benchmark unimodal and multimodal functions demonstrate superior results over most state-of-the-art methods regarding solution accuracy and convergence speed.

Optimal Control of Service Systems with Heterogeneous Servers and Priority Customers

We study service systems with parallel servers and random customer arrivals and focus on the waiting cost of customers. Using a Markov decision process (MDP) modeling approach, we analytically characterize the structures of the optimal dynamic server assignment policies for two important systems, one consisting of multiple homogeneous servers and two classes of customers and the other consisting of two heterogeneous servers and multiple classes of customers. Based on the obtained results, we propose a threshold-type heuristic policy for the generalized system consisting of multiple heterogeneous servers and multiple classes of customers. To design such a heuristic policy, we first develop techniques for the performance evaluation of general threshold-type policies with any given threshold values. We then construct a path to search for the optimal threshold values. We compare the performance of the best threshold-type heuristic policy with that of the optimal policy and show that our proposed heuristic policy is computationally efficient yet generates great performance. To derive additional managerial insights, we compare the system under our threshold-type dynamic server assignment policy with other commonly seen and simple systems, such as the dedicated system and the work-conserving flexible priority system. The clear performance advantage observed from extensive numerical experiments demonstrates the importance and usefulness of dynamic server assignment control for systems serving multiple classes of customer arrivals. Finally, we extend our analysis to incorporate customer-dependent service rates and sojourn-time minimization performance metrics. This paper was accepted by Chung Piaw Teo, optimization. Funding: This work was supported by the Hong Kong Research Grants Council [16500921, 16505819] and the National Natural Science Foundation of China [72201233, 72301222]. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2023.01228 .

Transdiagnostic Anxiety-Related Increases in Information Sampling are Associated With Altered Valuation.

Excessive information sampling in psychiatric patients characterized by high trait anxiety has been inconsistently linked with alterations in inferential and valuation processes. Methodological limitations could account in part for these inconsistencies. To address this, computational models of inference and valuation were applied to data collected from a transdiagnostic sample of adults with and without an anxiety or compulsive disorder using a version of the beads task with enhanced experimental controls. Participants diagnosed with an anxiety or compulsive disorder (n = 35) and healthy controls (n = 23) completed the beads task with three majority-to-minority ratios of blue versus green beads (60:40, 75:25, 90:10). First, a Bayesian belief-updating model was fit to quantify the iterative process by which new information (bead color) and prior beliefs were integrated to influence current beliefs about jar identity. Next, a parameterized partially observable Markov decision process model was used to parse the contribution of value-based decisions to sampling behavior and included a relative subjective cost parameter, Csub , for each bead-ratio condition. Higher trait anxiety was associated with more draws-to-decision, most robustly in the 90:10 bead-ratio condition. Only relative subjective cost of sampling decisions, and not inferential differences in weighting of new or old information, satisfactorily accounted for this relation. Specifically, lower Csub(0.9) was associated with more trait anxiety and more draws-to-decision. In a condition with high objective evidence strength, transdiagnostic trait-anxiety-related increases in information sampling were explained by a cost-benefit analysis where relatively higher subjective cost was assigned to an incorrect guess, highlighting valuation as a potential treatment target for future research.

Optimization of emergency frequency control strategy for power systems considering both source and load uncertainties

With the increasing integration of renewable energy sources and the presence of numerous controllable loads such as electric vehicles and energy storage in the modern power system, higher nonlinearities and uncertainty both sources and loads are introduced. These factors pose challenges in achieving fast and accurate emergency frequency control. Therefore, this paper addresses the issue of dual source-load uncertainties in power system and presents an optimization strategy based on the Soft Actor Critic (SAC) algorithm that involves the participation of controllable loads in emergency frequency control. Firstly, the spatio-temporal uncertainties of wind farm power output on power supply side and power demand on the load side are described using Weibull and normal probability distributions, respectively. Secondly, an improved Markov Decision Process (MDP) model for emergency frequency control is established, which considers the characteristics of the dual source-load uncertainties. Finally, an optimization of the SAC algorithm is conducted based on Deep Reinforcement Learning (DRL), aiming to achieve rapid system frequency recovery and minimize the cost of removing controllable loads. The presented approach in the paper enhances the emergency frequency control strategy for uncertain power systems and effectively addresses the issue of source-load uncertainty compounded by fault power shortages.

Optimal Timing of Organs-at-Risk-Sparing Adaptive Radiation Therapy for Head- and-Neck Cancer under Re-planning Resource Constraints.

Prior work on adaptive organ-at-risk (OAR)-sparing radiation therapy has typically reported outcomes based on fixed-number or fixed-interval re-planning, which represent one-size-fits-all approaches and do not account for the variable progression of individual patients' toxicities. The purpose of this study was to determine the personalized optimal timing for re-planning in adaptive OAR-sparing radiation therapy, considering limited re-planning resources, for patients with head and neck cancer (HNC). A novel Markov decision process (MDP) model was developed to determine optimal timing of re-planning based on the patient's expected toxicity, characterized by normal tissue complication probability (NTCP), for four toxicities. The MDP parameters were derived from a dataset comprising 52 HNC patients treated at the University of Texas MD Anderson Cancer Center between 2007 and 2013. Kernel density estimation was used to smooth the sample distributions. Optimal re-planning strategies were obtained when the permissible number of re-plans throughout the treatment was limited to 1, 2, and 3, respectively. The MDP (optimal) solution recommended re-planning when the difference between planned and actual NTCPs (ΔNTCP) was greater than or equal to 1%, 2%, 2%, and 4% at treatment fractions 10, 15, 20, and 25, respectively, exhibiting a temporally increasing pattern. The ΔNTCP thresholds remained constant across the number of re-planning allowances (1, 2, and 3). In limited-resource settings that impeded high-frequency adaptations, ΔNTCP thresholds obtained from an MDP model could derive optimal timing of re-planning to minimize the likelihood of treatment toxicities.

Medication recommendation for Parkinson’s disease based on dynamics of symptom progression

Parkinson’s Disease (PD) is a chronic condition, with extensive research on initial medication selection for treatment, but limited guidance on long-term medication management. This study aims to identify optimal medication adjustment strategies based on patient clusters, focusing on either maximizing time spent in favorable health states or minimizing time spent in unfavorable ones while avoiding adverse effects. To guide treatment, we developed decision models using prescription dosages converted into standardized units for various medications. Using data from the Parkinson’s Progression Markers Initiative, we employed a multivariate time-series clustering approach to capture symptom progression dynamics. This analysis identified four distinct clusters: two representing desirable and undesirable states, and two highlighting motor-focused and non-motor-focused failures across multiple domains. We developed two separate Markov Decision Process (MDP) models to address these dual objectives, which were then integrated into a comprehensive framework that suggests optimal actions and cautions against risky ones for each patient state. This model provides valuable insights for clinical decision-making by offering flexible guidance on adjusting medication intensity rather than prescribing specific medication types, enhancing its applicability in clinical practice.

Discounted fully probabilistic design of decision rules

Axiomatic fully probabilistic design (FPD) of optimal decision rules strictly extends the decision making (DM) theory represented by Markov decision processes (MDP). This means that any MDP task can be approximated by an explicitly found FPD task whereas many FPD tasks have no MDP equivalent. MDP and FPD model the closed loop — the coupling of an agent and its environment — via a joint probability density (pd) relating the involved random variables, referred to as behaviour. Unlike MDP, FPD quantifies agent's aims and constraints by an ideal pd. The ideal pd is high on the desired behaviours, small on undesired behaviours and zero on forbidden ones. FPD selects the optimal decision rules as the minimiser of Kullback-Leibler's divergence of the closed-loop-modelling pd to its ideal twin. The proximity measure choice follows from the FPD axiomatics.MDP minimises the expected total loss, which is usually the sum of discounted partial losses. The discounting reflects the decreasing importance of future losses. It also diminishes the influence of errors caused by:▪ the imperfection of the employed environment model;▪ roughly-expressed aims;▪ the approximate learning and decision-rules design.The established FPD cannot currently account for these important features. The paper elaborates the missing discounted version of FPD. This non-trivial filling of the gap in FPD also employs an extension of dynamic programming, which is of an independent interest.

Deep reinforcement learning-based spectrum resource allocation for the web of healthcare things with massive integrating wearable gadgets

With the development of the future Web of Healthcare Things (WoHT), there will be a trend of densely deploying medical sensors with massive simultaneous online communication requirements. The dense deployment and simultaneous online communication of massive medical sensors will inevitably generate overlapping interference. This will be extremely challenging to support data transmission at the medical-grade quality of service level. To handle the challenge, this paper proposes a hypergraph interference coordination-aided resource allocation based on the Deep Reinforcement Learning (DRL) method. Specifically, we build a novel hypergraph interference model for the considered WoHT by analyzing the impact of the overlapping interference. Due to the high complexity of directly solving the hypergraph interference model, the original resource allocation problem is converted into a sequential decision-making problem through the Markov Decision Process (MDP) modeling method. Then, a policy and value-based resource allocation algorithm is proposed to solve this problem under simultaneous online communication and dense deployment. In addition, to enhance the exploration ability of the optimal allocation strategy for the agent, we propose a resource allocation algorithm with an asynchronous parallel architecture. Simulation results verify that the proposed algorithms can achieve higher network throughput than the existing algorithms in the considered WoHT scenario.

A Deep Reinforcement Learning Approach to Injection Speed Control in Injection Molding Machines with Servomotor-Driven Constant Pump Hydraulic System

The control of the injection speed in hydraulic injection molding machines is critical to product quality and production efficiency. This paper analyzes servomotor-driven constant pump hydraulic systems in injection molding machines to achieve optimal tracking control of the injection speed. We propose an efficient reinforcement learning (RL)-based approach to achieve fast tracking control of the injection speed within predefined time constraints. First, we construct a precise Markov decision process model that defines the state space, action space, and reward function. Then, we establish a tracking strategy using the Deep Deterministic Policy Gradient RL method, which allows the controller to learn optimal policies by interacting with the environment. Careful attention is also paid to the network architecture and the definition of states/actions to ensure the effectiveness of the proposed method. Extensive numerical results validate the proposed approach and demonstrate accurate and efficient tracking of the injection velocity. The controller’s ability to learn and adapt in real time provides a significant advantage over the traditional Proportion Integration Differentiation controller. The proposed method provides a practical solution to the challenge of maintaining accurate control of the injection speed in the manufacturing process.

Optimal Learning and Management of Threatened Species

Amid an unprecedented loss of biodiversity, a pressing issue is how to improve the efficiency of conservation with limited resources and information. Collecting data on species with a small population is costly and time consuming, and many high-stakes decisions need to be made based on limited data. We develop a partially observable Markov decision processes model with unknown parameters to jointly optimize the information collection and protection efforts for threatened species. The model takes into account uncertainties about the state, detectability, and dynamics of the species, and it adaptively adjusts the efforts of surveying and protection in real time. Although the standard formulation is intractable, we exploit the structure of ecological problems to identify a hybrid belief state in low dimensions, and we reformulate the stochastic dynamic program as a piecewise deterministic optimal control problem. This enables us to obtain structural insights into the optimal policy (some are in closed form) and find a near-optimal approximate policy with performance guarantee. In certain situations, areas where the species has never been found may be more likely to contain the species than areas where it has been previously found. We also conduct a case study on the conservation of the Hainan gibbon, the rarest primate species, in which we extend the model to optimize the spatiotemporal allocation of limited resources. This paper was accepted by Chung Piaw Teo, optimization. Funding: This work was supported by the Natural Sciences and Engineering Research Council of Canada [Grant RGPIN-2019-05671]. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2023.01753 .

Biomanufacturing Harvest Optimization With Small Data

In biopharmaceutical manufacturing, fermentation processes play a critical role in productivity and profit. A fermentation process uses living cells with complex biological mechanisms, leading to high variability in the process outputs, namely, the protein and impurity levels. By building on the biological mechanisms of protein and impurity growth, we introduce a stochastic model to characterize the accumulation of the protein and impurity levels in the fermentation process. However, a common challenge in the industry is the availability of only a very limited amount of data, especially in the development and early stages of production. This adds an additional layer of uncertainty, referred to as model risk, due to the difficulty of estimating the model parameters with limited data. In this article, we study the harvesting decision for a fermentation process (i.e., when to stop the fermentation and collect the production reward) under model risk. We adopt a Bayesian approach to update the unknown parameters of the growth-rate distributions, and use the resulting posterior distributions to characterize the impact of model risk on fermentation output variability. The harvesting problem is formulated as a Markov decision process model with knowledge states that summarize the posterior distributions and hence incorporate the model risk in decision-making. Our case studies at MSD Animal Health demonstrate that the proposed model and solution approach improve the harvesting decisions in real life by achieving substantially higher average output from a fermentation batch along with lower batch-to-batch variability.

Heatmap Design for Probabilistic Driver Repositioning in Crowdsourced Delivery

This paper proposes the use of heatmaps as a control lever to manage the probabilistic repositioning of independent drivers in crowdsourced delivery platforms. The platform aims to maximize order fulfillment by dynamically matching drivers and orders and selecting heatmaps that trigger the probabilistic flow of unmatched drivers to balance driver supply and delivery requests across the service region. We develop a Markov decision process (MDP) model to sequentially select matching and heatmap decisions in which the repositioning behavior of drivers is captured by a multinomial logit discrete choice model. Because of the curse of dimensionality and the endogenous uncertainty of driver repositioning, the MDP model is solved using a rolling-horizon stochastic lookahead policy. This policy decomposes matching and heatmap decisions into two optimization problems: a two-stage stochastic programming upper bounding problem for matching decisions and a mixed-integer programming problem for heatmap decisions. We also propose a simple policy for efficiently solving large-scale problems. An extensive computational study on instances derived from the Chicago ride-hailing data set is conducted. Computational experiments demonstrate the value of heatmaps in improving order fulfillment beyond the level achieved by matching alone (up to 25%) and identify conditions that affect the benefit of using heatmaps to guide driver repositioning. Funding: The authors gratefully acknowledge the support of the Natural Sciences and Engineering Research Council of Canada through Discovery Grants [Grants RGPIN-2024-04881, RGPIN-2020-04498, and RGPIN-2019-06207] awarded to the first, second, and third authors, respectively. Supplemental Material: The online appendix is available at https://doi.org/10.1287/trsc.2022.0418 .

Dynamic tugboat deployment and scheduling with stochastic and time-varying service demands

Container ports serve as crucial logistics hubs in global supply chains, but navigating ships within such ports is complex due to restricted waterways. Tugboats play a critical role in ensuring safety and efficiency by escorting and towing ships under these conditions. However, the tugboat deployment and scheduling problem has received little attention. To fill the research gap, we propose a new research problem - the dynamic tugboat deployment and scheduling problem, in which not all requests are confirmed initially but dynamically confirmed over time and future tugging demands need to be anticipated when managing the utilization of tugboats. To formulate the problem, we propose an extended Markov decision process (MDP) that incorporates both reactive task assignment decisions and proactive tugboat waiting decisions, creating a reactive and proactive MDP. To solve the advanced MDP model efficiently for real-time decisions, we develop an anticipatory approximate dynamic programming method that incorporates appropriate task assignment and waiting strategies for deploying and scheduling a heterogeneous tugboat fleet and embed the method into an improved rollout algorithm to anticipate future scenarios. The effectiveness, efficiency, and performance sensitivity of the developed modeling and solution methods are demonstrated via extensive numerical experiments for the Singapore container port.

Resilience enhancement of power distribution system using fixed and mobile emergency generators based on deep reinforcement learning

Growing extreme weather-related events have increased the need for a resilient emergency response in the Power Distribution Systems (PDSs). This paper proposes a novel restoration scheme that involves forming Micro-Grids (MGs) and dispatching Mobile Emergency Generators (MEGs) between them. Due to the PDS variability and extensive decision space, conventional model-based approaches aren't suitable for implementing the proposed scheme. Therefore, a model-free framework is designed to control the restoration process. The proposed problem is formulated as a Markov Decision Process (MDP) considering uncertainties associated with load profile, and renewable resources. Deep Q-Network (DQN), a fundamental algorithm in Deep Reinforcement Learning (DRL), is employed to solve the proposed MDP model. DQN algorithm can learn optimal control strategy by interacting with the simulated environment at the training phase and then deploy the learned policy in a real-world application. The main objective of the designed DQN model is to maximize the restored loads and ensure a high level of post-restoration reliability for Critical Loads (CLs). MATLAB software is considered as the simulated environment to perform power flow calculation and MEGs routing. The efficiency and applicability of the proposed method are evaluated by conducting experiments on the modified IEEE 13-bus, 37-bus, and 123-bus networks.

Inventory rationing, admission control, and production capacity allocation in a make‐to‐stock/make‐to‐order manufacturing system

AbstractThis paper considers a manufacturing system in which products are produced in both make‐to‐stock (MTS) and make‐to‐order (MTO) modes. Production of MTS and MTO products is done in batches, incurs a setup cost, and is non‐preemptive. The inventory of MTS products fulfills the demand of multiple classes, and each class demand can be satisfied or rejected. Customer orders for MTO production can be accepted or rejected, and their size is the same as the production batch. The primary goal of this paper is to study a policy that coordinates inventory rationing, admission control, and production capacity allocation to maximize the system's profit. We formulate the problem as a Markov decision process model and identify the structure of optimal control policies. We investigate the effect of inventory rationing on the profit by comparing its performance to that of the system with a first‐come‐ first‐serve policy to allocate inventory to multiple demand classes and study the extent to which the benefit of inventory rationing can be affected by system parameter changes. We also propose a heuristic that manages control decisions from linear threshold functions. Our test results from numerical examples show that the average percentage difference between the optimal and heuristic policies is within 1.2%.

Solar–Hydrogen-Storage Integrated Electric Vehicle Charging Stations with Demand-Side Management and Social Welfare Maximization

The reliable operation of a power system requires a real-time balance between supply and demand. However, it is difficult to achieve this balance solely by relying on supply-side regulation. Therefore, it is necessary to cooperate with effective demand-side management, which is a key strategy within smart grid systems, encouraging end-users to actively engage and optimize their electricity usage. This paper proposes a novel bi-level optimization model for integrating solar, hydrogen, and battery storage systems with charging stations (SHS-EVCSs) to maximize social welfare. The first level employs a non-cooperative game theory model for each individual EVCS to minimize capital and operational costs. The second level uses a cooperative game framework with an internal management system to optimize energy transactions among multiple EVCSs while considering EV owners’ economic interests. A Markov decision process models uncertainties in EV charging times, and Monte Carlo simulations predict charging demand. Real-time electricity pricing based on the dual theory enables demand-side management strategies like peak shaving and valley filling. Case studies demonstrate the model’s effectiveness in reducing peak loads, balancing energy utilization, and enhancing overall system efficiency and sustainability through optimized renewable integration, energy storage, EV charging coordination, social welfare maximization, and cost minimization. The proposed approach offers a promising pathway toward sustainable energy infrastructure by harmonizing renewable sources, storage technologies, EV charging demands, and societal benefits.