Markov Decision Process Model Research Articles

Dynamic Home Care Routing and Scheduling with Uncertain Number of Visits per Referral

Despite the rapid growth of the home care industry, research on the scheduling and routing of home care visits in the presence of uncertainty is still limited. This paper investigates a dynamic version of this problem in which the number of referrals and their required number of visits are uncertain. We develop a Markov decision process (MDP) model for the single-nurse problem to minimize the expected weighted sum of the rejection, diversion, overtime, and travel time costs. Because optimally solving the MDP is intractable, we employ an approximate linear program (ALP) to obtain a feasible policy. The typical ALP approach can only solve very small-scale instances of the problem. We derive an intuitively explainable closed-form solution for the optimal ALP parameters in a special case of the problem. Inspired by this form, we provide two heuristic reduction techniques for the ALP model in the general problem to solve large-scale instances in an acceptable time. Numerical results show that the ALP policy outperforms a myopic policy that reflects current practice, and is better than a scenario-based policy in most instances considered. Funding: This work was supported by the Natural Sciences and Engineering Research Council of Canada [Grants RGPIN-2018-05225 and RGPIN-2020-210524] and by the Telfer School of Management SMRG Postdoctoral Research Fellowship Support [Grant 2020]. Supplemental Material: The electronic companion is available at https://doi.org/10.1287/trsc.2023.0120 .

Condition-based reallocation and maintenance for a 1-out-of-2 pairs balanced system

In a system consisting of multiple functionally exchangeable units, differences in units’ degradation levels can significantly affect the system’s performance. Dynamic reallocation of these units can improve performance and prolong the lifetime of the system. This study is the first to quantify the benefit of incorporating reallocation into a condition-based maintenance framework for a 1-out-of-2 pairs balanced system—a system with two pairs of units that functions if there is at least one functioning pair. The balance condition requires that the two units in the same pair should be active or inactive at the same time. Unit degradation is modeled as a Gamma process and is inspected periodically. A Markov decision process model is developed to determine the optimal integrated reallocation and maintenance policy that minimizes the long-run average cost per unit time. A numerical study illustrates that the proposed integrated reallocation and maintenance policy significantly outperforms other limited policies, including reallocate-only and maintain-only policies.

Reinforcement learning vs rule-based dynamic movement strategies in UAV assisted networks

Since resource allocation of cellular networks is not dynamic, some cells may experience unplanned high traffic demands due to unexpected events. Unmanned aerial vehicles (UAV) can be used to provide the additional bandwidth required for data offloading.Considering real-time and non-real-time traffic classes, our work is dedicated to optimize the placement of UAVs in cellular networks by two approaches. A first rule-based, low complexity method, that can be embedded in the UAV, while the other approach uses Reinforcement Learning (RL). It is based on Markov Decision Processes (MDP) for providing optimal results. The energy of the UAV battery and charging time constraints have been taken into account to cover a typical cellular environment consisting of many cells.We used an open dataset for the Milan cellular network provided by Telecom Italia to evaluate the performance of both proposed models. Considering this dataset, the MDP model outperforms the rule-based algorithm. Nevertheless, the rule-based one requires less processing complexity and can be used immediately without any prior data. This work makes a notable contribution to developing practical and optimal solutions for UAV deployment in modern cellular networks.

Computational mechanisms underlying motivation to earn symbolic reinforcers.

Reinforcement learning (RL) is a theoretical framework that describes how agents learn to select options that maximize rewards and minimize punishments over time. We often make choices, however, to obtain symbolic reinforcers (e.g. money, points) that are later exchanged for primary reinforcers (e.g. food, drink). Although symbolic reinforcers are ubiquitous in our daily lives, widely used in laboratory tasks because they can be motivating, mechanisms by which they become motivating are less understood. In the present study, we examined how monkeys learn to make choices that maximize fluid rewards through reinforcement with tokens. The question addressed here is how the value of a state, which is a function of multiple task features (e.g. current number of accumulated tokens, choice options, task epoch, trials since last delivery of primary reinforcer, etc.), drives value and affects motivation. We constructed a Markov decision process model that computes the value of task states given task features to then correlate with the motivational state of the animal. Fixation times, choice reaction times, and abort frequency were all significantly related to values of task states during the tokens task (n=5 monkeys, three male and two female). Furthermore, the model makes predictions for how neural responses could change on a moment-by-moment basis relative to changes in state value. Together, this task and model allow us to capture learning and behavior related to symbolic reinforcement.Significance statement Symbolic reinforcers, like money and points, play a critical role in our lives. However, we lack a mechanistic understanding of how symbolic reinforcement can be related to fluctuations in motivation. We investigated the effect of symbolic reinforcers on behaviors related to motivation during a token reinforcement learning task, using a novel reinforcement learning model and data from five monkeys. We designed a state-based model that can capture reward-predicting features and produce state values at sub-trial resolution. Our findings suggest that the value of a task state can affect willingness to initiate a trial, speed to choose, and persistence to complete a trial. Our model makes testable predictions for within trial fluctuations of neural activity related to values of task states.

Multi-agent deep Q-network-based metaheuristic algorithm for Nurse Rostering Problem

The Nurse Rostering Problem (NRP) aims to create an efficient and fair work schedule that balances both the needs of employees and the requirements of hospital operations. Traditional local search-based metaheuristic algorithms, such as adaptive neighborhood search (ANS) and variable neighborhood descent (VND), mainly focus on optimizing the current solution without considering potential long-term consequences, which may easily get stuck in local optima and limit the overall performance. Thus, we propose a multi-agent deep Q-network-based metaheuristic algorithm (MDQN-MA) for NRP to harness the strengths of various metaheuristics. Each agent encapsulates a metaheuristic algorithm, where its available actions represent different perspectives of the problem environment. By combining their strengths and various perspectives, these agents can work collaboratively to navigate and search for a broader range of potential solutions effectively. Furthermore, to improve the performance of an individual agent, we model its neighborhood search as a Markov Decision Process model and integrate a deep Q-network to consider long-term impacts for its neighborhood sequential decision-making. The experimental results clearly show that an individual agent in MDQN-MA can outperform ANS and VND, and multiple agents in MDQN-MA even perform better, achieving the best results among metaheuristic algorithms on the Second International Nurse Rostering Competition dataset.

On the Continuity of the Projection Mapping from Strategic Measures to Occupation Measures in Absorbing Markov Decision Processes

In this paper, we prove the following assertion for an absorbing Markov decision process (MDP) with the given initial distribution, which is also assumed to be semi-continuous: the continuity of the projection mapping from the space of strategic measures to the space of occupation measures, both endowed with their weak topologies, is equivalent to the MDP model being uniformly absorbing. An example demonstrates, among other interesting scenarios, that for an absorbing (but not uniformly absorbing) semi-continuous MDP with the given initial distribution, the space of occupation measures can fail to be compact in the weak topology.

Enhancing military medical evacuation dispatching with armed escort management: comparing model-based reinforcement learning approaches

The military medical evacuation (MEDEVAC) dispatching problem involves determining optimal policies for evacuating combat casualties to maximize patient survivability during military operations. This study explores a variation of the MEDEVAC dispatching problem, focusing on controlling armed escorts using a Markov decision process (MDP) model and model-based reinforcement learning (RL) approaches. A discounted, continuous-time MDP model over an infinite horizon is developed to maximize the expected total discounted reward of the system. Two model-based RL solution approaches are proposed: one utilizing semi-gradient descent Q-learning and another employing semi-gradient descent SARSA. A computational example, set in western and central Africa during contingency operations, assesses the performance of the RL-generated policies against the myopic policy, which military medical planners currently employ. Solution quality is derived from expected response time, a crucial determinant of life-saving potential in MEDEVAC operations. The research also explores sensitivity analysis and excursion scenarios to evaluate the RL-generated policies further. By explicitly controlling armed escort assets, dispatching authorities can better manage the location and allocation of these resources throughout combat operations. The findings of this study have the potential to inform military medical planning, operations, and tactics, ultimately leading to improved MEDEVAC system performance and higher patient survivability rates.

Multiagent Q-Learning Approach for the Recharging Scheduling of Electric Automated Guided Vehicles in Container Terminals

In recent years, advancements in battery technology have led to increased adoption of electric automated guided vehicles in container terminals. Given how critical these vehicles are to terminal operations, this trend requires efficient recharging scheduling for automated guided vehicles, and the main challenges arise from limited charging station capacity and tight vehicle schedules. Motivated by the dynamic nature of the problem, the recharging scheduling problem for an entire vehicle fleet given capacitated stations is formulated as a Markov decision process model. Then, it is solved using a multiagent Q-learning (MAQL) approach to produce a recharging schedule that minimizes the delay of jobs. Numerical experiments show that under a stochastic environment in terms of vehicle travel time, MAQL enables the exploration of better scheduling by coordinating across the entire vehicle fleet and charging facilities and outperforms various benchmark approaches, with an additional improvement of 18.8% on average over the best rule-based heuristic and 5.4% over the predetermined approach. Funding: This work was supported by the National Natural Science Foundation of China [Grant 72101203], the Shaanxi Provincial Key R&D Program, China [Grant 2022KW-02], and the Singapore Maritime Institute [Grant SMI-2017-SP-002]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/trsc.2022.0113 .

Distributed entropy-regularized multi-agent reinforcement learning with policy consensus

Sample efficiency is a limiting factor for existing distributed multi-agent reinforcement learning (MARL) algorithms over networked multi-agent systems. In this paper, the sample efficiency problem is tackled by formally incorporating the entropy regularization into the distributed MARL algorithm design. Firstly, a new entropy-regularized MARL problem is formulated under the model of networked multi-agent Markov decision processes with observation-based policies and homogeneous agents, where the policy parameter sharing among the agents provably preserves the optimality. Secondly, an on-policy distributed actor–critic algorithm is proposed, where each agent shares its parameters of both the critic and actor for consensus update. Then, the convergence analysis of the proposed algorithm is provided based on the stochastic approximation theory under the assumption of linear function approximation of the critic. Furthermore, a practical off-policy version of the proposed algorithm is developed which possesses scalability, data efficiency and learning stability. Finally, the proposed distributed algorithm is compared against the solid baselines including two classic centralized training algorithms in the multi-agent particle environment, whose learning performance is empirically demonstrated through extensive simulation experiments.

Scheduling of twin automated stacking cranes based on Deep Reinforcement Learning

Effective scheduling of twin automated stacking cranes (ASCs) in automated storage yard is critical to maximize operational efficiency. While Deep Reinforcement Learning (DRL) is promising in solving NP-hard scheduling problems, twin ASCs scheduling is challenging due to its unique properties including sequence-dependent setup and potential ASC interferences. In this paper, we propose a novel DRL method to learn high-quality policy for scheduling twin ASCs. We propose a Markov Decision Process model that enables the DRL agent to learn to minimize makespan and possible interferences. Based on the problem characteristics, we design a self-attention based neural architecture to effectively capture the relationships between containers under certain block state. Experiments show that the agent with the proposed feature extraction network can learn high-quality policies from training instances. These learned policies can be employed to produce effective scheduling solutions within seconds. Compared to traditional scheduling methods, the learned policy performs best in most problem sizes, and the performance improvement amplifies as the scales increase. Moreover, the policies show remarkable generalization ability on unseen instances with different distributions or scales.

Dynamic berth allocation under uncertainties based on deep reinforcement learning towards resilient ports

With the evolving global trade landscape and the post-pandemic effects, the resilience of ports has become paramount. The unforeseen disturbances bring substantial challenges, especially in berth allocation, a vital task ensuring seamless resilient port operations. The unpredictability of vessel arrivals and the variability in loading/unloading times intensify these issues, pushing traditional static allocation methods beyond their limits. Fortunately, the advent of smart ports has led in an era of big data availability, enabling the application of advanced deep reinforcement learning (DRL) techniques. To capitalize on this shift, this research presents a DRL-based methodology specially designed to solve the berth allocation problem with the uncertainties in vessel arrival and container handling time to enhance port resilience. A Markov Decision Process model (MDP) of the berth allocation problem is established to minimize the mean waiting time with tailored state space, rule-based action space, and reward function to address the issue. An offline training method is designed to train the agent in selecting the optimal action based on the current state of the port berth system at each decision point even in uncertain environments, deep Q-network (DQN) is implemented for this problem. Comprehensive experiments across different problem scales are conducted to validate the effectiveness and generality of the proposed method in solving berth allocation challenges under uncertain conditions. Furthermore, the trained model also performs better than other methods in different vessel congestion levels through learning.

Federated Offline Reinforcement Learning

Evidence-based or data-driven dynamic treatment regimes are essential for personalized medicine, which can benefit from offline reinforcement learning (RL). Although massive healthcare data are available across medical institutions, they are prohibited from sharing due to privacy constraints. Besides, heterogeneity exists in different sites. As a result, federated offline RL algorithms are necessary and promising to deal with the problems. In this article, we propose a multi-site Markov decision process model that allows for both homogeneous and heterogeneous effects across sites. The proposed model makes the analysis of the site-level features possible. We design the first federated policy optimization algorithm for offline RL with sample complexity. The proposed algorithm is communication-efficient, which requires only a single round of communication interaction by exchanging summary statistics. We give a theoretical guarantee for the proposed algorithm, where the suboptimality for the learned policies is comparable to the rate as if data is not distributed. Extensive simulations demonstrate the effectiveness of the proposed algorithm. The method is applied to a sepsis dataset in multiple sites to illustrate its use in clinical settings. Supplementary materials for this article are available online including a standardized description of the materials available for reproducing the work.

Federated Learning with Efficient Aggregation via Markov Decision Process in Edge Networks

Federated Learning (FL), as an emerging paradigm in distributed machine learning, has received extensive research attention. However, few works consider the impact of device mobility on the learning efficiency of FL. In fact, it is detrimental to the training result if heterogeneous clients undergo migration or are in an offline state during the global aggregation process. To address this issue, the Optimal Global Aggregation strategy (OGAs) is proposed. The OGAs first models the interaction between clients and servers of the FL as a Markov Decision Process (MDP) model, which jointly considers device mobility and data heterogeneity to determine local participants that are conducive to global aggregation. To obtain the optimal client participation strategy, an improved σ-value iteration method is utilized to solve the MDP, ensuring that the number of participating clients is maintained within an optimal interval in each global round. Furthermore, the Principal Component Analysis (PCA) is used to reduce the dimensionality of the original features to deal with the complex state space in the MDP. The experimental results demonstrate that, compared with other existing aggregation strategies, the OGAs has the faster convergence speed and the higher training accuracy, which significantly improves the learning efficiency of the FL.

Appointment Requests from Multiple Channels: Characterizing Optimal Set of Appointment Days to Offer with Patient Preferences

We consider the appointment scheduling for a physician in a healthcare facility. Patients, of two types differentiated by their revenues and day preferences, contact the facility through either a call center to be scheduled immediately or a website to be scheduled the following morning. The facility aims to maximize the long-run average revenue, while ensuring that a certain service level is satisfied for patients generating lower revenue. The facility has two decisions: offering a set of appointment days and choosing the patient type to prioritize while contacting the website patients. Model 1 is a periodic Markov Decision Process (MDP) model without the service-level constraint. We establish certain structural properties of Model 1, while providing sufficient conditions for the existence of a preferred patient type and for the nonoptimality of the commonly used offer-all policy. We also demonstrate the importance of patient preference in determining the preferred type. Model 2 is the constrained MDP model that accommodates the service-level constraint and has an optimal randomized policy with a special structure. This allows developing an efficient method to identify a well-performing policy. We illustrate the performance of this policy through numerical experiments, for systems with and without no-shows. Supplemental Material: The online appendix is available at https://doi.org/10.1287/stsy.2022.0029 .

Cancer Screening Benefits Maximization Using Markov Decision Process Models: A Systematic Review

Context: Due to the chronic nature of cancer, screening programs were a set of sequential decisions taken over time. Markov decision process (MDP) and partially observable Markov decision process (POMDP) models were the mathematical tools applied in studies, including sequential decision-making such as screening protocols for medical decision-making. Objectives: The main goal of this study was to investigate optimal policy for cancer screening using MDP and POMDP models. Methods: We performed a review of articles published within July 2000 to November 2022 in PubMed, Web of Science, and Scopus databases. The stopping age, the type of optimal strategy, the benefits of the optimal policy, and the relationship between age and risk threshold were extracted. Studies that did not use MDPs and POMDPs as the mathematical maximization models in cancer screening, review articles, editorials or commentaries, non-English articles, and those that did not focus on optimization were excluded. Results: From 532 articles, 6 studies met the study criteria. All studies suggested that the optimal policy was control-limit, and the cancer risk threshold was a non-decreasing function of age. Three studies specified a stopping age for cancer screening. In five studies, the optimal policies outperformed the guidelines or no screening strategy. Conclusions: Two essential factors in screening decisions were cancer risk and age, which were individual variables. The control-limit policy included these factors in decision-making for cancer screening. These policies highlighted personalized screening and showed that this type of screening could outperform cancer screening guidelines regarding economic and clinical benefits.

Field teams coordination for earthquake-damaged distribution system energization

The re-energization of electrical distribution systems in a post-disaster scenario is of grave importance as most modern infrastructure systems rely heavily on the presence of electricity. This paper introduces a method to coordinate the field teams for the optimal energization of an electrical distribution system after an earthquake-induced blackout. The proposed method utilizes a Markov Decision Process (MDP) to create an optimal energization strategy, which aims to minimize the expected time to energize each distribution system component. The travel duration of each team and the possible outcomes of the energization attempts are considered in the state transitions. The failure probabilities of the system components are computed using the fragility curves of structures and the Peak Ground Acceleration (PGA) values which are encoded to the MDP model via transition probabilities. Furthermore, the proposed solution offers several methods to determine the non-optimal actions during the construction of the MDP and eliminate them in order to improve the run-time performance without sacrificing the optimality of the solution.

Finding the optimal exploration-exploitation trade-off online through Bayesian risk estimation and minimization

We propose endogenous Bayesian risk minimization (EBRM) over policy sets as an approach to online learning across a wide range of settings. Many real-world online learning problems have complexities such as action- and belief-dependent rewards, time-discounting of reward, and heterogeneous costs for actions and feedback; we find that existing online learning heuristics cannot leverage most problem-specific information, to the detriment of their performance. We introduce a belief-space Markov decision process (BMDP) model that can capture these complexities, and further apply the concepts of aleatoric, epistemic, and process risks to online learning. These risk functions describe the risk inherent to the learning problem, the risk due to the agent's lack of knowledge, and the relative quality of its policy, respectively. We demonstrate how computing and minimizing these risk functions guides the online learning agent towards the optimal exploration-exploitation trade-off in any stochastic online learning problem, constituting the basis of the EBRM approach. We also show how Bayes' risk, the minimization objective in stochastic online learning problems, can be decomposed into the aforementioned aleatoric, epistemic, and process risks.In simulation experiments, EBRM algorithms achieve state-of-the-art performance across various classical online learning problems, including Gaussian and Bernoulli multi-armed bandits, best-arm identification, mixed objectives with action- and belief-dependent rewards, and dynamic pricing, a finite partial monitoring problem. To our knowledge, it is also the first computationally efficient online learning approach that can provide online bounds on an algorithm's Bayes' risk. Finally, because the EBRM approach is parameterized by a set of policy algorithms, it can be extended to incorporate new developments in online learning algorithms, and is thus well-suited as the foundation for developing real-world learning agents.

Efficient energy management in smart grids with finite horizon Q-learning

Efficient energy distribution in smart grids is an important problem driven by the need to manage increasing power consumption across the globe. This problem has been studied in the past using different frameworks including Markov Decision Processes (MDP) and Reinforcement Learning. However, existing algorithms in Reinforcement Learning Theory largely deal with infinite horizon decision making. On the other hand, smart grid problems are inherently finite horizon in nature and so can best be addressed using Finite Horizon algorithms. We therefore analyze the smart grid setup using a finite horizon MDP model and develop, for the first time, a finite horizon version of the popular Q-learning algorithm. We observe that our algorithm shows good empirical performance on the smart grid setting. We also theoretically analyze the full convergence of the algorithm by proving both the stability and the convergence of the same. Our analysis of stability and convergence of finite horizon Q-learning is based entirely on the ordinary differential equations (O.D.E) method. Apart from smart grids, we additionally demonstrate the performance of our algorithm on a setting of random MDPs indicating that the algorithm is more generally applicable and can be studied on other settings in the future.

Power Allocation Based on Multi-Agent Deep Deterministic Policy Gradient for Underwater Acoustic Communication Networks

This paper proposes a reinforcement learning-based power allocation for underwater acoustic communication networks (UACNs). The objective function is formulated as maximizing channel capacity under constraints of maximum power and minimum channel capacity. To solve this problem, a multi-agent deep deterministic policy gradient (MADDPG) approach is introduced, where each transmitter node is considered as an agent. Given the definition of a Markov decision process (MDP) model for this problem, the agents learn to collaboratively maximize the channel capacity by deep deterministic policy gradient (DDPG) learning. Specifically, the power allocation of each agent is obtained by a centralized training and distributed execution (CTDE) method. Simulation results show the sum rate achieved by the proposed algorithm approximates that of the fractional programming (FP) algorithm and improves by at least 5% compared with the DQN (deep Q-learning network) -based power allocation algorithm.

Priority-Aware Actuation Update Scheme in Heterogeneous Industrial Networks.

In heterogeneous wireless networked control systems (WNCSs), the age of information (AoI) of the actuation update and actuation update cost are important performance metrics. To reduce the monetary cost, the control system can wait for the availability of a WiFi network for the actuator and then conduct the update using a WiFi network in an opportunistic manner, but this leads to an increased AoI of the actuation update. In addition, since there are different AoI requirements according to the control priorities (i.e., robustness of AoI of the actuation update), these need to be considered when delivering the actuation update. To jointly consider the monetary cost and AoI with priority, this paper proposes a priority-aware actuation update scheme (PAUS) where the control system decides whether to deliver or delay the actuation update to the actuator. For the optimal decision, we formulate a Markov decision process model and derive the optimal policy based on Q-learning, which aims to maximize the average reward that implies the balance between the monetary cost and AoI with priority. Simulation results demonstrate that the PAUS outperforms the comparison schemes in terms of the average reward under various settings.