Markov Decision Process Approach Research Articles

Optimal Computation Resource Allocation in Energy-Efficient Edge IoT Systems With Deep Reinforcement Learning

This paper investigates a computation resource optimization problem of mobile edge computing (MEC)-aided Internet-of-Things (IoT) devices with a reinforcement learning (RL) solution. Specifically, we leverage the stochastic optimization method and formulate the Lyapunov optimization technique to maximize the long-term energy efficiency, taking into account the transmission power, network stability, and transmission latency. Based on the Markov decision process and model-free deep RL (DRL) approach, we propose a double DRL-based online computation offloading method to implement a deep neural network that learns from interactions to solve the computation offloading and transmission latency problem in the dynamic MEC-aided IoT environments. Furthermore, we design an adaptive method for continuous action-state spaces to minimize the completion time and total energy consumption of the IoT devices for stochastic computation offloading tasks. The proposed real-time Lyapunov optimization and DRL algorithms achieve low computational complexity and optimal processing time. Simulation results demonstrate that the proposed algorithm can achieve near-optimal control performance with enhanced energy efficiency performance compared to the baseline policy control algorithms.

Security-Oriented Cyber-Physical Risk Assessment for Cyberattacks on Distribution System

With the increasing deployment of advanced sensing and measurement devices, the modern distribution system is evolved into a cyber-physical power distribution system (CPPDS). Due to the extensive application of information and communication technology, CPPDS is prevalently exposed to a wide range of cybersecurity threats. In this paper, a novel security-oriented cyber-physical risk assessment method for CPPDS is proposed. Based on the information model composed of logical nodes, an attack graph of privilege promotion by exploiting the cyber vulnerabilities is constructed. The physical consequence caused by cyberattack is analyzed in detail. By using the Markov decision process (MDP) theory, the cyber-physical risk index (CPRI) is calculated. Furthermore, considering the allocation of the finite defense resource, the modified MDP approach with an attack–defense game is presented. The effectiveness of the proposed method is demonstrated with case studies on a four-feeder IEEE-RTBS test system.

Scheduling power-to-ammonia plants considering uncertainty and periodicity of electricity prices

Developing affordable and scalable energy storage solutions are essential to decarbonizing power systems. The conversion of renewable electricity into chemical energy carriers such as ammonia has attracted extensive attention from academia and industry. Many Power-to-Ammonia (PtA) plants have been conceptualized and developed worldwide in recent years. The PtA plant is an integration of multiple electrochemical processes, each with a distinct set of operational constraints and cost structure. One of the problems in the operation of PtA plants is the optimal scheduling of the hydrogen buffer in PtA plants considering the operational characteristics of electrochemical processes and the volatility and uncertainty of electricity prices. In this paper, a two-stage Markov-Decision-Process (MDP) approach is proposed. The computational challenges brought by the infinite optimization horizon and non-concavity of cost functions are resolved. The first stage solution is based on the periodic MDP approach, which captures the periodic structure of electricity prices. The second stage solution gives optimal real-time decisions based on a rolling-horizon MDP approach. Numerical results show that the accurate representations of the cost functions and the optimization horizon using the proposed method are necessary, while the linearization of cost functions and the truncation of the optimization horizon lead to notable deviations from the optimality.

Admission and pricing optimization of on-street parking with delivery bays

We analyze a parking lot, modeled as a loss queue, with passenger and delivery vehicles. The arrival process of delivery vehicles is exogenous, while that of passenger vehicles is a function of the parking price rate and accessibility. The aim of the parking operator is to maximize the revenue generated from passenger vehicles while providing a sufficient service level for delivery vehicles, in terms of their probability to find an available parking spot. Two levels of control are exercised: pricing and admission. From a Markov decision process approach, we prove that the optimal policy is a state-dependent reservation threshold policy that randomizes in at most one state. When some parking spots should be reserved for delivery vehicles, the price rate is selected to saturate the service level constraint, whereas when it is optimal not to restrict the parking lot accessibility, the price can also be selected as the unique local maximum of the revenue or to incentivize all potential passenger vehicles to arrive. Pricing should be used as a primary tool to control the flow of passenger vehicles. In complement, admission control is exercised with a limited use of reservation only when the service level guarantee for delivery vehicles is high.

Decision-Dependent Distributionally Robust Markov Decision Process Method in Dynamic Epidemic Control

In this paper, we present a Distributionally Robust Markov Decision Process (DRMDP) approach for addressing the dynamic epidemic control problem. The Susceptible-Exposed-Infectious-Recovered (SEIR) model is widely used to represent the stochastic spread of infectious diseases, such as COVID-19. While Markov Decision Processes (MDP) offers a mathematical framework for identifying optimal actions, such as vaccination and transmission-reducing intervention, to combat disease spreading according to the SEIR model. However, uncertainties in these scenarios demand a more robust approach that is less reliant on error-prone assumptions. The primary objective of our study is to introduce a new DRMDP framework that allows for an ambiguous distribution of transition dynamics. Specifically, we consider the worst-case distribution of these transition probabilities within a decision-dependent ambiguity set. To overcome the computational complexities associated with policy determination, we propose an efficient Real-Time Dynamic Programming (RTDP) algorithm that is capable of computing optimal policies based on the reformulated DRMDP model in an accurate, timely, and scalable manner. Comparative analysis against the classic MDP model demonstrates that the DRMDP achieves a lower proportion of infections and susceptibilities at a reduced cost.

Radar Anti-Jamming Countermeasures Intelligent Decision-Making: A Partially Observable Markov Decision Process Approach

Current electronic warfare jammers and radar countermeasures are characterized by dynamism and uncertainty. This paper focuses on a decision-making framework of radar anti-jamming countermeasures. The characteristics and implementation process of radar intelligent anti-jamming systems are analyzed, and a scheduling method for radar anti-jamming action based on the Partially Observable Markov Process (POMDP) is proposed. The sample-based belief distribution is used to reflect the radar’s cognition of the environment and describes the uncertainty of the recognition of jamming patterns in the belief state space. The belief state of jamming patterns is updated with Bayesian rules. The reward function is used as the evaluation criterion to select the best anti-jamming strategy, so that the radar is in a low threat state as often as possible. Numerical simulation combines the behavioral prior knowledge base of radars and jammers and obtains the behavioral confrontation benefit matrix from the past experience of experts. The radar controls the output according to the POMDP policy, and dynamically performs the best anti-jamming action according to the change of jamming state. The results show that the POMDP anti-jamming policy is better than the conventional policy. The POMDP approach improves the adaptive anti-jamming capability of the radar and can quickly realize the anti-jamming decision to jammers. This work provides some design ideas for the subsequent development of an intelligent radar.

Dynamic Multihop Routing in Terahertz Flow-Guided Nanosensor Networks: A Reinforcement Learning Approach

The Internet of Nano-Things (IoNT) is an emerging paradigm in which devices sized to the nanoscale (nanonodes) and transmitting in the terahertz (THz) band can become decisive actors in future medical applications. Flow-guided nanonetworks are well-known THz networks aimed at deploying the IoNT inside the human body, among other issues. In these networks, nanonodes flowing through the bloodstream monitor-sensitive biological/physical parameters and dispatch these data via electromagnetic (EM) waves to a nanorouter implanted in human tissue, which operates as a gateway to external Internet connectivity devices. Under these premises, two shortcomings arise. First, the use of the THz band greatly limits the nanonode’s communication range. Second, the nanonodes lack resources for processing, memory, and batteries. To minimize the impact of these concerns in EM nanocommunications, a novel dynamic multihop routing scheme is proposed to model in-body, flow-guided nanonetwork architecture. To this end, a reinforcement learning-based framework is conceived, combining the features of EM nanocommunications and hemodynamics or fluid dynamics applied to the bloodstream. A generic Markov decision process (MDP) approach is derived to maximize the throughput metric, analytically modeling: 1) the movement of the nanonodes in the bloodstream as laminar flow; 2) energy consumption (including energy-harvesting issues); and 3) prioritized events. A thoroughly THz flow-guided nanonetwork case of study is also defined. Under the umbrella of this case, diverse testbeds are planned to create a procedure of evaluation, validation, and discussion. Results reveal that multihop scenarios obtain better performance than direct nanonode-nanorouter communication, specifically, the two-hop scenario, which, for instance, quadrupled the throughput in a hand vein without sharply penalizing other aspects such as energy consumption.

Dynamic Routing in Stochastic Urban Air Mobility Networks: A Markov Decision Process Approach

Urban air mobility (UAM) is an emerging concept in short-range aviation transportation, where the aircraft will take off, land, and charge their batteries at a set of vertistops, and travel only through a set of flight corridors connecting these vertistops. We study the problem of routing an electric aircraft from its origin vertistop to its destination vertistop with the minimal expected total travel time. We first introduce a UAM network model that accounts for the limited battery capacity of aircraft, stochastic travel times of flight corridors, stochastic queueing delays, and a limited number of battery-charging stations at vertistops. Based on this model, we provide a sufficient condition for the existence of a routing strategy that avoids battery exhaustion. Furthermore, we show how to compute such a strategy by computing the optimal policy in a Markov decision process, a mathematical framework for decision-making in a stochastic dynamic environment. We illustrate our results using a case study with 29 vertistops and 137 flight corridors.

A Reinforcement Learning Method for Multiasset Roadway Improvement Scheduling Considering Traffic Impacts

Maintaining roadway pavements and bridge decks is key to providing high levels of service for road users. However, improvement actions incur downtime. These actions are typically scheduled by asset class, yet implemented on any asset type, they have network-wide impacts on traffic performance. This paper presents a bilevel program wherein the upper level involves a Markov decision process (MDP) through which potential roadway improvement actions across asset classes are prioritized and scheduled. The MDP approach considers uncertainty in component deterioration effects, while incorporating the benefits of implemented improvement actions. The upper level takes as input traffic flow estimates obtained from a lower-level user equilibrium traffic formulation that recognizes changes in capacities determined by decisions taken in the upper level. Because an exact solution of this bilevel, stochastic, dynamic program is formidable, a deep reinforcement learning (DRL) method is developed. The model and solution methodology were tested on a hypothetical problem from the literature. The importance of obtaining optimal activity plans that account for downtime effects, traffic congestion impacts, uncertainty in deterioration processes, and multiasset classes is demonstrated.

Dynamic policy for idling time preservation

AbstractThis study aims to determine and evaluate dynamic idling policies where an agent can idle while some customers remain waiting. This type of policies can be employed in situations where the flow of urgent customers does not allow the agent to spend sufficient time on back‐office tasks. We model the system as a single‐agent exponential queue with abandonment. The objective is to minimize the system's congestion while ensuring a certain proportion of idling time for the agent. Using a Markov decision process approach, we prove that the optimal policy is a threshold policy according to which the agent should idle above (below) a certain threshold on the queue length if the congestion‐related performance measure is concave (convex) with respect to the number of customers present. We subsequently obtain the stationary probabilities, performance measures, and idling time duration, expressed using complex integrals. We show how these integrals can be numerically computed and provide simpler expressions for fast‐agent and heavy‐traffic asymptotic cases. In practice, the most common way to regulate congestion is to control access to the service by rejecting some customers upon arrival. Our analysis reveals that idling policies allow high levels of idling probability that such rejection policies cannot reach. Furthermore, the greatest benefit of implementing an optimal idling policy occurs when the objective occupation rate is close to 50% in highly congested situations.

A Promising Approach to Optimizing Sequential Treatment Decisions for Depression: Markov Decision Process.

The most appropriate next step in depression treatment after the initial treatment fails is unclear. This study explores the suitability of the Markov decision process for optimizing sequential treatment decisions for depression. We conducted a formal comparison of a Markov decision process approach and mainstream state-transition models as used in health economic decision analysis to clarify differences in the model structure. We performed two reviews: the first to identify existing applications of the Markov decision process in the field of healthcare and the second to identify existing health economic models for depression. We then illustrated the application of a Markov decision process by reformulating an existing health economic model. This provided input for discussing the suitability of a Markov decision process for solving sequential treatment decisions in depression. The Markov decision process and state-transition models differed in terms of flexibility in modeling actions and rewards. In all, 23 applications of a Markov decision process within the context of somatic disease were included, 16 of which concerned sequential treatment decisions. Most existing health economic models relating to depression have a state-transition structure. The example application replicated the health economic model and enabled additional capacity to make dynamic comparisons of more interventions over time than was possible with traditional state-transition models. Markov decision processes have been successfully applied to address sequential treatment-decision problems, although the results have been published mostly in economics journals that are not related to healthcare. One advantage of a Markov decision process compared with state-transition models is that it allows extended action space: the possibility of making dynamic comparisons of different treatments over time. Within the context of depression, although existing state-transition models are too basic to evaluate sequential treatment decisions, the assumptions of a Markov decision process could be satisfied. The Markov decision process could therefore serve as a powerful model for optimizing sequential treatment in depression. This would require a sufficiently elaborate state-transition model at the cohort or patient level.Supplementary InformationThe online version contains supplementary material available at 10.1007/s40273-022-01185-z.

DeepAir: Deep Reinforcement Learning for Adaptive Intrusion Response in Software-Defined Networks

In this paper, we propose an adaptive intrusion response solution based on deep reinforcement learning, namely DeepAir, to effectively defend against cyber-attacks in Software-Defined Networks (SDN). Specifically, we first study an intrusion response system (IRS) that operates at the SDN control plane. Next, we propose a dynamic intrusion response solution to maximize the attack defense performance while minimizing the negative impact on benign traffic forwarding and the policy deployment cost in the SDN data plane. Then, we model the intrusion response system based on a Markov decision process (MDP) approach and formulate the related optimization problem. Afterward, we develop a Double Deep <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${Q}$ </tex-math></inline-formula> -Network based intrusion response control algorithm to assist the intrusion response system to quickly obtain the optimal intrusion response policy. In our case study, we consider denial-of-service (DoS) attacks—the performance evaluation results demonstrate that DeepAir can effectively prevent malicious packets from arriving at the victim in all considered DoS attack scenarios, i.e., approximately 85% of attack packets are dropped. Moreover, by applying the optimal intrusion response policy, DeepAir can significantly reduce the ratio of Quality-of-Service violated traffic flows compared to a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${Q}$ </tex-math></inline-formula> -learning based approach (by 70%), and to two existing solutions, i.e., GATE (by 75%) and GTAC-IRS (by 80%), respectively.

Multistage Energy Management of Coordinated Smart Buildings: A Multiagent Markov Decision Process Approach

Smart buildings provide an important opportunity for large-scale development of demand response, due to their existing flexibility that can be harvested through Internet-of-Things technologies with minimal cost of new equipment. However, after taking an energy management action, the resulting energy consumption of a building depends on several uncertain factors. Thus, the consumption of the smart building is not directly controllable and, contrary to the typical approach taken in the literature, it cannot be modeled as a decision variable in practice. In this paper, we consider the problem of coordinating the stochastic load control actions of multiple smart buildings under such endogenous uncertainties. We model the problem as a Multi-agent Markov Decision Process and, after reformulations, we bring it to a solvable decomposed form. Our simulations compare the proposed approach with a myopic approach that does not consider future uncertainties, and also quantify the trade-off between cost-effectiveness and computational time in terms of the look-ahead horizon length.

A Consumption and Investment Problem via a Markov Decision Processes Approach with Random Horizon

This work is devoted to a consumption and investment problem, in which there is an investor with certain initial wealth with the possibility of deciding how much of such wealth will be consumed and how much will be invested in each of a series of successive times. The key issue is to find a wealth assignation rule in order to maximize the performance criteria; such dilemma will be achieved by the dynamic programming technique for the Markov decision processes with random horizon.

Optimizing active surveillance for prostate cancer using partially observable Markov decision processes

We describe a finite-horizon partially observable Markov decision process (POMDP) approach to optimize decisions about whether and when to perform biopsies for patients on active surveillance for prostate cancer. The objective is to minimize a weighted combination of two criteria, the number of biopsies to conduct over a patient’s lifetime and the delay in detecting high-risk cancer that warrants more aggressive treatment. Our study also considers the impact of parameter ambiguity caused by variation across models fitted to different clinical studies and variation in the weights attributed to the reward criteria according to patients’ preferences. We introduce two fast approximation algorithms for the proposed model and describe some properties of the optimal policy, including the existence of a control-limit type policy. The numerical results show that our approximations perform well, and we use them to compare the model-based biopsy policies to published guidelines. Although our focus is on prostate cancer active surveillance, there are lessons to be learned for applications to other chronic diseases.

A Markov decision process approach for managing medical drone deliveries

Drone delivery is a fast and innovative method for delivering parcels, food, and medical supplies. Furthermore, this low-contact delivery mode contributes to reducing the spread of pandemic and vaccine-preventable diseases. Focusing on the delivery of medical supplies, this paper studies optimizing the distribution operations at a drone hub that dispatches drones to hospitals located at different geographic locations. Each hospital generates stochastic demands for medical supplies to be covered. This paper classifies stochastic demands based on the distance between hospitals and the drone hub. Satisfying the demands requires flying over different ranges, which is directly related to the amount of charge of the drone batteries. We develop a stochastic scheduling and allocation problem with multiple classes of demand and model the problem using a finite Markov decision process approach. We provide exact solutions for the modest sizes instances using backward induction and discuss that the problem suffers from the curses of dimensionality. Hence, we provide a reinforcement learning method capable of giving near-optimal solutions. We perform a set of computational tests using realistic data representing a prominent drone delivery company. Finally, we analyze the results to provide insights for managing drone hub operations and show that the reinforcement learning method has high performance compared with the exact and heuristic solution methods.

BLMDP: A new bi-level Markov decision process approach to joint bidding andtask-scheduling in cloud spot market

In the cloud computing market (CCM), computing services are traded between cloud providers and consumers in the form of the computing capacity of virtual machines (VMs). The Amazon spot market is one of the most well-known markets in which the surplus capacity of data centers is auctioned off in the form of VMs at relatively low prices. For each submitted task, the user can offer a price that is higher than the current price. However, uncertainty in the market environment confronts the user with challenges such as the variable price of VMs and the variable number of users. An appropriate strategy should both maximize the user?s utility and determine the best bid for him/her. In this paper, we aim to solve the problem of joint minimizing the cost of processing tasks and maximizing user satisfaction. Our proposed method, which we call bi-level Markov decision-making process (BLMDP), works on two levels. At the top level, it selects the most appropriate user bids and adjusts the spot price to minimize the cost of VMs on the cloud provider side. At a low level, it decides to admit tasks in such a way as to maximize the user side satisfaction. Performance evaluation based on real data collected from Amazon web services shows that BLMDP manages to minimize cloud provider's costs and maximize user gain more effectively compared to heuristic methods.

Optimization of the Recruitment Quota Allocation in Intra-Organizational Networks

In a decentralized organization such as a university, recruitment is critical for the development of departments. With the limited resources for recruitment, how to allocate the recruitment quotas to different departments is an essential problem for the human resource management center of the organization. Specifically, for a recruitment process that includes multiple phases, a proper number of quotas should be allocated to different departments at each recruitment phase to guarantee the recruitment performance of the whole organization. However, it is difficult since the performance of the whole recruitment process is prior unknown at each recruitment phase. Traditionally, a recruitment quota allocation scheme based on requirements raised by departments is generally adopted, which is somewhat subjective and may not achieve satisfying recruitment performance. To address the above problem, a recruitment quota allocation scheme is proposed in this paper, which is based on the theoretical modeling and optimization of the recruitment process, aiming to provide a more objective and theoretical method that may achieve optimal performance. We first define the recruitment utility as the performance metric for the recruitment process. The recruitment quota allocation problem is formulated using the Markov decision process approach, where the effect of the information of potential applicants is innovatively taken into account. Then, the recruitment quota allocation is optimized by maximizing the recruitment utility to achieve optimal recruitment performance. In addition, a recruitment quota allocation scheme based on the requirements of departments, which is traditionally adopted in the recruitment of a university, is used for comparison with the optimized scheme of the paper. Simulation results show that the proposed scheme can improve the efficiency of recruitment quota allocation significantly.

Privacy-Aware Load Ensemble Control: A Linearly-Solvable MDP Approach

Demand response (DR) programs engage distributed demand-side resources, e.g., controllable residential and commercial loads, in providing ancillary services for electric power systems. Ensembles of these resources can help reducing system load peaks and meeting operational limits by adjusting their electric power consumption. To equip utilities or load aggregators with adequate decision-support tools for ensemble dispatch, we develop a Markov Decision Process (MDP) approach to optimally control load ensembles in a privacy-preserving manner. To this end, the concept of differential privacy is internalized into the MDP routine to protect transition probabilities and, thus, privacy of DR participants. The proposed approach also provides a trade-off between solution optimality and privacy guarantees, and is analyzed using real-world data from DR events in the New York University microgrid in New York, NY.

Markov Decision Process approach in the estimation of raw material quality in incoming inspection process

The incoming inspection process in any manufacturing plant aims to control quality, reduce manufacturing costs, eliminate scrap, and process failure downtime due to defective raw materials. Prediction of the raw material acceptance rate can regulate the raw material supplier selection and improve the manufacturing process by filtering out non-conformities. This paper presents a raw material acceptance prediction model (RMAP) developed based on the Markov analysis. RFID tags are used to track the parts throughout the process. A secondary dataset can be derived from the raw RFID data. In this study, a dataset is simulated to reflect a typical incoming inspection process consisting of six substations (Packaging Inspection, Visual Inspection, Gauge Inspection, Rework1, and Rework2) are considered. The accepted parts are forwarded to the Pack and Store station and stored in the warehouse. The non-conforming parts are returned to the supplier. The proposed RMAP model estimates the probability of the raw material being accepted or rejected at each inspection station. The proposed model is evaluated using three test cases: case A (lower conformities), case B (higher conformities) and case C (equal chances of being accepted and rejected). Based on the outcome of the limiting matrix for the three test cases, the results are discussed. The steady-state matrix forecasts the probability of the raw material in a random state. This prediction and forecasting ability of the proposed model enables the industries to save time and cost.