Markov Decision Process Theory Research Articles

Markov decision processes with risk-sensitive criteria: an overview

AbstractThe paper provides an overview of the theory and applications of risk-sensitive Markov decision processes. The term ’risk-sensitive’ refers here to the use of the Optimized Certainty Equivalent as a means to measure expectation and risk. This comprises the well-known entropic risk measure and Conditional Value-at-Risk. We restrict our considerations to stationary problems with an infinite time horizon. Conditions are given under which optimal policies exist and solution procedures are explained. We present both the theory when the Optimized Certainty Equivalent is applied recursively as well as the case where it is applied to the cumulated reward. Discounted as well as non-discounted models are reviewed.

Security-Oriented Cyber-Physical Risk Assessment for Cyberattacks on Distribution System

With the increasing deployment of advanced sensing and measurement devices, the modern distribution system is evolved into a cyber-physical power distribution system (CPPDS). Due to the extensive application of information and communication technology, CPPDS is prevalently exposed to a wide range of cybersecurity threats. In this paper, a novel security-oriented cyber-physical risk assessment method for CPPDS is proposed. Based on the information model composed of logical nodes, an attack graph of privilege promotion by exploiting the cyber vulnerabilities is constructed. The physical consequence caused by cyberattack is analyzed in detail. By using the Markov decision process (MDP) theory, the cyber-physical risk index (CPRI) is calculated. Furthermore, considering the allocation of the finite defense resource, the modified MDP approach with an attack–defense game is presented. The effectiveness of the proposed method is demonstrated with case studies on a four-feeder IEEE-RTBS test system.

Electrical Vehicle Charging Load Mobility Analysis Based on a Spatial–Temporal Method in Urban Electrified-Transportation Networks

Charging load mobility evaluation becomes one of the main concerns for charging services and power system stability due to the stochastic nature of electrical vehicles (EVs) and is critical for the robust scheduling of economic operations at different intervals. Therefore, the EV spatial–temporal approach for load mobility forecasting is presented in this article. Furthermore, the reliability indicators of large-scale EV distribution network penetration are analyzed. The Markov decision process (MDP) theory and Monte Carlo simulation are applied to efficiently forecast the charging load and stochastic path planning. A spatial–temporal model is established to robustly forecast the load demand, stochastic path planning, traffic conditions, and temperatures under different scenarios to evaluate the charging load mobility and EV drivers’ behavior. In addition, the distribution network performance indicators are explicitly evaluated. A Monte Carlo simulation is adopted to examine system stability considering various charging scenarios. Urban coupled traffic-distribution networks comprising 30-node transportation and 33-bus distribution networks are considered as a test case to illustrate the proposed study. The results analysis reveals that the proposed method can robustly estimate the charging load mobility. Furthermore, significant EV penetrations, weather, and traffic congestion further adversely affect the performance of the power system.

Semi-uniform Feller Stochastic Kernels

This paper studies transition probabilities from a Borel subset of a Polish space to a product of two Borel subsets of Polish spaces. For such transition probabilities it introduces and studies the property of semi-uniform Feller continuity. This paper provides several equivalent definitions of semi-uniform Feller continuity and establishes its preservation under integration. The motivation for this study came from the theory of Markov decision processes with incomplete information, and this paper provides the fundamental results useful for this theory.

Automatic model training under restrictive time constraints

We develop a hyperparameter optimisation algorithm, Automated Budget Constrained Training, which balances the quality of a model with the computational cost required to tune it. The relationship between hyperparameters, model quality and computational cost must be learnt and this learning is incorporated directly into the optimisation problem. At each training epoch, the algorithm decides whether to terminate or continue training, and, in the latter case, what values of hyperparameters to use. This decision weighs optimally potential improvements in the quality with the additional training time and the uncertainty about the learnt quantities. The performance of our algorithm is verified on a number of machine learning problems encompassing random forests and neural networks. Our approach is rooted in the theory of Markov decision processes with partial information and we develop a numerical method to compute the value function and an optimal strategy.

Portfolio management under drawdown constraint in discrete-time financial markets

AbstractConsidering a representative agent in the market, we study the long-term optimal investment problem in a discrete-time financial market, introducing a set of restrictions in the admissible strategies. The drawdown constraints limit the size of possible losses of the portfolio and impose a floor-based performance measure. The optimal growth rate is characterized, and under suitable hypotheses it is proved that an optimal strategy exists. The approach to solving this problem is based on dynamic programming techniques and a fixed point argument adapted from the theory of Markov decision processes.

Resource allocation for UAV-assisted 5G mMTC slicing networks using deep reinforcement learning

The Internet of Things (IoT) application scenarios is becoming extensive due to the quick evolution of smart devices with fifth-generation (5G) network slicing technologies. Hence, IoTs are becoming significantly important in 5G/6G networks. However, communication with IoT devices is more sensitive in disasters because the network depends on the main power supply and devices are fragile. In this paper, we consider Unmanned Aerial Vehicles (UAV) as a flying base station (BS) for the emergency communication system with 5G mMTC Network Slicing to improve the quality of user experience. The UAV-assisted mMTC creates a base station selection method to maximize the system energy efficiency. Then, the system model is reduced to the stochastic optimization-based problem using Markov Decision Process (MDP) theory. We propose a reinforcement learning-based dueling-deep-Q-networks (DDQN) technique to maximise energy efficiency and resource allocation. We compare the proposed model with DQN and Q-Learning models and found that the proposed DDQN-based model performs better for resource allocation in terms of low transmission power and maximum energy efficiency.

A Green Hydrogen Energy System: Optimal control strategies for integrated hydrogen storage and power generation with wind energy

The intermittent nature of renewable energy resources such as wind and solar causes the energy supply to be less predictable leading to possible mismatches in the power network. To this end, hydrogen production and storage can provide a solution by increasing flexibility within the system. Stored hydrogen as compressed gas can either be converted back to electricity or it can be used as feed-stock for industry, heating for built environment, and as fuel for vehicles. This research is the first to examine optimal strategies for operating integrated energy systems consisting of renewable energy production and hydrogen storage with direct gas-based use-cases for hydrogen. Using Markov decision process theory, we construct optimal policies for day-to-day decisions on how much energy to store as hydrogen, or buy from or sell to the electricity market, and on how much hydrogen to sell for use as gas. We pay special emphasis to practical settings, such as contractually binding power purchase agreements, varying electricity prices, different distribution channels, green hydrogen offtake agreements, and hydrogen market price uncertainties. Extensive experiments and analysis are performed in the context of Northern Netherlands where Europe’s first Hydrogen Valley is being formed. Results show that gains in operational revenues of up to 51% are possible by introducing hydrogen storage units and competitive hydrogen market-prices. This amounts to a €126,000 increase in revenues per turbine per year for a 4.5 MW wind turbine. Moreover, our results indicate that hydrogen offtake agreements will be crucial in keeping the energy transition on track.

Mean-field Markov decision processes with common noise and open-loop controls

We develop an exhaustive study of Markov decision process (MDP) under mean field interaction both on states and actions in the presence of common noise, and when optimization is performed over open-loop controls on infinite horizon. Such model, called CMKV-MDP for conditional McKean–Vlasov MDP, arises and is obtained here rigorously with a rate of convergence as the asymptotic problem of N-cooperative agents controlled by a social planner/influencer that observes the environment noises but not necessarily the individual states of the agents. We highlight the crucial role of relaxed controls and randomization hypothesis for this class of models with respect to classical MDP theory. We prove the correspondence between CMKV-MDP and a general lifted MDP on the space of probability measures, and establish the dynamic programming Bellman fixed point equation satisfied by the value function, as well as the existence of ϵ-optimal randomized feedback controls. The arguments of proof involve an original measurable optimal coupling for the Wasserstein distance. This provides a procedure for learning strategies in a large population of interacting collaborative agents.

Multiply Accelerated Value Iteration for NonSymmetric Affine Fixed Point Problems and Application to Markov Decision Processes

We analyze a modified version of the Nesterov accelerated gradient algorithm, which applies to affine fixed point problems with non-self-adjoint matrices, such as the ones appearing in the theory of Markov decision processes with discounted or mean payoff criteria. We characterize the spectra of matrices for which this algorithm does converge with an accelerated asymptotic rate. We also introduce a $d$th-order algorithm and show that it yields a multiply accelerated rate under more demanding conditions on the spectrum. We subsequently apply these methods to develop accelerated schemes for nonlinear fixed point problems arising from Markov decision processes. This is illustrated by numerical experiments.

Throughput Maximization for Multiedge Multiuser Edge Computing Systems

The multiaccess edge computing/mobile-edge computing (MEC) is becoming a key technology toward “full 5G.” However, as it gets widely used, a fundamental problem is how to support as many service requests as possible under stringent Quality-of-Service (QoS) requirements and limited communications and computing resources. In this article, we study the long-term throughput maximization problem for multicell multiuser MEC systems. Different from most of the existing works that focus on energy or latency minimization problem for a single-edge system, a novel design is proposed from the service provider’s perspective to maximize the system-wide throughput under latency bounds by jointly taking user association and resource allocation for both communications and computing into account. To capture the stochastic nature of MEC environments, a Markov decision process (MDP) is employed to model the queuing states for both mobile devices and MEC servers. By combining MDP and matching theory, a joint user association and resource allocation algorithm is given, where the resource allocation policy under given user-server association is solved. Extensive numerical results demonstrate the superiority of the proposed scheme in comparison with several existing approaches.

Optimal Prefetching in Random Trees

We propose and analyze a model for optimizing the prefetching of documents, in the situation where the connection between documents is discovered progressively. A random surfer moves along the edges of a random tree representing possible sequences of documents, which is known to a controller only up to depth d. A quantity k of documents can be prefetched between two movements. The question is to determine which nodes of the known tree should be prefetched so as to minimize the probability of the surfer moving to a node not prefetched. We analyzed the model with the tools of Markov decision process theory. We formally identified the optimal policy in several situations, and we identified it numerically in others.

Deep Q-Network based resource allocation for UAV-assisted Ultra-Dense Networks

With the rapid development of the fifth-generation (5G) wireless communications, the number of users is increasing dramatically and Ultra-Dense Networks (UDN) are becoming more important for supporting numerous users and emerging mission-critical applications. In order to conquer the communication restrictions caused by natural disasters, an emergency communication system using Unmanned Aerial Vehicles (UAV) as a flying base station (BS) to assist UDN is proposed. By virtue of the resource allocation scheme of UAV-assisted UDN systems, communication resources can be reasonably and effectively allocated to improve the quality of user experience. Firstly, aiming to maximize the system energy efficiency (EE), a UDN system model including the BS selection is constructed. Secondly, Markov Decision Process (MDP) theory is applied to transform the system model into a stochastic optimization problem. Finally, by using deep reinforcement learning (DRL) technique, we propose a Deep Q-Network (DQN) based resource allocation scheme to maximize the system energy efficiency. Simulation results exhibit that the proposed DQN-based resource allocation scheme can significantly improve the system EE compared with the legacy Q-Learning, random and maximum resource allocation algorithms.

An Algorithmic Study to Maximize 5G Network Throughput Based on the Markov Decision Process

In this article, the network throughput optimization problem is investigated based on the theory of Markov decision process, combining the device-to-device direct selection problem with the finite stage discount MDP model problem. First, models for the device-to-device communication selection using MDP are built; second, the optimal mode selection strategy is derived using a finite stage backward iterative algorithm; and finally, the given mode selection strategy is evaluated by conducting a large number of simulation experiments. The results show that the MDP-based mode selection method proposed in this article has better performance in maximizing throughput and can yield better mode selection strategies with the advantage of obtaining larger system throughput.

A Bayesian Theory of Change Detection in Statistically Periodic Random Processes

A new class of stochastic processes called independent and periodically identically distributed (i.p.i.d.) processes is defined to capture periodically varying statistical behavior. A novel Bayesian theory is developed for detecting a change in the distribution of an i.p.i.d. process. It is shown that the Bayesian change point problem can be expressed as an optimal control problem of a Markov decision process (MDP) with periodic transition and cost structures. An optimal control theory is developed for periodic MDPs for discounted and undiscounted total cost criteria. A fixed-point equation is obtained that is satisfied by the optimal cost function. It is shown that a nonstationary but periodic policy is optimal. A value iteration algorithm is obtained to compute the optimal cost function. The results from the MDP theory are then applied to detect a change in the distribution of an i.p.i.d. process. It is shown that while a stopping rule based on a periodic sequence of thresholds is exactly optimal, a single-threshold policy is asymptotically optimal, as the probability of false alarm goes to zero. Numerical results are provided to demonstrate that the asymptotically optimal policy is not strictly optimal.

Comparative Effectiveness of Different Combinations of Treatment Interventions in Patients with Stroke at the Convalescence Stage Based on the Markov Decision Process.

Objective The objective of this study was to compare the effectiveness of different combinations of interventions in patients with stroke at the convalescence stage based on the electronic health records (EHRs) by using the Markov decision process (MDP) theory and explore the feasibility of the Markov model in the real-world study (RWS). Methods Screening was conducted for patients with stroke at the convalescence stage who were admitted to the Third Affiliated Hospital of Zhejiang Chinese Medical University from January 2012 to January 2017 based on the EHRs. The relevant clinical data were extracted, and the appropriate conversion was made (state-action-reward) according to the Markov model. The transformed data were analysed and solved by the MDP to obtain the best interventions for patients with various stroke recovery periods. Results 926 patients with stroke at the convalescence stage were initially selected. And according to the inclusion exclusion criteria, 854 patients were screened. Through the MDP, we obtained the following results: (1)when the patients with stroke at the convalescence stage have a medical history, but no complications, and mild neurological impairment, ≥66-year- and 18–45-year-old patients are advised to choose acupuncture treatment. 46–65-year-old patients are advised to choose rehabilitation treatment. When patients with moderate to severe neurological impairment, patients are advised to choose rehabilitation, Chinese herbal decoction, and acupuncture combined therapy. (2) Without complications or medical history, patients who are ≥ 66 years old are recommended to choose rehabilitation treatment when the nerve function impairment is mild; rehabilitation and acupuncture treatment are recommended when moderate and severe injuries are caused. (3) The combination of rehabilitation, Chinese herbal decoction, and acupuncture treatment is recommended for patients with phlegm and blood stasis. Acupuncture treatment is recommended for patients with mild impairment of nerve function in qi deficiency and blood stasis type. Rehabilitation, Chinese herbal decoction, and acupuncture treatment are recommended for moderate-severe injuries. Conclusions The MDP makes it possible to study the effectiveness of various treatment methods in stroke patients who are at the convalescence stage. Further exploratory studies using MDP theory in other areas in which complex interventions are common would be worthwhile.

Algorithmic trading in a microstructural limit order book model

We propose a microstructural modeling framework for studying optimal market-making policies in a FIFO (first in first out) limit order book (order book). In this context, the limit orders, market orders, and cancel orders arrivals in the order book are modeled as point processes with intensities that only depend on the state of the order book. These are high-dimensional models which are realistic from a micro-structure point of view and have been recently developed in the literature. In this context, we consider a market maker who stands ready to buy and sell stock on a regular and continuous basis at a publicly quoted price, and identifies the strategies that maximize their P&L penalized by their inventory. An extension of the methodology is proposed to solve market-making problems where the orders arrivals are modeled using Hawkes processes with exponential kernel. We apply the theory of Markov Decision Processes and dynamic programming method to characterize analytically the solutions to our optimal market-making problem. The second part of the paper deals with the numerical aspect of the high-dimensional trading problem. We use a control randomization method combined with quantization method to compute the optimal strategies. Several computational tests are performed on simulated data to illustrate the efficiency of the computed optimal strategy. In particular, we simulated an order book with constant/ symmetric/ asymmetrical/ state dependent intensities, and compared the computed optimal strategy with naive strategies. Some codes are available on https://github.com/comeh.

On value functions for FCFS queues with batch arrivals and general cost structures

We develop a unified framework for analyzing and optimizing costs for systems of FCFS queues with batch arrivals, setup delays and a general nonlinear cost structure that includes costs associated with energy used, setup times and Quality of Service (QoS) measures. We focus on the MX/G/1 and GeoX/G/1 queues with i.i.d. service times, but our results hold also for arbitrary i.i.d. batch structures where service times within a batch may depend on the batch size and have different, possibly dependent, distributions. We use the notion of value functions from the theory of Markov decision processes (MDPs), along with exponential cost functions, to develop the notion of, and simple expressions for, value generating functions. These can be used to find efficient energy control and job dispatching policies.

Charging Load Prediction and Distribution Network Reliability Evaluation Considering Electric Vehicles’ Spatial-Temporal Transfer Randomness

Integration of large-scale cluster electric vehicles (EVs) and their spatial-temporal transfer randomness are likely to affect the safety and economic operation of the distribution network. This paper investigates the spatial-temporal distribution prediction of EVs’ charging load and then evaluates the reliability of the distribution network penetrated with large-scale cluster EVs. To effectively predict the charging load, trip chain technology, Monte Carlo method and Markov decision process (MDP) theory are employed. Moreover, a spatial-temporal transfer model of EVs is established, and based on which, an EV energy consumption model and a charging load prediction model are constructed with consideration of temperature, traffic condition and EV owner’s subjective willingness in different scenarios. With the application of sequential Monte Carlo method, the paper further evaluates distribution network reliability in various charging scenarios. In the evaluation, indices including per unit value (PUV), fast voltage stability index (FVSI), loss of load probability (LOLP), system average interruption frequency index (SAIFI), system average interruption duration index (SAIDI), and expected energy not supplied (EENS) are incorporated. To validate the proposed prediction model and evaluation method, a series of numerical simulations are conducted on the basis of taking the traffic-distribution system of a typical city as an example. The result demonstrates that the proposed spatial-temporal transfer model is more practical in charging load prediction than the popularly used Dijkstra’s shortest path algorithm. Moreover, high temperature, congestion and the increment of EV penetration rate will further weaken distribution network reliability.

Online Learning-based Virtual Resource Allocation for Network Slicing in Virtualized Cloud Radio Access Network

To solve the problem of lacking efficient and dynamic resource allocation schemes for 5G Network Slicing (NS) in Cloud Radio Access Network (C-RAN) scenario in the existing researches, a virtual resource allocation algorithm for NS in virtualized C-RAN is proposed. Firstly, a stochastic optimization model in virtualized C-RAN network is established based on the Constrained Markov Decision Process (CMDP) theory, which maximizes the average sum rates of all slices as its objective, and is subject to the average delay constraint for each slice as well as the average network backhaul link bandwidth consumption constraint in the meantime. Secondly, in order to overcome the issue of having difficulties in acquiring the accurate transition probabilities of the system states in the proposed CMDP optimization problem, the concept of Post-Decision State (PDS) as an intermediate state” is introduced, which is used to describe the state of the system after the known dynamics, but before the unknown dynamics occur, and it incorporates all of the known information about the system state transition. Finally, an online learning based virtual resource allocation algorithm is presented for NS in virtualized C-RAN, where in each discrete resource scheduling slot, it will allocate appropriate Resource Blocks (RBs) and caching resource for each network slice according to the observed current system state. The simulation results reveal that the proposed algorithm can effectively satisfy the Quality of Service (QoS) demand of each individual network slice, reduce the pressure of backhaul link on bandwidth consumption and improve the system throughput.