Discounted Markov Decision Processes Research Articles

The effect of adherence on antihypertensive therapy plans in patients with diabetes

Patient adherence to a medication plan may significantly impact the benefits received by therapy. Thus, considering patient adherence to antihypertensive medication therapy in patients with diabetes, we investigate the impacts of adherence levels on patient health outcomes using a finite horizon, discounted Markov decision process. Health states are based on varying systolic blood pressure levels, cardiovascular complications, adherence levels, and the patient’s current hypertension medications. We model patient transitions through these health states by combining various models from the literature. The model maximizes the expected quality-adjusted life years (QALY). Experimentation on varying levels of patient adherence to medication plans emphasizes the importance of adherence to medication plans with respect to a quality of life metric. Furthermore, sensitivity analysis of model factors finds that smoking as an internal factor, and model parameters as external factors highly influence health outcomes. Furthermore, when patients maintain a higher than average adherence level, they will be able to noticeably increase their expected QALYs.

Discounted Markov Decision Processes with Constrained Costs: the decomposition approach

In this paper we consider a constrained optimization of discrete time Markov Decision Processes (MDPs) with finite state and action spaces, which accumulate both a reward and costs at each decision epoch. We will study the problem of finding a policy that maximizes the expected total discounted reward subject to the constraints that the expected total discounted costs are not greater than given values. Thus, we will investigate the decomposition method of the state space into the strongly communicating classes for computing an optimal or a nearly optimal stationary policy. The discounted criterion has many applications in several areas such that the Forest Management, the Management of Energy Consumption, the finance, the Communication System (Mobile Networks) and the artificial intelligence.

K-spin Hamiltonian for quantum-resolvable Markov decision processes

The Markov decision process is the mathematical formalization underlying the modern field of reinforcement learning when transition and reward functions are unknown. We derive a pseudo-Boolean cost function that is equivalent to a K-spin Hamiltonian representation of the discrete, finite, discounted Markov decision process with infinite horizon. This K-spin Hamiltonian furnishes a starting point from which to solve for an optimal policy using heuristic quantum algorithms such as adiabatic quantum annealing and the quantum approximate optimization algorithm on near-term quantum hardware. In arguing that the variational minimization of our Hamiltonian is approximately equivalent to the Bellman optimality condition for a prevalent class of environments we establish an interesting analogy with classical field theory. Along with proof-of-concept calculations to corroborate our formulation by simulated and quantum annealing against classical Q-Learning, we analyze the scaling of physical resources required to solve our Hamiltonian on quantum hardware.

Robust analysis of discounted Markov decision processes with uncertain transition probabilities

Optimal policies in Markov decision problems may be quite sensitive with regard to transition probabilities. In practice, some transition probabilities may be uncertain. The goals of the present study are to find the robust range for a certain optimal policy and to obtain value intervals of exact transition probabilities. Our research yields powerful contributions for Markov decision processes (MDPs) with uncertain transition probabilities. We first propose a method for estimating unknown transition probabilities based on maximum likelihood. Since the estimation may be far from accurate, and the highest expected total reward of the MDP may be sensitive to these transition probabilities, we analyze the robustness of an optimal policy and propose an approach for robust analysis. After giving the definition of a robust optimal policy with uncertain transition probabilities represented as sets of numbers, we formulate a model to obtain the optimal policy. Finally, we define the value intervals of the exact transition probabilities and construct models to determine the lower and upper bounds. Numerical examples are given to show the practicability of our methods.

Discounted Markov decision processes with fuzzy costs

Fuzzy theory is a discipline that has recently appeared in the mathematical literature. It generalizes classic situations. Therefore, its success continues to increase and to keep going up from time to time. In this work, we consider the model of Markov decision processes where the information on the costs includes imprecision. The fuzzy cost is represented by the fuzzy number set and the infinite horizon discounted cost is minimized from any stationary policy. This paper presents in the first part the notion of fuzzy sets and some axiomatic basis and relevant concepts with fuzzy theory in short. In second part, we propose a new definition of total discounted fuzzy cost in infinite planning horizon. We will compute an optimal stationary policy that minimizes the total fuzzy discounted cost by a new approach based on some standard algorithms of the dynamic programming using the ranking function concept. The last adapted criterion has many applications in several areas such that the forest management, the management of energy consumption, the finance, the communication system (mobile networks).

Online battery scheduling for grid-connected photo-voltaic systems

Abstract We study the problem of determining optimum policy for managing battery energy storage system (BESS) in grid-connected photo-voltaic (PV) systems, where the stochastic electricity demands from the load are met from three sources: grid, PV energy, and BESS. BESS is used either to store excess energy generated from PV systems for later use, or to purchase energy from the grid when the time-of-use (TOU) pricing is lower. The objective is to identify the optimum charging/discharging schedule of BESS so that the long-term cost of energy purchased from the grid is minimized. The stochastic variabilities in loads and PV energy are captured by employing probabilistic models of periodic stochastic process with parameters estimated using historical data. The optimization problem is formulated under the framework of periodic discounted Markov decision process (MDP), and the problem formulation includes the aging effects of batteries and solar panels. The online optimization problem is solved by adopting a policy iteration approach tailored for periodic MDP. The proposed online scheduling algorithm provides periodic policies for a period of 24-hour, where the system model is updated every day based on load and PV energy from the previous day in a rolling horizon fashion. Simulation results demonstrate that the proposed algorithm can achieve a 41.6% reduction in annual utility bills compared to conventional systems without PV and BESS, thus ascertaining the values of installing BESS and PV systems.

Complexity bounds for approximately solving discounted MDPs by value iterations

For an infinite-horizon discounted Markov decision process with a finite number of states and actions, this note provides upper bounds on the number of operations required to compute an approximately optimal policy by value iterations in terms of the discount factor, spread of the reward function, and desired closeness to optimality. One of the provided upper bounds on the number of iterations has the property that it is a non-decreasing function of the value of the discount factor.

Hamiltonian Cycles and Subsets of Discounted Occupational Measures

We study a certain polytope arising from embedding the Hamiltonian cycle problem in a discounted Markov decision process. The Hamiltonian cycle problem can be reduced to finding particular extreme points of a certain polytope associated with the input graph. This polytope is a subset of the space of discounted occupational measures. We characterize the feasible bases of the polytope for a general input graph G and determine the expected numbers of different types of feasible bases when the underlying graph is random. We utilize these results to demonstrate that augmenting certain additional constraints to reduce the polyhedral domain can eliminate a large number of feasible bases that do not correspond to Hamiltonian cycles. Finally, we develop a random walk algorithm on the feasible bases of the reduced polytope and present some numerical results. We conclude with a conjecture on the feasible bases of the reduced polytope.

Determining ambulance destinations when facing offload delays using a Markov decision process

When emergency departments (EDs) are crowded and cannot accept incoming ambulance patients immediately, paramedics commonly continue to provide patient care until an ED bed becomes available. This delay in transferring a patient to the ED is referred to as ambulance offload delay (AOD). AOD is a pressing problem for Emergency Medical Services (EMS) as it prolongs the time before paramedics are available to respond to other calls. This can negatively affect ambulance availability and patient safety. The objective of this study is to develop an ambulance destination policy to mitigate AOD, allowing patients to see physicians sooner, and returning ambulances to service more quickly. We formulate a discrete time, infinite-horizon, discounted Markov Decision Process (MDP) model to determine when it is advantageous to send appropriate patients to out-of-region EDs, which have longer transport times but shorter offload times. Based on the MDP model, an optimal ambulance destination policy is constructed using the policy iteration algorithm. A computational study is applied using 12-months of data from an EMS provider which experiences AOD regularly. We find that the optimal policies can significantly reduce AOD, time to bed for patients, and out-of-service time for paramedics at the expense of increased ambulances travel distances. The model can be generalized and used as a decision support tool for EMS systems to mitigate the impact of AOD on their operations.

A Moreau-Yosida regularization for Markov decision processes

This paper addresses a class of sequential optimization problems known as Markov decision processes. These kinds of processes are considered on Euclidean state and action spaces with the total expected discounted cost as the objective function. The main goal of the paper is to provide conditions to guarantee an adequate Moreau-Yosida regularization for Markov decision processes (named the original process). In this way, a new Markov decision process that conforms to the Markov control model of the original process except for the cost function induced via the Moreau-Yosida regularization is established. Compared to the original process, this new discounted Markov decision process has richer properties, such as the differentiability of its optimal value function, strictly convexity of the value function, uniqueness of optimal policy, and the optimal value function and the optimal policy of both processes, are the same. To complement the theory presented, an example is provided.

Towards solving 2-TBSG efficiently

Two-player turn-based stochastic game (2-TBSG) is a two-player game model which aims to find Nash equilibriums and is widely utilized in reinforcement learning and AI. Inspired by the fact that the simplex method for solving the deterministic discounted Markov decision processes is strongly polynomial independent of the discount factor, we are trying to answer an open problem whether there is a similar algorithm for 2-TBSG. We develop a simplex strategy iteration where one player updates its strategy with a simplex step while the other player finds an optimal counterstrategy in turn, and a modified simplex strategy iteration. Both of them belong to a class of geometrically converging algorithms. We establish the strongly polynomial property of these algorithms by considering a strategy combined from the current strategy and the equilibrium strategy. Moreover, we present a method to transform general 2-TBSGs into special 2-TBSGs where each state has exactly two actions.

Constrained discounted Markov decision processes with Borel state spaces

We study discrete-time discounted constrained Markov decision processes (CMDPs) with Borel state and action spaces. These CMDPs satisfy either weak (W) continuity conditions, that is, the transition probability is weakly continuous and the reward function is upper semicontinuous in state–action pairs, or setwise (S) continuity conditions, that is, the transition probability is setwise continuous and the reward function is upper semicontinuous in actions. Our main goal is to study models with unbounded reward functions, which are often encountered in applications, e.g., in consumption/investment problems. We provide some general assumptions under which the optimization problems in CMDPs are solvable in the class of randomized stationary policies and in the class of chattering policies introduced in this paper. If the initial distribution and transition probabilities are atomless, then using a general “purification result” of Feinberg and Piunovskiy we show the existence of a deterministic (stationary) optimal policy. Our main results are illustrated by examples.

Data-driven predictive maintenance scheduling policies for railways

Inspection and maintenance activities are essential to preserving safety and cost-effectiveness in railways. However, the stochastic nature of railway defect occurrence is usually ignored in literature; instead, defect stochasticity is considered independently of maintenance scheduling. This study presents a new approach to predict rail and geometry defects that relies on easy-to-obtain data and integrates prediction with inspection and maintenance scheduling activities. In the proposed approach, a novel use of risk-averse and hybrid prediction methodology controls the underestimation of defects. Then, a discounted Markov decision process model utilizes these predictions to determine optimal inspection and maintenance scheduling policies. Furthermore, in the presence of capacity constraints, Whittle indices via the multi-armed restless bandit formulation dynamically provide the optimal policies using the updated transition kernels. Results indicate a high accuracy rate in prediction and effective long-term scheduling policies that are adaptable to changing conditions.

Optimal forest management under financial risk aversion with discounted Markov decision process models

The common assumption of risk neutrality in forest decision making is generally inadequate because the stakeholders tend to be averse to fluctuations in the return criteria. In Markov decision processes (MDPs) of forest management, risk aversion and standard mean-variance analysis can be readily dealt with if the criteria are undiscounted expected values. However, with discounted criteria such as the fundamental net present value of financial returns, the classic mean-variance optimization is numerically intractable. In lieu of this, this paper (i) presents a linear-programming method to calculate the variance of discounted criteria conditional on any specific policy and (ii) adopts, as an alternative to the variance measure of risk, the “discount normalized variance” (DNV), an economically meaningful criterion consistent with income-smoothing behavior. The DNV is then used in procedures analogous to mean-variance analysis and certainty-equivalent optimization tractable by quadratic programming. The methods are applied to the management of uneven-aged, mixed-species forests in the southern United States. The results document the trade-off between the expected net present value and risk of financial returns, as well as the consequences for selected ecological criteria.

Optimal Power Allocation for Wireless Sensor Powered by Dedicated RF Energy Source

This paper studies a wireless-powered sensor network, where a sensor harvests energy from a dedicated radio-frequency (RF) energy source and transmits information to an information sink using the harvested energy. Two working modes are considered; One is the frequency division multiplexing (FDM) mode in which the sensor harvests RF energy and transmits information simultaneously over orthogonal frequency bands. The other is the time division multiplexing (TDM) mode in which energy harvesting and information transmission are implemented in the same frequency band but in different time slots. The energy harvesting channel and the information transmission channel are assumed to follow the Rician and the Rayleigh distributions, respectively, and are discretized and modeled as finite-state Markov chains. We formulate the process of energy harvesting and information transmission as an infinite-horizon discounted Markov decision process. The value iteration algorithm is used to find an asymptotically optimal energy harvesting and information transmission policy to optimize the long-term throughput. In the asymptotically optimal policy of the FDM mode, the energy transmitted from the sensor in one slot is proved to be non-decreasing with the battery state of the sensor. By contrast, such monotonicity between the transmitted energy and the battery state does not exist in the asymptotically optimal policy in the TDM mode. Simulation results verify the above findings and demonstrate that the proposed method outperforms the heuristic greedy method.

Mean–variance optimization of discrete time discounted Markov decision processes

In this paper, we study a mean–variance optimization problem in an infinite horizon discrete time discounted Markov decision process (MDP). The objective is to minimize the variance of system rewards with the constraint of mean performance. Different from most of works in the literature which require the mean performance already achieve optimum, we can let the discounted performance equal any constant. The difficulty of this problem is caused by the quadratic form of the variance function which makes the variance minimization problem not a standard MDP. By proving the decomposable structure of the feasible policy space, we transform this constrained variance minimization problem to an equivalent unconstrained MDP under a new discounted criterion and a new reward function. The difference of the variances of Markov chains under any two feasible policies is quantified by a difference formula. Based on the variance difference formula, a policy iteration algorithm is developed to find the optimal policy. We also prove the optimality of deterministic policy over the randomized policy generated in the mean-constrained policy space. Numerical experiments demonstrate the effectiveness of our approach.

Mean Variance Optimization in Markov Decision Processes with State-Dependent Risk Aversion

In many revenue management applications risk aversion is crucial. In dynamic decision problems it is challenging to find the right balance between maximizing expected profits and minimizing their variance. In this paper, we present an efficient dynamic programming approach to leverage mean variance optimization in discounted Markov decision processes with finite time horizon. We show how to derive optimal risk-sensitive policies which allow for state-dependent mean variance balancing. We compare our approach to risk-neutral policies as well as risk-averse policies of the exponential utility model. Based on state-probabilities, we analytically evaluate the expected evolution of accumulated profits as well as profit variances. Our general concepts are demonstrated for dynamic pricing applications.

Simplex Algorithm for Countable-State Discounted Markov Decision Processes

We consider discounted Markov decision processes (MDPs) with countably-infinite state spaces, finite action spaces, and unbounded rewards. Typical examples of such MDPs are inventory management and queueing control problems in which there is no specific limit on the size of inventory or queue. Existing solution methods obtain a sequence of policies that converges to optimality in value but may not improve monotonically, ie., a policy in the sequence may be worse than preceding policies. Our proposed approach considers countably-infinite linear programming (CILP) formulations of the MDPs (a CILP is defined as a linear program (LP) with countably-infinite numbers of variables and constraints). Under standard assumptions for analyzing MDPs with countably-infinite state spaces and unbounded rewards, we extend the major theoretical extreme point and duality results to the resulting CILPs. Under additional mild assumptions, which are satisfied by several applications of interest, we present a simplex-type algorithm that is implementable in the sense that each of its iterations requires only a finite amount of data and computation. We show that the algorithm finds a sequence of policies that improves monotonically and converges to optimality in value. Unlike existing simplex-type algorithms for CILPs, our proposed algorithm solves a class of CILPs in which each constraint may contain an infinite number of variables and each variable may appear in an infinite number of constraints. A numerical illustration for inventory management problems is also presented. The online appendix is available at https://doi.org/10.1287/opre.2017.1598 .

Accelerated decomposition techniques for large discounted Markov decision processes

Many hierarchical techniques to solve large Markov decision processes (MDPs) are based on the partition of the state space into strongly connected components (SCCs) that can be classified into some levels. In each level, smaller problems named restricted MDPs are solved, and then these partial solutions are combined to obtain the global solution. In this paper, we first propose a novel algorithm, which is a variant of Tarjan’s algorithm that simultaneously finds the SCCs and their belonging levels. Second, a new definition of the restricted MDPs is presented to ameliorate some hierarchical solutions in discounted MDPs using value iteration (VI) algorithm based on a list of state-action successors. Finally, a robotic motion-planning example and the experiment results are presented to illustrate the benefit of the proposed decomposition algorithms.

Outage Probability Minimization for Energy Harvesting Cognitive Radio Sensor Networks.

The incorporation of cognitive radio (CR) capability in wireless sensor networks yields a promising network paradigm known as CR sensor networks (CRSNs), which is able to provide spectrum efficient data communication. However, due to the high energy consumption results from spectrum sensing, as well as subsequent data transmission, the energy supply for the conventional sensor nodes powered by batteries is regarded as a severe bottleneck for sustainable operation. The energy harvesting technique, which gathers energy from the ambient environment, is regarded as a promising solution to perpetually power-up energy-limited devices with a continual source of energy. Therefore, applying the energy harvesting (EH) technique in CRSNs is able to facilitate the self-sustainability of the energy-limited sensors. The primary concern of this study is to design sensing-transmission policies to minimize the long-term outage probability of EH-powered CR sensor nodes. We formulate this problem as an infinite-horizon discounted Markov decision process and propose an ϵ-optimal sensing-transmission (ST) policy through using the value iteration algorithm. ϵ is the error bound between the ST policy and the optimal policy, which can be pre-defined according to the actual need. Moreover, for a special case that the signal-to-noise (SNR) power ratio is sufficiently high, we present an efficient transmission (ET) policy and prove that the ET policy achieves the same performance with the ST policy. Finally, extensive simulations are conducted to evaluate the performance of the proposed policies and the impaction of various network parameters.