Finite Markov Decision Processes Research Articles

Markov decision processes and stochastic games with total effective payoff

We consider finite Markov decision processes with undiscounted total effective payoff. We show that there exist uniformly optimal pure and stationary strategies that can be computed by solving a polynomial number of linear programs. This implies that in a two-player zero-sum stochastic game with perfect information and with total effective payoff there exists a stationary best response to any stationary strategy of the opponent. From this, we derive the existence of a uniformly optimal pure and stationary saddle point. Finally we show that mean payoff can be viewed as a special case of total payoff.

The dynamic shortest path problem with time-dependent stochastic disruptions

The dynamic shortest path problem with time-dependent stochastic disruptions consists of finding a route with a minimum expected travel time from an origin to a destination using both historical and real-time information. The problem is formulated as a discrete time finite horizon Markov decision process and it is solved by a hybrid Approximate Dynamic Programming (ADP) algorithm with a clustering approach using a deterministic lookahead policy and value function approximation. The algorithm is tested on a number of network configurations which represent different network sizes and disruption levels. Computational results reveal that the proposed hybrid ADP algorithm provides high quality solutions with a reduced computational effort.

A Dynamic pricing demand response algorithm for smart grid: Reinforcement learning approach

With the modern advanced information and communication technologies in smart grid systems, demand response (DR) has become an effective method for improving grid reliability and reducing energy costs due to the ability to react quickly to supply-demand mismatches by adjusting flexible loads on the demand side. This paper proposes a dynamic pricing DR algorithm for energy management in a hierarchical electricity market that considers both service provider’s (SP) profit and customers’ (CUs) costs. Reinforcement learning (RL) is used to illustrate the hierarchical decision-making framework, in which the dynamic pricing problem is formulated as a discrete finite Markov decision process (MDP), and Q-learning is adopted to solve this decision-making problem. Using RL, the SP can adaptively decide the retail electricity price during the on-line learning process where the uncertainty of CUs’ load demand profiles and the flexibility of wholesale electricity prices are addressed. Simulation results show that this proposed DR algorithm, can promote SP profitability, reduce energy costs for CUs, balance energy supply and demand in the electricity market, and improve the reliability of electric power systems, which can be regarded as a win-win strategy for both SP and CUs.

Compositional Synthesis of Finite Abstractions for Continuous-Space Stochastic Control Systems: A Small-Gain Approach

This paper is concerned with a compositional approach for constructing finite abstractions (a.k.a. finite Markov decision processes) of interconnected discrete-time stochastic control systems. The proposed framework is based on a notion of so-called stochastic simulation function enabling us to use an abstract system as a substitution of the original one in the controller design process with guaranteed error bounds. In the first part of the paper, we derive sufficient small-gain type conditions for the compositional quantification of the distance in probability between the interconnection of stochastic control subsystems and that of their (finite or infinite) abstractions. In the second part of the paper, we construct finite abstractions together with their corresponding stochastic simulation functions for the class of linear stochastic control systems. We apply our results to the temperature regulation in a circular building by constructing compositionally a finite abstraction of a network containing 1000 rooms. We use the constructed finite abstractions as substitutes to synthesize policies compositionally regulating the temperature in each room for a bounded time horizon.

Spare Parts Inventory Control under a Fixed-Term Contract with a Long-Down Constraint

We are interested in service contracts for spare parts. We introduce a new performance measure, XLD (extreme long down), that limits the number of deliveries that are later than an agreed threshold during the contract period. We consider a single item, single location stockpoint serving multiple systems where demand is satisfied in an alternative way if the stockpoint is out of stock. Using a finite horizon Markov decision process, we derive the optimal spare parts inventory policy for meeting the contract at minimum costs. We prove that a state-dependent and time-dependent base stock policy is optimal. We also prove that the optimal base stock level is non-increasing in the number of allowed extreme long downs in the remaining contract period and that the optimal base stock level decreases at most by one unit per step. We formulate three heuristics for our finite horizon problem. These heuristics are a translation of heuristics commonly used for other service measures than the XLD measure. We assess their performance in comparison to the optimal policy. The results of our numerical study show that Heuristic 3, a myopic heuristic, is the best performing with an average optimality gap of 4.5% and an optimality gap lower than 5% in 81% of the instances. The maximum optimality gap is very high for all three heuristics, showing that important savings can be made by taking into account the actual contract performance and the remaining contract duration in stocking decisions.

Energy Saving and Interference Coordination in HetNets Using Dynamic Programming and CEC

Energy efficiency in cellular networks has gained great relevance due to the increasing power supply demands in new generation heterogeneous network (HetNet). On the other hand, interference coordination is a key aspect in HetNets resource management which directly affects the performance and energy consumption. Although these two aspect are intertwined, they have been studied separately so far. In this paper, we address the joint problem of energy saving and interference coordination in HetNets. We formulate the problem as a finite horizon Markov decision process (MDP) leveraging two facts: the user traffic demands usually follow periodic patterns, and the knowledge and prediction of network load is crucial in order to select efficient network configurations. Quality of Service (QoS) in the network is defined as the ratio of users meeting a requirement specified by the operator, and allows us to account for a minimum QoS requirement by including a constraint in the formulation of the MDP. To address this MDP, we propose an approximate dynamic programming (ADP) algorithm which selects energy efficient farsighted configurations with QoS guarantees achieving near optimal performance. This ADP algorithm is built upon: 1) the certainty equivalent control principle, which simplifies the complexity of the MDP, and 2) machine learning techniques: a neural network and a polynomial regressor which allow us to predict the QoS and the consumption of the network in advance. We evaluate our proposal in a LTE-A network simulator following the 3GPP guidelines, and the results obtained show that a joint control of the energy saving and interference coordination mechanisms results in a notably performance improvement compared to a disjoint control, in terms of both energy savings and QoS guarantees. Moreover, our proposal has the advantage of being adaptable to the operator QoS requirements.

Stochastic control of observer trajectories in passive tracking with acoustic signal propagation optimisation

The authors present in this study a numerical method which computes the optimal trajectory of a underwater vehicle subject to some mission objectives. The method is applied to a submarine whose goal is to best detect one or several targets, or/and to minimise its own detection range perceived by the other targets. The signal considered is acoustic propagation attenuation. This approach is based on dynamic programming of a finite horizon Markov decision process. A quantisation method is applied to fully discretise the problem and allows a numerically tractable solution. Different scenarios are considered. The authors suppose at first that the position and the velocity of the targets are known and in the second they suppose that they are unknown and estimated by a Kalman type filter in a context of passive tracking.

Finite state estimation and control of a multi-input CSTR benchmark

The problem of curse-of-dimensionality in finite state and action Markov decision processes is considered using iterative clustering of closed-loop data and repeated discretization of the state space process model. The performance of the control design approach is demonstrated using a multi-input van der Vusse continuous stirred tank reactor control benchmark. It is demonstrated that the finite state description provides a simple implementation of Bayesian state estimation using cell filters, and dynamic programming gives means to conduct optimization of closed-loop performance of a nonlinear stochastic multidimensional chemical plant.

Network Coding for Video Distortion Reduction in Device-to-Device Communications

In this paper, we study the problem of distributing a real-time video sequence to a group of partially connected cooperative wireless devices using instantly decodable network coding (IDNC). In such a scenario, the coding conflicts occur to service multiple devices with an immediately decodable packet, and the transmission conflicts occur from simultaneous transmissions of multiple devices. To avoid these conflicts, we introduce a novel IDNC graph that represents all feasible coding and transmission conflict-free decisions in one unified framework. Moreover, a real-time video sequence has a hard deadline and unequal importance of video packets. Using these video characteristics and the new IDNC graph, we formulate the problem of minimizing the mean video distortion before the deadline as a finite horizon Markov decision process (MDP) problem. However, the backward induction algorithm that finds the optimal policy of the MDP formulation has high modeling and computational complexities. To reduce these complexities, we further design a two-stage maximal independent set selection algorithm, which can efficiently reduce the mean video distortion before the deadline. Simulation results over a real video sequence show that our proposed IDNC algorithms improve the received video quality compared with the existing IDNC algorithms.

Markov decision process model for patient admission decision at an emergency department under a surge demand

We study an admission control problem for patients arriving at an emergency department in the aftermath of a mass casualty incident. A finite horizon Markov decision process (MDP) model is formulated to determine patient admission decisions. In particular, our model considers the time-dependent arrival of patients and time-dependent reward function. We also consider a policy restriction that immediate-patients should be admitted as long as there is available beds. The MDP model has a continuous state space, and we solve the model by using a state discretization technique and obtain numerical solutions. Structural properties of an optimal policy are reviewed, and the structures observed in the numerical solutions are explained accordingly. Experimental results with virtual patient arrival scenarios demonstrates the performance and advantage of optimal policies obtained from the MDP model.

Traffic-Aware Green Cognitive Radio

Cognitive Radio Network (CRN) has emerged as an effective solution to the spectrum under-utilization problem, by providing secondary users (SUs) an opportunistic access to the unoccupied frequency bands of primary users (PUs). Most of the current research on CRN are based on the assumption that the SU always has a large amount of data to transmit. This leads to the objective of SU throughput maximization with a constraint on the allowable interference to the PU. However, in many of the practical scenarios, the data arrival process of the SU closely follows an ON–OFF traffic model, and thus the usual throughput optimization framework may no longer be suitable. In this paper, we propose an intelligent data scheduling strategy which minimizes the average transmission power of the SU while maintaining the transmission delay to be sufficiently small. The data scheduling problem has been formulated as a finite horizon Markov Decision Process (MDP) with an appropriate cost function. Dynamic programming approach has been adopted to arrive at an optimal solution. Our findings show that the average transmitted power for our proposed approach can be as small as 36.5% of the power required for usual throughput maximization technique with insignificant increase in average delay.

Integral temporal difference learning for continuous-time linear quadratic regulations

In this paper, we propose a temporal difference (TD) learning method, called integral TD learning that efficiently finds solutions to continuous-time (CT) linear quadratic regulation (LQR) problems in an online fashion where system matrix A is unknown. The idea originates from a computational reinforcement learning method known as TD(0), which is the simplest TD method in a finite Markov decision process. For the proposed integral TD method, we mathematically analyze the positive definiteness of the updated value functions, monotone convergence conditions, and stability properties concerning the locations of the closed-loop poles in terms of the learning rate and the discount factor. The proposed method includes the existing value iteration method for CT LQR problems as a special case. Finally, numerical simulations are carried out to verify the effectiveness of the proposed method and further investigate the aforementioned mathematical properties.

The booking problem of a diagnostic resource with multiple patient classes and emergency interruptions

CT department is an appointment based service system. All patients except emergency patients should be scheduled in advance. Regular patients arrive on time. Emergency patients arrive randomly and get non-preemptive priority. So the direct waiting time of regular patients should be explicitly described. The unit direct waiting cost gradually increases with the previous waiting time. On the one hand, too many regular patients will increase the direct waiting cost for them and overtime cost for medical resources. On the other hand, too few regular patients will cause idling of medical resources and penalty for delaying regular patients’ requests. So a reasonable reservation level should be determined in advance. The booking problem is modeled as a finite horizon Markov decision process. Some booking limit structures are given to improve the total expected reward. The numerical analysis shows that our policies are much better than the policy currently employed.

Approximate Safety Verification and Control of Partially Observable Stochastic Hybrid Systems

Assuring safety in stochastic hybrid systems is particularly difficult when only noisy or partial observations of the state are available. We first review a formulation of the probabilistic safety problem under partial hybrid observations as a dynamic program over an equivalent information state. Two methods for approximately solving the dynamic program are presented. The first approximates the hybrid system as a finite state Markov decision process, so that the information state is a probability mass function. The second method approximates an indicator function over the safe region using radial basis functions, to represent the information state as a Gaussian mixture. In both cases we discretize the hybrid observation process, then use point-based value iteration to under-approximate the safety probability and synthesize a safety-preserving control policy. We obtain error bounds and convergence results in both cases, assuming switched affine dynamics and additive Gaussian noise on the continuous states and observations. We compare the performance of the finite state and Gaussian mixture approaches on a simple numerical example.

Adaptive Transit Routing in Stochastic Time-Dependent Networks

We define an adaptive routing problem in a stochastic time-dependent transit network in which transit arc travel times are discrete random variables with known probability distributions. We formulate it as a finite horizon Markov decision process. Routing strategies are conditioned on the arrival time of the traveler at intermediate nodes and real-time information on arrival times of buses at stops along their routes. The objective is to find a strategy that minimizes the expected travel time, subject to a constraint that guarantees that the destination is reached within a certain threshold. Although this framework proves to be advantageous over a priori routing, it inherits the curse of dimensionality, and state space reduction through preprocessing is achieved by solving variants of the time-dependent shortest path problem. Numerical results on a network representing a part of the Austin, Texas, transit system indicate a promising reduction in the state space size and improved tractability of the dynamic program.

Process control using finite Markov chains with iterative clustering

Abstract Finite state Markov Decision Processes (MDP) for process control are considered. MDP provide robust tools to perform optimization in closed-loop, and their finite state description enables an easy implementation of Bayesian state estimation. An approach to tackle the curse of dimensionality problem, yet retaining the benefits of the finite state MDP in control and estimator design, is proposed. The suggested approach uses iterative re-discretization based on clustering of closed-loop data. An efficient modification of the k-means clustering technique is proposed. The performance of the approach is demonstrated using a challenging benchmark from chemical engineering, the van der Vusse continuous stirred tank reactor control problem. It is shown that the requirements of the benchmark are met, and that the suggested iterated clustering significantly improves the performance. It is concluded that the finite state MDP approach is a viable alternative for small-to-medium scale problems of practical process control and state estimation.

Risk minimising strategies for revenue management problems with target values

Consider a risk-averse decision maker in the setting of a single-leg dynamic revenue management problem with revenue controlled by limiting capacity for a fixed set of prices. Instead of focussing on maximising the expected revenue, the decision maker has the main objective of minimising the risk of failing to achieve a given target revenue. Interpreting the revenue management problem in the framework of finite Markov decision processes, we augment the state space of the risk-neutral problem definition and change the objective function to the probability of failing a certain specified target revenue. This enables us to obtain a dynamic programming solution that generates the policy minimising the risk of not attaining this target revenue. We compare this solution with recently proposed risk-sensitive policies in a numerical study and discuss advantages and limitations.

Sleeping experts and bandits approach to constrained Markov decision processes

This communique presents simple simulation-based algorithms for obtaining an approximately optimal policy in a given finite set in large finite constrained Markov decision processes. The algorithms are adapted from playing strategies for “sleeping experts and bandits” problem and their computational complexities are independent of state and action space sizes if the given policy set is relatively small. We establish convergence of their expected performances to the value of an optimal policy and convergence rates, and also almost-sure convergence to an optimal policy with an exponential rate for the algorithm adapted within the context of sleeping experts.

Optimal Marketing Strategies for Modeling Real Settings: Improving The Multichannel in Banking

This paper presents a dynamic model approach to analyze the utility generated by a customer's buying behavior dynamics. The dynamic of the model is represented by a class of controllable finite Markov Decision Process (MDP). The MDP model the trajectory of the vendor across their states (segments) governed by transition probabilities. We show that the system and the trajectory dynamics converge. For representing the properties of the dynamics we introduce optimal policies to maximize the one-step ahead increment of the utility function. At each time period the system state provides all the information necessary for choosing an action. We model the vendor in terms of the customer behavior over time considering the actions proposed by the vendor in the marketing campaigns. Strategies dictate how the customer makes his decisions (choose a strategy). As a result of choosing a strategy the customer moves to a new (possibly different) state whose probability distribution depends on the previous state and the actions chosen. Validity of the proposed method is successfully demonstrated both, theoretically and by a simulated experiment related to multichannel for a bank.

Random search for constrained Markov decision processes with multi-policy improvement

This communique first presents a novel multi-policy improvement method which generates a feasible policy at least as good as any policy in a given set of feasible policies in finite constrained Markov decision processes (CMDPs). A random search algorithm for finding an optimal feasible policy for a given CMDP is derived by properly adapting the improvement method. The algorithm alleviates the major drawback of solving unconstrained MDPs at iterations in the existing value-iteration and policy-iteration type exact algorithms. We establish that the sequence of feasible policies generated by the algorithm converges to an optimal feasible policy with probability one and has a probabilistic exponential convergence rate.