Average Reward Markov Decision Process Research Articles

Logarithmic Regret Bounds for Continuous-Time Average-Reward Markov Decision Processes

Logarithmic Regret Bounds for Continuous-Time Average-Reward Markov Decision Processes

Regret Analysis of Policy Gradient Algorithm for Infinite Horizon Average Reward Markov Decision Processes

In this paper, we consider an infinite horizon average reward Markov Decision Process (MDP). Distinguishing itself from existing works within this context, our approach harnesses the power of the general policy gradient-based algorithm, liberating it from the constraints of assuming a linear MDP structure. We propose a vanilla policy gradient-based algorithm and show its global convergence property. We then prove that the proposed algorithm has O(T^3/4) regret. Remarkably, this paper marks a pioneering effort by presenting the first exploration into regret bound computation for the general parameterized policy gradient algorithm in the context of average reward scenarios.

Relative Q-learning for Average-Reward Markov Decision Processes with Continuous States

Relative Q-learning for Average-Reward Markov Decision Processes with Continuous States

BATCH POLICY LEARNING IN AVERAGE REWARD MARKOV DECISION PROCESSES.

We consider the batch (off-line) policy learning problem in the infinite horizon Markov Decision Process. Motivated by mobile health applications, we focus on learning a policy that maximizes the long-term average reward. We propose a doubly robust estimator for the average reward and show that it achieves semiparametric efficiency. Further we develop an optimization algorithm to compute the optimal policy in a parameterized stochastic policy class. The performance of the estimated policy is measured by the difference between the optimal average reward in the policy class and the average reward of the estimated policy and we establish a finite-sample regret guarantee. The performance of the method is illustrated by simulation studies and an analysis of a mobile health study promoting physical activity.

Energy management for age of information control in solar-powered IoT end devices

In this paper, we propose several harvesting-aware energy management policies for solar-powered wireless IoT end devices that asynchronously send status updates for their surrounding environments to a network gateway device. For such devices, we aim at minimizing the average age of information (AoI) metric which has recently been investigated extensively for status update systems. The proposed energy management policies are obtained using discrete-time Markov chain-based modeling of the stochastic intra-day variations of the solar energy harvesting process in conjunction with the average reward Markov decision process formulation. With this approach, energy management policies are constructed by using the time of day and month of year information in addition to the instantaneous values of the age of information and the battery level. The effectiveness of the proposed energy management policies in terms of their capability to reduce the average AoI as well as improving upon the tail of the AoI distribution, is validated with empirical data for a wide range of system parameters.

Relative Q-Learning for Average-Reward Markov Decision Processes with Continuous States

Relative Q-Learning for Average-Reward Markov Decision Processes with Continuous States

Incremental Improvements of Heuristic Policies for Average-Reward Markov Decision Processes

Abstract Within the realm of Discrete Event Systems (DES) theory, the problem of performance optimization for many applications can be modeled as an infinite-horizon, average-reward Markov Decision Process (MDP) with a finite state space. In principle, these MDPs can be solved by various well-developed methods like value iteration, policy iteration and linear programming. But in reality, the tractability of these methods in the context of the aforementioned applications is compromised by the explosive size of the underlying state spaces, a problem that is known as “the curse of dimensionality”. Hence, the corresponding performance optimization problems are frequently addressed by heuristic control policies. The considered work uses results from (i) the sensitivity analysis of Markov reward processes and (ii) the ranking & selection theory in statistics in order to develop a methodology for assessing the optimality of isolated decisions in the context of any well-defined heuristic control policy for the aforementioned MDPs. It also determines an improved decision when the current one is found to be suboptimal. Hence, when embedded in an iterative scheme, this methodology can support the incremental enhancement of the original heuristic policy in a way that controls, both, the computational and also the representational complexity of the new policy. Finally, an additional important feature of the presented methodology is that it can be executed either in an “off-line” mode, using a simulation of the dynamics of the underlying DES, or in an “on-line” mode, based on the sample path that is defined by the real-time dynamics of the controlled system.

Balanced Dynamic Planning in Green Heterogeneous Cellular Networks

Dynamic planning has been used by network operators for energy saving by switching ON–OFF base stations (BSs) according to the traffic load condition while providing quality-of-service (QoS) guarantee for mobile users. In the literature, research efforts focus mainly on satisfying the QoS of downlink mobile users. However, the impact of dynamic planning on the QoS of uplink mobile users is overlooked. A dynamic planning approach that relies only on the downlink performance of mobile users to determine the BS switching decision can result in deactivating nearby small cells and associating uplink mobile users to a faraway macro BS. Consequently, uplink mobile users may suffer from QoS degradation due to the longer transmission distance. Hence, while saving energy for the network operators and satisfying the downlink QoS, the uplink QoS could be violated. In this paper, we propose a balanced dynamic planning approach that accounts for QoS requirements both in the uplink and downlink to specify the BS switching decision. The switching decision is formulated as a two-timescale average reward Markov decision process with finite horizon. Due to computational complexity, sub-optimal algorithms are proposed. Simulation results demonstrate the trade-off between energy saving and QoS guarantee.

Location Aware Opportunistic Bandwidth Sharing between Static and Mobile Users with Stochastic Learning in Cellular Networks

We consider location-dependent opportunistic bandwidth sharing between static and mobile downlink users in a cellular network. Each cell has some fixed number of static users. Mobile users enter the cell, move inside the cell for some time and then leave the cell. In order to provide higher data rate to mobile users, we propose to provide higher bandwidth to the mobile users at favourable times and locations, and provide higher bandwidth to the static users in other times. We formulate the problem as a long run average reward Markov decision process (MDP) where the per-step reward is a linear combination of instantaneous data volumes received by static and mobile users, and find the optimal policy. The transition structure of this MDP is not known in general. To alleviate this issue, we propose a learning algorithm based on single timescale stochastic approximation. Also, noting that the unconstrained MDP can be used to solve a constrained problem, we provide a learning algorithm based on multi-timescale stochastic approximation. The results are extended to address the issue of fair bandwidth sharing between the two classes of users. Numerical results demonstrate performance improvement by our scheme, and also the trade-off between performance gain and fairness.

Variance-constrained actor-critic algorithms for discounted and average reward MDPs

In many sequential decision-making problems we may want to manage risk by minimizing some measure of variability in rewards in addition to maximizing a standard criterion. Variance related risk measures are among the most common risk-sensitive criteria in finance and operations research. However, optimizing many such criteria is known to be a hard problem. In this paper, we consider both discounted and average reward Markov decision processes. For each formulation, we first define a measure of variability for a policy, which in turn gives us a set of risk-sensitive criteria to optimize. For each of these criteria, we derive a formula for computing its gradient. We then devise actor-critic algorithms that operate on three timescales--a TD critic on the fastest timescale, a policy gradient (actor) on the intermediate timescale, and a dual ascent for Lagrange multipliers on the slowest timescale. In the discounted setting, we point out the difficulty in estimating the gradient of the variance of the return and incorporate simultaneous perturbation approaches to alleviate this. The average setting, on the other hand, allows for an actor update using compatible features to estimate the gradient of the variance. We establish the convergence of our algorithms to locally risk-sensitive optimal policies. Finally, we demonstrate the usefulness of our algorithms in a traffic signal control application.

Mode Selection and Resource Allocation in Device-to-Device Communications With User Arrivals and Departures

The pervasive increasing mobile devices and explosively increasing data traffic pose imminent challenges on wireless network design. Device-to-device (D2D) communication is envisioned to play a key role in the fifth generation cellular networks to efficiently support much larger and more diverse set of devices. This paper investigates the mode selection and resource allocation for D2D communications with dynamic user arrivals and departures. We formulate the optimal resource control problem to minimize the average energy consumption of flow transmission into an infinite horizon average reward Markov decision process. In order to deal with the well-known curse of dimensionality problem and facilitate distributed implementation, we approximate the mode selection $Q$ -factor by the sum of per-queue mode selection $Q$ -factors. Moreover, we apply distributive stochastic online learning to estimate the per-queue $Q$ -factors. Simulation results show that the proposed approach outperforms various existing baseline algorithms.

Delay-Aware Two-Hop Cooperative Relay Communications via Approximate MDP and Stochastic Learning

In this paper, a low-complexity delay-aware cross-layer scheduling algorithm for two-hop relay communication systems is proposed. The complex interactions of the queues at the source node and the M relay nodes (RSs) are modeled as an infinite horizon average reward Markov decision process (MDP), whose state space involves the joint queue state information (QSI) of the queues at the source node and the M RSs as well as the joint channel state information (CSI) of all S-R and R-D links. To address the curse of dimensionality, an equivalent MDP formulation is first proposed, where the system state depends only on global QSI. Furthermore, using approximate MDP and stochastic learning, an auction-based distributed online learning algorithm is derived, where each node iteratively estimates a per-node value function based on real-time observations of the local CSI and local QSI as well as signaling between relays. The combined distributed learning converges almost surely to a global optimal solution for large arrivals. Finally, it is showed by simulations that the proposed scheme achieves significant gain compared with various baselines such as the conventional CSIT-only control and the throughput optimal control (in stability sense).

Delay-Optimal Distributed Scheduling in Multi-User Multi-Relay Cellular Wireless Networks

We propose a novel scheme for delay-optimal scheduling in multi-user multi-relay cellular wireless networks. The cell area is divided into several sectors, each serviced by an individual relay station (RS). In order to have simultaneous transmissions by the users in neighbouring sectors, we assume that users of each individual sector use separate set of orthogonal channels to communicate with the RS and the base station (BS). Moreover, a separate orthogonal channel is shared among relays for transmission to the BS. For uplink communication, users are allowed to choose between two modes of transmission, namely, direct transmission mode and relayed transmission mode through a simple transmission mode selection algorithm. Users are allocated fractions of the time-slot for the first phase of transmission (from the users to the BS and the RSs) in a time-division multiple access (TDMA) fashion. For the second phase of transmission (from the RSs to the BS), each RS is allocated a fraction of the time-slot. We model the problem of end-to-end (e2e) delay-optimal scheduling as an infinite-horizon average reward Markov decision process (MDP) for users and relays in two separate stages. An online learning approach is then employed to solve the problem in a distributed manner for both users and relays in each phase of transmission. The proposed online stochastic learning solution converges to the optimal solution almost surely (with probability 1) under some realistic conditions. Simulation results show that the proposed approach outperforms the conventional scheduling schemes.

Stochastic Dominance-Constrained Markov Decision Processes

We are interested in risk constraints for infinite horizon discrete time Markov decision processes (MDPs). Starting with average reward MDPs, we show that increasing concave stochastic dominance constraints on the empirical distribution of reward lead to linear constraints on occupation measures. An optimal policy for the resulting class of dominance-constrained MDPs is obtained by solving a linear program. We compute the dual of this linear program to obtain average dynamic programming optimality equations that reflect the dominance constraint. In particular, a new pricing term appears in the optimality equations corresponding to the dominance constraint. We show that many types of stochastic orders can be used in place of the increasing concave stochastic order. We also carry out a parallel development for discounted reward MDPs with stochastic dominance constraints. A portfolio optimization example is used to motivate the paper.

Adaptive aggregation for reinforcement learning in average reward Markov decision processes

We present an algorithm which aggregates online when learning to behave optimally in an average reward Markov decision process. The algorithm is based on the reinforcement learning algorithm UCRL and uses confidence intervals for aggregating the state space. We derive bounds on the regret our algorithm suffers with respect to an optimal policy. These bounds are only slightly worse than the original bounds for UCRL.

Policy Iteration for Continuous-Time Average Reward Markov Decision Processes in Polish Spaces

We study thepolicy iteration algorithm(PIA) for continuous-time jump Markov decision processes in general state and action spaces. The corresponding transition rates are allowed to beunbounded, and the reward rates may haveneither upper nor lower bounds. The criterion that we are concerned with isexpected average reward. We propose a set of conditions under which we first establish the average reward optimality equation and present the PIA. Then under twoslightlydifferent sets of conditions we show that the PIA yields the optimal (maximum) reward, an average optimal stationary policy, and a solution to the average reward optimality equation.

Fuzzy optimality relation for perceptive MDPs—the average case

In this paper, the fuzzy perceptive model for average reward Markov decision processes is defined and a method of computing the corresponding fuzzy perceptive values is proposed. Under the minorization condition for fuzzy perceptive transition matrices, it is characterized by the optimal average expected reward, called the average perceptive value, using a fuzzy optimality relation. Also, we give a simple numerical example.

Achieving Target State-Action Frequencies in Multichain Average-Reward Markov Decision Processes

In this paper we address a basic problem that arises naturally in average-reward Markov decision processes with constraints and/or nonstandard payoff criteria: Given a feasible state-action frequency vector (“the target”), construct a policy whose state-action frequencies match those of the target vector. While it is well known that the solution to this problem cannot, in general, be found in the space of stationary randomized policies, we construct a solution that has “ultimately stationary” structure: It consists of two stationary policies where the first one is used initially, and then the switch to the second one is made at a certain random switching time. The computational effort required to construct this solution is minimal. We also show that our problem can always be solved by a stationary policy if the original MDP is “extended” by adding certain states and actions. The solution in the original MDP is obtained by mapping the solution in the extended MDP back to the original process.

The LP approach in average reward MDPs with multiple cost constraints: The countable state case

The objective of this paper is to study the linear programming for countable state average reward Markov decision processes with multiple cost constraints. A corresponding linear programming and its dual are formulated and their solvability and absence of a duality gap are proved under some reasonable conditions. Also, the ergodic assumption derives the existence of a constrained optimal stationary policy.

The policy iteration algorithm for average reward Markov decision processes with general state space

The average cost optimal control problem is addressed for Markov decision processes with unbounded cost. It is found that the policy iteration algorithm generates a sequence of policies which are c-regular, where c is the cost function under consideration. This result only requires the existence of an initial c-regular policy and an irreducibility condition on the state space. Furthermore, under these conditions the sequence of relative value functions generated by the algorithm is bounded from below and "nearly" decreasing, from which it follows that the algorithm is always convergent. Under further conditions, it is shown that the algorithm does compute a solution to the optimality equations and hence an optimal average cost policy. These results provide elementary criteria for the existence of optimal policies for Markov decision processes with unbounded cost and recover known results for the standard linear-quadratic-Gaussian problem. In particular, in the control of multiclass queueing networks, it is found that there is a close connection between optimization of the network and optimal control of a far simpler fluid network model.