Average Reward Markov Decision Research Articles

Regret Analysis of Policy Gradient Algorithm for Infinite Horizon Average Reward Markov Decision Processes

In this paper, we consider an infinite horizon average reward Markov Decision Process (MDP). Distinguishing itself from existing works within this context, our approach harnesses the power of the general policy gradient-based algorithm, liberating it from the constraints of assuming a linear MDP structure. We propose a vanilla policy gradient-based algorithm and show its global convergence property. We then prove that the proposed algorithm has O(T^3/4) regret. Remarkably, this paper marks a pioneering effort by presenting the first exploration into regret bound computation for the general parameterized policy gradient algorithm in the context of average reward scenarios.

Relative Q-learning for Average-Reward Markov Decision Processes with Continuous States

Relative Q-learning for Average-Reward Markov Decision Processes with Continuous States

BATCH POLICY LEARNING IN AVERAGE REWARD MARKOV DECISION PROCESSES.

We consider the batch (off-line) policy learning problem in the infinite horizon Markov Decision Process. Motivated by mobile health applications, we focus on learning a policy that maximizes the long-term average reward. We propose a doubly robust estimator for the average reward and show that it achieves semiparametric efficiency. Further we develop an optimization algorithm to compute the optimal policy in a parameterized stochastic policy class. The performance of the estimated policy is measured by the difference between the optimal average reward in the policy class and the average reward of the estimated policy and we establish a finite-sample regret guarantee. The performance of the method is illustrated by simulation studies and an analysis of a mobile health study promoting physical activity.

Energy management for age of information control in solar-powered IoT end devices

In this paper, we propose several harvesting-aware energy management policies for solar-powered wireless IoT end devices that asynchronously send status updates for their surrounding environments to a network gateway device. For such devices, we aim at minimizing the average age of information (AoI) metric which has recently been investigated extensively for status update systems. The proposed energy management policies are obtained using discrete-time Markov chain-based modeling of the stochastic intra-day variations of the solar energy harvesting process in conjunction with the average reward Markov decision process formulation. With this approach, energy management policies are constructed by using the time of day and month of year information in addition to the instantaneous values of the age of information and the battery level. The effectiveness of the proposed energy management policies in terms of their capability to reduce the average AoI as well as improving upon the tail of the AoI distribution, is validated with empirical data for a wide range of system parameters.

Relative Q-Learning for Average-Reward Markov Decision Processes with Continuous States

Relative Q-Learning for Average-Reward Markov Decision Processes with Continuous States

Incremental Improvements of Heuristic Policies for Average-Reward Markov Decision Processes

Abstract Within the realm of Discrete Event Systems (DES) theory, the problem of performance optimization for many applications can be modeled as an infinite-horizon, average-reward Markov Decision Process (MDP) with a finite state space. In principle, these MDPs can be solved by various well-developed methods like value iteration, policy iteration and linear programming. But in reality, the tractability of these methods in the context of the aforementioned applications is compromised by the explosive size of the underlying state spaces, a problem that is known as “the curse of dimensionality”. Hence, the corresponding performance optimization problems are frequently addressed by heuristic control policies. The considered work uses results from (i) the sensitivity analysis of Markov reward processes and (ii) the ranking & selection theory in statistics in order to develop a methodology for assessing the optimality of isolated decisions in the context of any well-defined heuristic control policy for the aforementioned MDPs. It also determines an improved decision when the current one is found to be suboptimal. Hence, when embedded in an iterative scheme, this methodology can support the incremental enhancement of the original heuristic policy in a way that controls, both, the computational and also the representational complexity of the new policy. Finally, an additional important feature of the presented methodology is that it can be executed either in an “off-line” mode, using a simulation of the dynamics of the underlying DES, or in an “on-line” mode, based on the sample path that is defined by the real-time dynamics of the controlled system.

Randomized Linear Programming Solves the Markov Decision Problem in Nearly Linear (Sometimes Sublinear) Time

We propose a novel randomized linear programming algorithm for approximating the optimal policy of the discounted-reward and average-reward Markov decision problems. By leveraging the value–policy duality, the algorithm adaptively samples state–action–state transitions and makes exponentiated primal–dual updates. We show that it finds an ɛ-optimal policy using nearly linear runtime in the worst case for a fixed value of the discount factor. When the Markov decision process is ergodic and specified in some special data formats, for fixed values of certain ergodicity parameters, the algorithm finds an ɛ-optimal policy using sample size and time linear in the total number of state–action pairs, which is sublinear in the input size. These results provide a new venue and complexity benchmarks for solving stochastic dynamic programs.

Balanced Dynamic Planning in Green Heterogeneous Cellular Networks

Dynamic planning has been used by network operators for energy saving by switching ON–OFF base stations (BSs) according to the traffic load condition while providing quality-of-service (QoS) guarantee for mobile users. In the literature, research efforts focus mainly on satisfying the QoS of downlink mobile users. However, the impact of dynamic planning on the QoS of uplink mobile users is overlooked. A dynamic planning approach that relies only on the downlink performance of mobile users to determine the BS switching decision can result in deactivating nearby small cells and associating uplink mobile users to a faraway macro BS. Consequently, uplink mobile users may suffer from QoS degradation due to the longer transmission distance. Hence, while saving energy for the network operators and satisfying the downlink QoS, the uplink QoS could be violated. In this paper, we propose a balanced dynamic planning approach that accounts for QoS requirements both in the uplink and downlink to specify the BS switching decision. The switching decision is formulated as a two-timescale average reward Markov decision process with finite horizon. Due to computational complexity, sub-optimal algorithms are proposed. Simulation results demonstrate the trade-off between energy saving and QoS guarantee.

Location Aware Opportunistic Bandwidth Sharing between Static and Mobile Users with Stochastic Learning in Cellular Networks

We consider location-dependent opportunistic bandwidth sharing between static and mobile downlink users in a cellular network. Each cell has some fixed number of static users. Mobile users enter the cell, move inside the cell for some time and then leave the cell. In order to provide higher data rate to mobile users, we propose to provide higher bandwidth to the mobile users at favourable times and locations, and provide higher bandwidth to the static users in other times. We formulate the problem as a long run average reward Markov decision process (MDP) where the per-step reward is a linear combination of instantaneous data volumes received by static and mobile users, and find the optimal policy. The transition structure of this MDP is not known in general. To alleviate this issue, we propose a learning algorithm based on single timescale stochastic approximation. Also, noting that the unconstrained MDP can be used to solve a constrained problem, we provide a learning algorithm based on multi-timescale stochastic approximation. The results are extended to address the issue of fair bandwidth sharing between the two classes of users. Numerical results demonstrate performance improvement by our scheme, and also the trade-off between performance gain and fairness.

Variance-constrained actor-critic algorithms for discounted and average reward MDPs

In many sequential decision-making problems we may want to manage risk by minimizing some measure of variability in rewards in addition to maximizing a standard criterion. Variance related risk measures are among the most common risk-sensitive criteria in finance and operations research. However, optimizing many such criteria is known to be a hard problem. In this paper, we consider both discounted and average reward Markov decision processes. For each formulation, we first define a measure of variability for a policy, which in turn gives us a set of risk-sensitive criteria to optimize. For each of these criteria, we derive a formula for computing its gradient. We then devise actor-critic algorithms that operate on three timescales--a TD critic on the fastest timescale, a policy gradient (actor) on the intermediate timescale, and a dual ascent for Lagrange multipliers on the slowest timescale. In the discounted setting, we point out the difficulty in estimating the gradient of the variance of the return and incorporate simultaneous perturbation approaches to alleviate this. The average setting, on the other hand, allows for an actor update using compatible features to estimate the gradient of the variance. We establish the convergence of our algorithms to locally risk-sensitive optimal policies. Finally, we demonstrate the usefulness of our algorithms in a traffic signal control application.

Mode Selection and Resource Allocation in Device-to-Device Communications With User Arrivals and Departures

The pervasive increasing mobile devices and explosively increasing data traffic pose imminent challenges on wireless network design. Device-to-device (D2D) communication is envisioned to play a key role in the fifth generation cellular networks to efficiently support much larger and more diverse set of devices. This paper investigates the mode selection and resource allocation for D2D communications with dynamic user arrivals and departures. We formulate the optimal resource control problem to minimize the average energy consumption of flow transmission into an infinite horizon average reward Markov decision process. In order to deal with the well-known curse of dimensionality problem and facilitate distributed implementation, we approximate the mode selection $Q$ -factor by the sum of per-queue mode selection $Q$ -factors. Moreover, we apply distributive stochastic online learning to estimate the per-queue $Q$ -factors. Simulation results show that the proposed approach outperforms various existing baseline algorithms.

A model for equilibrium in some service-provider user-set interactions

We propose a model for interaction between a user-set (market) and a service-provider (firm) when the offered demand is sensitive to the offered Quality of Service which in turn depends on the offered demand. We show that, under fairly mild conditions, an equilibrium point or an equilibrium set exists either when the service-provider bases its operational decisions as per a revenue optimal policy of an average reward Markov decision model or when it uses a discounted reward Markov decision model. We also consider the cases when the firm has to meet functional constraints on consumption of resources for its operations. If an equilibrium set exists, then the interaction can suitably be interpreted as one explanation of the business cycles phenomena. We illustrate these and related issues via a model for vehicle relocation system which may be of independent interest.

Delay-Aware Two-Hop Cooperative Relay Communications via Approximate MDP and Stochastic Learning

In this paper, a low-complexity delay-aware cross-layer scheduling algorithm for two-hop relay communication systems is proposed. The complex interactions of the queues at the source node and the M relay nodes (RSs) are modeled as an infinite horizon average reward Markov decision process (MDP), whose state space involves the joint queue state information (QSI) of the queues at the source node and the M RSs as well as the joint channel state information (CSI) of all S-R and R-D links. To address the curse of dimensionality, an equivalent MDP formulation is first proposed, where the system state depends only on global QSI. Furthermore, using approximate MDP and stochastic learning, an auction-based distributed online learning algorithm is derived, where each node iteratively estimates a per-node value function based on real-time observations of the local CSI and local QSI as well as signaling between relays. The combined distributed learning converges almost surely to a global optimal solution for large arrivals. Finally, it is showed by simulations that the proposed scheme achieves significant gain compared with various baselines such as the conventional CSIT-only control and the throughput optimal control (in stability sense).

Delay-Optimal Distributed Scheduling in Multi-User Multi-Relay Cellular Wireless Networks

We propose a novel scheme for delay-optimal scheduling in multi-user multi-relay cellular wireless networks. The cell area is divided into several sectors, each serviced by an individual relay station (RS). In order to have simultaneous transmissions by the users in neighbouring sectors, we assume that users of each individual sector use separate set of orthogonal channels to communicate with the RS and the base station (BS). Moreover, a separate orthogonal channel is shared among relays for transmission to the BS. For uplink communication, users are allowed to choose between two modes of transmission, namely, direct transmission mode and relayed transmission mode through a simple transmission mode selection algorithm. Users are allocated fractions of the time-slot for the first phase of transmission (from the users to the BS and the RSs) in a time-division multiple access (TDMA) fashion. For the second phase of transmission (from the RSs to the BS), each RS is allocated a fraction of the time-slot. We model the problem of end-to-end (e2e) delay-optimal scheduling as an infinite-horizon average reward Markov decision process (MDP) for users and relays in two separate stages. An online learning approach is then employed to solve the problem in a distributed manner for both users and relays in each phase of transmission. The proposed online stochastic learning solution converges to the optimal solution almost surely (with probability 1) under some realistic conditions. Simulation results show that the proposed approach outperforms the conventional scheduling schemes.

Adaptive aggregation for reinforcement learning in average reward Markov decision processes

We present an algorithm which aggregates online when learning to behave optimally in an average reward Markov decision process. The algorithm is based on the reinforcement learning algorithm UCRL and uses confidence intervals for aggregating the state space. We derive bounds on the regret our algorithm suffers with respect to an optimal policy. These bounds are only slightly worse than the original bounds for UCRL.

Distributive Stochastic Learning for Delay-Optimal OFDMA Power and Subband Allocation

In this paper, we consider the distributive queue-aware power and subband allocation design for a delay-optimal OFDMA uplink system with one base station, <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K</i> users and <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">N</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</sub> independent subbands. Each mobile has an uplink queue with heterogeneous packet arrivals and delay requirements. We model the problem as an infinite horizon average reward Markov decision problem (MDP) where the control actions are functions of the instantaneous channel state information (CSI) as well as the joint queue state information (QSI). To address the distributive requirement and the issue of exponential memory requirement and computational complexity, we approximate the subband allocation <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Q</i> -factor by the sum of the per-user subband allocation <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Q</i> -factor and derive a distributive online stochastic learning algorithm to estimate the per-user <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Q</i> -factor and the Lagrange multipliers (LM) simultaneously and determine the control actions using an auction mechanism. We show that under the proposed auction mechanism, the distributive online learning converges almost surely (with probability 1). For illustration, we apply the proposed distributive stochastic learning framework to an application example with exponential packet size distribution. We show that the delay-optimal power control has the <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">multilevel water-filling</i> structure where the CSI determines the instantaneous power allocation and the QSI determines the water-level. The proposed algorithm has linear signaling overhead and computational complexity <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">O</i> ( <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">KN</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</sub> ), which is desirable from an implementation perspective.

Delay-optimal power and subcarrier allocation for OFDMA systems via stochastic approximation

In this paper, we consider delay-optimal power and subcarrier allocation design for OFDMA systems with N <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</sub> subcarriers, K mobiles and one base station. There are K queues at the base station for the downlink traffic to the K mobiles with heterogeneous packet arrivals and delay requirements. We shall model the problem as a K-dimensional infinite horizon average reward Markov Decision Problem (MDP) where the control actions are assumed to be a function of the instantaneous Channel State Information (CSI) as well as the joint Queue State Information (QSI). We propose an online stochastic value iteration solution using stochastic approximation. The proposed power control algorithm, which is a function of both the CSI and the QSI, takes the form of multi-level water-filling. We prove that under two mild conditions in Theorem 1, the proposed solution converges to the optimal solution almost surely (with probability 1) and the proposed framework offers a possible solution to the general stochastic NUM problem. By exploiting the birth-death structure of the queue dynamics, we obtain a reduced complexity decomposed solution with linear O(KN <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</sub> ) complexity and O(K) memory requirement.

Policy Iteration for Continuous-Time Average Reward Markov Decision Processes in Polish Spaces

We study thepolicy iteration algorithm(PIA) for continuous-time jump Markov decision processes in general state and action spaces. The corresponding transition rates are allowed to beunbounded, and the reward rates may haveneither upper nor lower bounds. The criterion that we are concerned with isexpected average reward. We propose a set of conditions under which we first establish the average reward optimality equation and present the PIA. Then under twoslightlydifferent sets of conditions we show that the PIA yields the optimal (maximum) reward, an average optimal stationary policy, and a solution to the average reward optimality equation.

Fuzzy optimality relation for perceptive MDPs—the average case

In this paper, the fuzzy perceptive model for average reward Markov decision processes is defined and a method of computing the corresponding fuzzy perceptive values is proposed. Under the minorization condition for fuzzy perceptive transition matrices, it is characterized by the optimal average expected reward, called the average perceptive value, using a fuzzy optimality relation. Also, we give a simple numerical example.

A Reinforcement Learning Algorithm Based on Policy Iteration for Average Reward: Empirical Results with Yield Management and Convergence Analysis

We present a Reinforcement Learning (RL) algorithm based on policy iteration for solving average reward Markov and semi-Markov decision problems. In the literature on discounted reward RL, algorithms based on policy iteration and actor-critic algorithms have appeared. Our algorithm is an asynchronous, model-free algorithm (which can be used on large-scale problems) that hinges on the idea of computing the value function of a given policy and searching over policy space. In the applied operations research community, RL has been used to derive good solutions to problems previously considered intractable. Hence in this paper, we have tested the proposed algorithm on a commercially significant case study related to a real-world problem from the airline industry. It focuses on yield management, which has been hailed as the key factor for generating profits in the airline industry. In the experiments conducted, we use our algorithm with a nearest-neighbor approach to tackle a large state space. We also present a convergence analysis of the algorithm via an ordinary differential equation method.