Relative Value Iteration Research Articles

Robust Average-Reward Markov Decision Processes

In robust Markov decision processes (MDPs), the uncertainty in the transition kernel is addressed by finding a policy that optimizes the worst-case performance over an uncertainty set of MDPs. While much of the literature has focused on discounted MDPs, robust average-reward MDPs remain largely unexplored. In this paper, we focus on robust average-reward MDPs, where the goal is to find a policy that optimizes the worst-case average reward over an uncertainty set. We first take an approach that approximates average-reward MDPs using discounted MDPs. We prove that the robust discounted value function converges to the robust average-reward as the discount factor goes to 1, and moreover when it is large, any optimal policy of the robust discounted MDP is also an optimal policy of the robust average-reward. We further design a robust dynamic programming approach, and theoretically characterize its convergence to the optimum. Then, we investigate robust average-reward MDPs directly without using discounted MDPs as an intermediate step. We derive the robust Bellman equation for robust average-reward MDPs, prove that the optimal policy can be derived from its solution, and further design a robust relative value iteration algorithm that provably finds its solution, or equivalently, the optimal robust policy.

A Reinforcement Learning Approach for Optimizing the Age-of-Computing-Enabled IoT

Age of Information (AoI) is a newly rising metric for measuring the freshness of information. In this article, we consider a multidevice computing-enabled Internet of Things (IoT) system with a common destination, in which the status update sampled by the device can be offloaded directly to the destination for computing or computed by the device and then delivered to the destination, and jointly design offloading and scheduling policies to minimize the average weighted sum of AoI and energy consumption. The challenge lies in computing mode selection and its strong coupling with scheduling decisions. To address this issue, we formulate the optimization problem as a bilevel discrete-time Markov decision process (MDP) and approximate the optimal solution by relative value iteration. Furthermore, the threshold structure of the MDP policy is shown. However, with the expansion of the system scale, the MDP policy will suffer from the curse of dimensionality. In light of this, we develop a learning-based algorithm based on emerging deep reinforcement learning (DRL) to reduce the dimensionality of state space and utilize a late experience storage method to train two heterogeneous artificial neural networks (ANNs) synchronously during the training process. Simulation results show the structure of the MDP policy and verify the performance of the DRL policy is near-optimal.

Optimal Sampling and Scheduling for Timely Status Updates in Multi-Source Networks

We consider a joint sampling and scheduling problem for optimizing data freshness in multi-source systems. Data freshness is measured by a non-decreasing penalty function of age of information, where all sources have the same age-penalty function. Sources take turns to generate update packets, and forward them to their destinations one-by-one through a shared channel with random delay. There is a scheduler, that chooses the update order of the sources, and a sampler, that determines when a source should generate a new packet in its turn. We aim to find the optimal scheduler-sampler pairs that minimize the total-average age-penalty at delivery times (Ta-APD) and the total-average age-penalty (Ta-AP). We prove that the Maximum Age First (MAF) scheduler and the zero-wait sampler are jointly optimal for minimizing the Ta-APD. Meanwhile, the MAF scheduler and a relative value iteration with reduced complexity (RVI-RC) sampler are jointly optimal for minimizing the Ta-AP. The RVI-RC sampler is based on a relative value iteration algorithm whose complexity is reduced by exploiting a threshold property in the optimal sampler. Finally, a low-complexity threshold-type sampler is devised via an approximate analysis of Bellman's equation. This threshold-type sampler reduces to a simple water-filling sampler for a linear age-penalty function.

User association and power allocation for UAV-assisted networks: A distributed reinforcement learning approach

Unmanned aerial vehicles (UAVs) can be employed as aerial base stations (BSs) due to their high mobility and flexible deployment. This paper focuses on a UAV-assisted wireless network, where users can be scheduled to get access to either an aerial BS or a terrestrial BS for uplink transmission. In contrast to state-of-the-art designs focusing on the instantaneous cost of the network, this paper aims at minimizing the long-term average transmit power consumed by the users by dynamically optimizing user association and power allocation in each time slot. Such a joint user association scheduling and power allocation problem can be formulated as a Markov decision process (MDP). Unfortunately, solving such an MDP problem with the conventional relative value iteration (RVI) can suffer from the curses of dimensionality, in the presence of a large number of users. As a countermeasure, we propose a distributed RVI algorithm to reduce the dimension of the MDP problem, such that the original problem can be decoupled into multiple solvable small-scale MDP problems. Simulation results reveal that the proposed algorithm can yield lower long-term average transmit power consumption than both the conventional RVI algorithm and a baseline algorithm with myopic policies.

Optimal Energy Cooperation Policy in Fusion Center-Based Sustainable Wireless Sensor Networks

In this paper, in order to maximize a long-term average system transmission rate, we propose an optimal energy cooperation policy for the sustainable wireless sensor networks where each pair of nodes can harvest energy and transfer energy in the pair. We formulate this optimization problem into a Markov decision process (MDP). We define a node pairing metric-relative energy distance ratio (REDR)-to design a node pairing algorithm which can decouple the original MDP into a multiple low dimension MDP. Then, value iteration and relative value iteration algorithms are brought to find optimal energy cooperation policies over finite and infinite horizons for each pair of nodes respectively. Simulation results show that the system transmission rate of the proposed policy with REDR based pairing algorithm outperforms the non-cooperation policies as well as the policies with straightforward pairing algorithms.

Low Complexity Online Radio Access Technology Selection Algorithm in LTE-WiFi HetNet

In an offload-capable Long Term Evolution (LTE)- Wireless Fidelity (WiFi) Heterogeneous Network (HetNet), we consider the problem of maximization of the total system throughput under voice user blocking probability constraint. The optimal policy is threshold in nature. However, computation of optimal policy requires the knowledge of the statistics of system dynamics, viz., arrival processes of voice and data users, which may be difficult to obtain in reality. Motivated by the Post-Decision State (PDS) framework to learn the optimal policy under unknown statistics of system dynamics, we propose, in this paper, an online Radio Access Technology (RAT) selection algorithm using Relative Value Iteration Algorithm (RVIA). However, the convergence speed of this algorithm can be further improved if the underlying threshold structure of the optimal policy can be exploited. To this end, we propose a novel structure-aware online RAT selection algorithm which reduces the feasible policy space, thereby offering lesser storage and computational complexity and faster convergence. This algorithm provides a novel framework for designing online learning algorithms for other problems and hence is of independent interest. We prove that both the algorithms converge to the optimal policy. Simulation results demonstrate that the proposed algorithms converge faster than a traditional scheme. Also, the proposed schemes perform better than other benchmark algorithms under realistic network scenarios.

On the relative value iteration with a risk-sensitive criterion

A multiplicative relative value iteration algorithm for solving the dynamic programming equation for the risk-sensitive control problem is studied for discrete time controlled Markov chains with a compact Polish state space, and controlled diffusions in on the whole Euclidean space. The main result is a proof of convergence to the desired limit in each case.

Open Problem—Convergence and Asymptotic Optimality of the Relative Value Iteration in Ergodic Control

The relative value iteration scheme (RVI) for Markov decision processes (MDP) dates back to White (1963) , a seminal work, which introduced an algorithm for solving the ergodic dynamic programming equation for the finite state, finite action case. Its ramifications have given rise to popular learning algorithms (Q-learning). More recently, this algorithm gained prominence because of its implications for model predictive control (MPC). For stochastic control problems on an infinite time horizon, especially for problems that seek to optimize the average performance (ergodic control), obtaining the optimal policy in explicit form is only possible for a few classes of well-structured models. What is often used in practice is a heuristic method called the rolling horizon, or receding horizon, or MPC. This works as follows: one solves the finite horizon problem for a given number of steps N, or for an interval [0,T] in the case of a continuous time problem. The result is a nonstationary Markov policy, which is optimal for the finite horizon problem. We fix the initial action (this is the action determined at the Nth step of the value iteration (VI) algorithm) and apply it as a stationary Markov control. We refer to this Markov control as the rolling horizon control. This of course depends on the length of the horizon N. One expects that for well-structured problems, if N is sufficiently large, then the rolling horizon control is near optimal. Of course, this is a heuristic. The rolling horizon control might not even be stable. For a good discussion on this problem, we refer the reader to Della Vecchia et al. (2012) . Obtaining such solutions is further complicated by the fact that the value of the ergodic cost required in the successive iteration scheme is not known. This is the reason for the RVI.

CMDP-based intelligent transmission for wireless body area network in remote health monitoring

Remote health monitoring is one kind of E-health service, which transfer the users’ physiological data to the medical data center for analysis or diagnosis. Wireless body area network (WBAN) is a promising technology to achieve physiological information acquiring and delivering and thus has been widely adopted in remote health-monitoring applications. For WBAN, energy consumption is the major concern which has been addressed in many researches. Different from existing works, this work studies a joint scheduling and admission control problem with objective of optimizing the energy efficiency of both intra- and beyond-WBAN link. The problem is formulated as constrained Markov decision processes, and the relative value iteration and Lagrange multiplier approach are used to derive the optimal intelligent algorithm. Simulation results show the proposed algorithm is capable of, in comparison with greedy scheme, achieving nearly 100% throughput improvement in various power consumption budgets. Moreover, the proposed algorithm can achieve up to 5.5× power consumption saving for sensor node in comparison with other scheduling algorithms.

Strong convergence and dynamic economic models

Morton and Wecker (1977) stated that the value iteration algorithm solves a dynamic program's policy function faster than its value function when the limiting Markov chain is ergodic. I show that their proof is incomplete, and provide a new proof of this classic result. I use this result to accelerate the estimation of Markov decision processes and the solution of Markov perfect equilibria. Markov decision process Markov perfect equilibrium strong convergence relative value iteration dynamic discrete choice nested fixed point nested pseudo‐likelihood C01 C13 C15 C61 C63 C65

Markov Decision Processes with Exogenous Variables

I present two algorithms for solving dynamic programs with exogenous variables: endogenous value iteration and endogenous policy iteration. These algorithms are like relative value iteration and relative policy iteration, except they discard the variation in the value function due solely to the exogenous variables (this variation doesn't affect the policy function). My algorithms are always at least as fast as relative value iteration and relative policy iteration, and are faster when the endogenous variables converge to their stationary distributions faster than the exogenous variables.

Structural Estimation Using Parametric Mathematical Programming with Equilibrium Constraints and Homotopy Path Continuation

In this paper, we formulate the likelihood function of structural models as a parametric optimization problem, where the model equations enter as constraints, forming a mathematical program with equilibrium constraints (Su and Judd, 2012). We trace the solution to its first-order conditions in dependence on a controlled parameter using homotopy continuation, delivering a relation from the controlled parameter to the corresponding maximum likelihood estimates and their confidence intervals. This enables us to estimate models with identification issues, multiplicity of equilibria, etc. As applications, we first trace the parameter estimates of the bus engine replacement model of Rust (1987), a dynamic discrete choice model, in dependence of the discount factor β. Using relative value iteration, we find that β is well identified and statistically significantly larger than 1. Second, for a simple static binary choice model, we demonstrate how the effects of multiplicity of equilibria and a lack of identification can be mitigated by the tracing method.

A Continuous-Time Markov decision process-based resource allocation scheme in vehicular cloud for mobile video services

The development of vehicular network technologies boosts the wide deployment of mobile video applications with high requirements of Quality of Experience (QoE) in the Fifth-Generation (5G) era. However, the limitation of computing capabilities of intelligent vehicles makes it difficult to meet the QoE demands. The offloading technique that is put forward in vehicular cloud can extend such limitations largely by offloading video processing tasks to cloud or other vehicles. On the other hand, the emerging mobile social networks create new patterns for mobile applications to serve people on the basis of social relations. The mobile video offloading services can also be influenced by social relations of users inside a cloudlet. Therefore, in this paper we study the impact of social graphs on mobile video offloading services and propose a Continuous-Time Markov Decision Process (CTMDP) based resource allocation scheme considering social graphs as constraints. By using relative value iteration algorithm, an optimal policy can be obtained, which aims at maximizing the average system rewards. Simulation results show that our CTMDP based scheme achieves an enhanced performance against Greedy benchmark under different metrics.

Optimal Dynamic Multicast Scheduling for Cache-Enabled Content-Centric Wireless Networks

Caching and multicasting at base stations are two promising approaches to support massive content delivery over wireless networks. However, existing scheduling designs do not make full use of the advantages of the two approaches. In this paper, we consider the optimal dynamic multicast scheduling to jointly minimize the average delay, power, and fetching costs for cache-enabled content-centric wireless networks. We formulate this stochastic optimization problem as an infinite horizon average cost Markov decision process (MDP). It is well-known to be a difficult problem due to the curse of dimensionality, and there generally only exist numerical solutions. By using relative value iteration algorithm and the special structures of the request queue dynamics, we analyze the properties of the value function and the state-action cost function of the MDP for both the uniform and nonuniform channel cases. Based on these properties, we show that the optimal policy, which is adaptive to the request queue state, has a switch structure in the uniform case and a partial switch structure in the nonuniform case. Moreover, in the uniform case with two contents, we show that the switch curve is monotonically non-decreasing. Then, by exploiting these structural properties of the optimal policy, we propose two low-complexity optimal algorithms. Motivated by the switch structures of the optimal policy, to further reduce the complexity, we also propose a low-complexity suboptimal policy, which possesses similar structural properties to the optimal policy, and develop a low-complexity algorithm to compute this policy.

Markov Decision Processes with Exogenous Variables

I present two algorithms for solving dynamic programs with exogenous variables: endogenous value iteration and endogenous policy iteration. These algorithms are always at least as fast as relative value iteration and relative policy iteration, and they are faster when the endogenous variables converge to their stationary distributions sooner than the exogenous variables. This paper was accepted by Yinyu Ye, optimization.

A Correction to “A Relative Value Iteration Algorithm for Nondegenerate Controlled Diffusions''

In A Relative Value Iteration Algorithm for Nondegenerate Controlled Diffusions, [SIAM J. Control Optim., 50 (2012), pp. 1886--1902], convergence of the relative value iteration for the ergodic control problem for a nondegenerate diffusion controlled through its drift was established, under the assumption of geometric ergodicity, using two methods: (a) the theory of monotone dynamical systems and (b) the theory of reverse martingales. However, in the proof using (a) it is wrongly claimed that the semiflow is strong order preserving. In this note, we provide a simple generic proof and also comment on how to relax the uniform geometric ergodicity hypothesis.

SMDP-Based Radio Resource Allocation Scheme in Software-Defined Internet of Things Networks

With rapid development of the Internet of Things (IoT), various machine-to-machine communications technologies have emerged in recent years to provide ubiquitous wireless connections for a massive number of IoT devices. This poses significant challenges to network control and management of large-scale IoT networks. Software-defined networking (SDN) is considered a promising technology to streamline network management due to dynamic reconfigurable network elements. Thus, the integration of SDN and IoT provides a potentially feasible solution to strengthening management and control capabilities of the IoT network. Benefit from the SDN technology, resource utilization in the IoT network can be further enhanced. In this paper, we first propose a software-defined network architecture for IoT. Then, the resource allocation problem in the proposed SDN-based IoT network is investigated. The optimal problem of maximizing the expected average rewards of the network is formulated as a semi-Markov decision process (SMDP). The optimal solution is obtained through solving the SMDP problem using a relative value iteration algorithm. Simulation results demonstrate that the proposed resource allocation scheme is able to improve the system rewards compared with other comparative resource allocation schemes.

Stochastic Content-Centric Multicast Scheduling for Cache-Enabled Heterogeneous Cellular Networks

Caching at small base stations (SBSs) has demonstrated significant benefits in alleviating the backhaul requirement in heterogeneous cellular networks (HetNets). While many existing works focus on what contents to cache at each SBS, an equally important problem is what contents to deliver so as to satisfy dynamic user demands given the cache status. In this paper, we study optimal content delivery in cache-enabled HetNets by taking into account the inherent multicast capability of wireless medium. We consider stochastic content multicast scheduling to jointly minimize the average network delay and power costs under a multiple access constraint. We establish a content-centric request queue model and formulate this stochastic optimization problem as an infinite horizon average cost Markov decision process (MDP). By using \emph{relative value iteration} and special properties of the request queue dynamics, we characterize some properties of the value function of the MDP. Based on these properties, we show that the optimal multicast scheduling policy is of threshold type. Then, we propose a structure-aware optimal algorithm to obtain the optimal policy. We also propose a low-complexity suboptimal policy, which possesses similar structural properties to the optimal policy, and develop a low-complexity algorithm to obtain this policy.

Stochastic Throughput Optimization for Two-Hop Systems With Finite Relay Buffers

Optimal queueing control of multi-hop networks remains a challenging problem even in the simplest scenarios. In this paper, we consider a two-hop half-duplex relaying system with random channel connectivity. The relay is equipped with a finite buffer. We focus on stochastic link selection and transmission rate control to maximize the average system throughput subject to a half-duplex constraint. We formulate this stochastic optimization problem as an infinite horizon average cost Markov decision process (MDP), which is well-known to be a difficult problem. By using sample-path analysis and exploiting the specific problem structure, we first obtain an \emph{equivalent Bellman equation} with reduced state and action spaces. By using \emph{relative value iteration algorithm}, we analyze the properties of the value function of the MDP. Then, we show that the optimal policy has a threshold-based structure by characterizing the \emph{supermodularity} in the optimal control. Based on the threshold-based structure and Markov chain theory, we further simplify the original complex stochastic optimization problem to a static optimization problem over a small discrete feasible set and propose a low-complexity algorithm to solve the simplified static optimization problem by making use of its special structure. Furthermore, we obtain the closed-form optimal threshold for the symmetric case. The analytical results obtained in this paper also provide design insights for two-hop relaying systems with multiple relays equipped with finite relay buffers.

Generalized Queue-Aware Resource Management and Scheduling for Wireless Communications

The general problem of a queue-aware radio resource management and scheduling design is investigated for wireless communications under quasi-static fading channel conditions. Based on an analysis of the source buffer queuing system, the problem is formulated as a constrained nonlinear discrete programming problem. The state transition matrix of the queuing system determined by the queue-aware scheduler is shown to have a highly dynamic structure, so that the conventional matrix analysis and optimization tools are not applicable. By reformulating the problem into a nonlinear integer programming problem on an integer convex set, a direct search approach is considered. Two types of search algorithms, gradient based and gradient-free, are investigated. An integer steepest-descent search with a sub-sequential interval search algorithm and a constrained discrete Rosenbrock search (CDRS) algorithm is proposed to solve the nonlinear integer problem. Both algorithms are shown to have low complexity and good convergence. The numerical results for a single user resource allocation are presented, which show that both algorithms outperform equal partitioning and random partitioning queue-aware scheduling. The dynamic programming (DP) solution given by the relative value iteration algorithm, which provides the true optima but has high complexity, is used as a benchmark. In the majority of the numerical examples, the performance of the CDRS algorithm is almost identical to that of the DP approach in terms of both the average queue length minimization and the average packet blocking plus packet retransmission minimization, but it is less complex, and thus has better scalability.