Constrained Markov Decision Problem Research Articles

Deep reinforcement learning with reward shaping for tracking control and vibration suppression of flexible link manipulator

This paper puts forward a novel deep reinforcement learning control using deep deterministic policy gradient (DRLC-DDPG) framework to address the reference tracking and vibration suppression problem of rotary flexible link (RFL) manipulator. Specifically, this study attempts to address the continuous action space DRLC problem through DDPG algorithm and presents a Lyapunov function based reward shaping approach for guaranteed deep reinforcement learning (DRL) convergence and enhanced speed of training. The proposed approach synthesizes the hard and soft constraints of the flexible manipulator as a constrained Markov decision problem (MDP) and evaluates the performance of DRLC-DDPG framework through hardware in loop (HIL) testing to realize precise servo tracking and suppressed vibration of the flexible manipulator. For identifying the dynamical model of the RFL, an empirical Auto-Regressive eXogenous (ARX) model using the closed loop identification technique is built. Moreover, to extract the true states (servo angle and deflection angle) from the actual measurements, which typically have the influence of sensor noise, an adaptive Kalman filter (AKF) is augmented with the DRLC scheme. The experimental results of DRLC-DDPG scheme compared with those of the model predictive control (MPC) for several test cases reveal that the proposed scheme is superior to MPC both in terms of trajectory tracking and robustness against the external disturbances and model uncertainty.

Event-Triggered Communication Network With Limited-Bandwidth Constraint for Multi-Agent Reinforcement Learning.

Communicating agents with each other in a distributed manner and behaving as a group are essential in multi-agent reinforcement learning. However, real-world multi-agent systems suffer from restrictions on limited bandwidth communication. If the bandwidth is fully occupied, some agents are not able to send messages promptly to others, causing decision delay and impairing cooperative effects. Recent related work has started to address the problem but still fails in maximally reducing the consumption of communication resources. In this article, we propose an event-triggered communication network (ETCNet) to enhance communication efficiency in multi-agent systems by communicating only when necessary. For different task requirements, two paradigms of the ETCNet framework, event-triggered sending network (ETSNet) and event-triggered receiving network (ETRNet), are proposed for learning efficient sending and receiving protocols, respectively. Leveraging the information theory, the limited bandwidth is translated to the penalty threshold of an event-triggered strategy, which determines whether an agent at each step participates in communication or not. Then, the design of the event-triggered strategy is formulated as a constrained Markov decision problem and reinforcement learning finds the feasible and optimal communication protocol that satisfies the limited bandwidth constraint. Experiments on typical multi-agent tasks demonstrate that ETCNet outperforms other methods in reducing bandwidth occupancy and still preserves the cooperative performance of multi-agent systems at the most.

Detecting State Transitions of a Markov Source: Sampling Frequency and Age Trade-off

We consider a finite-state Discrete-Time Markov Chain (DTMC) source that can be sampled for detecting the events when the DTMC transits to a new state. Our goal is to study the trade-off between sampling frequency and staleness in detecting the events. We argue that, for the problem at hand, using Age of Information (AoI) for quantifying the staleness of a sample is conservative and therefore, study another freshness metric <i>age penalty</i>, which is defined as the time elapsed since the first transition out of the most recently observed state. We study two optimization problems: minimize average age penalty subject to an average sampling frequency constraint, and minimize average sampling frequency subject to an average age penalty constraint; both are Constrained Markov Decision Problems. We solve them using the Lagrangian MDP approach, where we also provide structural results that reduce the search space. Our numerical results demonstrate that the computed Markov policies not only outperform optimal periodic sampling policies, but also achieve sampling frequencies close to or lower than that of an optimal clairvoyant (non-causal) sampling policy, if a small age penalty is allowed.

On the Design of Temporal Compression Strategies for Energy Harvesting Sensor Networks

Recent advances in energy harvesting devices and low-power embedded systems are enabling energetically self-sustainable wireless sensing systems able to sense, process, and wirelessly transmit environmental data. In such systems, energy resources need to be judiciously allocated to processing and transmission tasks to guarantee high-fidelity reconstruction of the phenomenon under observation while keeping the system operational over extended periods of time. Within this context, this paper addresses the problem of designing efficient policies to control the task of lossy data compression for wireless transmission over fading channels in the presence of a stochastic energy input process and a replenishable energy buffer. As a first contribution, the transmission and energy dynamics of a sensor node implementing practical lossy compression methods are modeled as a constrained Markov decision problem (CMDP). Then, an algorithm is designed to derive optimal compression/transmission policies through a Lagrangian relaxation approach combined with a dichotomic search for the Lagrangian multiplier, while also obtaining theoretical results on the optimal policy structure. Furthermore, a thorough numerical evaluation of optimal and heuristic policies is conducted under different scenarios. Finally, the impact of practical operating conditions, including perfect versus delayed channel state information and power control, is evaluated.

Optimal Primary-Secondary User Cooperation Policies in Cognitive Radio Networks

In cognitive radio networks, secondary users (SUs) may cooperate with the primary user (PU) so that the success probability of PU transmissions are improved, while SUs obtain more transmission opportunities. However, SUs have limited power resources and, therefore, they have to take intelligent decisions on whether to cooperate or not and at which power level, to maximize their throughput. Cooperation policies in this framework require the solution of a constrained Markov decision problem with infinite state space. In our work, we restrict attention to the class of stationary policies that take randomized decisions of an SU activation and its transmit power in every time slot based only on spectrum sensing. Assuming infinitely backlogged SUs queues, the proposed class of policies is shown to achieve the maximum throughput for the SUs, while significantly enlarging the stability region of PU queue. The structure of the optimal policies remains the same even if the assumption of infinitely backlogged SU queues is relaxed. Furthermore, the model is extended for the case of imperfect channel sensing. Finally, a lightweight distributed protocol for the implementation of the proposed policies is presented, which is applicable to realistic scenarios.

Toward Optimal Admission Control and Resource Allocation for LTE-A Femtocell Uplink

The Third-Generation Partnership Project (3GPP) has incorporated femtocell (FC) technology in the Long-Term Evolution Advanced (LTE-A) standard to enhance the quality of service of indoor mobile users and extend the coverage area of existing macrocells (MCs). In such two-tier LTE-A MC/FC systems, cotier and cross-tier interference exists in cochannel deployment, exerting adverse effects on system performance. In this paper, we study the single-carrier frequency-division multiple-access (SC-FDMA)-based LTE-A FC uplink. We propose the use of transport-layer data admission control (AC) in femto user equipment (FUE) and interference-aware resource allocation (RA) in each base station (BS) to manage the intercell interference (ICI). We first formulate the problem as a constrained Markov decision problem (CMDP) that aims at maximizing the time-average throughput of the entire FC tier subject to the queue stability constraint for each FUE. Then, we propose a joint AC and RA (JACRA) algorithm to obtain the optimal AC and RA policies. In light of the NP-hardness of the RA subproblem, we further propose an iterative heuristic with polynomial time complexity. Simulation studies show that the proposed JACRA algorithm is throughput optimal, outperforming alternative proportional fair (PF) and round robin (RR) scheduling schemes. Moreover, the proposed heuristic achieves near-optimal throughput with substantial improvement in computational complexity.

Dynamic Optimization and Learning for Renewal Systems

This paper considers optimization of time averages in systems with variable length renewal frames. Applications include power-aware and profit-aware scheduling in wireless networks, peer-to-peer networks, and transportation systems. Every frame, a new policy is implemented that affects the frame size and that creates a vector of attributes. The policy can be a single decision in response to a random event observed on the frame, or a sequence of such decisions. The goal is to choose policies on each frame in order to maximize a time average of one attribute, subject to additional time average constraints on the others. Two algorithms are developed, both based on Lyapunov optimization concepts. The first makes decisions to minimize a “drift-plus-penalty” ratio over each frame. The second is similar but does not involve a ratio. For systems that make only a single decision on each frame, both algorithms are shown to learn efficient behavior without a-priori statistical knowledge. The framework is also applicable to large classes of constrained Markov decision problems. Such problems are reduced to finding an approximate solution to a simpler unconstrained stochastic shortest path problem on each frame. Approximations for the simpler problem may still suffer from a curse of dimensionality for systems with large state space. However, our results are compatible with any approximation method, and demonstrate an explicit tradeoff between performance and convergence time.

Opportunistic Cooperation in Cognitive Femtocell Networks

We investigate opportunistic cooperation between secondary (femtocell) users and primary (macrocell) users in cognitive femtocell networks. We consider two models for such cooperation. In the first model, called the Cooperative Relay Model, a secondary user cannot transmit its own data concurrently with a primary user. However, it can employ cooperative relaying of primary user data in order to improve the latter's effective transmission rate. In the second model, called the Interference Model, a secondary user is allowed to transmit its data concurrently with a primary user. However, the secondary user can "cooperate" by deferring its transmissions when the primary user is busy. In both models, the secondary users must make intelligent cooperation decisions as they seek to maximize their own throughput subject to average power constraints. The decision options are different during idle and busy periods of the primary user, and the decisions in turn influence the durations of these periods according to a controllable infinite state Markov chain. Such problems can be formulated as constrained Markov decision problems, and conventional solution techniques require either extensive knowledge of the system dynamics or learning based approaches that suffer from large convergence times. However, using a generalized Lyapunov optimization technique, we design a novel greedy and online control algorithm that overcomes these challenges. Remarkably, this algorithm does not require any knowledge of the network arrival rates and is provably optimal.

Markov chains, Hamiltonian cycles and volumes of convex bodies

In this note the Hamiltonian cycle problem is mapped into an infinite horizon discounted cost constrained Markov decision problem. The occupation measure based linear polytope associated with this control problem defines a convex set which either strictly contains or is equal to another convex set, depending on whether the underlying graph has a Hamiltonian cycle or not. This allows us to distinguish Hamiltonian graphs from non-Hamiltonian graphs by comparing volumes of two convex sets.

Power efficient scheduling over fading channel for cross‐layer optimization

ABSTRACTWe consider the minimization of long‐term average power consumption for packet transmission between a mobile station and the base station over Nakagami‐m fading channel. Power consumption is minimized by intelligent transmission scheduling design, with the average queuing delay and joint packet loss across MAC and physical layers being confined below certain levels. The problem is formulated as an infinite horizon constrained Markov decision problem and solved by linear programming (LP) method. The primary intention of this paper is to provide a visible paradigm on using LP method to optimize the performance of mobile wireless communication systems. We elaborate the detailed mathematical solution with consistent simulation experiments and emphasize the effectiveness of adaptive transmission scheduling for cross‐layer QoS provisioning. Copyright © 2010 John Wiley &amp; Sons, Ltd.

A stochastic control approach for scheduling multimedia transmissions over a polled multiaccess fading channel

We develop scheduling strategies for carrying multimedia traffic over a polled multiple access wireless network with fading. We consider a slotted system with three classes of traffic (voice, streaming media and file transfers). A Markov model is used for the fading and also for modeling voice packet arrivals and streaming arrivals. The performance objectives are a loss probability for voice, mean network delay for streaming media, and time average throughput for file transfers. A central scheduler (e.g., the access point in a single cell IEEE 802.11 wireless local area network (WLAN)) is assumed to be able to keep track of all the available state information and make the scheduling decision in each slot (e.g., as would be the case for PCF mode operation of the IEEE 802.11 WLAN). The problem is modeled as a constrained Markov decision problem. By using constraint relaxations (a linear relaxation and Whittle type relaxations) an index based policy is obtained. For the file transfers the decision problem turns out to be one with partial state information. Numerical comparisons are provided with the performance obtained from some simple policies.

Efficient QoS Provisioning for Adaptive Multimedia in Mobile Communication Networks by Reinforcement Learning

The scarcity and large fluctuations of link bandwidth in wireless networks have motivated the development of adaptive multimedia services in mobile communication networks, where it is possible to increase or decrease the bandwidth of individual ongoing flows. This paper studies the issues of quality of service (QoS) provisioning in such systems. In particular, call admission control and bandwidth adaptation are formulated as a constrained Markov decision problem. The rapid growth in the number of states and the difficulty in estimating state transition probabilities in practical systems make it very difficult to employ classical methods to find the optimal policy. We present a novel approach that uses a form of discounted reward reinforcement learning known as Q-learning to solve QoS provisioning for wireless adaptive multimedia. Q-learning does not require the explicit state transition model to solve the Markov decision problem; therefore more general and realistic assumptions can be applied to the underlying system model for this approach than in previous schemes. Moreover, the proposed scheme can efficiently handle the large state space and action set of the wireless adaptive multimedia QoS provisioning problem. Handoff dropping probability and average allocated bandwidth are considered as QoS constraints in our model and can be guaranteed simultaneously. Simulation results demonstrate the effectiveness of the proposed scheme in adaptive multimedia mobile communication networks.

Stability and singular perturbations in constrained Markov decision problems

Constrained Markov decision problems (CMDPs) with the average cost criterion and a single ergodic chain, or the discounted cost with a general multichain structure, are considered. Conditions for stability of the optimal value and control to changes of the parameters of the problem, such as immediate costs, transition probabilities, and the discount factor, are established. Singular constrained problems, for which the optimal value and controls exhibit discontinuities, are studied. >

Stochastic approximations for finite-state Markov chains

This paper develops an a.s. convergence theory for a class of projected stochastic approximations driven by finite-state Markov chains. The conditions are mild and are given explicitly in terms of the model data, mainly the Lipschitz continuity of the one-step transition probabilities. The approach used here is a version of the ODE method as proposed by Métivier and Priouret. It combines the Kushner-Clark Lemma with properties of the Poisson equation associated with the underlying family of Markov chains. The class of algorithms studied here was motivated by implementation issues for constrained Markov decision problems, where the policies of interest often depend on quantities not readily available due either to insufficient knowledge of the model parameters or to computational difficulties. This naturally leads to the on-line estimation (or computation) problem investigated here. Several examples from the area of queueing systems are discussed.