Constrained Markov Decision Process Research Articles

An AoTI-Driven Joint Sampling Frequency and Access Selection Optimization for Industrial Wireless Sensor Networks

The freshness of system information has attracted extensive attention and has become one of the most critical Quality of Service (QoS) indicators in industrial wireless sensor networks (IWSNs). For practical IWSNs composed of numerous components, the system freshness for a specific task is usually related to multiple and multitype sensing data. For example, in a chemical plant fire alarm system, the system controller (SC) cannot determine the fire notification until analyzing multiple types of ambient information, such as temperature, flame, carbon dioxide, and smoke. However, most existing research on freshness metrics, such as Age of Information (AoI) or Age of Processing (AoP), only considers a single-package setting with a single type of data. Therefore, to measure the system freshness accurately in IWSNs, we propose the Age of Task-oriented Information (AoTI) for industrial tasks. It measures the time elapsed of the latest analyzed results before arriving at the receiver since the generation of any type of sampling data belonging to an industrial task. Considering the time-varying and wireless environments, we aim to minimize the long-term AoTI for IWSNs applications by jointly optimizing access selections and sampling frequencies for all sensors. We first formulate this problem as a Mixed Integer Nonlinear Programming (MINLP) problem and then transform it into a Constrained Markov Decision Process (CMDP), which is further relaxed as an MDP using the Lagrangian method. Finally, we develop a Learning-based Access selection and Sampling frequency Control (LASC) algorithm and verify its superiority through extensive simulations.

A Distributed and Privacy-Aware High-Throughput Transaction Scheduling Approach for Scaling Blockchain

Payment channel networks (PCNs) are considered as a prominent solution for scaling blockchain, where users can establish payment channels and complete transactions in an off-chain manner. However, it is non-trivial to schedule transactions in PCNs and most existing routing algorithms suffer from the following challenges: 1) one-shot optimization, 2) privacy-invasive channel probing, 3) vulnerability to DoS attacks. To address these challenges, we propose a privacy-aware transaction scheduling algorithm with defence against DoS attacks based on deep reinforcement learning (DRL), namely PTRD. Specifically, considering both the privacy preservation and long-term throughput into the optimization criteria, we formulate the transaction-scheduling problem as a Constrained Markov Decision Process. We then design PTRD, which extends off-the-shelf DRL algorithms to constrained optimization with an additional cost critic-network and an adaptive Lagrangian multiplier. Moreover, considering the distribution nature of PCNs, in which each user schedules transactions independently, we develop a distributed training framework to collect the knowledge learned by each agent so as to enhance learning effectiveness. With the customized network design and the distributed training framework, PTRD achieves a good balance between the optimization of the throughput and the minimization of privacy risks. Evaluations show that PTRD outperforms the state-of-the-art PCN routing algorithms by 2.7%–62.5% in terms of the long-term throughput while satisfying privacy constraints.

Safe-State Enhancement Method for Autonomous Driving via Direct Hierarchical Reinforcement Learning

Reinforcement learning (RL) has shown excellent performance in the sequential decision-making problem, where safety in the form of state constraints is of great significance in the design and application of RL. Simple constrained end-to-end RL methods might lead to significant failure in a complex system like autonomous vehicles. In contrast, some hierarchical RL (HRL) methods generate driving goals directly, which could be closely combined with motion planning. With safety requirements, some safe-enhanced RL methods add post-processing modules to avoid unsafe goals or achieve expectation-based safety, which accepts the existence of unsafe states and allows some violations of safe constraints. However, ensuring state safety is vital for autonomous vehicles. Therefore, this paper proposes a state-based safety enhancement method for autonomous driving via direct hierarchical reinforcement learning. Finally, we design a constrained reinforcement learner based on the State-based Constrained Markov Decision Process (SCMDP), where a learnable safety module could adjust the constraint strength adaptively. We integrate a dynamic module in the policy training and generate future goals considering safety, temporal-spatial continuity, and dynamic feasibility, which could eliminate dependence on the prior model. Simulations in the typical highway scenes with uncertainties show that the proposed method has better training performance, higher driving safety in interactive scenes, more decision intelligence in traffic congestions, and better economic driving ability on roads with changing slopes.

A safe reinforcement learning-based charging strategy for electric vehicles in residential microgrid

With the growing popularity of electric vehicles (EVs), it is a new challenge for the residential microgrid system to conduct charging scheduling to meet the charging demands of EVs while maximizing its profit. In this work, a safe reinforcement learning (RL)-based charging scheduling strategy is proposed to meet this challenge. We construct a complete microgrid system equipped with a large charging station and consider different types of EVs, as well as the vehicle-to-grid (V2G) mode and nonlinear charging characteristics of EVs. Subsequently, the charging scheduling problem is formulated as a constrained Markov decision process (CMDP) due to the various limitations of power and demands. To effectively capture the uncertainties of the supply side and demand side of the microgrid, a model-free RL framework is employed. However, the curse of dimensionality of the action space is inevitable as EVs increase. To solve this problem, a charging and discharging strategy based on a general ladder electricity pricing scheme is designed. Different EVs are divided into different sets according to their states under this strategy, and the agent gives control signals of different sets instead of controlling each EV individually, which effectively reduces the dimension of the action space. Subsequently, a constrained soft actor-critic (CSAC) algorithm is designed to solve the established CMDP, and a safety filter is introduced to ensure safety. In the end, a numerical case is conducted to verify the effectiveness of the proposed method.

Scheduling Policies for AoI Minimization With Timely Throughput Constraints

In 5G and beyond communication systems, the notion of latency gets great momentum in wireless connectivity as a metric for serving real-time communications requirements. However, in many applications, research has pointed out that latency could be inefficient to handle applications with data freshness requirements. Recently, Age of Information (AoI) metric, which can capture the freshness of the data, has attracted a lot of attention. In this work, we consider mixed traffic with time-sensitive users; a deadline-constrained user, and an AoI-oriented user. To develop an efficient scheduling policy, we cast a novel optimization problem formulation for minimizing the average AoI while satisfying the timely throughput constraints. The formulated problem is cast as a Constrained Markov Decision Process (CMDP). We relax the constrained problem to an unconstrained Markov Decision Process (MDP) problem by utilizing the Lyapunov optimization theory and it can be proved that it is solved per frame by applying backward dynamic programming algorithms with optimality guarantees. In addition, we provide a low-complexity algorithm guaranteeing that the timely-throughput constraint is satisfied. Simulation results show that the timely throughput constraints are satisfied while minimizing the average AoI. Simulation results show the convergence of the algorithms for different values of the weighted factor and the trade-off between the AoI and the timely throughput.

Minimizing Age of Information in Downlink Wireless Networks With Time-Varying Channels and Peak Power Constraint

The rapid development of real-time applications such as the Internet of Things (IoT) puts forward high requirements for the data freshness of wireless networks. The metric Age of Information (AoI) is adopted to measure the data freshness from the user side [1]. In this paper, we investigate the AoI-minimization problem of wireless networks with time-varying channels and the peak power constrained base station (BS). Firstly we model the time-varying channels as a multi-state ergodic Markov chain and formulate the multi-user scheduling problem. Next, we relax the original strict power constraint and formulate the relaxed problem as a constrained Markov decision process (CMDP). To reduce the computational complexity, we exploit the threshold structure of the optimal policy for the CMDP and search for the optimal policy via Linear programming (LP). To satisfy the original strict power constraint, we propose an truncated multi-user scheduling policy, which is proven asymptotic optimal. Finally, we simulate our proposed scheduling policy and compare the AoI performance with other policies. Simulation results verify the theoretical analysis and indicate that our policy can significantly reduce the AoI of users with bad channel quality, thus outperforming other policies.

Provably Efficient Primal-Dual Reinforcement Learning for CMDPs with Non-stationary Objectives and Constraints

We consider primal-dual-based reinforcement learning (RL) in episodic constrained Markov decision processes (CMDPs) with non-stationary objectives and constraints, which plays a central role in ensuring the safety of RL in time-varying environments. In this problem, the reward/utility functions and the state transition functions are both allowed to vary arbitrarily over time as long as their cumulative variations do not exceed certain known variation budgets. Designing safe RL algorithms in time-varying environments is particularly challenging because of the need to integrate the constraint violation reduction, safe exploration, and adaptation to the non-stationarity. To this end, we identify two alternative conditions on the time-varying constraints under which we can guarantee the safety in the long run. We also propose the Periodically Restarted Optimistic Primal-Dual Proximal Policy Optimization (PROPD-PPO) algorithm that can coordinate with both two conditions. Furthermore, a dynamic regret bound and a constraint violation bound are established for the proposed algorithm in both the linear kernel CMDP function approximation setting and the tabular CMDP setting under two alternative conditions. This paper provides the first provably efficient algorithm for non-stationary CMDPs with safe exploration.

AutoCost: Evolving Intrinsic Cost for Zero-Violation Reinforcement Learning

Safety is a critical hurdle that limits the application of deep reinforcement learning to real-world control tasks. To this end, constrained reinforcement learning leverages cost functions to improve safety in constrained Markov decision process. However, constrained methods fail to achieve zero violation even when the cost limit is zero. This paper analyzes the reason for such failure, which suggests that a proper cost function plays an important role in constrained RL. Inspired by the analysis, we propose AutoCost, a simple yet effective framework that automatically searches for cost functions that help constrained RL to achieve zero-violation performance. We validate the proposed method and the searched cost function on the safety benchmark Safety Gym. We compare the performance of augmented agents that use our cost function to provide additive intrinsic costs to a Lagrangian-based policy learner and a constrained-optimization policy learner with baseline agents that use the same policy learners but with only extrinsic costs. Results show that the converged policies with intrinsic costs in all environments achieve zero constraint violation and comparable performance with baselines.

Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Conservative Natural Policy Gradient Primal-Dual Algorithm

We consider the problem of constrained Markov decision process (CMDP) in continuous state actions spaces where the goal is to maximize the expected cumulative reward subject to some constraints. We propose a novel Conservative Natural Policy Gradient Primal Dual Algorithm (CNPGPD) to achieve zero constraint violation while achieving state of the art convergence results for the objective value function. For general policy parametrization, we prove convergence of value function to global optimal upto an approximation error due to restricted policy class. We improve the sample complexity of existing constrained NPGPD algorithm. To the best of our knowledge, this is the first work to establish zero constraint violation with Natural policy gradient style algorithms for infinite horizon discounted CMDPs. We demonstrate the merits of proposed algorithm via experimental evaluations.

Policy-Based Primal-Dual Methods for Convex Constrained Markov Decision Processes

We study convex Constrained Markov Decision Processes (CMDPs) in which the objective is concave and the constraints are convex in the state-action occupancy measure. We propose a policy-based primal-dual algorithm that updates the primal variable via policy gradient ascent and updates the dual variable via projected sub-gradient descent. Despite the loss of additivity structure and the nonconvex nature, we establish the global convergence of the proposed algorithm by leveraging a hidden convexity in the problem, and prove the O(T^-1/3) convergence rate in terms of both optimality gap and constraint violation. When the objective is strongly concave in the occupancy measure, we prove an improved convergence rate of O(T^-1/2). By introducing a pessimistic term to the constraint, we further show that a zero constraint violation can be achieved while preserving the same convergence rate for the optimality gap. This work is the first one in the literature that establishes non-asymptotic convergence guarantees for policy-based primal-dual methods for solving infinite-horizon discounted convex CMDPs.

Planning and Learning for Non-markovian Negative Side Effects Using Finite State Controllers

Autonomous systems are often deployed in the open world where it is hard to obtain complete specifications of objectives and constraints. Operating based on an incomplete model can produce negative side effects (NSEs), which affect the safety and reliability of the system. We focus on mitigating NSEs in environments modeled as Markov decision processes (MDPs). First, we learn a model of NSEs using observed data that contains state-action trajectories and severity of associated NSEs. Unlike previous works that associate NSEs with state-action pairs, our framework associates NSEs with entire trajectories, which is more general and captures non-Markovian dependence on states and actions. Second, we learn finite state controllers (FSCs) that predict NSE severity for a given trajectory and generalize well to unseen data. Finally, we develop a constrained MDP model that uses information from the underlying MDP and the learned FSC for planning while avoiding NSEs. Our empirical evaluation demonstrates the effectiveness of our approach in learning and mitigating Markovian and non-Markovian NSEs.

A Reinforcement Learning Framework for Vehicular Network Routing Under Peak and Average Constraints

Providing provable performance guarantees in vehicular network routing problems is crucial to ensure safely and timely delivery of information in an environment characterized by high mobility, dynamic network conditions, and frequent topology changes. While Reinforcement Learning (RL) has shown great promise in network routing, existing RL-based solutions typically support decision-making with either peak constraints or average constraints, but not both. For network routing in intelligent transportation, such as advanced vehicle control and safety, both peak constraints (e.g., maximum latency or minimum bandwidth guarantees) and average constraints (e.g., average transmit power or data rate constraints) must be satisfied. In this paper, we propose a holistic framework for RL-based vehicular network routing, which maximizes routing decisions under both average and peak constraints. The routing problem is modeled as a Constrained Markov Decision Process and recast into an optimization based on Constraint Satisfaction Problems (CSPs). We prove that the optimal policy of a given CSP can be learned by an extended Q-learning algorithm while satisfying both peak and average latency constraints. To improve the scalability of our framework, we further turn it into a decentralized implementation through a cluster-based learning structure. Applying the proposed RL algorithm to vehicular network routing problems under both peak and average latency constraints, simulation results show that our algorithm achieves much higher rewards than heuristic baselines with over 40% improvement in average transmission rate, while resulting in zero violation in terms of both peak and average constraints.

Optimal Control of Wireless Powered Edge Computing System for Balance Between Computation Rate and Energy Harvested

Wireless powered edge computing system (WPECS) enhances the computing power and extends the lifetime of wireless devices (WDs). This paper studies the WPECS with multiple WDs, in which the access point (AP) provides some transmission channels which differ from each other in the channel gain, and the WD powered through the wireless power transfer (WPT) technology has some indivisible tasks and adopts binary task-offloading actions. More energy harvested strengthens the WDs with more computing power, while corresponding to more energy consumption. Therefore, how to make the optimal tradeoff between computation rate and energy harvested arises as an interesting issue. To address this issue, this paper first formulates the switch process of transmission channel as a constrained Markov decision process (CMDP), and then proposed an effective algorithm to maximize the sum of computation rates of all WDs in terms of task data bits computed, within the required level of accumulative energy harvested. Theoretical analysis, simulations and field experiments jointly document and illustrate its performance. Note to Practitioners—This paper addresses the interesting tradeoff between computation rate and energy harvested in a wireless powered edge computing system that operates in the environments with limited available energy. It helps to improve the operation efficiency of the edge computing systems in the area of Internet of Things (IoT) or Cyber-Physical Systems (CPS) that employ wireless power transfer technology to power the wireless devices through the access point over the air to maximize the sum of computations rates of all WDs in terms of task data bits computed, while keeping the accumulative energy harvested within a range. Simulations and experimental investigations show that the solution proposed here outperforms existing solutions.

Query-Age-Optimal Scheduling Under Sampling and Transmission Constraints

This letter provides query-age-optimal joint sampling and transmission scheduling policies for a heterogeneous status update system, consisting of a stochastic arrival and a generate-at-will source, with an unreliable channel. Our main goal is to minimize the average query age of information (QAoI) subject to average sampling, average transmission, and per-slot transmission constraints. To this end, an optimization problem is formulated and solved by casting it into a linear program. We also provide a low-complexity near-optimal policy using the notion of weakly-coupled constrained Markov decision processes. The numerical results show up to 32% performance improvement by the proposed policies compared with a benchmark policy.

Delay optimal for reliability-guaranteed concurrent transmissions with raptor code in multi-access 6G edge network

This paper proposes RCMEN, a solution for Concurrent Multipath Transfer (CMT) of Raptor-Coded (RC) Protocol Data Units (PDU) in a Multi-Access 6G Edge Network (MEN) to provide low-latency and reliable transmission for 6G User Equipment (UE) and Data Network (DN). While 3GPP Release 16 aims to improve the reliability of a single service based on CMT, replication-based reliability causes high resource and signaling overhead, and asymmetric bandwidth and latency lead to excessive latency. RCMEN addresses these issues by distributing data across multiple PDU sessions and using a queue-aware variable-block-length Raptor encoder to optimize resource utilization under random arrivals. We analyze RCMEN’s delay-reliability performance using the Constrained Markov Decision Process (CMDP), reliability is guaranteed by the reliability function, which takes the amount of data sent as a parameter. Through Linear Programming, we obtain the optimal delay-reliability trade-off and optimize delay metrics for two classical RCMEN. Experiments show that both classical RCMEN optimize the long-tail distribution of latency while ensuring reliability, with the entire system being compatible with the current 5G network infrastructure.

Energy-Efficient Space–Air–Ground Integrated Edge Computing for Internet of Remote Things: A Federated DRL Approach

Space–air–ground integrated edge computing is expecting to provide pervasive computation services for Internet of Things (IoT), especially in remote areas. However, the offloading process of power-limited IoT devices is a challenge issue due to unreliable communications in an aerial environment. In this article, we propose an energy-efficient space–air–ground integrated edge computing network architecture, in which the IoT devices choose the most appropriate LEO satellites or unmanned aerial vehicles (UAVs) for task offloading according to their energy level, communication conditions and computing capabilities. In order to providing efficient task offloading and energy-saving policy under an uncertainty aerial environment, a constrained Markov decision process is employed to formulate the task offloading decision problem and a deep reinforcement learning (DRL)-based algorithm is devised to solve the proposed problem. An adaptive federated DRL-based offloading method is further proposed to find suboptimal offloading decisions by considering the privacy protection and communication failure in the proposed network. Numerical results confirm the effectiveness of the proposed schemes on energy saving and computation efficiency.

A Transfer Learning Approach for UAV Path Design With Connectivity Outage Constraint

The connectivity-aware path design is crucial in the effective deployment of autonomous unmanned aerial vehicles (UAVs). Recently, reinforcement learning (RL) algorithms have become the popular approach to solving this type of complex problem, but RL algorithms suffer slow convergence. In this article, we propose a transfer learning (TL) approach, where we use a teacher policy previously trained in an old domain to boost the path learning of the agent in the new domain. As the exploration processes and the training continue, the agent refines the path design in the new domain based on the subsequent interactions with the environment. We evaluate our approach considering an old domain at sub-6 GHz and a new domain at millimeter-wave (mmWave). The teacher path policy, previously trained at the sub-6 GHz path, is the solution to a connectivity-aware path problem that we formulate as a constrained Markov decision process (CMDP). We employ a Lyapunov-based model-free deep <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -network (DQN) to solve the path design at sub-6 GHz that guarantees connectivity constraint satisfaction. We empirically demonstrate the effectiveness of our approach for different urban environment scenarios. The results demonstrate that our proposed approach is capable of reducing the training time considerably at mmWave.

Age of Processed Information Minimization Over Fading Multiple Access Channels

In many real-time applications, timely accessibility to actionable insights is an important requirement. We consider users equipped with sensors and actuators, where the sensors generate updates which need to be processed for driving the actuators. The processing task can be carried out locally or at an edge server located at a common base station (BS). The users access the BS via a fading multiple access channel (MAC) using a non-orthogonal multiple access technique. To measure the timeliness of processed information, we adopt a metric called age of processed information (AoPI), defined as the time elapsed since the generation of the last successfully processed packet available to the user. In this setting, we formulate an average AoPI minimization problem, when users are subjected to average power constraints. The BS has to decide when the users should generate status updates and whether to process them locally or at the edge server. We cast the problem as a constrained Markov decision process (CMDP) and obtain its solution via Lagrangian relaxation. We then obtain a simpler policy via Lyapunov optimization and bound its performance. Via numerical simulations, we finally illustrate properties of the CMDP solution and study behavior of average AoPIs under different policies when problem parameters are varied.

Learning in Constrained Markov Decision Processes

We consider reinforcement learning (RL) in Markov Decision Processes in which an agent repeatedly interacts with an environment that is modeled by a controlled Markov process. At each time step $t$, it earns a reward, and also incurs a cost-vector consisting of $M$ costs. We design model-based RL algorithms that maximize the cumulative reward earned over a time horizon of $T$ time-steps, while simultaneously ensuring that the average values of the $M$ cost expenditures are bounded by agent-specified thresholds $c^{ub}_i,i=1,2,\ldots,M$. In order to measure the performance of a reinforcement learning algorithm that satisfies the average cost constraints, we define an $M+1$ dimensional regret vector that is composed of its reward regret, and $M$ cost regrets. The reward regret measures the sub-optimality in the cumulative reward, while the $i$-th component of the cost regret vector is the difference between its $i$-th cumulative cost expense and the expected cost expenditures $Tc^{ub}_i$. We prove that the expected value of the regret vector of UCRL-CMDP, is upper-bounded as $\tilde{O}\left(T^{2\slash 3}\right)$, where $T$ is the time horizon. We further show how to reduce the regret of a desired subset of the $M$ costs, at the expense of increasing the regrets of rewards and the remaining costs. To the best of our knowledge, ours is the only work that considers non-episodic RL under average cost constraints, and derive algorithms that can~\emph{tune the regret vector} according to the agent's requirements on its cost regrets.

Online Radio Access Technology Selection Algorithms in a 5G Multi-RAT Network

In today's wireless networks, a variety of Radio Access Technologies (RATs) are present. However, each RAT being controlled individually leads to suboptimal utilization of network resources. Due to the remarkable growth of data traffic, interworking among different RATs is becoming necessary to overcome the problem of suboptimal resource utilization. Users can be offloaded from one RAT to another based on loads of different networks, channel conditions and priority of users. We consider the optimal RAT selection problem in a Fifth Generation (5G) New Radio (NR)-Wireless Fidelity (WiFi) network where we aim to maximize the total system throughput subject to constraints on the blocking probability of high priority users and the offloading probability of low priority users. The problem is formulated as a Constrained Markov Decision Process (CMDP). We reduce the effective dimensionality of the action space by eliminating the provably suboptimal actions. We propose low-complexity online heuristics for RAT selection which can operate without the knowledge regarding the statistics of system dynamics. Network Simulator-3 (ns-3) simulations reveal that the proposed algorithms outperform traditional RAT selection algorithms under realistic network scenarios including user mobility.