Constrained Markov Decision Process Research Articles

Deep Reinforcement Learning-based Mining Task Offloading Scheme for Intelligent Connected Vehicles in UAV-aided MEC

The convergence of unmanned aerial vehicle (UAV)-aided mobile edge computing (MEC) networks and blockchain transforms the existing mobile networking paradigm. However, in the temporary hotspot scenario for intelligent connected vehicles (ICVs) in UAV-aided MEC networks, deploying blockchain-based services and applications in vehicles is generally impossible due to its high computational resource and storage requirements. One possible solution is to offload part of all the computational tasks to MEC servers wherever possible. Unfortunately, due to the limited availability and high mobility of the vehicles, there is still lacking simple solutions that can support low-latency and higher reliability networking services for ICVs. In this article, we study the task offloading problem of minimizing the total system latency and the optimal task offloading scheme, subject to constraints on the hover position coordinates of the UAV, the fixed bonuses, flexible transaction fees, transaction rates, mining difficulty, costs and battery energy consumption of the UAV. The problem is confirmed to be a challenging linear integer planning problem, we formulate the problem as a constrained Markov decision process. Deep Reinforcement Learning (DRL) has excellently solved sequential decision-making problems in dynamic ICVs environment, therefore, we propose a novel distributed DRL-based P-D3QN approach by using Prioritized Experience Replay strategy and the dueling double deep Q-network (D3QN) algorithm to solve the optimal task offloading policy effectively. Finally, experiment results show that compared with the benchmark scheme, the P-D3QN algorithm can bring about 26.24% latency improvement and increase about 42.26% offloading utility.

Adaptive Cruise Control Based on Safe Deep Reinforcement Learning.

Adaptive cruise control (ACC) enables efficient, safe, and intelligent vehicle control by autonomously adjusting speed and ensuring a safe following distance from the vehicle in front. This paper proposes a novel adaptive cruise system, namely the Safety-First Reinforcement Learning Adaptive Cruise Control (SFRL-ACC). This system aims to leverage the model-free nature and high real-time inference efficiency of Deep Reinforcement Learning (DRL) to overcome the challenges of modeling difficulties and lower computational efficiency faced by current optimization control-based ACC methods while simultaneously maintaining safety advantages and optimizing ride comfort. Firstly, we transform the ACC problem into a safe DRL formulation Constrained Markov Decision Process (CMDP) by carefully designing state, action, reward, and cost functions. Subsequently, we propose the Projected Constrained Policy Optimization (PCPO)-based ACC Algorithm SFRL-ACC, which is specifically tailored to solve the CMDP problem. PCPO incorporates safety constraints that further restrict the trust region formed by the Kullback-Leibler (KL) divergence, facilitating DRL policy updates that maximize performance while keeping safety costs within their limit bounds. Finally, we train an SFRL-ACC policy and compare its computation time, traffic efficiency, ride comfort, and safety with state-of-the-art MPC-based ACC control methods. The experimental results prove the superiority of the proposed method in the aforementioned performance aspects.

Safe Reinforcement Learning for Arm Manipulation with Constrained Markov Decision Process

Safe Reinforcement Learning for Arm Manipulation with Constrained Markov Decision Process

Autonomous spacecraft collision avoidance with a variable number of space debris based on safe reinforcement learning

The avoidance of multiple space debris collisions by autonomous spacecraft has garnered significant interests worldwide. Applying deep reinforcement learning (DRL) to autonomous spacecraft collision avoidance problems is still difficult because of limitations on constraint satisfaction and environment state perception, even if DRL is a suitable model-free and data-driven framework. In this research, a state-of-the-art penalized proximal policy optimization (P3O) method is applied to address the spacecraft's autonomous collision avoidance problem, which is formalized as a constrained Markov decision process (CMDP). In contrast with traditional DRL methods, P3O promises to satisfy multiple constraints in actual spaceship operations while also facilitating efficient learning in multi-dimensional, continuous state and action spaces. The scalability of the P3O algorithm is enhanced by combining the feature extraction capabilities and variable-length input capabilities of the long short-term memory (LSTM) networks, enabling the P3O to adapt to a variable number of space debris without the need for network retraining. The performance of the proposed method is compared with other five methods through simulation cases, which verifies the superior performance of the proposed method in terms of scalability, energy consumption, collision probability and constraint satisfaction.

Rank-1 transition uncertainties in constrained Markov decision processes

We consider an infinite-horizon discounted constrained Markov decision process (CMDP) with uncertain transition probabilities. We assume that the uncertainty in transition probabilities has a rank-1 matrix structure and the underlying uncertain parameters belong to a polytope. We formulate the uncertain CMDP problem using a robust optimization framework. To derive reformulation of the robust CMDP problem, we restrict to the class of stationary policies and show that it is equivalent to a bilinear programming problem. We provide a simple example where a Markov policy performs better than the optimal policy in the class of stationary policies, implying that, unlike in classical CMDP problem, an optimal policy of the robust CMDP problem need not be present in the class of stationary policies. For the case of a single uncertain parameter, we propose sufficient conditions under which an optimal policy of the restricted robust CMDP problem is unaffected by uncertainty. The numerical experiments are performed on randomly generated instances of a machine replacement problem and a well-known class of problems called Garnets.

GRU-integrated constrained soft actor-critic learning enabled fully distributed scheduling strategy for residential virtual power plant

In this study, a novel residential virtual power plant (RVPP) scheduling method that leverages a gate recurrent unit (GRU)-integrated deep reinforcement learning (DRL) algorithm is proposed. In the proposed scheme, the GRU- integrated DRL algorithm guides the RVPP to participate effectively in both the day-ahead and real-time markets, lowering the electricity purchase costs and consumption risks for end-users. The Lagrangian relaxation technique is introduced to transform the constrained Markov decision process (CMDP) into an unconstrained optimization problem, which guarantees that the constraints are strictly satisfied without determining the penalty coefficients. Furthermore, to enhance the scalability of the constrained soft actor-critic (CSAC)-based RVPP scheduling approach, a fully distributed scheduling architecture was designed to enable plug-and-play in the residential distributed energy resources (RDER). Case studies performed on the constructed RVPP scenario validated the performance of the proposed methodology in enhancing the responsiveness of the RDER to power tariffs, balancing the supply and demand of the power grid, and ensuring customer comfort.

Long-Term Safe Reinforcement Learning with Binary Feedback

Safety is an indispensable requirement for applying reinforcement learning (RL) to real problems. Although there has been a surge of safe RL algorithms proposed in recent years, most existing work typically 1) relies on receiving numeric safety feedback; 2) does not guarantee safety during the learning process; 3) limits the problem to a priori known, deterministic transition dynamics; and/or 4) assume the existence of a known safe policy for any states. Addressing the issues mentioned above, we thus propose Long-term Binary-feedback Safe RL (LoBiSaRL), a safe RL algorithm for constrained Markov decision processes (CMDPs) with binary safety feedback and an unknown, stochastic state transition function. LoBiSaRL optimizes a policy to maximize rewards while guaranteeing long-term safety that an agent executes only safe state-action pairs throughout each episode with high probability. Specifically, LoBiSaRL models the binary safety function via a generalized linear model (GLM) and conservatively takes only a safe action at every time step while inferring its effect on future safety under proper assumptions. Our theoretical results show that LoBiSaRL guarantees the long-term safety constraint, with high probability. Finally, our empirical results demonstrate that our algorithm is safer than existing methods without significantly compromising performance in terms of reward.

HP3O algorithm-based all electric ship energy management strategy integrating demand-side adjustment

To tackle the energy management challenge that integrates power generation scheduling and demand-side adjustment for all-electric ship in uncertain marine environment, a hybrid penalized proximal policy optimization algorithm (HP3O)-based energy management strategy is proposed. First, demand-side adjustment, which involves adjusting the power of the ship's electric propulsion motors and flexible service loads, is integrated into the energy management problem. Second, HP3O algorithm is employed to obtain both continuous and discrete variables simultaneously. It utilizes a continuous actor network to obtain continuous variables, such as the generator's power and ship cruising speed, while employing a discrete actor network to determine discrete variables, i.e., the on/off status of the generators. Third, to handle complex constraints reasonably, the energy management problem is formulated as a constrained Markov decision process (CMDP), and an action mask mechanism is also integrated into the energy management framework to make agent's actions more reliable. The simulation results of an all-electric cruise ship validate the effectiveness and superiority of the proposed strategy in achieving near-optimal scheduling while satisfying operation constraints. Furthermore, a case study on a hybrid diesel-electric ferry confirms its generalization performance.

Faster algorithm and sharper analysis for constrained Markov decision process

The problem of constrained Markov decision process (CMDP) is investigated, where an agent aims to maximize the expected accumulated reward subject to constraints on its utilities/costs. We propose a new primal-dual approach with a novel integration of entropy regularization and Nesterov's accelerated gradient method. The proposed approach is shown to converge to the global optimum with a complexity of O˜(1/ϵ) in terms of the optimality gap and the constraint violation, which improves the complexity of the existing primal-dual approaches by a factor of O(1/ϵ).

Fuzzy-based predictive deep reinforcement learning for robust and constrained optimal control of industrial solar thermal plants

Integrating distributed solar fields (DSFs) into conventional heat and power plants (CHPs) of industries is mostly constrained by the availability of a real-time capable control scheme. Safe and efficient operation of industrial DSF requires the supply of a fluctuating and periodically available energy at the required temperature while reducing losses and ensuring operational constraint. Reinforcement learning (RL), specifically Q-learning methods, are being recently applied for such control tasks. However, existing RL approaches cannot be directly applied to continuous domain control tasks or tasks with operation constraints. While using deep learning in model-free RL can remove these limitations, tractability and scalability boundaries on performing the deep learning are becoming significant, with scarce data and multi-objective concerns. In contrast, model-based deep RL schemes can be applied for sample efficiency and satisfying operational constraints, but their modeling framework is not general and accurate. To address these challenges, the work here develops a hybrid fuzzy convolution model (HFCM) that takes full advantages of data, models (dynamic and steady-state), and prior knowledge on industrial DSF. The HFCM is then extended for use in deep deterministic policy gradient (DDPG) algorithm to learn the DSF control task. This is done so by solving a multi-objective optimization problem, which is formulated as a constrained Markov decision process (CMDP) with continuous state and actions. Some practical and relevant findings were made with this HFCM. Firstly, it allowed the DDPG agent to learn independently a temperature state and a disturbance state, using only temperature measurement and single realizations of state vector. It also permitted the testing of an operational state violation strategy on a DDPG actions, and at the same time, the correction of the effects at a safety layer, if needed. Furthermore, the proposed control strategy simultaneously reduced mean temperature tracking error by 24%–51% and energy gain by 13.9%–17.35% when compared respectively to model-free DDPG and MPC baselines.

Age of information minimization in hybrid cognitive radio networks under a timely throughput constraint

Exchanging time-critical information is prevalent in various industrial applications where low latency and timely delivery are paramount. Through this work, we consider a cognitive radio network comprised of multiple secondary users with time-sensitive traffic, and they can access the licensed channel under the hybrid interweave/underlay scheme to enhance spectrum utilization. Traffic in the secondary system is divided into two distinct categories: deadline-constrained data and status updates. Quality of service of data with expiration time, such as multimedia streams, is assessed through the timely throughput metric. However, the age of information metric is used to characterize the freshness of the status update packets, which is vital in several emerging applications. Within an interference constraint imposed by the primary user, a dynamic scheduling policy is proposed to optimize the weighted sum of the average age of information of the status update users under a strict timely throughput requirement for each user with deadline-constrained traffic. We formulate the optimization problem as a constrained Markov decision process. Then, through the drift-plus-penalty method, the problem is reduced into a series of unconstrained Markov decision problems. Finally, each subproblem is tackled using the backward dynamic programming technique. Simulation results illustrate the effect of the main system parameters, such as the PU transmitted power and transmission rate level, on the performance of the secondary system. Moreover, the model feasibility regarding the fulfillment of the constraints against PU activity is experimentally investigated under the proposed hybrid mode and classical interweave mode. The performance of the proposed policy is compared to two other low-complexity scheduling schemes, which ensure the satisfaction of the constraints; results show the performance superiority of our proposed policy.

Rule-based shields embedded safe reinforcement learning approach for electric vehicle charging control

The optimal and secure electric vehicles charging control can be formulated as a large-scale partially-observable constrained Markov decision process (PO-CMDP) problem with high levels of security risks and uncertainties. Such a problem is very difficult to be handled by optimization-theory-based methods. Data-driven methods, especially reinforcement learning (RL), are suitable for handling uncertainties but are poor in ensuring safety, which is unacceptable in power systems. Motivated by the fact that humans could be supervised by an expert when learning a new skill involving risk, this paper proposes a safe RL framework that embeds rule-based local and global shields in the loop of RL for supervising the actions of agents. The proposed framework not only strictly guarantees local and global security in the training and execution phases, but also helps the agent to find a more near-optimal policy. The effectiveness and efficiency are demonstrated by comparison with multiple baseline methods in the IEEE-33 node system.

Online Preventive Control for Transmission Overload Relief Using Safe Reinforcement Learning With Enhanced Spatial-Temporal Awareness

The risk of transmission overload (TO) in power grids is increasing with the large-scale integration of intermittent renewable energy sources. An effective online preventive control schemes proves to be vital in safeguarding the security of power systems. In this paper, we formulate the online preventive control problem for TO alleviation as a constrained Markov decision process (CMDP), targeted to reduce the load rate of overloaded lines by implementing generation re-dispatch, transmission and busbar switching actions. The CMDP is solved with a state-of-the-art safe deep reinforcement learning method, based on the computationally efficient interior-point policy optimization (IPO), which facilitates desirable learning behavior towards constraint satisfaction and policy improvement simultaneously. The performance of IPO method is further improved with an enhanced perception of spatial-temporal correlations in power gird nodal and edge features, combining the strength of edge conditioned convolutional network and long short-term memory network, fostering more effective and robust preventive control policies to be devised. Case studies on a real-world system and a large-scale system validate the superior performance of the proposed method in TO alleviation, constraint handling, uncertainty adaptability and stability preservation, as well as its favorable computational performance, through benchmarking against both model-based and reinforcement learning-based baseline methods.

A Gradient-Aware Search Algorithm for Constrained Markov Decision Processes.

The canonical solution methodology for finite constrained Markov decision processes (CMDPs), where the objective is to maximize the expected infinite-horizon discounted rewards subject to the expected infinite-horizon discounted costs' constraints, is based on convex linear programming (LP). In this brief, we first prove that the optimization objective in the dual linear program of a finite CMDP is a piecewise linear convex (PWLC) function with respect to the Lagrange penalty multipliers. Next, we propose a novel, provably optimal, two-level gradient-aware search (GAS) algorithm which exploits the PWLC structure to find the optimal state-value function and Lagrange penalty multipliers of a finite CMDP. The proposed algorithm is applied in two stochastic control problems with constraints for performance comparison with binary search (BS), Lagrangian primal-dual optimization (PDO), and LP. Compared with the benchmark algorithms, it is shown that the proposed GAS algorithm converges to the optimal solution quickly without any hyperparameter tuning. In addition, the convergence speed of the proposed algorithm is not sensitive to the initialization of the Lagrange multipliers.

Multi-Agent Safe Graph Reinforcement Learning for PV Inverters-Based Real-Time Decentralized Volt/Var Control in Zoned Distribution Networks

To realize real-time voltage/var control (VVC) in active distribution networks (ADNs), this paper proposes a new multi-agent safe graph reinforcement learning method to optimize reactive power output from PV inverters. The network is divided into several zones, and a decentralized framework is proposed for coordinated control of reactive power output in each zone to regulate voltage profiles and minimize network energy loss. The VVC problem is formulated as a multi-agent decentralized partially observable constrained Markov decision process. Each zone has a central control agent that embeds graph convolution networks (GCNs) in the policy network to improve the decision-making capability. The GCN extracts graph-structured features from the ADN topology, reflecting the relationship between VVC and grid topology, and can filter noise and impute missing data. The training process includes primal-dual policy optimization to rigorously satisfy voltage safety constraints. Simulations on a 141-bus distribution system demonstrate that the proposed method can effectively minimize network energy loss and reduce voltage deviations, even in the presence of noisy or incomplete input measurements.

Deep Reinforcement Learning-Based Scheduling for NR-U/WiGig Coexistence in Unlicensed mmWave Bands

This paper investigates the coexistence of the New Radio-based access to Unlicensed spectrum (NR-U) network and the Wireless Gigabit (WiGig) network in unlicensed millimeter-wave (mmWave) bands. To enable the NR-U network to achieve equitable and harmonious spectrum sharing with WiGig systems, we develop two new classes of user equipment (UE) scheduling schemes by exploiting the deep reinforcement learning (DRL) technique. Specifically, we first propose the distributed deep reinforcement learning scheduling (DeepDS) scheme, wherein multiple deep neural networks (DNNs) are used to make decisions for different panels in an independent fashion. Thereafter, to reduce the computational cost of adopting multiple DNNs, we design the centralized deep reinforcement learning scheduling (DeepCS) scheme that introduces the shared DNN framework to perform decisions for all panels in parallel at one time. The objective of both DeepDS and DeepCS is to maximize the total data rate of the NR-U network with as little interference to WiGig systems as possible, while satisfying the quality of service (QoS) requirement for each UE. We first formulate this problem into the constrained Markov decision process framework. To address the multi-constraint issue, we put forth a new DRL algorithm that incorporates the Lagrangian primal-dual optimization into the deep Q-network framework, referred to as adaptive multi-constraint deep Q-network (AMC-DQN). With AMC-DQN, both DeepDS and DeepCS can achieve their goals even without acquiring prior operations about the WiGig network. Simulation results show that compared with the state-of-the-art omniLBT and dirLBT, both DeepDS and DeepCS yield significant performance benefits in terms of the total network data rate. We also demonstrate the ability of DeepDS and DeepCS to satisfy the QoS requirements of different UEs and their robustness against various simulation setups. Furthermore, compared with DeepDS, DeepCS can save a large amount of computational cost although at the expense of a slightly lower data rate.

Risk-Averse Decision Making Under Uncertainty

A large class of decision making under uncertainty problems can be described via Markov decision processes (MDPs) or partially observable MDPs (POMDPs), with application to artificial intelligence and operations research, among others. In this paper, we consider the problem of designing policies for MDPs and POMDPs with objectives and constraints in terms of dynamic coherent risk measures rather than the traditional total expectation, which we refer to as the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">constrained risk-averse problem</i> . Our contributions can be described as follows: For MDPs, under some mild assumptions, we propose an optimization-based method to synthesize Markovian policies. We then demonstrate that such policies can be found by solving difference convex programs (DCPs). We show that our formulation generalize linear programs for constrained MDPs with total discounted expected costs and constraints; For POMDPs, we show that, if the coherent risk measures can be defined as a Markov risk transition mapping, an infinite-dimensional optimization can be used to design Markovian belief-based policies. For POMDPs with stochastic finite-state controllers (FSCs), we show that the latter optimization simplifies to a (finite-dimensional) DCP. We incorporate these DCPs in a policy iteration algorithm to design risk-averse FSCs for POMDPs. We demonstrate the efficacy of the proposed method with numerical experiments involving conditional-value-at-risk (CVaR) and entropic-value-at-risk (EVaR) risk measures.

Joint user plane function instance and base station scheduling in mobile networks

AbstractTo guarantee a high data transmission rate in heterogeneous mobile networks, sufficient small base stations (SBSs) and user plane function (UPF) instances should be active. However, the excessive operation of SBSs and UPF instances can increase the operating expenditure (OPEX) for the network operator. To balance the data rate and OPEX, we propose a joint UPF instance–SBS scheduling algorithm (J‐UBSA). In the proposed J‐UBSA, a controller periodically determines the appropriate number of active SBSs and UPF instances based on estimated variations in session requests. The decision‐making process is formulated as a constrained Markov decision process and converted into a linear programming model to obtain the optimal solution using a traditional algorithm with low complexity. Evaluation results demonstrate that J‐UBSA can substantially reduce the OPEX while guaranteeing a suitable average data rate for mobile devices.

Application of Deep Learning in Rayleigh Dispersion Curves Inversion

The inversion of the Rayleigh wave dispersion curve is a fundamental step in determining the shear wave velocity profile. However, existing inversion methods, such as linear and nonlinear approaches, have inherent limitations. In order to overcome these challenges and enhance the speed and accuracy of the inversion, this paper proposes a novel deep learning-based approach. We establish a method to construct sample data that ensures ergodicity and evolutive orderliness of the near-surface velocity profile using a constrained Markov decision process to address these limitations. We then design a deep learning model to capture the nonlinear mapping relationship between the dispersion sequence data and velocity structure. Training the model on the generated sample data accurately predicts the subsurface velocity profile based on the dispersion curve. The proposed method is evaluated through theoretical model experiments, demonstrating high computational accuracy.

On Distributed Detection in EH-WSNs With Finite-State Markov Channel and Limited Feedback

We consider a network of N sensors, tasked with solving binary distributed detection, a fusion center (FC), and a feedback channel from the FC to sensors. Each sensor is capable of harvesting energy and is equipped with a finite-size battery to store randomly arrived energy. Sensors process their observations and transmit their symbols to the FC over orthogonal and Markovian time correlated fading channels. The FC fuses the received symbols and makes a global binary decision. We aim at developing adaptive channel-dependent transmit power control policies such that J-divergence based detection metric is maximized at the FC, subject to total transmit power constraint. Modeling quantized fading channel, energy arrival, and battery dynamics as time-homogeneous finite-state Markov chains, and the network lifetime as a geometric random variable, we formulate our power control optimization problem as a discounted infinite-horizon constrained Markov decision process (MDP) problem, where sensors’ transmit powers are functions of the battery states, quantized channel gains, and the arrived energies. We utilize stochastic dynamic programming and Lagrangian approach to find the optimal and sub-optimal power control policies. We demonstrate that our sub-optimal policy provides a close-to-optimal performance with a reduced computational complexity and without imposing signaling overhead on sensors.