Constrained Reinforcement Learning Research Articles

Data-driven constrained reinforcement learning algorithm for path tracking control of hovercraft

This paper presents a novel Constrained Backstepping Reinforcement Learning (CBRL) approach designed for path tracking of hovercraft. The approach addresses the challenges posed by complex modeling, multiple constraints, and large sideslip angles during high-speed maneuvers of underactuated hovercraft. Initially, a unique state constraint function is formulated, incorporating constraint boundaries related to navigation speed and path curvature. Additionally, transformations are applied to the yaw angular velocity and virtual yaw control law, ensuring that the yaw angular velocity remains within safety limits. Subsequently, a data-driven backstepping reinforcement learning optimal approach, coupled with a line-of-sight guidance law, is employed to guide the hovercraft along the intended path by regulating its yaw angle. Simultaneously, the backstepping reinforcement learning optimal control scheme is used to regulate the surge speed of the hovercraft above the resistance peak speed, thereby preventing excessive sideslip angles during path tracking. Neural network observers are integrated into the design process of both the yaw and surge controllers to monitor system dynamics in the presence of interference. Through simulation, the effectiveness and feasibility of the proposed CBRL algorithm are demonstrated.

Exoatmospheric Evasion Guidance Law with Total Energy Limit via Constrained Reinforcement Learning

AbstractDue to the lack of aerodynamic forces, the available propulsion for exoatmospheric pursuit-evasion problem is strictly limited, which has not been thoroughly investigated. This paper focuses on the evasion guidance in an exoatmospheric environment with total energy limit. A Constrained Reinforcement Learning (CRL) method is proposed to solve the problem. Firstly, the acceleration commands of the evader are defined as cost and an Actor-Critic-Cost (AC2) network structure is established to predict the accumulated cost of a trajectory. The learning objective of the agent becomes to maximize cumulative rewards while satisfying the cost constraint. Secondly, a Maximum-Minimum Entropy Learning (M2EL) method is proposed to minimize the randomness of acceleration commands while preserving the agent’s exploration capability. Our approaches address two challenges in the application of reinforcement learning: constraint specification and precise control. The well-trained agent is capable of generating accurate commands while satisfying the specified constraints. The simulation results indicate that the CRL and M2EL methods can effectively control the agent’s energy consumption within the specified constraints. The robustness of the agent under information error is also validated.

Enhancing Off-Policy Constrained Reinforcement Learning through Adaptive Ensemble C Estimation

In the domain of real-world agents, the application of Reinforcement Learning (RL) remains challenging due to the necessity for safety constraints. Previously, Constrained Reinforcement Learning (CRL) has predominantly focused on on-policy algorithms. Although these algorithms exhibit a degree of efficacy, their interactivity efficiency in real-world settings is sub-optimal, highlighting the demand for more efficient off-policy methods. However, off-policy CRL algorithms grapple with challenges in precise estimation of the C-function, particularly due to the fluctuations in the constrained Lagrange multiplier. Addressing this gap, our study focuses on the nuances of C-value estimation in off-policy CRL and introduces the Adaptive Ensemble C-learning (AEC) approach to reduce these inaccuracies. Building on state-of-the-art off-policy algorithms, we propose AEC-based CRL algorithms designed for enhanced task optimization. Extensive experiments on nine constrained robotics tasks reveal the superior interaction efficiency and performance of our algorithms in comparison to preceding methods.

Reward Penalties on Augmented States for Solving Richly Constrained RL Effectively

Constrained Reinforcement Learning employs trajectory-based cost constraints (such as expected cost, Value at Risk, or Conditional VaR cost) to compute safe policies. The challenge lies in handling these constraints effectively while optimizing expected reward. Existing methods convert such trajectory-based constraints into local cost constraints, but they rely on cost estimates, leading to either aggressive or conservative solutions with regards to cost. We propose an unconstrained formulation that employs reward penalties over states augmented with costs to compute safe policies. Unlike standard primal-dual methods, our approach penalizes only infeasible trajectories through state augmentation. This ensures that increasing the penalty parameter always guarantees a feasible policy, a feature lacking in primal-dual methods. Our approach exhibits strong empirical performance and theoretical properties, offering a fresh paradigm for solving complex Constrained RL problems, including rich constraints like expected cost, Value at Risk, and Conditional Value at Risk. Our experimental results demonstrate superior performance compared to leading approaches across various constraint types on multiple benchmark problems.

Constrained Reinforcement Learning for Predictive Control in Real-Time Stochastic Dynamic Optimal Power Flow

Constrained Reinforcement Learning for Predictive Control in Real-Time Stochastic Dynamic Optimal Power Flow

Learn Zero-Constraint-Violation Safe Policy in Model-Free Constrained Reinforcement Learning.

We focus on learning the zero-constraint-violation safe policy in model-free reinforcement learning (RL). Existing model-free RL studies mostly use the posterior penalty to penalize dangerous actions, which means they must experience the danger to learn from the danger. Therefore, they cannot learn a zero-violation safe policy even after convergence. To handle this problem, we leverage the safety-oriented energy functions to learn zero-constraint-violation safe policies and propose the safe set actor-critic (SSAC) algorithm. The energy function is designed to increase rapidly for potentially dangerous actions, locating the safe set on the action space. Therefore, we can identify the dangerous actions prior to taking them and achieve zero-constraint violation. Our major contributions are twofold. First, we use the data-driven methods to learn the energy function, which releases the requirement of known dynamics. Second, we formulate a constrained RL problem to solve the zero-violation policies. We prove that our Lagrangian-based constrained RL solutions converge to the constrained optimal zero-violation policies theoretically. The proposed algorithm is evaluated on the complex simulation environments and a hardware-in-loop (HIL) experiment with a real autonomous vehicle controller. Experimental results suggest that the converged policies in all environments achieve zero-constraint violation and comparable performance with model-based baseline.

Motion Planner With Fixed-Horizon Constrained Reinforcement Learning for Complex Autonomous Driving Scenarios

In autonomous driving, behavioral decision-making and trajectory planning remain huge challenges due to the large amount of uncertainty in environments and complex interaction relationships between the ego vehicle and other traffic participants. In this paper, we propose a novel fixed-horizon constrained reinforcement learning (RL) framework to solve decision-making and planning problems. Firstly, to introduce lane-level global navigation information into the lane state representation and avoid constant lane changes, we propose the constrained A-star algorithm, which can get the optimal path without constant lane changes. The optimality of the algorithm is also theoretically guaranteed. Then, to balance safety, comfort, and goal completion (reaching targets), we construct the planning problem as a constrained RL problem, in which the reward function is designed for goal completion, and two fixed-horizon constraints are developed for safety and comfort, respectively. Subsequently, a motion planning policy network (planner) with vectorized input is constructed. Finally, a dual ascent optimization method is proposed to train the planner network. With the advantage of being able to fully explore in the environment, the agent can learn an efficient decision-making and planning policy. In addition, benefiting from modeling the safety and comfort of the ego vehicle as constraints, the learned policy can guarantee the safety of the ego vehicle and achieve a good balance between goal completion and comfort. Experiments demonstrate that the proposed algorithm can achieve superior performance than existing rule-based, imitation learning-based, and typical RL-based methods.

CVaR-Constrained Policy Optimization for Safe Reinforcement Learning.

Current constrained reinforcement learning (RL) methods guarantee constraint satisfaction only in expectation, which is inadequate for safety-critical decision problems. Since a constraint satisfied in expectation remains a high probability of exceeding the cost threshold, solving constrained RL problems with high probabilities of satisfaction is critical for RL safety. In this work, we consider the safety criterion as a constraint on the conditional value-at-risk (CVaR) of cumulative costs, and propose the CVaR-constrained policy optimization algorithm (CVaR-CPO) to maximize the expected return while ensuring agents pay attention to the upper tail of constraint costs. According to the bound on the CVaR-related performance between two policies, we first reformulate the CVaR-constrained problem in augmented state space using the state extension procedure and the trust-region method. CVaR-CPO then derives the optimal update policy by applying the Lagrangian method to the constrained optimization problem. In addition, CVaR-CPO utilizes the distribution of constraint costs to provide an efficient quantile-based estimation of the CVaR-related value function. We conduct experiments on constrained control tasks to show that the proposed method can produce behaviors that satisfy safety constraints, and achieve comparable performance to most safe RL (SRL) methods.

State Augmented Constrained Reinforcement Learning: Overcoming the Limitations of Learning With Rewards

State Augmented Constrained Reinforcement Learning: Overcoming the Limitations of Learning With Rewards

Stability Constrained Reinforcement Learning for Decentralized Real-Time Voltage Control

Stability Constrained Reinforcement Learning for Decentralized Real-Time Voltage Control

Uniformly constrained reinforcement learning

Uniformly constrained reinforcement learning

Gradient-Adaptive Pareto Optimization for Constrained Reinforcement Learning

Constrained Reinforcement Learning (CRL) burgeons broad interest in recent years, which pursues maximizing long-term returns while constraining costs. Although CRL can be cast as a multi-objective optimization problem, it is still facing the key challenge that gradient-based Pareto optimization methods tend to stick to known Pareto-optimal solutions even when they yield poor returns (e.g., the safest self-driving car that never moves) or violate the constraints (e.g., the record-breaking racer that crashes the car). In this paper, we propose Gradient-adaptive Constrained Policy Optimization (GCPO for short), a novel Pareto optimization method for CRL with two adaptive gradient recalibration techniques. First, to find Pareto-optimal solutions with balanced performance over all targets, we propose gradient rebalancing which forces the agent to improve more on under-optimized objectives at every policy iteration. Second, to guarantee that the cost constraints are satisfied, we propose gradient perturbation that can temporarily sacrifice the returns for costs. Experiments on the SafetyGym benchmarks show that our method consistently outperforms previous CRL methods in reward while satisfying the constraints.

Constrained Reinforcement Learning in Hard Exploration Problems

One approach to guaranteeing safety in Reinforcement Learning is through cost constraints that are dependent on the policy. Recent works in constrained RL have developed methods that ensure constraints are enforced even at learning time while maximizing the overall value of the policy. Unfortunately, as demonstrated in our experimental results, such approaches do not perform well on complex multi-level tasks, with longer episode lengths or sparse rewards. To that end, we propose a scalable hierarchical approach for constrained RL problems that employs backward cost value functions in the context of task hierarchy and a novel intrinsic reward function in lower levels of the hierarchy to enable cost constraint enforcement. One of our key contributions is in proving that backward value functions are theoretically viable even when there are multiple levels of decision making. We also show that our new approach, referred to as Hierarchically Limited consTraint Enforcement (HiLiTE) significantly improves on state of the art Constrained RL approaches for many benchmark problems from literature. We further demonstrate that this performance (on value and constraint enforcement) clearly outperforms existing best approaches for constrained RL and hierarchical RL.

AutoCost: Evolving Intrinsic Cost for Zero-Violation Reinforcement Learning

Safety is a critical hurdle that limits the application of deep reinforcement learning to real-world control tasks. To this end, constrained reinforcement learning leverages cost functions to improve safety in constrained Markov decision process. However, constrained methods fail to achieve zero violation even when the cost limit is zero. This paper analyzes the reason for such failure, which suggests that a proper cost function plays an important role in constrained RL. Inspired by the analysis, we propose AutoCost, a simple yet effective framework that automatically searches for cost functions that help constrained RL to achieve zero-violation performance. We validate the proposed method and the searched cost function on the safety benchmark Safety Gym. We compare the performance of augmented agents that use our cost function to provide additive intrinsic costs to a Lagrangian-based policy learner and a constrained-optimization policy learner with baseline agents that use the same policy learners but with only extrinsic costs. Results show that the converged policies with intrinsic costs in all environments achieve zero constraint violation and comparable performance with baselines.

Data-driven constrained reinforcement learning for optimal control of a multistage evaporation process

It is challenging for reinforcement learning to solve the optimal control problem of industrial processes with constraints under uncertain operating conditions. In this context, this paper proposes a novel data-driven constrained reinforcement learning algorithm for the optimal control of a multistage evaporation process consisting of multiple evaporators in series with coupled liquid levels. We first formulate the optimal control problem as a constrained Markov decision process. Then, with the cumulative tracking error of the outlet liquor density taken as the cumulative constraint, a Lagrangian-based constrained policy optimization is developed. The fast setpoint tracking is achieved by gradient iteration of the policy and the dual variable. An action correction layer based on the online sequential version of random vector functional-link networks is built on the output of the policy network to address the instantaneous constraints of the liquid levels. The infeasible action is corrected in real-time so as to keep the liquid levels in each evaporator within operating range. Finally, we utilize both on-policy and off-policy data generated by the interaction between the constrained policy and the evaporation environment to update our algorithm, which is more data-efficient. Experiments have been carried out on a multistage evaporation system, and the results validate the effectiveness of the proposed algorithm.

Constrained Reinforcement Learning for Vehicle Motion Planning with Topological Reachability Analysis

Rule-based traditional motion planning methods usually perform well with prior knowledge of the macro-scale environments but encounter challenges in unknown and uncertain environments. Deep reinforcement learning (DRL) is a solution that can effectively deal with micro-scale unknown and uncertain environments. Nevertheless, DRL is unstable and lacks interpretability. Therefore, it raises a new challenge: how to combine the effectiveness and overcome the drawbacks of the two methods while guaranteeing stability in uncertain environments. In this study, a multi-constraint and multi-scale motion planning method is proposed for automated driving with the use of constrained reinforcement learning (RL), named RLTT, and comprising RL, a topological reachability analysis used for vehicle path space (TPS), and a trajectory lane model (TLM). First, a dynamic model of vehicles is formulated; then, TLM is developed on the basis of the dynamic model, thus constraining RL action and state space. Second, macro-scale path planning is achieved through TPS, and in the micro-scale range, discrete routing points are achieved via RLTT. Third, the proposed motion planning method is designed by combining sophisticated rules, and a theoretical analysis is provided to guarantee the efficiency of our method. Finally, related experiments are conducted to evaluate the effectiveness of the proposed method; our method can reduce 19.9% of the distance cost in the experiments as compared to the traditional method. Experimental results indicate that the proposed method can help mitigate the gap between data-driven and traditional methods, provide better performance for automated driving, and facilitate the use of RL methods in more fields.

Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Primal-Dual Approach

Reinforcement learning is widely used in applications where one needs to perform sequential decisions while interacting with the environment. The problem becomes more challenging when the decision requirement includes satisfying some safety constraints. The problem is mathematically formulated as constrained Markov decision process (CMDP). In the literature, various algorithms are available to solve CMDP problems in a model-free manner to achieve epsilon-optimal cumulative reward with epsilon feasible policies. An epsilon-feasible policy implies that it suffers from constraint violation. An important question here is whether we can achieve epsilon-optimal cumulative reward with zero constraint violations or not. To achieve that, we advocate the use of a randomized primal-dual approach to solve the CMDP problems and propose a conservative stochastic primal-dual algorithm (CSPDA) which is shown to exhibit O(1/epsilon^2) sample complexity to achieve epsilon-optimal cumulative reward with zero constraint violations. In the prior works, the best available sample complexity for the epsilon-optimal policy with zero constraint violation is O(1/epsilon^5). Hence, the proposed algorithm provides a significant improvement compared to the state of the art.

Reinforcement learning-based integrated active fault diagnosis and tracking control

Reliable and real-time active diagnosis of system faults with uncertainties is strongly dependent on the input design. This paper establishes a data-driven framework for integrated design of active fault diagnosis and control while ensuring the tracking performance. To be specific, the input design is formulated as a constrained optimization problem that can be solved with the aid of constrained reinforcement learning algorithms. Moreover, based on the maximum mean discrepancy metric, a novel active fault isolation scheme is proposed to implement model discrimination using system outputs. At the end, the effectiveness of the proposed approach is evaluated in two case studies in the presence of probabilistic disturbances and uncertainties.

Model-Free design of control systems over wireless fading channels

Wireless control systems rely on wireless networks to exchange information between spatially distributed actuators, plants and sensors. The noise in wireless channels renders traditional control policies suboptimal, and their performance is moreover directly dependent on the allocation of wireless resources between control loops sharing the wireless medium, making the design of control and resource allocation policies critical to achieve reliable performance of the wireless control system. The co-design of control-aware resource allocation policies and communication-aware controllers, however, is a challenging problem due to its infinite dimensionality, existence of constraints on the use of communication resources or performance of individual plants, and need for explicit knowledge of the plants and wireless network models. To overcome those challenges, we propose a constrained reinforcement learning approach to design model-free control-aware communication policies and communication-aware control policies for wireless control systems. We demonstrate the near optimality of control system performance and stability using near-universal policy parametrizations and present a practical model-free algorithm to learn the co-design policy. Numerical experiments show the strong performance of learned policies over baseline solutions.

An SDN Controller-Based Network Slicing Scheme Using Constrained Reinforcement Learning

In order to meet the strong diversification of services that demand network flexibility that will be able to serve the dire need for transmission resources, network slicing was embraced as a plausible solution. Reinforcement learning (RL) has been applied in resource allocation (RA) problems, but has not yet marked the translation from traditional optimization approaches primarily due to its inability to satisfy state constraints. The aim of this article is to address this challenge. This article proposes a logical architecture for network slicing based on software-defined networking (SDN), where an SDN controller controls the network slicing process in a centralized fashion, and manages the resource allocation (RA) process with the help of the slice manager. The considered problem jointly addresses power and channel allocation using a hybrid access mode for ultra-reliable low-latency communication (URLLC) and enhanced mobile broadband (eMBB) slices. Proper assumptions on the arrival rates, packet length distributions, as well as power and delay constraints were used to design the behavior of the reward function to realize a constrained RL approach. Here, the Bellman optimality equation was reformulated into a primal-dual optimization problem through the use of Nesterov’s smoothing technique and the Legendre-Fenchel transformation. The proposed algorithm shows favorable performance over the traditional RL strategy in attributes favoring eMBB services, i.e., the average bit rate, and significantly outperforms both baselines in attributes favoring URLLC services, i.e., average latency. Systematically, on the power-delay performance evaluation, it shows that it can adapt very well in rapidly time-varying non-Markovian environments and still successfully satisfy the delay constraints of the applications hosted on a slice.