Safe Reinforcement Learning Research Articles

Safe deep reinforcement learning in diesel engine emission control.

A deep reinforcement learning application is investigated to control the emissions of a compression ignition diesel engine. The main purpose of this study is to reduce the engine-out nitrogen oxide emissions and to minimize fuel consumption while tracking a reference engine load. First, a physics-based engine simulation model is developed in GT-Power and calibrated using experimental data. Using this model and a GT-Power/Simulink co-simulation, a deep deterministic policy gradient is developed. To reduce the risk of an unwanted output, a safety filter is added to the deep reinforcement learning. Based on the simulation results, this filter has no effect on the final trained deep reinforcement learning; however, during the training process, it is crucial to enforce constraints on the controller output. The developed safe reinforcement learning is then compared with an iterative learning controller and a deep neural network-based nonlinear model predictive controller. This comparison shows that the safe reinforcement learning is capable of accurately tracking an arbitrary reference input while the iterative learning controller is limited to a repetitive reference. The comparison between the nonlinear model predictive control and reinforcement learning indicates that for this case reinforcement learning is able to learn the optimal control output directly from the experiment without the need for a model. However, to enforce output constraint for safe learning reinforcement learning, a simple model of system is required. In this work, reinforcement learning was able to reduce emissions more than the nonlinear model predictive control; however, it suffered from slightly higher error in load tracking and a higher fuel consumption.

Safe reinforcement learning under temporal logic with reward design and quantum action selection

This paper proposes an advanced Reinforcement Learning (RL) method, incorporating reward-shaping, safety value functions, and a quantum action selection algorithm. The method is model-free and can synthesize a finite policy that maximizes the probability of satisfying a complex task. Although RL is a promising approach, it suffers from unsafe traps and sparse rewards and becomes impractical when applied to real-world problems. To improve safety during training, we introduce a concept of safety values, which results in a model-based adaptive scenario due to online updates of transition probabilities. On the other hand, a high-level complex task is usually formulated via formal languages, including Linear Temporal Logic (LTL). Another novelty of this work is using an Embedded Limit-Deterministic Generalized Büchi Automaton (E-LDGBA) to represent an LTL formula. The obtained deterministic policy can generalize the tasks over infinite and finite horizons. We design an automaton-based reward, and the theoretical analysis shows that an agent can accomplish task specifications with the maximum probability by following the optimal policy. Furthermore, a reward shaping process is developed to avoid sparse rewards and enforce the RL convergence while keeping the optimal policies invariant. In addition, inspired by quantum computing, we propose a quantum action selection algorithm to replace the existing varepsilon-greedy algorithm for the balance of exploration and exploitation during learning. Simulations demonstrate how the proposed framework can achieve good performance by dramatically reducing the times to visit unsafe states while converging optimal policies.

Secure energy management of multi-energy microgrid: A physical-informed safe reinforcement learning approach

The large-scale integration of distributed energy resources into the energy industry enables the fast transition to a decarbonized future but raises some potential challenges of insecure and unreliable operations. Multi-energy Microgrids (MEMGs), as localized small multi-energy systems, can effectively integrate a variety of energy components with multiple energy sectors, which have been recently recognized as a valid solution to improve the operational security and reliability. As a result, a massive amount of research has been conducted to investigate MEMG energy management problems, including both model-based optimization and model-free learning approaches. Compared to optimization approaches, reinforcement learning is being widely deployed in MEMG energy management problems owing to its ability to handle highly dynamic and stochastic processes without knowing any system knowledge. However, it is still difficult for conventional model-free reinforcement learning methods to capture the physical constraints of the MEMG model, which may therefore destroy its secure operation. To address this research challenge, this paper proposes a novel safe reinforcement learning method by learning a dynamic security assessment rule to abstract a physical-informed safety layer on top of the conventional model-free reinforcement learning energy management policy, which can respect all the physical constraints through mathematically solving an action correction formulation. In this setting, the secure energy management of the MEMG can be guaranteed for both training and test procedures. Extensive case studies based on two integrated systems (i.e., a small 6-bus power and 7-node gas network, and a large 33-bus power and 20-node gas network) are carried out to verify the superior performance of the proposed physical-informed reinforcement learning method in achieving a cost-effective MEMG energy management performance while respecting all the physical constraints, compared to conventional reinforcement learning and optimization approaches.

Knowledge-based integrated optimization design of agile imaging satellites' attitude controller and vibration isolator

The integrated design of modern agile imaging satellites' systems demands multidisciplinary optimization to ensure the satisfaction of image quality and acquisition efficiency requirements. Beyond the scope of traditional integrated design methods that optimize parameters, the policy to vary parameters over time is required to be optimized especially for changing environments and objectives. Though modern optimization methods provide powerful tools, knowledge about the dynamic system is indispensable to guarantee credibility for truly intelligent integrated design. This paper proposes an integrated control and structural optimization design scheme to optimize variables of the full control-structure system in order to meet conflicting constraints on the attitude control stability requirement and the vibration isolation requirement. To reduce the workload of the iterative design, a safe reinforcement learning algorithm is proposed to learn the time-varying parameters of the camera's vibration isolator and the attitude controller. Our design philosophy is to combine priori knowledge about the non-collocation control stability into the reinforcement learning optimization to safely optimize the control-structure design. The constraint satisfaction is achieved with an improved safety layer technique applied to the output policy of the reinforcement learning agent, which solves a nonlinear constraint programming problem to timely confine learned parameters. We exemplify the proposed method on a push-broom imaging case and a backward-forward slewing case, successfully optimizing the effective imaging time under constraints.

Safe Reinforcement Learning with Contextual Information: Theory and Application to Personalized Comorbidity Management

Safe Reinforcement Learning with Contextual Information: Theory and Application to Personalized Comorbidity Management

Safe Reinforcement Learning for Autonomous Navigation of a Driveable Vertical Mast Lift

In this work, we consider the issue of controlling a Driveable Vertical Mast Lift (DVML) to autonomously navigate an environment while ensuring required safety constraints. DMVL are industrial vehicles used in several applications, f.i. in logistics and smart agriculture, to allow operators placed in a basket accessing elevated worksites. When driving such machines, operators are exposed to hazards that could lead to potentially serious accidents. Reinforcement Learning (RL) is a data-driven approach that is increasingly being used to control complex systems. This work investigates the advancements in the field of Safe RL from a practical perspective, employing several state-of-art algorithms to equip a DVML with autonomous driving capabilities. We highlight how benchmark environments, while satisfactorily affirming Safe RL methodologies as proof-of-concepts, can widen the gap that prevents such methodologies from both being applied in real scenarios and becoming much more popular in industrial use-cases.

Multi-Agent Partial Observable Safe Reinforcement Learning for Counter Uncrewed Aerial Systems

Multi-Agent Partial Observable Safe Reinforcement Learning for Counter Uncrewed Aerial Systems

Provably Safe Reinforcement Learning via Action Projection Using Reachability Analysis and Polynomial Zonotopes

While reinforcement learning produces very promising results for many applications, its main disadvantage is the lack of safety guarantees, which prevents its use in safety-critical systems. In this work, we address this issue by a safety shield for nonlinear continuous systems that solve reach-avoid tasks. Our safety shield prevents applying potentially unsafe actions from a reinforcement learning agent by projecting the proposed action to the closest safe action. This approach is called action projection and is implemented via mixed-integer optimization. The safety constraints for action projection are obtained by applying parameterized reachability analysis using polynomial zonotopes, which enables to accurately capture the nonlinear effects of the actions on the system. In contrast to other state-of-the-art approaches for action projection, our safety shield can efficiently handle input constraints and dynamic obstacles, eases incorporation of the spatial robot dimensions into the safety constraints, guarantees robust safety despite process noise and measurement errors, and is well suited for high-dimensional systems, as we demonstrate on several challenging benchmark systems.

Safe Reinforcement Learning for Single Train Trajectory Optimization via Shield SARSA

The single train trajectory optimization, also known as speed profile optimization (SPO), is a traditional problem to minimize the traction energy consumption of trains. As a kind of optimal method, reinforcement learning (RL) has been used to solve the SPO problem. In the learning process of a common RL algorithm, a soft constraint (punishment) is always used to keep the agent away from unsafe states. However, a soft constraint can not guarantee and explain the safety of the result. For the SPO problem, it means that the optimized speed profile obtained by a simple RL may break the speed limit which is unacceptable in reality. This paper proposes a protection mechanism called Shield and constructs a Shield SARSA ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${S}$ </tex-math></inline-formula> -SARSA) algorithm to protect the learning process of the high-speed train. Four different reward functions are used to compare the protective efficacy between the proposed algorithm and the soft constraint. The numerical experiments based on the line data from Wuxi East to Suzhou North verify the protective efficacy and effectiveness.

Safe reinforcement learning for high-speed autonomous racing

Safe reinforcement learning for high-speed autonomous racing

Learning safety in model-based Reinforcement Learning using MPC and Gaussian Processes

This paper proposes a method to encourage safety in Model Predictive Control (MPC)-based Reinforcement Learning (RL) via Gaussian Process (GP) regression. The framework consists of 1) a parametric MPC scheme that is employed as model-based controller with approximate knowledge on the real system's dynamics, 2) an episodic RL algorithm tasked with adjusting the MPC parametrization in order to increase its performance, and 3) GP regressors used to estimate, directly from data, constraints on the MPC parameters capable of predicting, up to some probability, whether the parametrization is likely to yield a safe or unsafe policy. These constraints are then enforced onto the RL updates in an effort to enhance the learning method with a probabilistic safety mechanism. Compared to other recent publications combining safe RL with MPC, our method does not require further assumptions on, e.g., the prediction model in order to retain computational tractability. We illustrate the results of our method in a numerical example on the control of a quadrotor drone in a safety-critical environment.

Safe reinforcement learning for multi-energy management systems with known constraint functions

Reinforcement learning (RL) is a promising optimal control technique for multi-energy management systems. It does not require a model a priori - reducing the upfront and ongoing project-specific engineering effort and is capable of learning better representations of the underlying system dynamics. However, vanilla RL does not provide constraint satisfaction guarantees — resulting in various potentially unsafe interactions within its environment. In this paper, we present two novel online model-free safe RL methods, namely SafeFallback and GiveSafe, where the safety constraint formulation is decoupled from the RL formulation. These provide hard-constraint satisfaction guarantees both during training and deployment of the (near) optimal policy. This is without the need of solving a mathematical program, resulting in less computational power requirements and more flexible constraint function formulations. In a simulated multi-energy systems case study we have shown that both methods start with a significantly higher utility compared to a vanilla RL benchmark and Optlayer benchmark (94,6% and 82,8% compared to 35,5% and 77,8%) and that the proposed SafeFallback method even can outperform the vanilla RL benchmark (102,9% to 100%). We conclude that both methods are viably safety constraint handling techniques applicable beyond RL, as demonstrated with random policies while still providing hard-constraint guarantees.

Safe reinforcement learning for dynamical systems using barrier certificates

Safety control is a fundamental problem in policy design. Basic reinforcement learning is effective at learning policy with goal-reaching property. However, it does not guarantee safety property of the learned policy. This paper integrates barrier certificates into actor-critic-based reinforcement learning methods in a feedback-driven framework to learn safe policies for dynamical systems. The safe reinforcement learning framework is composed of two interactive parts: Learner and Verifier. Learner trains the policy to satisfy goal-reaching and safety properties. Since the policy is trained on training datasets, the two properties may not be retained on the whole system. Verifier validates the learned policy on the whole system. If the validation fails, Verifier returns the counterexamples to Learner for retraining the policy in the next iteration. We implement a safe policy learning tool SRLBC and evaluate its performance on three control tasks. Experimental results show that SRLBC achieves safety with no more than 0.5× time overhead compared to the baseline reinforcement learning method, showing the feasibility and effectiveness of our framework.

Optimal dispatch of an energy hub with compressed air energy storage: A safe reinforcement learning approach

With the advancements in renewable energy and energy storage technologies, the energy hubs (EH) have been emerging in recent years. The scheduling of EH is a challenging task due to the need to incorporate uncertainties at energy supply and load side. The existing model-based optimization approaches proposed for the above purpose have limitations in terms of solution accuracy and computational efficiency, which makes hinders their applications. This paper proposes a model-free, safe deep reinforcement learning (DRL) approach, using primal-dual optimization and imitation learning, for optimal scheduling of an EH that includes a tri-generative advanced adiabatic compressed air energy storage (AA-CAES). First, the operation of an AA-CAES under off-design conditions is modeled and linearized using a mixed-integer linear programming (MILP). Then, a safe DRL approach is proposed with training and testing steps considering a case study. The performance of the proposed approach in reducing operational cost and satisfying constraints is compared to the state-of-the-art DRL algorithms as well as a deterministic MILP approach. In addition, the generalization of the proposed approach is examined on a test-set. Finally, the effect of off-design conditions of a tri-generative AA-CAES on the optimal dispatch strategy is investigated. The results indicate that the proposed approach can effectively reduce the operational cost and satisfy the operational constraints.

Safe reinforcement learning for affine nonlinear systems with state constraints and input saturation using control barrier functions

This paper provides a novel safe reinforcement learning (RL) control algorithm to solve safe optimal problems for discrete-time affine nonlinear systems, while the safety and convergence of the control algorithm are proven. The algorithm is proposed based on an adjusted policy iteration (PI) framework using only the measured data along the system trajectories in the environment. The adjusted PI algorithm combines with the system predictive information. Unlike most PI algorithms, an effective method of obtaining an initial safe and stable control policy is given here. In addition, control barrier functions (CBFs) and an input constraint function are introduced to augment reward functions. And the monotonically nonincreasing property of the iterative value function maintains the safe set forward invariant in the PI framework. Moreover, the safety and convergence of the proposed algorithm are proven in theory. Then, the design and implementation of the proposed algorithm are presented based on the identifier-actor-critic structure, where neural networks are employed to approximate the system dynamics, the iterative control policy, and the iterative value function, respectively. Finally, the simulation results illustrate the effectiveness and safety of the proposed algorithm.

Safe reinforcement learning for discrete-time fully cooperative games with partial state and control constraints using control barrier functions

In this paper, a novel safe reinforcement learning is proposed for fully cooperative games of discrete-time multi-player systems with partial state and control constraints. The fully cooperative game is a special case of the nonzero-sum games where all players cooperate to accomplish the common task. However, there are few works for fully cooperative game issues of discrete-time systems with partial state and control constraints. The issue is addressed by our algorithm based on the constrained value iteration framework using the measured data along the system trajectories, and the Nash equilibrium of the constrained fully cooperative game is achieved. Compared to previous methods for fully cooperative game issues, neither the accurate system dynamics nor the initial admissible control policies are required via the algorithm. Meanwhile, the discrete-time exponential control barrier functions are adopted to address the issue of state constraints. Moreover, the convergence of the proposed algorithm is proven in theory. Then, the system dynamics, the control policies and the value function are approximated by the three-layer neural networks, respectively. Finally, two experiments are presented to demonstrate the safety and effectiveness of the proposed algorithm.

Sim-to-Lab-to-Real: Safe reinforcement learning with shielding and generalization guarantees

Safety is a critical component of autonomous systems and remains a challenge for learning-based policies to be utilized in the real world. In particular, policies learned using reinforcement learning often fail to generalize to novel environments due to unsafe behavior. In this paper, we propose Sim-to-Lab-to-Real to bridge the reality gap with a probabilistically guaranteed safety-aware policy distribution. To improve safety, we apply a dual policy setup where a performance policy is trained using the cumulative task reward and a backup (safety) policy is trained by solving the Safety Bellman Equation based on Hamilton-Jacobi (HJ) reachability analysis. In Sim-to-Lab transfer, we apply a supervisory control scheme to shield unsafe actions during exploration; in Lab-to-Real transfer, we leverage the Probably Approximately Correct (PAC)-Bayes framework to provide lower bounds on the expected performance and safety of policies in unseen environments. Additionally, inheriting from the HJ reachability analysis, the bound accounts for the expectation over the worst-case safety in each environment. We empirically study the proposed framework for ego-vision navigation in two types of indoor environments with varying degrees of photorealism. We also demonstrate strong generalization performance through hardware experiments in real indoor spaces with a quadrupedal robot. See https://sites.google.com/princeton.edu/sim-to-lab-to-real for supplementary material.

Proactive Load Balancing Through Constrained Policy Optimization for Ultra-Dense Networks

Designing an intelligent self-organizing network (SON) architecture is challenging for future wireless networks. To meet the needs of SON, the reactive self-organizing model of the traditional network needs to be transformed into an active and interactive one. Due to the user mobility and small coverage of cells in ultra-dense networks (UDNs), the network load usually becomes unbalanced, leading to deteriorated network performance, such as low throughput, radio link failure, and poor user experience. Therefore, the technique of mobility load balancing (MLB) is critical to ensuring a seamless user experience among cells. This letter proposes an active and interactive MLB strategy for UDNs, which transforms the original reactive MLB into a forward-aware and active one. In particular, user mobility is first predicted based on the Bayesian additive regression tree (BART). Then, with the mobility predictions, the joint mobility robust optimization and MLB problem subject to users’ rate constraint is solved via safe reinforcement learning. The pertaining simulation results show that the proposed method can improve the network performance and realize intelligent mobile management for future UDNs.

Safe Reinforcement Learning Using Black-Box Reachability Analysis

Reinforcement learning (RL) is capable of sophisticated motion planning and control for robots in uncertain environments. However, state-of-the-art deep RL approaches typically lack safety guarantees, especially when the robot and environment models are unknown. To justify widespread deployment, robots must respect safety constraints without sacrificing performance. Thus, we propose a Black-box Reachability-based Safety Layer (BRSL) with three main components: (1) data-driven reachability analysis for a black-box robot model, (2) a trajectory rollout planner that predicts future actions and observations using an ensemble of neural networks trained online, and (3) a differentiable polytope collision check between the reachable set and obstacles that enables correcting unsafe actions. In simulation, BRSL outperforms other state-of-the-art safe RL methods on a Turtlebot 3, a quadrotor, a trajectory-tracking point mass, and a hexarotor in wind with an unsafe set adjacent to the area of highest reward.

Decentralized safe reinforcement learning for inverter-based voltage control

Inverter-based distributed energy resources provide the possibility for fast time-scale voltage control by quickly adjusting their reactive power. The power-electronic interfaces allow these resources to realize almost arbitrary control law, but designing these decentralized controllers is nontrivial. Reinforcement learning (RL) approaches are becoming increasingly popular to search for policy parameterized by neural networks. It is difficult, however, to enforce that the learned controllers are safe, in the sense that they may introduce instabilities into the system.This paper proposes a safe learning approach for voltage control. We prove that the system is guaranteed to be exponentially stable if each controller satisfies certain Lipschitz constraints. The set of Lipschitz bound is optimized to enlarge the search space for neural network controllers. We explicitly engineer the structure of neural network controllers such that they satisfy the Lipschitz constraints by design. A decentralized RL framework is constructed to train local neural network controller at each bus in a model-free setting.