Robust Reinforcement Learning Research Articles

Trustworthy autonomous driving via defense-aware robust reinforcement learning against worst-case observational perturbations

Despite the substantial advancements in reinforcement learning (RL) in recent years, ensuring trustworthiness remains a formidable challenge when applying this technology to safety-critical autonomous driving domains. One pivotal bottleneck is that well-trained driving policy models may be particularly vulnerable to observational perturbations or perceptual uncertainties, potentially leading to severe failures. In view of this, we present a novel defense-aware robust RL approach tailored for ensuring the robustness and safety of autonomous vehicles in the face of worst-case attacks on observations. The proposed paradigm primarily comprises two crucial modules: an adversarial attacker and a robust defender. Specifically, the adversarial attacker is devised to approximate the worst-case observational perturbations that attempt to induce safety violations (e.g., collisions) in the RL-driven autonomous vehicle. Additionally, the robust defender is developed to facilitate the safe RL agent to learn robust optimal policies that maximize the return while constraining the policy and cost perturbed by the adversarial attacker within specified bounds. Finally, the proposed technique is assessed across three distinct traffic scenarios: highway, on-ramp, and intersection. The simulation and experimental results indicate that our scheme enables the agent to execute trustworthy driving policies, even in the presence of the worst-case observational perturbations.

Improve Robustness of Reinforcement Learning against Observation Perturbations via l∞ Lipschitz Policy Networks

Deep Reinforcement Learning (DRL) has achieved remarkable advances in sequential decision tasks. However, recent works have revealed that DRL agents are susceptible to slight perturbations in observations. This vulnerability raises concerns regarding the effectiveness and robustness of deploying such agents in real-world applications. In this work, we propose a novel robust reinforcement learning method called SortRL, which improves the robustness of DRL policies against observation perturbations from the perspective of the network architecture. We employ a novel architecture for the policy network that incorporates global $l_\infty$ Lipschitz continuity and provide a convenient method to enhance policy robustness based on the output margin. Besides, a training framework is designed for SortRL, which solves given tasks while maintaining robustness against $l_\infty$ bounded perturbations on the observations. Several experiments are conducted to evaluate the effectiveness of our method, including classic control tasks and video games. The results demonstrate that SortRL achieves state-of-the-art robustness performance against different perturbation strength.

Active Reinforcement Learning for Robust Building Control

Reinforcement learning (RL) is a powerful tool for optimal control that has found great success in Atari games, the game of Go, robotic control, and building optimization. RL is also very brittle; agents often overfit to their training environment and fail to generalize to new settings. Unsupervised environment design (UED) has been proposed as a solution to this problem, in which the agent trains in environments that have been specially selected to help it learn. Previous UED algorithms focus on trying to train an RL agent that generalizes across a large distribution of environments. This is not necessarily desirable when we wish to prioritize performance in one environment over others. In this work, we will be examining the setting of robust RL building control, where we wish to train an RL agent that prioritizes performing well in normal weather while still being robust to extreme weather conditions. We demonstrate a novel UED algorithm, ActivePLR, that uses uncertainty-aware neural network architectures to generate new training environments at the limit of the RL agent's ability while being able to prioritize performance in a desired base environment. We show that ActivePLR is able to outperform state-of-the-art UED algorithms in minimizing energy usage while maximizing occupant comfort in the setting of building control.

Tube-based robust reinforcement learning for autonomous maneuver decision for UCAVs

Reinforcement Learning (RL) algorithms enhance intelligence of air combat Autonomous Maneuver Decision (AMD) policy, but they may underperform in target combat environments with disturbances. To enhance the robustness of the AMD strategy learned by RL, this study proposes a Tube-based Robust RL (TRRL) method. First, this study introduces a tube to describe reachable trajectories under disturbances, formulates a method for calculating tubes based on sum-of-squares programming, and proposes the TRRL algorithm that enhances robustness by utilizing tube size as a quantitative indicator. Second, this study introduces offline techniques for regressing the tube size function and establishing a tube library before policy learning, aiming to eliminate complex online tube solving and reduce the computational burden during training. Furthermore, an analysis of the tube library demonstrates that the mitigated AMD strategy achieves greater robustness, as smaller tube sizes correspond to more cautious actions. This finding highlights that TRRL enhances robustness by promoting a conservative policy. To effectively balance aggressiveness and robustness, the proposed TRRL algorithm introduces a “laziness factor” as a weight of robustness. Finally, combat simulations in an environment with disturbances confirm that the AMD policy learned by the TRRL algorithm exhibits superior air combat performance compared to selected robust RL baselines.

Reinforcement learning-based event-triggered secondary control of DC microgrids

In this paper, a reinforcement learning (RL)-based event-triggered mechanism (ETM) for employing in the secondary control layer (SCL) of DC microgrids is developed. The proposed RL-based ETM satisfies the SCL objectives, which is overcoming the disadvantages of primary control (such as voltage deviation and inappropriate current sharing among the distributed generating units). More importantly, it also aids in reducing the amount of transmitted data exchanged within all the distributed generators (DGs). The design parameters of the ETM scheme are regulated through a robust RL approach to provide adaptive ETM parameter tuning, enabling the ETM error vector threshold to quickly adapt to changes in the MG. The suggested RL-based ETM approach is implemented in a DC microgrid, and utilizing hardware in the loop (HIL) real-time OPAL-RT experimental tests, its performance in the SCL of DC microgrids is investigated. Experimental validations have confirmed the merits of the proposed approach.

On Practical Robust Reinforcement Learning: Adjacent Uncertainty Set and Double-Agent Algorithm.

Robust reinforcement learning (RRL) aims to seek a robust policy by optimizing the worst case performance over an uncertainty set. This set contains some perturbed Markov decision processes (MDPs) from a nominal MDP (N-MDP) that generate samples for training, which reflects some potential mismatches between the training simulator (i.e., N-MDP) and real-world settings (i.e., the testing environments). Unfortunately, existing RRL algorithms are only applied to the tabular setting and it is still an open problem to extend them into more general continuous state space. We contribute to this subject in the following ways. We first construct an elaborated uncertainty set, which contains plausible (perturbed) MDPs only compared with the existing sets. Based on this, we propose a sample-based RRL algorithm named adjacent robust Q-learning (ARQ-Learning) for the tabular setting and characterize its finite-time error bound. Also, it is proved that ARQ-Learning converges as fast as the standard Q-learning and robust Q-learning (Robust-Q) while guaranteeing better robustness. Our major contribution is to introduce an additional pessimistic agent that can address the major hurdle for the extension of ARQ-Learning into cases with large or continuous state spaces. Leveraging this double-agent approach, we for the first time develop (model-free) RRL algorithms for continuous state/action spaces. Via experiments, we demonstrate the effectiveness of our algorithms.

Robust reinforcement learning with UUB guarantee for safe motion control of autonomous robots

Robust reinforcement learning with UUB guarantee for safe motion control of autonomous robots

Toward Trustworthy Decision-Making for Autonomous Vehicles: A Robust Reinforcement Learning Approach with Safety Guarantees

While autonomous vehicles are vital components of intelligent transportation systems, ensuring the trustworthiness of decision-making remains a substantial challenge in realizing autonomous driving. Therefore, we present a novel robust reinforcement learning approach with safety guarantees to attain trustworthy decision-making for autonomous vehicles. The proposed technique ensures decision trustworthiness in terms of policy robustness and collision safety. Specifically, an adversary model is learned online to simulate the worst-case uncertainty by approximating the optimal adversarial perturbations on the observed states and environmental dynamics. In addition, an adversarial robust actor-critic algorithm is developed to enable the agent to learn robust policies against perturbations in observations and dynamics. Moreover, we devise a safety mask to guarantee the collision safety of the autonomous driving agent during both the training and testing processes using an interpretable knowledge model known as the Responsibility-Sensitive Safety Model. Finally, the proposed approach is evaluated through both simulations and experiments. These results indicate that the autonomous driving agent can make trustworthy decisions and drastically reduce the number of collisions through robust safety policies.

Digital twin-enabled domain adaptation for zero-touch UAV networks: Survey and challenges

In existing wireless networks, the control programs have been designed manually and for certain predefined scenarios. This process is complicated and error-prone, and the resulting control programs are not resilient to disruptive changes. Data-driven control based on Artificial Intelligence and Machine Learning (AI/ML) has been envisioned as a key technique to automate the modeling, optimization and control of complex wireless systems. However, existing AI/ML techniques rely on sufficient well-labeled data and may suffer from slow convergence and poor generalizability. In this article, focusing on digital twin-assisted wireless unmanned aerial vehicle (UAV) systems, we provide a survey of emerging techniques that can enable fast-converging data-driven control of wireless systems with enhanced generalization capability to new environments. These include simultaneous localization and sensing (SLAM)-based sensing and network softwarization for digital twin construction, robust reinforcement learning and system identification for domain adaptation, and testing facility sharing and federation. The corresponding research opportunities are also discussed.

Enhancing construction robot learning for collaborative and long-horizon tasks using generative adversarial imitation learning

The development and deployment of robots on construction sites are integral to the industrialization of construction, known as Construction 4.0. Tele-operated and pre-programmed robots have enhanced construction efficiency and safety. However, their utilization on-site remains limited due to the need for expert remote control and the lack of adaptability in dynamic environments. Reinforcement learning (RL) has emerged as a promising solution, as RL-controlled robots possess inherent self-learning abilities to adapt to diverse situations. Nevertheless, manual design of RL reward functions for complex tasks poses challenges. To address this issue, inverse reinforcement learning (IRL) methods, such as Generative Adversarial Imitation Learning (GAIL), have been proposed to learn optimal actions through expert demonstration and self-exploration, without explicitly defined reward functions. In this study, we propose an innovative approach integrating GAIL and virtual reality (VR) integrated robot control approach to control robots for long-horizon collaborative construction tasks involving multiple sub-tasks. We employ VR expert demonstrations as input for GAIL training, enabling a team of robots, including an Unmanned Ground Vehicle (UGV) and two robot arms, to interact with the designed RL environment and perform tasks such as transporting, picking, and installing window panels. Handle long-horizon collaborative construction tasks (i.e., a long sequence of several sub-tasks performed by multiple robots). For evaluation, we compare the performance of our VR-GAIL model with a prevalent and robust RL baseline model, Proximal Policy Optimization (PPO). The results demonstrate that our reward-free VR-GAIL model achieves, on average, a 4.5% higher success rate than the PPO counterpart equipped with carefully designed reward functions across all three sub-tasks and their randomized variations. Furthermore, the performance gap between GAIL and PPO widens as the task difficulty increases. These findings indicate that our approach effectively enhances RL agent performance in tackling complex construction tasks while expediting development by eliminating reward function design requirements.

User-Oriented Robust Reinforcement Learning

User-Oriented Robust Reinforcement Learning

Curricular Robust Reinforcement Learning via GAN-Based Perturbation Through Continuously Scheduled Task Sequence

Reinforcement learning (RL), one of three branches of machine learning, aims for autonomous learning and is now greatly driving the artificial intelligence development, especially in autonomous distributed systems, such as cooperative Boston Dynamics robots. However, robust RL has been a challenging problem of reliable aspects due to the gap between laboratory simulation and real world. Existing efforts have been made to approach this problem, such as performing random environmental perturbations in the learning process. However, one cannot guarantee to train with a positive perturbation as bad ones might bring failures to RL. In this work, we treat robust RL as a multi-task RL problem, and propose a curricular robust RL approach. We first present a generative adversarial network (GAN) based task generation model to iteratively output new tasks at the appropriate level of difficulty for the current policy. Furthermore, with these progressive tasks, we can realize curricular learning and finally obtain a robust policy. Extensive experiments in multiple environments demonstrate that our method improves the training stability and is robust to differences in training/test conditions.

Active Gamma-Ray Log Pattern Localization With Distributionally Robust Reinforcement Learning

Accurately localizing 1D signal patterns, such as Gamma-ray well-log depth matching, is crucial in the oilfield service industry as it directly affects the quality of oil and gas exploration. However, traditional methods such as well-log curve analysis and pattern hand-picking matching are labor-intensive and heavily rely on human expertise, leading to inconsistent results. Although attempts have been made to automate this process, challenges such as low computational performance, non-robustness, and non-generalization remain unsolved. To address these challenges, we have developed a data-driven AI system that learns an active signal pattern localization strategy inspired by human attention. Our artificial intelligence system uses an offline reinforcement learning (RL) framework as its central component, which solves a highly abstracted Markov decision process problem via offline training on human-labeled historical data. The RL agent uses top-down reasoning to determine the location of target signal fragments by deforming a bounding window using simple transformation actions. To overcome distribution shifts between logged data and real and ensure generalization, we propose a discrete distributionally robust soft actor-critic RL framework (DRSAC-Discrete) to solve the Markov decision process problem under uncertainty. By exploring unfamiliar environments in a restrictive manner, the DRSAC-Discrete algorithm provides a safe solution that can be used when data is limited during the early stage of this industrial application. We evaluated the reinforcement learning-based localization system on augmented field Gamma-ray well-log datasets, and the results showed promising localization capability. Furthermore, the DRSAC-Discrete algorithm demonstrated relatively robust performance guarantees when facing data shortage.

Robust Reinforcement Learning Based Visual Servoing with Convolutional Features

Image-based visual servoing poses a significant challenge for robotic systems, as it involves detecting the object and controlling the robot arm based on image feedback. These tasks are further complicated by various interferences such as changes in ambient lighting, distractions, and background clutter. Recent research suggests that reinforcement learning is a promising approach to learning efficient control policies for such tasks. In this paper, we propose a data-driven approach for closed-loop visual servoing based on a reinforcement learning algorithm that does not require any prior knowledge of the task object or intrinsic camera parameters. Our method utilizes a convolutional neural network for object detection and a servoing strategy that enables the robot to determine the relative camera motion and position the camera at the desired pose. Our experimental results demonstrate that the proposed approach successfully steers the camera using only a single template image of the task object.

Performance Analysis of Reinforcement Learning Techniques for Augmented Experience Training Using Generative Adversarial Networks

This paper aims at analyzing the performance of reinforcement learning (RL) agents when trained in environments created by a generative adversarial network (GAN). This is a first step towards the greater goal of developing fast-learning and robust RL agents by leveraging the power of GANs for environment generation. The RL techniques that we tested were exact Q-learning, approximate Q-learning, approximate SARSA and a heuristic agent. The task for the agents was to learn how to play the game Super Mario Bros (SMB). This analysis will be helpful in suggesting which RL techniques are best suited for augmented experience training (with synthetic environments). This would further help in establishing a reinforcement learning framework using the agents that can learn faster by bringing a greater variety in environment exploration.

Integrating safety constraints into adversarial training for robust deep reinforcement learning

The ability to resist interference is the key to the widespread application of reinforcement learning. Although adversarial training is a promising method for robust promotion, standard adversarial training leads to unstable results or performance deterioration due to the presence of perturbation. To address the problem, a robust reinforcement learning method which integrates safety constraints that are modelled by environment termination conditions into adversarial training is proposed, where safety constraints are adopted to restrict agent’s actions and guide the training process. For better modelling the robust reinforcement learning problem, a modified constrained Markov Decision Process (MDP) that considers perturbation for robust reinforcement learning, named Constrained Markov Decision Process (CMDP) with Perturbation (CMDPP) is also introduced. The proposed safe robust reinforcement learning method based on CMDPP utilizes the penalty function to solve CMDP and generates perturbation from the gradient of state for adversarial training. Tests on the robustness of the proposed method under several attack methods and evaluation of generalization through changing environment dynamics were carried out on the OpenAI gym and Roboschool environments. The results demonstrate that our method not only has a better performance confronting the attack but also has a higher generalization capability with reference to the changing environment dynamics.

Variational multiscale reinforcement learning for discovering reduced order closure models of nonlinear spatiotemporal transport systems

A central challenge in the computational modeling and simulation of a multitude of science applications is to achieve robust and accurate closures for their coarse-grained representations due to underlying highly nonlinear multiscale interactions. These closure models are common in many nonlinear spatiotemporal systems to account for losses due to reduced order representations, including many transport phenomena in fluids. Previous data-driven closure modeling efforts have mostly focused on supervised learning approaches using high fidelity simulation data. On the other hand, reinforcement learning (RL) is a powerful yet relatively uncharted method in spatiotemporally extended systems. In this study, we put forth a modular dynamic closure modeling and discovery framework to stabilize the Galerkin projection based reduced order models that may arise in many nonlinear spatiotemporal dynamical systems with quadratic nonlinearity. However, a key element in creating a robust RL agent is to introduce a feasible reward function, which can be constituted of any difference metrics between the RL model and high fidelity simulation data. First, we introduce a multi-modal RL to discover mode-dependant closure policies that utilize the high fidelity data in rewarding our RL agent. We then formulate a variational multiscale RL (VMRL) approach to discover closure models without requiring access to the high fidelity data in designing the reward function. Specifically, our chief innovation is to leverage variational multiscale formalism to quantify the difference between modal interactions in Galerkin systems. Our results in simulating the viscous Burgers equation indicate that the proposed VMRL method leads to robust and accurate closure parameterizations, and it may potentially be used to discover scale-aware closure models for complex dynamical systems.

Explore the weakness: Instructive exploration adversarial robust reinforcement learning

Although reinforcement learning has been proved to be effective in many simulated platforms, it may still fail in environments due to the difference between simulation environment and real world environment, as well as being subjected to unexcepted attacks that objectively exist. Therefore, it calls for improving the robustness of the agent to increase its stability. To address the problem, an algorithm that uses the curiosity mechanism to improve the model exploration, referred to as instructive exploration adversarial robust reinforcement learning(Iearrl), is proposed, which enhances the adaption ability of agents through adversary learning, ensuring that the agent chooses a better action in practical environments with different settings from the training environment. At the same time, in order to increase the efficiency of exploration and reduce the cost, a model used to evaluate the competency of the agent is built for mentoring internal rewards determining whether further exploration is needed by analyzing the agent’s action in the current state space. The experiments in MuJoCo platforms verified the effectiveness of the proposed method.

Robust optimization for quantum reinforcement learning control using partial observations

The current quantum reinforcement learning control models often assume that the quantum states are known a priori for control optimization. However, full observation of the quantum state is experimentally infeasible due to the exponential scaling of the number of required quantum measurements on the number of qubits. In this paper, we investigate a robust reinforcement learning method using partial observations to overcome this difficulty. This control scheme is compatible with near-term quantum devices, where the noise is prevalent and predetermining the dynamics of the quantum state is practically impossible. We show that this simplified control scheme can achieve similar or even better performance when compared to the conventional methods relying on full observation. We demonstrate the effectiveness of this scheme on examples of quantum state control and the quantum approximate optimization algorithm (QAOA). It has been shown that high-fidelity state control can be achieved even if the noise amplitude is at the same level as the control amplitude. Besides, an acceptable level of optimization accuracy can be achieved for a QAOA with a noisy control Hamiltonian. This robust control optimization model can be trained to compensate for the uncertainties in practical quantum computing.

Robust Reinforcement Learning: A Review of Foundations and Recent Advances

Reinforcement learning (RL) has become a highly successful framework for learning in Markov decision processes (MDP). Due to the adoption of RL in realistic and complex environments, solution robustness becomes an increasingly important aspect of RL deployment. Nevertheless, current RL algorithms struggle with robustness to uncertainty, disturbances, or structural changes in the environment. We survey the literature on robust approaches to reinforcement learning and categorize these methods in four different ways: (i) Transition robust designs account for uncertainties in the system dynamics by manipulating the transition probabilities between states; (ii) Disturbance robust designs leverage external forces to model uncertainty in the system behavior; (iii) Action robust designs redirect transitions of the system by corrupting an agent’s output; (iv) Observation robust designs exploit or distort the perceived system state of the policy. Each of these robust designs alters a different aspect of the MDP. Additionally, we address the connection of robustness to the risk-based and entropy-regularized RL formulations. The resulting survey covers all fundamental concepts underlying the approaches to robust reinforcement learning and their recent advances.