Deep Reinforcement Learning Policies Research Articles

Continuous control of structural vibrations using hybrid deep reinforcement learning policy

An efficient active structural control scheme by utilizing the collective concepts of data-driven and physics-inspired reinforcement learning (RL) approaches is being introduced here. The controller is designed to deliver optimal feedback forces based on the full-state information, wherein the training process involves the use of deep neural networks (NNs) and a specially designed gradient descent-based sequence within the RL framework. This integration of algorithms results in a unique active control policy that accelerates the learning process, thereby demands considerably less computational resources to develop an optimal and stable controller as compared to existing data-driven approaches. Most importantly, the mentioned data-driven and physics-inspired approaches encompass deep deterministic policy gradient (DDPG) and an iterative gradient-based state feedback control (SFSC) algorithms, respectively, to establish the architecture of the hybrid control policy, referred to as hybrid RL-controller. Additional advantage of the hybrid RL-controller is its ability to operate in continuous state–action spaces, which allows the designed controller to address diverse structural control problems. The staggered performance of the hybrid RL-controller, both in continuous and discrete time, is evaluated in three case studies that involve structures in linear and nonlinear regimes. The outcomes include a detailed comparison of the hybrid RL-controller with the individually designed DDPG and SFSC RL control strategies, as well as the uncontrolled scenario. Furthermore, this study thoroughly investigates the real-life implementation concerns of control strategies, such as perturbations in model parameters and input forces, and time delays in the feedback loop. Finally, the results from this study corroborate that the designed RL-controller showcases superior performance and faster execution time in the feedback loop, making it suitable for the vibration control of multidimensional complex structures.

Deep reinforcement learning sensor scheduling for effective monitoring of dynamical systems

Advances in technology have enabled the use of sensors with varied modalities to monitor different parts of systems, each providing diverse levels of information about the underlying system. However, resource limitations and computational power restrict the number of sensors/data that can be processed in real-time in most complex systems. These challenges necessitate the need for selecting/scheduling a subset of sensors to obtain measurements that guarantee the best monitoring objectives. This paper focuses on sensor scheduling for systems modeled by hidden Markov models. Despite the development of several sensor selection and scheduling methods, existing methods tend to be greedy and do not take into account the long-term impact of selected sensors on monitoring objectives. This paper formulates optimal sensor scheduling as a reinforcement learning problem defined over the posterior distribution of system states. Further, the paper derives a deep reinforcement learning policy for offline learning of the sensor scheduling policy, which can then be executed in real-time as new information unfolds. The proposed method applies to any monitoring objective that can be expressed in terms of the posterior distribution of the states (e.g. state estimation, information gain, etc.). The performance of the proposed method in terms of accuracy and robustness is investigated for monitoring the security of networked systems and the health monitoring of gene regulatory networks.

Unleashing mixed-reality capability in Deep Reinforcement Learning-based robot motion generation towards safe human–robot collaboration

The integration of human–robot collaboration yields substantial benefits, particularly in terms of enhancing flexibility and efficiency within a range of mass-personalized manufacturing tasks, for example, small-batch customized product inspection and assembly/disassembly. Meanwhile, as human–robot collaboration lands broader in manufacturing, the unstructured scene and operator uncertainties are increasingly involved and considered. Consequently, it becomes imperative for robots to execute in a safe and adaptive manner rather than solely relying on pre-programmed instructions. To tackle it, a systematic solution for safe robot motion generation in human–robot collaborative activities is proposed, leveraging mixed-reality technologies and Deep Reinforcement Learning. This solution covers the entire process of collaboration starting with an intuitive interface that facilitates bare-hand task command transmission and scene coordinate transformation before the collaboration begins. In particular, mixed-reality devices are employed as effective tools for representing the state of humans, robots, and scenes. This enables the learning of an end-to-end Deep Reinforcement Learning policy that addresses both the uncertainties in robot perception and decision-making in an integrated manner. The proposed solution also implements policy simulation-to-reality deployment, along with motion preview and collision detection mechanisms, to ensure safe robot motion execution. It is hoped that this work could inspire further research in human–robot collaboration to unleash and exploit the powerful capabilities of mixed reality.

π-Light: Programmatic Interpretable Reinforcement Learning for Resource-Limited Traffic Signal Control

The recent advancements in Deep Reinforcement Learning (DRL) have significantly enhanced the performance of adaptive Traffic Signal Control (TSC). However, DRL policies are typically represented by neural networks, which are over-parameterized black-box models. As a result, the learned policies often lack interpretability, and cannot be deployed directly in the real-world edge hardware due to resource constraints. In addition, the DRL methods often exhibit limited generalization performance, struggling to generalize the learned policy to other geographical regions. These factors limit the practical application of learning-based approaches. To address these issues, we suggest the use of an inherently interpretable program for representing the control policy. We present a new approach, Programmatic Interpretable reinforcement learning for traffic signal control (π-light), designed to autonomously discover non-differentiable programs. Specifically, we define a Domain Specific Language (DSL) and transformation rules for constructing programs, and utilize Monte Carlo Tree Search (MCTS) to find the optimal program in a discrete space. Extensive experiments demonstrate that our method consistently outperforms baseline approaches. Moreover, π-Light exhibits superior generalization capabilities compared to DRL, enabling training and evaluation across intersections from different cities. Finally, we analyze how the learned program policies can directly deploy on edge devices with extremely limited resources.

Evolving interpretable decision trees for reinforcement learning

In recent years, reinforcement learning (RL) techniques have achieved great success in many different applications. However, their heavy reliance on complex deep neural networks makes most RL models uninterpretable, limiting their application in domains where trust and security are important. To address this challenge, we propose MENS-DT-RL, an algorithm capable of constructing interpretable models for RL via the evolution of decision tree (DT) models. MENS-DT-RL uses a multi-method ensemble algorithm to evolve univariate DTs, guiding the process with a fitness metric that prioritizes interpretability and consistent high performance. Three different initializations for the MENS-DT-RL are proposed, including the use of Imitation Learning (IL) techniques, and a novel pruning approach that reduces solution size without compromising performance. To evaluate the proposed approach, we compare it with other models from the literature on three benchmark tasks from the OpenAI Gym library, as well as on a fertilization problem inspired by real-world crop management. To the best of our knowledge, the proposed scheme is the first to solve the Lunar Lander benchmark with both interpretability and a high confidence rate (90% of episodes are successful), as well as the first to solve the Mountain Car environment with a tree of only 7 nodes. On the real-world task, the proposed MENS-DT-RL is able to produce solutions with the same quality as deep RL policies, with the added bonus of interpretability. We also analyze the best solutions found by the algorithm and show that they are not only interpretable but also diverse in their behavior, empowering the end user with the choice of which model to apply. Overall, the findings show that the proposed approach is capable of producing high-quality transparent models for RL, achieving interpretability without losing performance.

Multi-agent reinforcement learning satellite guidance for triangulation of a moving object in a relative orbit frame

Multi-agent systems are of ever-increasing importance in a contested space environment—use of multiple, cooperative satellites potentially increases positive mission outcomes on orbit, while autonomy becomes an ever-increasing requirement to increase reaction time to dynamic situations and lower the burden on space operators. This research explores multi-agent satellite swarm Guidance, Navigation, and Control (GNC) using deep reinforcement learning (DRL). DRL policies are trained to provide guidance inputs to agents in multi-agent swarm environments for completing complex, teamwork-focused objectives in geosynchronous orbit. An example scenario is explored for a group of satellite agents maneuvering to triangulate an object that is non-stationary in the relative orbit frame. Reward shaping is used to encourage learning guidance that positions swarm members to maximize triangulation accuracy, using angles-only observations for navigation relative to the target. Results show the policies successfully learn guidance through reward shaping to improve triangulation accuracy by a significant factor.

Pretty Darn Good Control: When are Approximate Solutions Better than Approximate Models.

Existing methods for optimal control struggle to deal with the complexity commonly encountered in real-world systems, including dimensionality, process error, model bias and data heterogeneity. Instead of tackling these system complexities directly, researchers have typically sought to simplify models to fit optimal control methods. But when is the optimal solution to an approximate, stylized model better than an approximate solution to a more accurate model? While this question has largely gone unanswered owing to the difficulty of finding even approximate solutions for complex models, recent algorithmic and computational advances in deep reinforcement learning (DRL) might finally allow us to address these questions. DRL methods have to date been applied primarily in the context of games or robotic mechanics, which operate under precisely known rules. Here, we demonstrate the ability for DRL algorithms using deep neural networks to successfully approximate solutions (the "policy function" or control rule) in a non-linear three-variable model for a fishery without knowing or ever attempting to infer a model for the process itself. We find that the reinforcement learning agent discovers a policy that outperforms both constant escapement and constant mortality policies-the standard family of policies considered in fishery management. This DRL policy has the shape of a constant escapement policy whose escapement values depend on the stock sizes of other species in the model.

Spatio-Temporal Graph Convolutional Neural Networks for Physics-Aware Grid Learning Algorithms

This paper proposes novel architectures for spatio-temporal graph convolutional and recurrent neural networks whose structure is inspired by the physics of power systems. The key insight behind our design consists in deriving the so-called graph shift operator (GSO), which is the cornerstone of Graph Convolutional Neural Network (GCN) and Graph Recursive Neural Network (GRN) designs, from the power flow equations. We demonstrate the effectiveness of the proposed architectures in two applications: in forecasting the power grid state and in finding a stochastic policy for foresighted voltage control using deep reinforcement learning. Since our design can be adopted in single-phase as well as three-phase unbalanced systems, we test our architecture in both environments. For state forecasting experiments we consider the single phase IEEE 118-bus case systems; for voltage regulation, we illustrate the performance of deep reinforcement learning policy on the unbalanced three-phase IEEE 123-bus feeder system. In both cases the physics based GCN and GRN learning algorithms we propose outperform the state of the art.

Deep reinforcement learning and 3D physical environments applied to crowd evacuation in congested scenarios

ABSTRACT To avoid crowd evacuation simulations depending on 2D environments and real data, we propose a framework for crowd evacuation modeling and simulation by applying deep reinforcement learning (DRL) and 3D physical environments (3DPEs). In 3DPEs, we construct simulation scenarios from the aspects of geometry, semantics and physics, which include the environment, the agents and their interactions, and provide training samples for DRL. In DRL, we design a double branch feature extraction combined actor and critic network as the DRL policy and value function and use a clipped surrogate objective with polynomial decay to update the policy. With a unified configuration, we conduct evacuation simulations. In scenarios with one exit, we reproduce and verify the bottleneck effect of congested crowds and explore the impact of exit width and agent characteristics (number, mass and height) on evacuation. In scenarios with two exits and a uniform (nonuniform) distribution of agents, we explore the impact of exit characteristics (width and relative position) and agent characteristics (height, initial location and distribution) on agent exit selection and evacuation. Overall, interactive 3DPEs and unified DRL enable agents to adapt to different evacuation scenarios to simulate crowd evacuation and explore the laws of crowd evacuation.

Industrial Cross-Robot Transfer Learning

With increased complexity and individualization of processes and products, the manufacturing industry experiences growing pressure to automate. To meet these constantly growing requirements regarding the flexibility of production and logistic systems, intelligent robotic systems that can adapt dynamically to changing process conditions are essential. Deep reinforcement learning provides a solution and offers a paradigm of learning adaptive control strategies through autonomous agents. However, the corresponding training procedure is based on trial-and-error interactions with an environment and is thus very data inefficient and costly in real-world scenarios. Transfer learning approaches can mitigate these costs and improve learning by leveraging control capabilities acquired in other settings, such as simulations, or from previously learning a different task. In this paper, we investigate the transfer of trajectories between different robot models. We propose a methodology that maps trajectory demonstrations from a source robot into trajectories of a target robot by using forward and inverse kinematics models. We introduce a similarity measure, that one the one hand compares different robot models using key poses along the kinematic chains, and on the other hand captures the efficiency of transitions. This similarity measure is used to select from the mapped trajectories those which are feasible, efficient and retain the general shape of the robotic arm. Finally, we perform behavioral cloning to pre-train a deep reinforcement learning policy for the target domain and demonstrate significant benefits compared to learning the target policy from scratch.

Preparing for the next pandemic: Simulation-based deep reinforcement learning to discover and test multimodal control of systemic inflammation using repurposed immunomodulatory agents.

Preparation to address the critical gap in a future pandemic between non-pharmacological measures and the deployment of new drugs/vaccines requires addressing two factors: 1) finding virus/pathogen-agnostic pathophysiological targets to mitigate disease severity and 2) finding a more rational approach to repurposing existing drugs. It is increasingly recognized that acute viral disease severity is heavily driven by the immune response to the infection ("cytokine storm" or "cytokine release syndrome"). There exist numerous clinically available biologics that suppress various pro-inflammatory cytokines/mediators, but it is extremely difficult to identify clinically effective treatment regimens with these agents. We propose that this is a complex control problem that resists standard methods of developing treatment regimens and accomplishing this goal requires the application of simulation-based, model-free deep reinforcement learning (DRL) in a fashion akin to training successful game-playing artificial intelligences (AIs). This proof-of-concept study determines if simulated sepsis (e.g. infection-driven cytokine storm) can be controlled in the absence of effective antimicrobial agents by targeting cytokines for which FDA-approved biologics currently exist. We use a previously validated agent-based model, the Innate Immune Response Agent-based Model (IIRABM), for control discovery using DRL. DRL training used a Deep Deterministic Policy Gradient (DDPG) approach with a clinically plausible control interval of 6 hours with manipulation of six cytokines for which there are existing drugs: Tumor Necrosis Factor (TNF), Interleukin-1 (IL-1), Interleukin-4 (IL-4), Interleukin-8 (IL-8), Interleukin-12 (IL-12) and Interferon-γ(IFNg). DRL trained an AI policy that could improve outcomes from a baseline Recovered Rate of 61% to one with a Recovered Rate of 90% over ~21 days simulated time. This DRL policy was then tested on four different parameterizations not seen in training representing a range of host and microbe characteristics, demonstrating a range of improvement in Recovered Rate by +33% to +56. The current proof-of-concept study demonstrates that significant disease severity mitigation can potentially be accomplished with existing anti-mediator drugs, but only through a multi-modal, adaptive treatment policy requiring implementation with an AI. While the actual clinical implementation of this approach is a projection for the future, the current goal of this work is to inspire the development of a research ecosystem that marries what is needed to improve the simulation models with the development of the sensing/assay technologies to collect the data needed to iteratively refine those models.

Sim–Real Mapping of an Image-Based Robot Arm Controller Using Deep Reinforcement Learning

Models trained with Deep Reinforcement learning (DRL) have been deployed in various areas of robotics with varying degree of success. To overcome the limitations of data gathering in the real world, DRL training utilizes simulated environments and transfers the learned policy to real-world scenarios, i.e., sim–real transfer. Simulators fail to accurately capture the entire dynamics of the real world, so simulation-trained policies often fail when applied to reality, termed a reality gap (RG). In this paper, we propose a search (mapping) algorithm that takes in real-world observation (images) and maps them to the policy-equivalent images in the simulated environment using a convolution neural network (CNN) model. The two-step training process, DRL policy and a mapping model, overcomes the RG problem with simulated data only. We evaluated the proposed system with a gripping task of a custom-made robot arm in the real world and compared the performance against a conventional DRL sim–real transfer system. The conventional system achieved a 15–57% success rate in gripping operation depending on the position of the target object while the mapping-based sim–real system achieved 100%. The experimental results demonstrated that the proposed DRL with mapping method appropriately corresponded the real world to the simulated environment, confirming that the scheme can achieve high sim–real generalization at significantly low training costs.

A DRL-Driven Intelligent Optimization Strategy for Resource Allocation in Cloud-Edge-End Cooperation Environments

Complex dynamic services and heterogeneous network environments make the asymmetrical control a curial issue to handle on the Internet. With the advent of the Internet of Things (IoT) and the fifth generation (5G), the emerging network applications lead to the explosive growth of mobile traffic while bringing forward more challenging service requirements to future radio access networks. Therefore, how to effectively allocate limited heterogeneous network resources to improve content delivery for massive application services to ensure network quality of service (QoS) becomes particularly urgent in heterogeneous network environments. To cope with the explosive mobile traffic caused by emerging Internet services, this paper designs an intelligent optimization strategy based on deep reinforcement learning (DRL) for resource allocation in heterogeneous cloud-edge-end collaboration environments. Meanwhile, the asymmetrical control problem caused by complex dynamic services and heterogeneous network environments is discussed and overcome by distributed cooperation among cloud-edge-end nodes in the system. Specifically, the multi-layer heterogeneous resource allocation problem is formulated as a maximal traffic offloading model, where content caching and request aggregation mechanisms are utilized. A novel DRL policy is proposed to improve content distribution by making cache replacement and task scheduling for arriving content requests in accordance with the information about users’ history requests, in-network cache capacity, available link bandwidth and topology structure. The performance of our proposed solution and its similar counterparts are analyzed in different network conditions.

RSAC: A Robust Deep Reinforcement Learning Strategy for Dimensionality Perturbation

Artificial agents are used in autonomous systems such as autonomous vehicles, autonomous robotics, and autonomous drones to make predictions based on data generated by fusing the values from many sources such as different sensors. Malfunctioning of sensors was noticed in the robotics domain. The correct observation from sensors corresponds to the true estimate of the dimension value of the state vector in deep reinforcement learning (DRL). Hence, noisy estimates from these sensors lead to dimensionality impairment in the state. DRL policies have shown to stagger its decision by the wrong choice of action in case of adversarial attack or modeling error. Hence, it is necessary to examine the effect of dimensionality perturbation on neural policy. In this regard, we analyze whether subtle dimensionality perturbation that occurs due to the noise in the source of input at the testing time distracts agent decisions. Also, we propose an RSAC (robust soft actor-critic) approach that uses a noisy state for prediction, and estimates target from nominal observation. We find that the injection of such noisy input during training will not hamper learning. We have done our simulation in the OpenAI gym MuJoCo (Walker2d-V2) environment and our empirical results demonstrate that the proposed approach competes for SAC’s performance and makes it robust to test time dimensionality perturbation.

Deep-Reinforcement-Learning-Based Resource Allocation for Content Distribution in Fog Radio Access Networks

With the rapid development of wireless communication technologies, the emerging multimedia applications make mobile Internet traffic grow explosively while putting forward higher service requirements for the next-generation wireless networks. Therefore, how to achieve low-latency content transmission by effectively allocating heterogeneous network resources to improve the network quality of service and end-user quality of experience is a key issue to be solved urgently in the current Internet. In this article, we propose a deep reinforcement learning (DRL)-based resource allocation scheme to improve content distribution in a layered fog radio access network (FRAN). We formulate the optimal resource allocation problem as a minimal delay model, where in-network caching is deployed and the same content requests from mobile users can be aggregated in the queue of each base station. To cope with the increasing user requests and overcome capacity constraints of the FRAN, moreover, a cloud–edge cooperation offloading scheme is utilized in our model, where the integrated allocation of caching, computing, and communication resources and joint optimization between in-network caching and routing are considered to promote resource utilization and content delivery. In our solution, a new DRL policy is designed to make cross-layer cooperative caching and routing decisions for the arriving content requests according to request history information and available network resources in the system. Simulation results demonstrate that our proposed model can performs much better than the existing cloud–edge cooperation schemes in the FRAN.

Frame-Correlation Transfers Trigger Economical Attacks on Deep Reinforcement Learning Policies.

Adversarial attack can be deemed as a necessary prerequisite evaluation procedure before the deployment of any reinforcement learning (RL) policy. Most existing approaches for generating adversarial attacks are gradient based and are extensive, viz., perturbing every pixel of every frame. In contrast, recent advances show that gradient-free selective perturbations (i.e., attacking only selected pixels and frames) could be a more realistic adversary. However, these attacks treat every frame in isolation, ignoring the relationship between neighboring states of a Markov decision process; thus resulting in high computational complexity that tends to limit their real-world plausibility due to the tight time constraint in RL. Given the above, this article showcases the first study of how transferability across frames could be exploited for boosting the creation of minimal yet powerful attacks in image-based RL. To this end, we introduce three types of frame-correlation transfers (FCTs) (i.e., anterior case transfer, random projection-based transfer, and principal components-based transfer) with varying degrees of computational complexity in generating adversaries via a genetic algorithm. We empirically demonstrate the tradeoff between the complexity and potency of the transfer mechanism by exploring four fully trained state-of-the-art policies on six Atari games. Our FCTs dramatically speed up the attack generation compared to existing methods, often reducing the computation time required to nearly zero; thus, shedding light on the real threat of real-time attacks in RL.

Deep Reinforcement Learning Policies Learn Shared Adversarial Features across MDPs

The use of deep neural networks as function approximators has led to striking progress for reinforcement learning algorithms and applications. Yet the knowledge we have on decision boundary geometry and the loss landscape of neural policies is still quite limited. In this paper, we propose a framework to investigate the decision boundary and loss landscape similarities across states and across MDPs. We conduct experiments in various games from Arcade Learning Environment, and discover that high sensitivity directions for neural policies are correlated across MDPs. We argue that these high sensitivity directions support the hypothesis that non-robust features are shared across training environments of reinforcement learning agents. We believe our results reveal fundamental properties of the environments used in deep reinforcement learning training, and represent a tangible step towards building robust and reliable deep reinforcement learning agents.

Detect, Understand, Act: A Neuro-symbolic Hierarchical Reinforcement Learning Framework

In this paper we introduce Detect, Understand, Act (DUA), a neuro-symbolic reinforcement learning framework. The Detect component is composed of a traditional computer vision object detector and tracker. The Act component houses a set of options, high-level actions enacted by pre-trained deep reinforcement learning (DRL) policies. The Understand component provides a novel answer set programming (ASP) paradigm for symbolically implementing a meta-policy over options and effectively learning it using inductive logic programming (ILP). We evaluate our framework on the Animal-AI (AAI) competition testbed, a set of physical cognitive reasoning problems. Given a set of pre-trained DRL policies, DUA requires only a few examples to learn a meta-policy that allows it to improve the state-of-the-art on multiple of the most challenging categories from the testbed. DUA constitutes the first holistic hybrid integration of computer vision, ILP and DRL applied to an AAI-like environment and sets the foundations for further use of ILP in complex DRL challenges.

Minimalistic Attacks: How Little It Takes to Fool Deep Reinforcement Learning Policies

Recent studies have revealed that neural-network-based policies can be easily fooled by adversarial examples. However, while most prior works analyze the effects of perturbing every pixel of every frame assuming white-box policy access, in this article, we take a more restrictive view toward adversary generation—with the goal of unveiling the limits of a model’s vulnerability. In particular, we explore minimalistic attacks by defining <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">three key settings</i> : 1) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Black-Box Policy Access</i> : where the attacker only has access to the input (state) and output (action probability) of an RL policy; 2) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Fractional-State Adversary</i> : where only several pixels are perturbed, with the extreme case being a single-pixel adversary; and 3) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Tactically Chanced Attack</i> : where only significant frames are tactically chosen to be attacked. We formulate the adversarial attack by accommodating the three key settings, and explore their potency on six Atari games by examining four fully trained state-of-the-art policies. In Breakout, for example, we surprisingly find that: 1) all policies showcase significant performance degradation by merely modifying 0.01% of the input state and 2) the policy trained by DQN is totally deceived by perturbing only 1% frames.

Mobile‐fog‐cloud assisted deep reinforcement learning and blockchain‐enable IoMT system for healthcare workflows

AbstractThe Internet of Medical Things (IoMT) is increasingly being used to secure blockchain technology to operate healthcare applications in a distributed network. The applications are mobile and can move from one place to another with different wireless connectivity. However, there are a lot of challenges that are investigated further. For instance, dynamic content values changed during mobile applications during any business goal. The workflow healthcare applications are complex as compared to coarse‐grained and fine‐grained workload in IoMT. In this article, the study analyzed offloading and scheduling problems for healthcare workflows in IoMT fog‐cloud network. Therefore, the study considered the problem as an offloading and scheduling problem formulated deep reinforcement learning as Markov problem. The study devises the novel deep reinforcement learning and blockchain‐enabled system, consisting of multi‐criteria offloading based on deep reinforcement learning policies and blockchain task scheduling with task sequencing and research matching methods for healthcare workloads in the IoMT system. The simulation results suggested strategies that reduced the communication and computation time for each application in the system.