Deep Reinforcement Learning Policies Research Articles

Multi-agent long-distance end-to-end indoor navigation: using imitation learning pre-training and global map

Abstract To address the navigation challenges of mobile robot swarm coordination in complex environments, this paper presents a distributed navigation approach for mobile robot swarm. The study initially formulates the problem of robot swarm navigation as a partially observable Markov decision process, employing a distributed proximal policy optimization algorithm to train decision models mapping observations to actions directly. Furthermore, it adopts imitation learning pre-training to maximize expert decisions in the initial training phase, thereby reducing exploration space. Lastly, an improved path point generation algorithm along with a spatiotemporal reachability calculation method is proposed to guide the Deep Reinforcement Learning policy in accomplishing long-distance indoor navigation. Comprehensive performance evaluations are conducted in simulated environments to compare the proposed method with existing approaches. Results demonstrate that the proposed approach effectively enhances both convergence speed and model performance, while significantly improving the navigation capabilities of robot swarms in complex environments.

A versatile door opening system with mobile manipulator through adaptive position-force control and reinforcement learning

The ability of robots to navigate through doors is crucial for their effective operation in indoor environments. Consequently, extensive research has been conducted to develop robots capable of opening specific doors. However, the diverse combinations of door handles and opening directions necessitate a more versatile door opening system for robots to successfully operate in real-world environments. In this paper, we propose a mobile manipulator system that can autonomously open various doors without prior knowledge. By using convolutional neural networks, point cloud extraction techniques, and external force measurements during exploratory motion, we obtained information regarding handle types, poses, and door characteristics. Through two different approaches, adaptive position-force control and deep reinforcement learning, we successfully opened doors without precise trajectory or excessive external force. The adaptive position-force control method involves moving the end-effector in the direction of the door opening while responding compliantly to external forces, ensuring safety and manipulator workspace. Meanwhile, the deep reinforcement learning policy minimizes applied forces and eliminates unnecessary movements, enabling stable operation across doors with different poses and widths. The RL-based approach outperforms the adaptive position-force control method in terms of compensating for external forces, ensuring smooth motion, and achieving efficient speed. It reduces the maximum force required by 3.27 times and improves motion smoothness by 1.82 times. However, the non-learning-based adaptive position-force control method demonstrates more versatility in opening a wider range of doors, encompassing revolute doors with four distinct opening directions and varying widths.

Fill-and-Spill: Deep Reinforcement Learning Policy Gradient Methods for Reservoir Operation Decision and Control

Fill-and-Spill: Deep Reinforcement Learning Policy Gradient Methods for Reservoir Operation Decision and Control

Continuous control of structural vibrations using hybrid deep reinforcement learning policy

An efficient active structural control scheme by utilizing the collective concepts of data-driven and physics-inspired reinforcement learning (RL) approaches is being introduced here. The controller is designed to deliver optimal feedback forces based on the full-state information, wherein the training process involves the use of deep neural networks (NNs) and a specially designed gradient descent-based sequence within the RL framework. This integration of algorithms results in a unique active control policy that accelerates the learning process, thereby demands considerably less computational resources to develop an optimal and stable controller as compared to existing data-driven approaches. Most importantly, the mentioned data-driven and physics-inspired approaches encompass deep deterministic policy gradient (DDPG) and an iterative gradient-based state feedback control (SFSC) algorithms, respectively, to establish the architecture of the hybrid control policy, referred to as hybrid RL-controller. Additional advantage of the hybrid RL-controller is its ability to operate in continuous state–action spaces, which allows the designed controller to address diverse structural control problems. The staggered performance of the hybrid RL-controller, both in continuous and discrete time, is evaluated in three case studies that involve structures in linear and nonlinear regimes. The outcomes include a detailed comparison of the hybrid RL-controller with the individually designed DDPG and SFSC RL control strategies, as well as the uncontrolled scenario. Furthermore, this study thoroughly investigates the real-life implementation concerns of control strategies, such as perturbations in model parameters and input forces, and time delays in the feedback loop. Finally, the results from this study corroborate that the designed RL-controller showcases superior performance and faster execution time in the feedback loop, making it suitable for the vibration control of multidimensional complex structures.

Deep reinforcement learning sensor scheduling for effective monitoring of dynamical systems

Advances in technology have enabled the use of sensors with varied modalities to monitor different parts of systems, each providing diverse levels of information about the underlying system. However, resource limitations and computational power restrict the number of sensors/data that can be processed in real-time in most complex systems. These challenges necessitate the need for selecting/scheduling a subset of sensors to obtain measurements that guarantee the best monitoring objectives. This paper focuses on sensor scheduling for systems modeled by hidden Markov models. Despite the development of several sensor selection and scheduling methods, existing methods tend to be greedy and do not take into account the long-term impact of selected sensors on monitoring objectives. This paper formulates optimal sensor scheduling as a reinforcement learning problem defined over the posterior distribution of system states. Further, the paper derives a deep reinforcement learning policy for offline learning of the sensor scheduling policy, which can then be executed in real-time as new information unfolds. The proposed method applies to any monitoring objective that can be expressed in terms of the posterior distribution of the states (e.g. state estimation, information gain, etc.). The performance of the proposed method in terms of accuracy and robustness is investigated for monitoring the security of networked systems and the health monitoring of gene regulatory networks.

Unleashing mixed-reality capability in Deep Reinforcement Learning-based robot motion generation towards safe human–robot collaboration

The integration of human–robot collaboration yields substantial benefits, particularly in terms of enhancing flexibility and efficiency within a range of mass-personalized manufacturing tasks, for example, small-batch customized product inspection and assembly/disassembly. Meanwhile, as human–robot collaboration lands broader in manufacturing, the unstructured scene and operator uncertainties are increasingly involved and considered. Consequently, it becomes imperative for robots to execute in a safe and adaptive manner rather than solely relying on pre-programmed instructions. To tackle it, a systematic solution for safe robot motion generation in human–robot collaborative activities is proposed, leveraging mixed-reality technologies and Deep Reinforcement Learning. This solution covers the entire process of collaboration starting with an intuitive interface that facilitates bare-hand task command transmission and scene coordinate transformation before the collaboration begins. In particular, mixed-reality devices are employed as effective tools for representing the state of humans, robots, and scenes. This enables the learning of an end-to-end Deep Reinforcement Learning policy that addresses both the uncertainties in robot perception and decision-making in an integrated manner. The proposed solution also implements policy simulation-to-reality deployment, along with motion preview and collision detection mechanisms, to ensure safe robot motion execution. It is hoped that this work could inspire further research in human–robot collaboration to unleash and exploit the powerful capabilities of mixed reality.

π-Light: Programmatic Interpretable Reinforcement Learning for Resource-Limited Traffic Signal Control

The recent advancements in Deep Reinforcement Learning (DRL) have significantly enhanced the performance of adaptive Traffic Signal Control (TSC). However, DRL policies are typically represented by neural networks, which are over-parameterized black-box models. As a result, the learned policies often lack interpretability, and cannot be deployed directly in the real-world edge hardware due to resource constraints. In addition, the DRL methods often exhibit limited generalization performance, struggling to generalize the learned policy to other geographical regions. These factors limit the practical application of learning-based approaches. To address these issues, we suggest the use of an inherently interpretable program for representing the control policy. We present a new approach, Programmatic Interpretable reinforcement learning for traffic signal control (π-light), designed to autonomously discover non-differentiable programs. Specifically, we define a Domain Specific Language (DSL) and transformation rules for constructing programs, and utilize Monte Carlo Tree Search (MCTS) to find the optimal program in a discrete space. Extensive experiments demonstrate that our method consistently outperforms baseline approaches. Moreover, π-Light exhibits superior generalization capabilities compared to DRL, enabling training and evaluation across intersections from different cities. Finally, we analyze how the learned program policies can directly deploy on edge devices with extremely limited resources.

Cooperative Deep Reinforcement Learning Policies for Autonomous Navigation in Complex Environments

Cooperative Deep Reinforcement Learning Policies for Autonomous Navigation in Complex Environments

Evolving interpretable decision trees for reinforcement learning

In recent years, reinforcement learning (RL) techniques have achieved great success in many different applications. However, their heavy reliance on complex deep neural networks makes most RL models uninterpretable, limiting their application in domains where trust and security are important. To address this challenge, we propose MENS-DT-RL, an algorithm capable of constructing interpretable models for RL via the evolution of decision tree (DT) models. MENS-DT-RL uses a multi-method ensemble algorithm to evolve univariate DTs, guiding the process with a fitness metric that prioritizes interpretability and consistent high performance. Three different initializations for the MENS-DT-RL are proposed, including the use of Imitation Learning (IL) techniques, and a novel pruning approach that reduces solution size without compromising performance. To evaluate the proposed approach, we compare it with other models from the literature on three benchmark tasks from the OpenAI Gym library, as well as on a fertilization problem inspired by real-world crop management. To the best of our knowledge, the proposed scheme is the first to solve the Lunar Lander benchmark with both interpretability and a high confidence rate (90% of episodes are successful), as well as the first to solve the Mountain Car environment with a tree of only 7 nodes. On the real-world task, the proposed MENS-DT-RL is able to produce solutions with the same quality as deep RL policies, with the added bonus of interpretability. We also analyze the best solutions found by the algorithm and show that they are not only interpretable but also diverse in their behavior, empowering the end user with the choice of which model to apply. Overall, the findings show that the proposed approach is capable of producing high-quality transparent models for RL, achieving interpretability without losing performance.

Multi-agent reinforcement learning satellite guidance for triangulation of a moving object in a relative orbit frame

Multi-agent systems are of ever-increasing importance in a contested space environment—use of multiple, cooperative satellites potentially increases positive mission outcomes on orbit, while autonomy becomes an ever-increasing requirement to increase reaction time to dynamic situations and lower the burden on space operators. This research explores multi-agent satellite swarm Guidance, Navigation, and Control (GNC) using deep reinforcement learning (DRL). DRL policies are trained to provide guidance inputs to agents in multi-agent swarm environments for completing complex, teamwork-focused objectives in geosynchronous orbit. An example scenario is explored for a group of satellite agents maneuvering to triangulate an object that is non-stationary in the relative orbit frame. Reward shaping is used to encourage learning guidance that positions swarm members to maximize triangulation accuracy, using angles-only observations for navigation relative to the target. Results show the policies successfully learn guidance through reward shaping to improve triangulation accuracy by a significant factor.

Pretty Darn Good Control: When are Approximate Solutions Better than Approximate Models.

Existing methods for optimal control struggle to deal with the complexity commonly encountered in real-world systems, including dimensionality, process error, model bias and data heterogeneity. Instead of tackling these system complexities directly, researchers have typically sought to simplify models to fit optimal control methods. But when is the optimal solution to an approximate, stylized model better than an approximate solution to a more accurate model? While this question has largely gone unanswered owing to the difficulty of finding even approximate solutions for complex models, recent algorithmic and computational advances in deep reinforcement learning (DRL) might finally allow us to address these questions. DRL methods have to date been applied primarily in the context of games or robotic mechanics, which operate under precisely known rules. Here, we demonstrate the ability for DRL algorithms using deep neural networks to successfully approximate solutions (the "policy function" or control rule) in a non-linear three-variable model for a fishery without knowing or ever attempting to infer a model for the process itself. We find that the reinforcement learning agent discovers a policy that outperforms both constant escapement and constant mortality policies-the standard family of policies considered in fishery management. This DRL policy has the shape of a constant escapement policy whose escapement values depend on the stock sizes of other species in the model.

Spatio-Temporal Graph Convolutional Neural Networks for Physics-Aware Grid Learning Algorithms

This paper proposes novel architectures for spatio-temporal graph convolutional and recurrent neural networks whose structure is inspired by the physics of power systems. The key insight behind our design consists in deriving the so-called graph shift operator (GSO), which is the cornerstone of Graph Convolutional Neural Network (GCN) and Graph Recursive Neural Network (GRN) designs, from the power flow equations. We demonstrate the effectiveness of the proposed architectures in two applications: in forecasting the power grid state and in finding a stochastic policy for foresighted voltage control using deep reinforcement learning. Since our design can be adopted in single-phase as well as three-phase unbalanced systems, we test our architecture in both environments. For state forecasting experiments we consider the single phase IEEE 118-bus case systems; for voltage regulation, we illustrate the performance of deep reinforcement learning policy on the unbalanced three-phase IEEE 123-bus feeder system. In both cases the physics based GCN and GRN learning algorithms we propose outperform the state of the art.

Deep reinforcement learning and 3D physical environments applied to crowd evacuation in congested scenarios

ABSTRACT To avoid crowd evacuation simulations depending on 2D environments and real data, we propose a framework for crowd evacuation modeling and simulation by applying deep reinforcement learning (DRL) and 3D physical environments (3DPEs). In 3DPEs, we construct simulation scenarios from the aspects of geometry, semantics and physics, which include the environment, the agents and their interactions, and provide training samples for DRL. In DRL, we design a double branch feature extraction combined actor and critic network as the DRL policy and value function and use a clipped surrogate objective with polynomial decay to update the policy. With a unified configuration, we conduct evacuation simulations. In scenarios with one exit, we reproduce and verify the bottleneck effect of congested crowds and explore the impact of exit width and agent characteristics (number, mass and height) on evacuation. In scenarios with two exits and a uniform (nonuniform) distribution of agents, we explore the impact of exit characteristics (width and relative position) and agent characteristics (height, initial location and distribution) on agent exit selection and evacuation. Overall, interactive 3DPEs and unified DRL enable agents to adapt to different evacuation scenarios to simulate crowd evacuation and explore the laws of crowd evacuation.

Industrial Cross-Robot Transfer Learning

With increased complexity and individualization of processes and products, the manufacturing industry experiences growing pressure to automate. To meet these constantly growing requirements regarding the flexibility of production and logistic systems, intelligent robotic systems that can adapt dynamically to changing process conditions are essential. Deep reinforcement learning provides a solution and offers a paradigm of learning adaptive control strategies through autonomous agents. However, the corresponding training procedure is based on trial-and-error interactions with an environment and is thus very data inefficient and costly in real-world scenarios. Transfer learning approaches can mitigate these costs and improve learning by leveraging control capabilities acquired in other settings, such as simulations, or from previously learning a different task. In this paper, we investigate the transfer of trajectories between different robot models. We propose a methodology that maps trajectory demonstrations from a source robot into trajectories of a target robot by using forward and inverse kinematics models. We introduce a similarity measure, that one the one hand compares different robot models using key poses along the kinematic chains, and on the other hand captures the efficiency of transitions. This similarity measure is used to select from the mapped trajectories those which are feasible, efficient and retain the general shape of the robotic arm. Finally, we perform behavioral cloning to pre-train a deep reinforcement learning policy for the target domain and demonstrate significant benefits compared to learning the target policy from scratch.

Preparing for the next pandemic: Simulation-based deep reinforcement learning to discover and test multimodal control of systemic inflammation using repurposed immunomodulatory agents.

Preparation to address the critical gap in a future pandemic between non-pharmacological measures and the deployment of new drugs/vaccines requires addressing two factors: 1) finding virus/pathogen-agnostic pathophysiological targets to mitigate disease severity and 2) finding a more rational approach to repurposing existing drugs. It is increasingly recognized that acute viral disease severity is heavily driven by the immune response to the infection ("cytokine storm" or "cytokine release syndrome"). There exist numerous clinically available biologics that suppress various pro-inflammatory cytokines/mediators, but it is extremely difficult to identify clinically effective treatment regimens with these agents. We propose that this is a complex control problem that resists standard methods of developing treatment regimens and accomplishing this goal requires the application of simulation-based, model-free deep reinforcement learning (DRL) in a fashion akin to training successful game-playing artificial intelligences (AIs). This proof-of-concept study determines if simulated sepsis (e.g. infection-driven cytokine storm) can be controlled in the absence of effective antimicrobial agents by targeting cytokines for which FDA-approved biologics currently exist. We use a previously validated agent-based model, the Innate Immune Response Agent-based Model (IIRABM), for control discovery using DRL. DRL training used a Deep Deterministic Policy Gradient (DDPG) approach with a clinically plausible control interval of 6 hours with manipulation of six cytokines for which there are existing drugs: Tumor Necrosis Factor (TNF), Interleukin-1 (IL-1), Interleukin-4 (IL-4), Interleukin-8 (IL-8), Interleukin-12 (IL-12) and Interferon-γ(IFNg). DRL trained an AI policy that could improve outcomes from a baseline Recovered Rate of 61% to one with a Recovered Rate of 90% over ~21 days simulated time. This DRL policy was then tested on four different parameterizations not seen in training representing a range of host and microbe characteristics, demonstrating a range of improvement in Recovered Rate by +33% to +56. The current proof-of-concept study demonstrates that significant disease severity mitigation can potentially be accomplished with existing anti-mediator drugs, but only through a multi-modal, adaptive treatment policy requiring implementation with an AI. While the actual clinical implementation of this approach is a projection for the future, the current goal of this work is to inspire the development of a research ecosystem that marries what is needed to improve the simulation models with the development of the sensing/assay technologies to collect the data needed to iteratively refine those models.

Sim–Real Mapping of an Image-Based Robot Arm Controller Using Deep Reinforcement Learning

Models trained with Deep Reinforcement learning (DRL) have been deployed in various areas of robotics with varying degree of success. To overcome the limitations of data gathering in the real world, DRL training utilizes simulated environments and transfers the learned policy to real-world scenarios, i.e., sim–real transfer. Simulators fail to accurately capture the entire dynamics of the real world, so simulation-trained policies often fail when applied to reality, termed a reality gap (RG). In this paper, we propose a search (mapping) algorithm that takes in real-world observation (images) and maps them to the policy-equivalent images in the simulated environment using a convolution neural network (CNN) model. The two-step training process, DRL policy and a mapping model, overcomes the RG problem with simulated data only. We evaluated the proposed system with a gripping task of a custom-made robot arm in the real world and compared the performance against a conventional DRL sim–real transfer system. The conventional system achieved a 15–57% success rate in gripping operation depending on the position of the target object while the mapping-based sim–real system achieved 100%. The experimental results demonstrated that the proposed DRL with mapping method appropriately corresponded the real world to the simulated environment, confirming that the scheme can achieve high sim–real generalization at significantly low training costs.

A DRL-Driven Intelligent Optimization Strategy for Resource Allocation in Cloud-Edge-End Cooperation Environments

Complex dynamic services and heterogeneous network environments make the asymmetrical control a curial issue to handle on the Internet. With the advent of the Internet of Things (IoT) and the fifth generation (5G), the emerging network applications lead to the explosive growth of mobile traffic while bringing forward more challenging service requirements to future radio access networks. Therefore, how to effectively allocate limited heterogeneous network resources to improve content delivery for massive application services to ensure network quality of service (QoS) becomes particularly urgent in heterogeneous network environments. To cope with the explosive mobile traffic caused by emerging Internet services, this paper designs an intelligent optimization strategy based on deep reinforcement learning (DRL) for resource allocation in heterogeneous cloud-edge-end collaboration environments. Meanwhile, the asymmetrical control problem caused by complex dynamic services and heterogeneous network environments is discussed and overcome by distributed cooperation among cloud-edge-end nodes in the system. Specifically, the multi-layer heterogeneous resource allocation problem is formulated as a maximal traffic offloading model, where content caching and request aggregation mechanisms are utilized. A novel DRL policy is proposed to improve content distribution by making cache replacement and task scheduling for arriving content requests in accordance with the information about users’ history requests, in-network cache capacity, available link bandwidth and topology structure. The performance of our proposed solution and its similar counterparts are analyzed in different network conditions.

RSAC: A Robust Deep Reinforcement Learning Strategy for Dimensionality Perturbation

Artificial agents are used in autonomous systems such as autonomous vehicles, autonomous robotics, and autonomous drones to make predictions based on data generated by fusing the values from many sources such as different sensors. Malfunctioning of sensors was noticed in the robotics domain. The correct observation from sensors corresponds to the true estimate of the dimension value of the state vector in deep reinforcement learning (DRL). Hence, noisy estimates from these sensors lead to dimensionality impairment in the state. DRL policies have shown to stagger its decision by the wrong choice of action in case of adversarial attack or modeling error. Hence, it is necessary to examine the effect of dimensionality perturbation on neural policy. In this regard, we analyze whether subtle dimensionality perturbation that occurs due to the noise in the source of input at the testing time distracts agent decisions. Also, we propose an RSAC (robust soft actor-critic) approach that uses a noisy state for prediction, and estimates target from nominal observation. We find that the injection of such noisy input during training will not hamper learning. We have done our simulation in the OpenAI gym MuJoCo (Walker2d-V2) environment and our empirical results demonstrate that the proposed approach competes for SAC’s performance and makes it robust to test time dimensionality perturbation.

Deep-Reinforcement-Learning-Based Resource Allocation for Content Distribution in Fog Radio Access Networks

With the rapid development of wireless communication technologies, the emerging multimedia applications make mobile Internet traffic grow explosively while putting forward higher service requirements for the next-generation wireless networks. Therefore, how to achieve low-latency content transmission by effectively allocating heterogeneous network resources to improve the network quality of service and end-user quality of experience is a key issue to be solved urgently in the current Internet. In this article, we propose a deep reinforcement learning (DRL)-based resource allocation scheme to improve content distribution in a layered fog radio access network (FRAN). We formulate the optimal resource allocation problem as a minimal delay model, where in-network caching is deployed and the same content requests from mobile users can be aggregated in the queue of each base station. To cope with the increasing user requests and overcome capacity constraints of the FRAN, moreover, a cloud–edge cooperation offloading scheme is utilized in our model, where the integrated allocation of caching, computing, and communication resources and joint optimization between in-network caching and routing are considered to promote resource utilization and content delivery. In our solution, a new DRL policy is designed to make cross-layer cooperative caching and routing decisions for the arriving content requests according to request history information and available network resources in the system. Simulation results demonstrate that our proposed model can performs much better than the existing cloud–edge cooperation schemes in the FRAN.

Frame-Correlation Transfers Trigger Economical Attacks on Deep Reinforcement Learning Policies.

Adversarial attack can be deemed as a necessary prerequisite evaluation procedure before the deployment of any reinforcement learning (RL) policy. Most existing approaches for generating adversarial attacks are gradient based and are extensive, viz., perturbing every pixel of every frame. In contrast, recent advances show that gradient-free selective perturbations (i.e., attacking only selected pixels and frames) could be a more realistic adversary. However, these attacks treat every frame in isolation, ignoring the relationship between neighboring states of a Markov decision process; thus resulting in high computational complexity that tends to limit their real-world plausibility due to the tight time constraint in RL. Given the above, this article showcases the first study of how transferability across frames could be exploited for boosting the creation of minimal yet powerful attacks in image-based RL. To this end, we introduce three types of frame-correlation transfers (FCTs) (i.e., anterior case transfer, random projection-based transfer, and principal components-based transfer) with varying degrees of computational complexity in generating adversaries via a genetic algorithm. We empirically demonstrate the tradeoff between the complexity and potency of the transfer mechanism by exploring four fully trained state-of-the-art policies on six Atari games. Our FCTs dramatically speed up the attack generation compared to existing methods, often reducing the computation time required to nearly zero; thus, shedding light on the real threat of real-time attacks in RL.