Deep Reinforcement Learning Training Research Articles

Addressing environmental stochasticity in reconfigurable intelligent surface aided unmanned aerial vehicle networks: Multi-task deep reinforcement learning based optimization for physical layer security

Deep Reinforcement Learning (DRL) is a powerful tool for optimizing communications in reconfigurable intelligent surface (RIS)-assisted millimeter-wave unmanned aerial vehicle (UAV) systems, particularly for ensuring physical layer security by dynamically maximizing signal strength for legitimate users while minimizing leakage towards eavesdroppers. However, existing approaches often overfit the DRL agent to a non-stochastic environment, where the trajectories of users and eavesdroppers are fixed. This results in a single-task DRL agent that cannot generalize beyond the training environment/task. In response, this study introduces a novel multi-task DRL framework to address this limitation. Firstly, recognizing that not all tasks share the same solution, we proposed a reward-driven clustering method to group similar tasks based on their reward scores. This enables the training of a shared multi-task DRL agent for each cluster. Each cluster can be treated as an expertise, and the cluster-specific multi-task DRL agents are considered the 'experts'. Meanwhile, we leverage federated learning (FL) to speed up the training of the shared multi-task DRL agent. By combining these two techniques, our reward-driven clustered FL autonomously groups similar tasks together rapidly. Simulation results demonstrate a reduction in the number of agents from 50 to 10 specialized 'experts', while achieving a 10X speedup compared to conventional single-task DRL training. Furthermore, the study presents a mixture of experts (MoE) model that unifies these 'experts' into a single model. The MoE is trained to select the most suitable 'expert' when encountering new tasks. Our MoE exhibits robustness by effectively generalizing to 50 novel tasks without fine-tuning.

Framework for Federated Learning and Edge Deployment of Real-Time Reinforcement Learning Decision Engine on Software Defined Radio

Machine learning promises to empower dynamic resource allocation requirements of Next Generation (NextG) wireless networks including 6G and tactical networks. Recently, we have seen the impact machine learning can make on various aspects of wireless networks. Yet, in most cases, the progress has been limited to simulations and/or relies on large processing units to run the decision engines as opposed to deploying it on the radio at the edge. While relying on simulations for rapid and efficient training of deep reinforcement learning (DRL) may be necessary, it is key to mitigate the sim-real gap while trying to improve the generalization capability. To mitigate these challenges, we developed the Marconi-Rosenblatt Framework for Intelligent Networks (MR-iNet Gym), an open-source architecture designed for accelerating the deployment of novel DRL for NextG wireless networks. To demonstrate its impact, we tackled the problem of distributed frequency and power allocation while emphasizing the generalization capability of DRL decision engine. The end-to-end solution was implemented on the GPU-embedded software-defined radio and validated using over-the-air evaluation. To the best of our knowledge, these were the first instances that established the feasibility of deploying DRL for optimized distributed resource allocation for next-generation of GPU-embedded radios.

Optimal parallelization strategies for active flow control in deep reinforcement learning-based computational fluid dynamics

Deep reinforcement learning (DRL) has emerged as a promising approach for handling highly dynamic and nonlinear active flow control (AFC) problems. However, the computational cost associated with training DRL models presents a significant performance bottleneck. To address this challenge and enable efficient scaling on high-performance computing architectures, this study focuses on optimizing DRL-based algorithms in parallel settings. We validate an existing state-of-the-art DRL framework used for AFC problems and discuss its efficiency bottlenecks. Subsequently, by deconstructing the overall framework and conducting extensive scalability benchmarks for individual components, we investigate various hybrid parallelization configurations and propose efficient parallelization strategies. Moreover, we refine input/output (I/O) operations in multi-environment DRL training to tackle critical overhead associated with data movement. Finally, we demonstrate the optimized framework for a typical AFC problem where near-linear scaling can be obtained for the overall framework. We achieve a significant boost in parallel efficiency from around 49% to approximately 78%, and the training process is accelerated by approximately 47 times using 60 central processing unit (CPU) cores. These findings are expected to provide valuable insight for further advancements in DRL-based AFC studies.

Learn to flap: foil non-parametric path planning via deep reinforcement learning

To optimize flapping foil performance, in the current study we apply deep reinforcement learning (DRL) to plan foil non-parametric motion, as the traditional control techniques and simplified motions cannot fully model nonlinear, unsteady and high-dimensional foil–vortex interactions. Therefore, a DRL training framework is proposed based on the proximal policy optimization algorithm and the transformer architecture, where the policy is initialized from the sinusoidal expert display. We first demonstrate the effectiveness of the proposed DRL-training framework, learning the coherent foil flapping motion to generate thrust. Furthermore, by adjusting reward functions and action thresholds, DRL-optimized foil trajectories can gain significant enhancement in both thrust and efficiency compared with the sinusoidal motion. Last, through visualization of wake morphology and instantaneous pressure distributions, it is found that DRL-optimized foil can adaptively adjust the phases between motion and shedding vortices to improve hydrodynamic performance. Our results give a hint of how to solve complex fluid manipulation problems using the DRL method.

Cheaper and Faster: Distributed Deep Reinforcement Learning with Serverless Computing

Deep reinforcement learning (DRL) has gained immense success in many applications, including gaming AI, robotics, and system scheduling. Distributed algorithms and architectures have been vastly proposed (e.g., actor-learner architecture) to accelerate DRL training with large-scale server-based clusters. However, training on-policy algorithms with the actor-learner architecture unavoidably induces resource wasting due to synchronization between learners and actors, thus resulting in significantly extra billing. As a promising alternative, serverless computing naturally fits on-policy synchronization and alleviates resource wasting in distributed DRL training with pay-as-you-go pricing. Yet, none has leveraged serverless computing to facilitate DRL training. This paper proposes MinionsRL, the first serverless distributed DRL training framework that aims to accelerate DRL training- and cost-efficiency with dynamic actor scaling. We prototype MinionsRL on top of Microsoft Azure Container Instances and evaluate it with popular DRL tasks from OpenAI Gym. Extensive experiments show that MinionsRL reduces total training time by up to 52% and training cost by 86% compared to latest solutions.

Testing of Deep Reinforcement Learning Agents with Surrogate Models

Deep Reinforcement Learning (DRL) has received a lot of attention from the research community in recent years. As the technology moves away from game playing to practical contexts, such as autonomous vehicles and robotics, it is crucial to evaluate the quality of DRL agents. In this article, we propose a search-based approach to test such agents. Our approach, implemented in a tool called Indago , trains a classifier on failure and non-failure environment (i.e., pass) configurations resulting from the DRL training process. The classifier is used at testing time as a surrogate model for the DRL agent execution in the environment, predicting the extent to which a given environment configuration induces a failure of the DRL agent under test. The failure prediction acts as a fitness function, guiding the generation towards failure environment configurations, while saving computation time by deferring the execution of the DRL agent in the environment to those configurations that are more likely to expose failures. Experimental results show that our search-based approach finds 50% more failures of the DRL agent than state-of-the-art techniques. Moreover, such failures are, on average, 78% more diverse; similarly, the behaviors of the DRL agent induced by failure configurations are 74% more diverse.

An optimal solutions-guided deep reinforcement learning approach for online energy storage control

As renewable energy becomes more prevalent in the power grid, energy storage systems (ESSs) are playing an ever-increasingly crucial role in mitigating short-term supply–demand imbalances. However, the operation and control of ESS are not straightforward, given the ever-changing electricity prices in the market environment and the stochastic and intermittent nature of renewable energy generations, which respond to real-time load variations. In this paper, we propose a deep reinforcement learning (DRL) approach to address the electricity arbitrage problem associated with optimal ESS management. First, we analyze the structure of the optimal offline ESS control problem using the mixed-integer linear programming (MILP) formulation. This formulation identifies optimal control actions to absorb excess renewable energy and perform price arbitrage strategies. To tackle the uncertainties inherent in the prediction data, we then recast the online ESS control problem into a Markov Decision Process (MDP) framework and develop the DRL approach, which involves integrating the optimal offline control solution obtained from the training data into the training process and introducing noise to the state transitions. Unlike typical offline DRL training over a long time interval, we employ the Deep Deterministic Policy Gradient (DDPG) and Proximal Policy Optimization (PPO) algorithms with smaller neural networks training over a short time interval. Numerical studies demonstrate the promising potential of the proposed DRL-enabled approach for achieving better online control performance than the model predictive control (MPC) method under different price errors. This highlights the sample efficiency and robustness of our DRL approaches in managing ESS for electricity arbitrage.

Enhancing efficiency and propulsion in bio-mimetic robotic fish through end-to-end deep reinforcement learning

Aquatic organisms are known for their ability to generate efficient propulsion with low energy expenditure. While existing research has sought to leverage bio-inspired structures to reduce energy costs in underwater robotics, the crucial role of control policies in enhancing efficiency has often been overlooked. In this study, we optimize the motion of a bio-mimetic robotic fish using deep reinforcement learning (DRL) to maximize propulsion efficiency and minimize energy consumption. Our novel DRL approach incorporates extended pressure perception, a transformer model processing sequences of observations, and a policy transfer scheme. Notably, significantly improved training stability and speed within our approach allow for end-to-end training of the robotic fish. This enables agiler responses to hydrodynamic environments and possesses greater optimization potential compared to pre-defined motion pattern controls. Our experiments are conducted on a serially connected rigid robotic fish in a free stream with a Reynolds number of 6000 using computational fluid dynamics simulations. The DRL-trained policies yield impressive results, demonstrating both high efficiency and propulsion. The policies also showcase the agent's embodiment, skillfully utilizing its body structure and engaging with surrounding fluid dynamics, as revealed through flow analysis. This study provides valuable insights into the bio-mimetic underwater robots optimization through DRL training, capitalizing on their structural advantages, and ultimately contributing to more efficient underwater propulsion systems.

Deep Reinforcement Learning With Reward Design for Quantum Control

Deep reinforcement learning (DRL) has been recognized as a powerful tool in quantum physics, where DRL's reward design is non-trivial but crucial for quantum control tasks. To address the problem of over-reliance on human empirical knowledge to design DRL's rewards, we propose a deep reinforcement learning approach with a novel reward paradigm designed by the learning process information (DRL-LPI), where the learning process information (LPI) comprises the state information and the experiences. In DRL-LPI, the state information after being classified by a fidelity threshold, and the experiences are first stored simultaneously in the respective sequences, and this process is repeated until a similar-segment ends. Then, the stored state information is converted to the real value and used to design the reward value by applying a self-amplitude function. Next, the designed reward values are integrated with the stored experiences to compose transitions for DRL's training. Through comparisons to five representative reward schemes, the proposed DRL-LPI is evaluated on two typical quantum control tasks, i.e., the spin-(1/2) quantum state control and many-coupled qubits state control, and the experimental results show the superior learning efficiency and control performance of the proposed approach. More results show that DRL-LPI exhibits the ability to learn the control strategy with few control actions compared to stochastic gradient descent (SGD) and genetic algorithm (GA).

Variance Reduced Domain Randomization for Reinforcement Learning With Policy Gradient.

By introducing randomness on the environments, domain randomization (DR) imposes diversity to the policy training of deep reinforcement learning, and thus improves its capability of generalization. The randomization of environments, however, introduces another source of variability for the estimate of policy gradients, in addition to the already high variance incurred by trajectory sampling. Therefore, with standard state-dependent baselines, the policy gradient methods may still suffer high variance, causing a low sample efficiency during the training of DR. In this paper, we theoretically derive a bias-free and state/environment-dependent optimal baseline for DR, and analytically show its ability to achieve further variance reduction over the standard constant and state-dependent baselines for DR. Based on our theory, we further propose a variance reduced domain randomization (VRDR) approach for policy gradient methods, to strike a tradeoff between the variance reduction and computational complexity for the practical implementation. By dividing the entire space of environments into some subspaces and then estimating the state/subspace-dependent baseline, VRDR enjoys a theoretical guarantee of variance reduction and faster convergence than the state-dependent baselines. Empirical evaluations on six robot control tasks with randomized dynamics demonstrate that VRDR not only accelerates the convergence of policy training, but can consistently achieve a better eventual policy with improved training stability.

Dynamic slicing of multidimensional resources in DCI-EON with penalty-aware deep reinforcement learning

With the increasing demand for dynamic cloud computing services, data center interconnections based on elastic optical networks (DCI-EON) require efficient allocation methods for spectrum, access IP bandwidth, and compute resources. Dynamic slicing of multidimensional resources in DCI-EON has emerged as a promising solution. However, improper reallocation of resources can diminish the benefits of slice reconfiguration, and different resource reconfiguration techniques can lead to varying degrees of service degradation for existing services. In this paper, we propose a prediction-based dynamic slicing approach (DS-DRL-RW) that leverages penalty-aware deep reinforcement learning (DRL) to optimize resource allocation while considering the trade-off between the benefits and penalties of slice reconfiguration. DS-DRL-RW employs statistical prediction to obtain a coarse-grained solution for dynamic slicing that does not differentiate among multidimensional resources. Subsequently, through focused DRL training based on the coarse-grained solution, the accurate result for multidimensional resource slicing is achieved. Moreover, DS-DRL-RW comprehensively considers the benefits and penalties associated with different reconfiguration techniques after slice reconfiguration, enabling the determination of a suitable reconfiguration strategy. Simulation results demonstrate that DS-DRL-RW improves training efficiency and reduces the blocking rate of dynamic services by integrating slice traffic prediction and DRL. It effectively addresses both direct penalties from reconfiguration and indirect penalties from resource waste, thereby enhancing multidimensional resource utilization. DS-DRL-RW effectively handles the diverse penalties associated with various reconfiguration techniques and selects the appropriate reconfiguration strategy. Furthermore, DS-DRL-RW prioritizes the different quality requirements of services in slices, such as completion time, to avoid service degradation.

Intelligent active flow control of long-span bridge deck using deep reinforcement learning integrated transfer learning

Aerodynamic forces of the Great Belt Bridge are mitigated using deep reinforcement learning (DRL) based active flow control (AFC) techniques at a Reynolds number of 5×105 in this study. The flow control method involves placing a suction slit at the bottom of the trailing edge of the bridge. The DRL agent as a controller is able to optimize the velocity of the suction to reduce the fluctuating coefficients of bending moment, drag, and lift by 99.1%, 73.7%, and 95.8% respectively. To reduce the high computational demands associated with DRL-based AFC training, this study integrates transfer learning technique into DRL, which calls transfer learning based deep reinforcement learning (TL-DRL) method. Specifically, the transfer learning method based on a DRL pre-trained model trained with a coarse mesh scheme is implemented. Results indicate that with TL-DRL method, the training cost can be reduced by 53%, while achieving the same control strategy and control effect as the DRL training from scratch. This study shows that the TL-DRL based flow control method is highly effective in reducing aerodynamic forces on long-span bridges. Furthermore, TL-DRL based flow control approach can adapt flexibly to different flow field environments, ultimately enhancing energy utilization efficiency.

Federated Reinforcement Learning for Collective Navigation of Robotic Swarms

The recent advancement of Deep Reinforcement Learning (DRL) contributed to robotics by allowing automatic controller design. The automatic controller design is a crucial approach for designing swarm robotic systems, which require more complex controllers than a single robot system to lead a desired collective behaviour. Although the DRL-based controller design method showed its effectiveness for swarm robotic systems, the reliance on the central training server is a critical problem in real-world environments where robot-server communication is unstable or limited. We propose a novel Federated Learning (FL) based DRL training strategy (FLDDPG) for use in swarm robotic applications. Through the comparison with baseline strategies under a limited communication bandwidth scenario, it is shown that the FLDDPG method resulted in higher robustness and generalisation ability into a different environment and real robots, while the baseline strategies suffer from the limitation of communication bandwidth. This result suggests that the proposed method can benefit swarm robotic systems operating in environments with limited communication bandwidth, e.g., in high-radiation, underwater, or subterranean environments.

DSMC Evaluation Stages: Fostering Robust and Safe Behavior in Deep Reinforcement Learning – Extended Version

Neural networks (NN) are gaining importance in sequential decision-making. Deep reinforcement learning (DRL), in particular, is extremely successful in learning action policies in complex and dynamic environments. Despite this success, however, DRL technology is not without its failures, especially in safety-critical applications: (i) the training objective maximizes average rewards, which may disregard rare but critical situations and hence lack local robustness; (ii) optimization objectives targeting safety typically yield degenerated reward structures, which, for DRL to work, must be replaced with proxy objectives. Here, we introduce a methodology that can help to address both deficiencies. We incorporate evaluation stages (ES) into DRL, leveraging recent work on deep statistical model checking (DSMC), which verifies NN policies in Markov decision processes. Our ES apply DSMC at regular intervals to determine state space regions with weak performance. We adapt the subsequent DRL training priorities based on the outcome, (i) focusing DRL on critical situations and (ii) allowing to foster arbitrary objectives. We run case studies on two benchmarks. One of them is the Racetrack, an abstraction of autonomous driving that requires navigating a map without crashing into a wall. The other is MiniGrid, a widely used benchmark in the AI community. Our results show that DSMC-based ES can significantly improve both (i) and (ii).

Surrogate model-based deep reinforcement learning for experimental study of active flow control of circular cylinder

The flow around a circular cylinder is a classical problem in fluid mechanics, and the reduction of drag and lift has been a long-standing research focus in flow control. In this study, we apply deep reinforcement learning (DRL) to intelligently determine suction flow rate on a circular cylinder model in wind tunnel, aiming to minimize aerodynamic forces while considering energy dissipation efficiency. However, DRL has been criticized for its low data utilization rate and long training period, leading to high experimental training cost. To address these issues, this study employs a surrogate model to optimize the reward function and hyperparameters, and this method is called SM-DRL. This SM-DRL method efficiently expedites the DRL training process, significantly reducing the experimental training cost. In addition, DRL training was conducted in a variable flow field, and the robustness of the obtained DRL model was tested. The results indicate that the DRL agent can determine the optimal control strategy, i.e., automatically select the optimal suction flow rate in terms of the incoming wind velocity, resulting in a significant reduction in lift fluctuations. For Reynolds number of 1.65×104, the reduction in lift fluctuations of the circular cylinder exceeds 50%.

Deep reinforcement learning for financial trading using multi-modal features

Deep Reinforcement Learning (DRL) has several successful applications in various fields. One of these fields is financial trading, in which an agent interacts with its environment in order to maximize profit by purchasing and selling financial assets. Besides price-related features, sentiment-related features have recently been used for this task. Sentiment information reflects the public emotional reaction to these assets. This type of information has been shown to have a beneficial impact on the performance of the agents in recent works. However, directly incorporating such information into trading agents is not always trivial, often leading to sub-optimal results, e.g., due to overfitting. In this work, we propose a novel multi-modal financial embedding-based approach, as an easy and efficient way of combining various modalities from multiple sources of information, such as price and sentiment data. In addition to effectively combining multi-modal information, the proposed method allows for (a) quantifying the impact of different modalities in the decisions of the model, as well as (b) altering the importance of them on-the-fly without performing retraining. We evaluate the agents’ performance using the Profit and Loss (PnL) metric and figure that the impact of sentiment utilization in combination with embeddings is advantageous for DRL training.

Deep reinforcement learning-aided autonomous navigation with landmark generators.

Mobile robots are playing an increasingly significant role in social life and industrial production, such as searching and rescuing robots, autonomous exploration of sweeping robots, and so on. Improving the accuracy of autonomous navigation of mobile robots is a hot issue to be solved. However, traditional navigation methods are unable to realize crash-free navigation in an environment with dynamic obstacles, more and more scholars are gradually using autonomous navigation based on deep reinforcement learning (DRL) to replace overly conservative traditional methods. But on the other hand, DRL's training time is too long, and the lack of long-term memory easily leads the robot to a dead end, which makes its application in the actual scene more difficult. To shorten training time and prevent mobile robots from getting stuck and spinning around, we design a new robot autonomous navigation framework which combines the traditional global planning and the local planning based on DRL. Therefore, the entire navigation process can be transformed into first using traditional navigation algorithms to find the global path, then searching for several high-value landmarks on the global path, and then using the DRL algorithm to move the mobile robot toward the designated landmarks to complete the final navigation, which makes the robot training difficulty greatly reduced. Furthermore, in order to improve the lack of long-term memory in deep reinforcement learning, we design a feature extraction network containing memory modules to preserve the long-term dependence of input features. Through comparing our methods with traditional navigation methods and reinforcement learning based on end-to-end depth navigation methods, it shows that while the number of dynamic obstacles is large and obstacles are rapidly moving, our proposed method is, on average, 20% better than the second ranked method in navigation efficiency (navigation time and navigation paths' length), 34% better than the second ranked method in safety (collision times), 26.6% higher than the second ranked method in success rate, and shows strong robustness.

Learn-to-supervise: Causal reinforcement learning for high-level control in industrial processes

Possessing efficient supervisory control systems is crucial for maintaining the desired operational performance of complex industrial processes. Several challenges face the developers of these systems, such as requiring accurate physical models, dealing with the variability and uncertainty of process operating conditions and coordinating between local controllers to reach desired global performance. This paper proposes an intelligent supervisory control approach based on causal reinforcement learning (CRL) to effectively manipulate the controllers’ setpoints of the process in a way that optimizes its key performance indicators (KPIs), thereby improving the energy efficiency of the process. The approach adopts deep reinforcement learning (DRL) to develop an efficient control policy through interaction with a process simulation. The DRL training history is then exploited using interpretable machine learning and process mining to build a discrete event system (DES) model, in the form of a state-event graph. The DES model identifies causal relationships between events and provides interpretability to the control policy developed by the DRL method. The DES discovered is exploited as a Markov decision process to apply the Q-learning algorithm as a CRL supervisor. The supervisor incorporates causal knowledge into its training process, thus improving the DRL control policy developed and identifying the event paths that optimize the process’s KPIs. The proposed approach is validated using two heat recovery systems in a pulp & paper mill. It successfully achieves a control policy that reduces energy consumption by up to 15.6% for the first system and 5.02% for the second, compared to the expert’s baseline methods.

Fast DRL-based scheduler configuration tuning for reducing tail latency in edge-cloud jobs

Edge-cloud applications are rapidly prevailing in recent years and pose the challenge of using both resource-strenuous edge devices and elastic cloud resources under dynamic workloads. Efficient resource allocation on edge-cloud jobs via cluster schedulers (e.g. Kubernetes/Volcano scheduler) is essential to guarantee their performance, e.g. tail latency, and such allocation is sensitive to scheduler configurations such as applied scheduling algorithms and task restart/discard policy. Deep reinforcement learning (DRL) is increasingly applied to optimize scheduling decisions. However, DRL faces the conundrum of achieving high rewards at a dauntingly long training time (e.g. hours or days), making it difficult to tune the scheduler configurations online in accordance to dynamically changing edge-cloud workloads and resources. For such an issue, this paper proposes EdgeTuner, a fast scheduler configuration tuning approach that efficiently leverages DRL to reduce tail latency of edge-cloud jobs. The enabling feature of EdgeTuner is to effectively simulate the execution of edge-cloud jobs under different scheduler configurations and thus quickly estimate these configurations’ influence on job performance. The simulation results allow EdgeTuner to timely train a DRL agent in order to properly tune scheduler configurations in dynamic edge-cloud environment. We implement EdgeTuner in both Kubernetes and Volcano schedulers and extensively evaluate it on real workloads driven by Alibaba production traces. Our results show that EdgeTuner outperforms prevailing scheduling algorithms by achieving much lower tail latency while accelerating DRL training speed by an average of 151.63x.

Deep-Reinforcement-Learning-Based IRS for Cooperative Jamming Networks Under Edge Computing

Effective energy design and secure communications are critical in the Internet of Things (IoT). A cooperative jamming (CJ) scheme based on deep reinforcement learning (DRL) is proposed for intelligent reflecting surface (IRS) aided secure cooperative network with eavesdroppers. Our goal is to maximize the average secrecy rate, or energy efficiency with secrecy constraints by jointly optimising the transmit, jamming beamforming matrices and IRS phase-shift matrix. In order to figure out the challenging non-convex optimization problem, the deep deterministic policy gradient (DDPG) optimization algorithm is proposed to solve the energy efficiency maximization problem. In order to speed up task processing and reduce service delay, the edge computing server is employed for DRL training to improve computing speed and promote secure communication efficiency. By taking advantages of the features of proximity to end devices and computing resources, the edge devices provide a low latency efficient training support to the legal users and real-time control to the IRS units. By this way, a high security and low energy consumption IoT system is built. Numerical results show that the novel scheme improves the secrecy rate and energy efficiency compared with the benchmark scheme.