Deep Reinforcement Learning Strategy Research Articles

Federated deep reinforcement learning for mobile robot navigation

Navigation of a mobile robot in an unknown environment ensuring the safety of the robot and its surroundings is of utmost importance. Traditional methods, such as pathplanning algorithms, simultaneous localization and mapping, computer vision, and fuzzy techniques, have been employed to address this challenge. However, to achieve better generalization and self-improvement capabilities, reinforcement learning has gained significant attention. The concern of privacy issues in sharing data is also rising in various domains. In this study, a deep reinforcement learning strategy is applied to the mobile robot to move from its initial position to a destination. Specifically, the Deep Q-Learning algorithm has been used for this purpose. This strategy is trained using a federated learning approach to overcome privacy issues and to set a foundation for further analysis of distributed learning. The application scenario considered in this work involves the navigation of a mobile robot to a charging point within a greenhouse environment. The results obtained indicate that both the traditional deep reinforcement learning and federated deep reinforcement learning frameworks are providing 100% success rate. However federated deep reinforcement learning could be a better alternate since it overcomes the privacy issue along with other advantages discussed in this paper.

A Deep Reinforcement Learning Strategy for MEC Enabled Virtual Reality in Telecommunication Networks

A Deep Reinforcement Learning Strategy for MEC Enabled Virtual Reality in Telecommunication Networks

Mathematical Model-Driven Deep Learning Enables Personalized Adaptive Therapy.

Standard-of-care treatment regimens have long been designed for maximal cell killing, yet these strategies often fail when applied to metastatic cancers due to the emergence of drug resistance. Adaptive treatment strategies have been developed as an alternative approach, dynamically adjusting treatment to suppress the growth of treatment-resistant populations and thereby delay, or even prevent, tumor progression. Promising clinical results in prostate cancer indicate the potential to optimize adaptive treatment protocols. Here, we applied deep reinforcement learning (DRL) to guide adaptive drug scheduling and demonstrated that these treatment schedules can outperform the current adaptive protocols in a mathematical model calibrated to prostate cancer dynamics, more than doubling the time to progression. The DRL strategies were robust to patient variability, including both tumor dynamics and clinical monitoring schedules. The DRL framework could produce interpretable, adaptive strategies based on a single tumor burden threshold, replicating and informing optimal treatment strategies. The DRL framework had no knowledge of the underlying mathematical tumor model, demonstrating the capability of DRL to help develop treatment strategies in novel or complex settings. Finally, a proposed five-step pathway, which combined mechanistic modeling with the DRL framework and integrated conventional tools to improve interpretability compared to traditional "black-box" DRL models, could allow translation of this approach to the clinic. Overall, the proposed framework generated personalized treatment schedules that consistently outperformed clinical standard-of-care protocols.

Enhancing Stability and Performance in Mobile Robot Path Planning with PMR-Dueling DQN Algorithm.

Path planning for mobile robots in complex circumstances is still a challenging issue. This work introduces an improved deep reinforcement learning strategy for robot navigation that combines dueling architecture, Prioritized Experience Replay, and shaped Rewards. In a grid world and two Gazebo simulation environments with static and dynamic obstacles, the Dueling Deep Q-Network with Modified Rewards and Prioritized Experience Replay (PMR-Dueling DQN) algorithm is compared against Q-learning, DQN, and DDQN in terms of path optimality, collision avoidance, and learning speed. To encourage the best routes, the shaped Reward function takes into account target direction, obstacle avoidance, and distance. Prioritized replay concentrates training on important events while a dueling architecture separates value and advantage learning. The results show that the PMR-Dueling DQN has greatly increased convergence speed, stability, and overall performance across conditions. In both grid world and Gazebo environments the PMR-Dueling DQN achieved higher cumulative rewards. The combination of deep reinforcement learning with reward design, network architecture, and experience replay enables the PMR-Dueling DQN to surpass traditional approaches for robot path planning in complex environments.

Heating ventilation air‐conditioner system for multi‐regional commercial buildings based on deep reinforcement learning

AbstractIn an era of significant energy consumption by commercial building HVAC systems, this study introduces a Deep Reinforcement Learning (DRL) approach to optimize these systems in multi‐zone commercial buildings, targeting reduced energy usage and enhanced user comfort. The research begins with the development of an energy consumption model for multi‐zone HVAC systems, considering the complexity and uncertainty of system parameters. This model informs the creation of a novel DRL‐based optimization algorithm, which incorporates multi‐stage training and a multi‐agent attention mechanism, enhancing stability and scalability. Comparative analysis against traditional control methods shows the proposed algorithm's effectiveness in reducing energy consumption while maintaining indoor comfort. The study presents an innovative DRL strategy for energy management in commercial HVAC systems, offering substantial potential for sustainable practices in building management.

Abstract PR001: Learning to adapt: Rational personalization of adaptive therapy using deep reinforcement learning

Abstract Standard-of-care treatment regimens have long been designed for maximal cell kill, yet these strategies often fail when applied to metastatic cancers due to the emergence of drug resistance. Adaptive treatment strategies have been developed as an alternative approach, dynamically adjusting treatment to suppress the growth of treatment-resistant populations, delaying, or even preventing, tumor progression. Promising clinical results in prostate cancer indicate the potential to optimize adaptive treatment protocols. We propose the application of deep reinforcement learning (DRL) to guide adaptive drug scheduling, and demonstrate that these treatment schedules can outperform the current adaptive protocols in a mathematical model calibrated to prostate cancer dynamics, more than doubling the time to progression. We show that the DRL strategies are robust to patient variability, including both tumor dynamics and clinical monitoring schedules. We demonstrate that the DRL framework can produce interpretable, adaptive strategies based on a single tumor burden threshold, replicating and informing optimal treatment strategies. Given the DRL framework has no knowledge of the underlying mathematical tumor model, this demonstrates the capability of DRL to help develop treatment strategies in novel or complex settings. Finally, we propose a five-step pathway combining mechanistic modeling with the DRL framework to translate this approach to the clinic with limited information for a given patient. Integrating conventional tools to ensure the interpretability of traditionally `black-box' DRL models, we generate personalized treatment schedules that consistently outperform clinical standard-of-care protocols. Citation Format: Kit Gallagher, Maximilian Strobl, Robert Gatenby, Philip Maini, Alexander Anderson. Learning to adapt: Rational personalization of adaptive therapy using deep reinforcement learning [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Translating Cancer Evolution and Data Science: The Next Frontier; 2023 Dec 3-6; Boston, Massachusetts. Philadelphia (PA): AACR; Cancer Res 2024;84(3 Suppl_2):Abstract nr PR001.

A Deep Reinforcement Learning Strategy for Surrounding Vehicles-Based Lane-Keeping Control.

As autonomous vehicles (AVs) are advancing to higher levels of autonomy and performance, the associated technologies are becoming increasingly diverse. Lane-keeping systems (LKS), corresponding to a key functionality of AVs, considerably enhance driver convenience. With drivers increasingly relying on autonomous driving technologies, the importance of safety features, such as fail-safe mechanisms in the event of sensor failures, has gained prominence. Therefore, this paper proposes a reinforcement learning (RL) control method for lane-keeping, which uses surrounding object information derived through LiDAR sensors instead of camera sensors for LKS. This approach uses surrounding vehicle and object information as observations for the RL framework to maintain the vehicle's current lane. The learning environment is established by integrating simulation tools, such as IPG CarMaker, which incorporates vehicle dynamics, and MATLAB Simulink for data analysis and RL model creation. To further validate the applicability of the LiDAR sensor data in real-world settings, Gaussian noise is introduced in the virtual simulation environment to mimic sensor noise in actual operational conditions.

COLREGs-Based Path Planning for USVs Using the Deep Reinforcement Learning Strategy

This research introduces a two-stage deep reinforcement learning approach for the cooperative path planning of unmanned surface vehicles (USVs). The method is designed to address cooperative collision-avoidance path planning while adhering to the International Regulations for Preventing Collisions at Sea (COLREGs) and considering the collision-avoidance problem within the USV fleet and between USVs and target ships (TSs). To achieve this, the study presents a dual COLREGs-compliant action-selection strategy to effectively manage the vessel-avoidance problem. Firstly, we construct a COLREGs-compliant action-evaluation network that utilizes a deep learning network trained on pre-recorded TS avoidance trajectories by USVs in compliance with COLREGs. Then, the COLREGs-compliant reward-function-based action-selection network is proposed by considering various TS encountering scenarios. Consequently, the results of the two networks are fused to select actions for cooperative path-planning processes. The path-planning model is established using the multi-agent proximal policy optimization (MAPPO) method. The action space, observation space, and reward function are tailored for the policy network. Additionally, a TS detection method is introduced to detect the motion intentions of TSs. The study conducted Monte Carlo simulations to demonstrate the strong performance of the planning method. Furthermore, experiments focusing on COLREGs-based TS avoidance were carried out to validate the feasibility of the approach. The proposed TS detection model exhibited robust performance within the defined task.

Impact-Angle Constraint Guidance and Control Strategies Based on Deep Reinforcement Learning

In this study, two different impact-angle-constrained guidance and control strategies using deep reinforcement learning (DRL) are proposed. The proposed strategies are based on the dual-loop and integrated guidance and control types. To address comprehensive flying object dynamics and the control mechanism, a Markov decision process is used to solve the guidance and control problem, and a real-time impact-angle error in the state vector is used to improve the model applicability. In addition, a reasonable reward mechanism is designed based on the state component which reduces both the miss distance and the impact-angle error and solves the problem of sparse rewards in DRL. Further, to overcome the negative effects of unbounded distributions on bounded action spaces, a Beta distribution is used instead of a Gaussian distribution in the proximal policy optimization algorithm for policy sampling. The state initialization is then realized using a sampling method adjusted to engineering backgrounds, and the control strategy is adapted to a wide range of operational scenarios with different impact angles. Simulation and Monte Carlo experiments in various scenarios show that, compared with other methods mentioned in the experiment in this paper, the proposed DRL strategy has smaller impact-angle errors and miss distance, which demonstrates the method’s effectiveness, applicability, and robustness.

A deep reinforcement learning strategy for autonomous robot flocking

<p>Social behaviors in animals such as bees, ants, and birds have shown high levels of intelligence from a multi-agent system perspective. They present viable solutions to real-world problems, particularly in navigating constrained environments with simple robotic platforms. Among these behaviors is swarm flocking, which has been extensively studied for this purpose. Flocking algorithms have been developed from basic behavioral rules, which often require parameter tuning for specific applications. However, the lack of a general formulation for tuning has made these strategies difficult to implement in various real conditions, and even to replicate laboratory behaviors. In this paper, we propose a flocking scheme for small autonomous robots that can self-learn in dynamic environments, derived from a deep reinforcement learning process. Our approach achieves flocking independently of population size and environmental characteristics, with minimal external intervention. Our multi-agent system model considers each agent’s action as a linear function dynamically adjusting the motion according to interactions with other agents and the environment. Our strategy is an important contribution toward real-world flocking implementation. We demonstrate that our approach allows for autonomous flocking in the system without requiring specific parameter tuning, making it ideal for applications where there is a need for simple robotic platforms to navigate in dynamic environments.</p>

A hybrid decision support system for adaptive trading strategies: Combining a rule-based expert system with a deep reinforcement learning strategy

Stock trading strategies pose challenging applications of machine learning for significant commercial yields in the finance industry, drawing the attention of both economists and computer scientists. Until now, many researchers have proposed various methods to implement intelligent trading strategy systems that can support decisions regarding stock trading. Some studies have shown that the problem of trading strategies can be successfully addressed by applying hybrid approaches. Motivated by this, we propose a hybrid decision support system for adaptive trading strategies that combines a rule-based system with deep reinforcement learning to self-improve by learning with human expertise. This study overcomes the limitations of previous hybrid models that mainly have focused on optimizing trading decisions and improving forecasting accuracy. The proposed hybrid model combines decision-making information from a rule-based model to enable the agent of reinforcement learning to capture more trading opportunities. In addition, the investor's available balance states facilitate adaptive learning by interacting with the environment. Moreover, the proposed trading mechanism adjusts the volume size using the policy gradient algorithm's action probabilities, resulting in improved risk-adjusted returns. The proposed hybrid model has the potential to be a reliable trading system in real-world applications through its ability to adapt to different market scenarios, withstand stressful market conditions, reduce transaction costs, scale to various index funds, and extend the proposed hybrid structure. This study highlights the applicability of more advanced machine learning in financial areas, and we also suggest expanding this approach to adaptive decision-making systems in other fields.

Deep Reinforcement Learning for Flipper Control of Tracked Robots in Urban Rescuing Environments

Tracked robots equipped with flippers and LiDAR sensors have been widely used in urban search and rescue. Achieving autonomous flipper control is important in enhancing the intelligent operation of tracked robots within complex urban rescuing environments. While existing methods mainly rely on the heavy work of manual modeling, this paper proposes a novel Deep Reinforcement Learning (DRL) approach named ICM-D3QN for autonomous flipper control in complex urban rescuing terrains. Specifically, ICM-D3QN comprises three modules: a feature extraction and fusion module for extracting and integrating robot and environment state features, a curiosity module for enhancing the efficiency of flipper action exploration, and a deep Q-Learning control module for learning robot-control policy. In addition, a specific reward function is designed, considering both safety and passing smoothness. Furthermore, simulation environments are constructed using the Pymunk and Gazebo physics engine for training and testing. The learned policy is then directly transferred to our self-designed tracked robot in a real-world environment for quantitative analysis. The consistently high performance of the proposed approach validates its superiority over hand-crafted control models and state-of-the-art DRL strategies for crossing complex terrains.

Sequential and Correlated Image Hash Code Generation with Deep Reinforcement Learning

Image hashing is an algorithm used to represent an image with a unique value. Hashing methods, which are generally developed to search for similar examples of an image, have gained a new dimension with the use of deep network structures and better results have started to be obtained with the methods. The developed deep network models generally consider hash functions independently and do not take into account the correlation between them. In addition, most of the existing data-dependent hashing methods use pairwise/triplet similarity metrics that capture data relationships from a local perspective. In this study, the Central similarity metric, which can achieve better results, is adapted to the deep reinforcement learning method with sequential learning strategy, and successful results are obtained in learning binary hash codes. By taking into account the errors of previous hash functions in the deep reinforcement learning strategy, a new model is presented that performs interrelated and central similarity based learning.

Leveraging Minimum Nodes for Optimum Key Player Identification in Complex Networks: A Deep Reinforcement Learning Strategy with Structured Reward Shaping

The problem of finding key players in a graph, also known as network dismantling, or network disintegration, aims to find an optimal removal sequence of nodes (edges, substructures) through a certain algorithm, ultimately causing functional indicators such as the largest connected component (GCC) or network pair connectivity in the graph to rapidly decline. As a typical NP-hard problem on graphs, recent methods based on reinforcement learning and graph representation learning have effectively solved such problems. However, existing reinforcement-learning-based key-player-identification algorithms often need to remove too many nodes in order to achieve the optimal effect when removing the remaining network until no connected edges remain. The use of a minimum number of nodes while maintaining or surpassing the performance of existing methods is a worthwhile research problem. To this end, a novel algorithm called MiniKey was proposed to tackle such challenges, which employs a specific deep Q-network architecture for reinforcement learning, a novel reward-shaping mechanism based on network functional indicators, and the graph-embedding technique GraphSage to transform network nodes into latent representations. Additionally, a technique dubbed ‘virtual node technology’ is integrated to grasp the overarching feature representation of the whole network. This innovative algorithm can be effectively trained on small-scale simulated graphs while also being scalable to large-scale real-world networks. Importantly, experiments from both six simulated datasets and six real-world datasets demonstrates that MiniKey can achieve optimal performance, striking a perfect balance between the effectiveness of key node identification and the minimization of the number of nodes that is utilized, which holds potential for real-world applications such as curbing misinformation spread in social networks, optimizing traffic in transportation systems, and identifying key targets in biological networks for targeted interventions.

A Multi-Task Fusion Strategy-Based Decision-Making and Planning Method for Autonomous Driving Vehicles.

The autonomous driving technology based on deep reinforcement learning (DRL) has been confirmed as one of the most cutting-edge research fields worldwide. The agent is enabled to achieve the goal of making independent decisions by interacting with the environment and learning driving strategies based on the feedback from the environment. This technology has been widely used in end-to-end driving tasks. However, this field faces several challenges. First, developing real vehicles is expensive, time-consuming, and risky. To further expedite the testing, verification, and iteration of end-to-end deep reinforcement learning algorithms, a joint simulation development and validation platform was designed and implemented in this study based on VTD-CarSim and the Tensorflow deep learning framework, and research work was conducted based on this platform. Second, sparse reward signals can cause problems (e.g., a low-sample learning rate). It is imperative for the agent to be capable of navigating in an unfamiliar environment and driving safely under a wide variety of weather or lighting conditions. To address the problem of poor generalization ability of the agent to unknown scenarios, a deep deterministic policy gradient (DDPG) decision-making and planning method was proposed in this study in accordance with a multi-task fusion strategy. The main task based on DRL decision-making planning and the auxiliary task based on image semantic segmentation were cross-fused, and part of the network was shared with the main task to reduce the possibility of model overfitting and improve the generalization ability. As indicated by the experimental results, first, the joint simulation development and validation platform built in this study exhibited prominent versatility. Users were enabled to easily substitute any default module with customized algorithms and verify the effectiveness of new functions in enhancing overall performance using other default modules of the platform. Second, the deep reinforcement learning strategy based on multi-task fusion proposed in this study was competitive. Its performance was better than other DRL algorithms in certain tasks, which improved the generalization ability of the vehicle decision-making planning algorithm.

Virtual Inertia Control of Isolated Microgrids Using an NN-Based VFOPID Controller

Reduction in system inertia and maintaining the frequency at the nominal value is a staple of today's and future power systems since their operation, stability, and resiliency are degraded by frequency oscillation and cascading failures. Consequently, designing a stable, scalable, and robust virtual inertia control system is highly relevant to skillfully diminishing the deviations during major contingencies. Therefore, considering the potential problems in predesigned nonflexible control systems with offline tuning techniques, we propose a variable fractional-order PID controller for virtual inertia control applications, which is tuned online using a modified neural network-based algorithm. The new proposed tuner algorithm is trained using a deep reinforcement learning strategy with a simplified deep deterministic policy gradient, which considers microgrid uncertainties. Compared with existing methods, all the tuning knobs of the discrete type and fully tunable variable FOPID controller (for both gain and order) can be captured based on the proposed hybrid algorithm, which inherits features from both classical and advanced techniques. To demonstrate the effectiveness of the training of the proposed controller, a comparative analysis with the standard FOPID and PID controllers is given under three different scenarios with a smooth (dis)connection of renewable energy sources and loads.

Real-time planning and collision avoidance control method based on deep reinforcement learning

Real time planning and collision avoidance is an important part of the motion planning for USVs, its core is to design effective planning and collision avoidance methods. This paper proposes a real-time planning and collision avoidance method based on deep reinforcement learning for USVs in complex environments. Firstly, a novel reward function is designed, which transforms the collision avoidance problem into a deep reinforcement learning strategy solution problem that includes distance, direction, speed, and GOLREGs constraints, it ensures the smooth and safe operation of the USV. Secondly, inefficient exploration and exploitation methods make it difficult for intelligent agents to discover key information in the environment, high value experience is obtained through the designed layered sampling exploration mechanism in this article, the intelligent agent learns effective control strategies through this mechanism, which accelerates the convergence of the strategies and reduces the waste of computing resources. Finally, a real-time collision avoidance simulation platform for the USV is established, and experimental results of different exploration mechanisms are compared, the efficiency of the layered sampling exploration mechanism has been verified. The simulation scenarios from easy to difficult are designed, and the results indicate that the USV can complete real-time planning and collision avoidance tasks in complex environments.

LFDC: Low-Energy Federated Deep Reinforcement Learning for Caching Mechanism in Cloud–Edge Collaborative

The optimization of caching mechanisms has long been a crucial research focus in cloud–edge collaborative environments. Effective caching strategies can substantially enhance user experience quality in these settings. Deep reinforcement learning (DRL), with its ability to perceive the environment and develop intelligent policies online, has been widely employed for designing caching strategies. Recently, federated learning, when combined with DRL, has been in gaining popularity for optimizing caching strategies and protecting data training privacy from eavesdropping attacks. However, online federated deep reinforcement learning algorithms face high environmental dynamics, and real-time training can result in increased training energy consumption despite improving caching efficiency. To address this issue, we propose a low-energy federated deep reinforcement learning strategy for caching mechanisms (LFDC) that balances caching efficiency and training energy consumption. The LFDC strategy encompasses a novel energy efficiency model, a deep reinforcement learning mechanism, and a dynamic energy-saving federated policy. Our experimental results demonstrate that the proposed LFDC strategy significantly outperforms existing benchmarks in terms of energy efficiency.

A Dueling Deep Q-Network method for low-carbon traffic signal control

Improper traffic signal control will lead to long delays for vehicles and produce massive carbon emissions. The vast vehicle exhaust emissions will pollute the environment and exacerbate the earth’s greenhouse effect. Intersection signal optimization tends to start from the traditional view of improving traffic efficiency but ignores the perspective of reducing vehicle carbon emissions. Under the framework of a deep reinforcement learning strategy, this study proposes a novel signal control method to minimize the carbon emissions of vehicles at the intersection. To associate with carbon emissions and signal control plans, the method employs the negative value of vehicle’s carbon dioxide emissions as the reward and takes the feature vectors at different time points in the two decision action intervals as the state features. The fully connected neural network, convolutional neural network, and long short-term memory network are respectively adopted to extract the state features of the decision-making period and compare their Q-value estimation effects. Through the SUMO simulation platform, the proposed signal control method is comprehensively evaluated and compared with different baseline models. It has been proved that the proposed signal control approach can not only directly reduce vehicle carbon emissions but also improve the operational efficiency of the intersection.

Dynamic Offloading Loading Optimization in Distributed Fault Diagnosis System with Deep Reinforcement Learning Approach

Artificial intelligence and distributed algorithms have been widely used in mechanical fault diagnosis with the explosive growth of diagnostic data. A novel intelligent fault diagnosis system framework that allows intelligent terminals to offload computational tasks to Mobile edge computing (MEC) servers is provided in this paper, which can effectively address the problems of task processing delays and enhanced computational complexity. As the resources at the MEC and intelligent terminals are limited, performing reasonable resource allocation optimization can improve the performance, especially for a multi-terminals offloading system. In this study, to minimize the task computation delay, we jointly optimize the local content splitting ratio, the transmission/computation power allocation, and the MEC server selection under a dynamic environment with stochastic task arrivals. The challenging dynamic joint optimization problem is formulated as a reinforcement learning (RL) problem, which is designed as the computational offloading policies to minimize the long-term average delay cost. Two deep RL strategies, deep Q-learning network (DQN) and deep deterministic policy gradient (DDPG), are adopted to learn the computational offloading policies adaptively and efficiently. The proposed DQN strategy takes the MEC selection as a unique action while using the convex optimization approach to obtain the local content splitting ratio and the transmission/computation power allocation. Simultaneously, the actions of the DDPG strategy are selected as all dynamic variables, including the local content splitting ratio, the transmission/computation power allocation, and the MEC server selection. Numerical results demonstrate that both proposed strategies perform better than the traditional non-learning schemes. The DDPG strategy outperforms the DQN strategy in all simulation cases exhibiting minimal task computation delay due to its ability to learn all variables online.