Application Of Deep Reinforcement Learning Research Articles

A DRL-Based Satellite Service Allocation Method in LEO Satellite Networks

Satellite computing represents a recent computational paradigm in the development of low Earth orbit (LEO) satellites. It aims to augment the capabilities of LEO satellites beyond their current transparent relay functions by enabling real-time processing, thereby providing low-latency computational services to end users. In LEO constellations, a significant deployment of computationally capable satellites is orchestrated to offer enhanced computational resources. Challenges arise in the optimal allocation of terminal services to the most suitable satellite due to overlapping coverage among neighboring satellites, compounded by constraints on satellite energy and computational resources. The satellite service allocation (SSA) problem is recognized as NP-hard, yet assessing allocation methods through results allows for the application of deep reinforcement learning (DRL) to obtain improved solutions, partially addressing the SSA challenge. In this paper, we introduce a satellite computing capability model to quantify satellite computational resources. A DRL model is proposed to address service demands, computational resources, and resolve service allocation conflicts, strategically placing each service on appropriate servers. Through simulation experiments, numerical results demonstrate the superiority of our proposed method over baseline approaches in service allocation and satellite resource utilization, showcasing advancements in this field.

Application of deep reinforcement learning in electricity demand response market: Demand response decision-making of load aggregator

With the large-scale integration of renewable energy generation, developing the potential of demand response is becoming more and more significant. However, the number of consumers is huge, the electricity consumption behaviors are various and the information of consumers is incomplete, thus there are great difficulties in modeling and processing demand response by the conventional mathematical methods. Therefore, how to accurately predict the response behaviors of consumers and select the appropriate consumers for demand response is worthy of in-depth discussion. In this paper, the decision-making method of the load aggregators who participant in demand response market on behalf of the small and medium-sized consumers based on Deep Q-network algorithm is proposed, supporting the aggregators to properly guide the potential demand response consumers. At the same time, a dynamic consumer classification method based on self-organizing maps algorithm is also proposed, supporting the aggregators to accurately predict the response behaviors of the consumers. Simulation results show that the proposed method can effectively realize the classification of the consumers and result in a more beneficial demand response.•The proposed method does not require complete information, such as demand response behavior parameters of the consumers.•Self-organizing maps can realize a dynamic classification of demand response consumers as the case may be.•DQN algorithm can effectively realize the demand response decision-making of aggregators under the condition of incomplete information.

Promoting sustainable tourism by recommending sequences of attractions with deep reinforcement learning

AbstractDeveloping Recommender Systems (RSs) is particularly interesting in the tourist domain, where one or more attractions have to be suggested to users based on preferences, contextual dimensions, and several other constraints. RSs usually rely on the availability of a vast amount of historical information about users’ past activities. However, this is not usually the case in the tourist domain, where acquiring complete and accurate information about the user’s behavior is complex, and providing personalized suggestions is frequently practically impossible. Moreover, even though most available Touristic RSs (T-RSs) are user-focused, the touristic domain also requires the development of systems that can promote a more sustainable form of tourism. The concept of sustainable tourism covers many aspects, from economic, social, and environmental issues to the attention to improving tourists’ experience and the needs of host communities. In this regard, one of the most important aspects is the prevention of overcrowded situations in attractions or locations (over-tourism). For this reason, this paper proposes a different kind of T-RS, which focuses more on the tourists’ impact on the destinations, trying to improve their experiences by offering better visit conditions. Moreover, instead of suggesting the next Point of Interest (PoI) to visit in a given situation, it provides a suggestion about a complete sequence of PoIs (tourist itinerary) that covers an entire day or vacation period. The proposed technique is based on the application of Deep Reinforcement Learning, where the tourist’s reward depends on the specific spatial and temporal context in which the itinerary has to be performed. The solution has been evaluated with a real-world dataset regarding the visits conducted by tourists in Verona (Italy) from 2014 to 2023 and compared with three baselines.

Knowledge transfer for adaptive maintenance policy optimization in engineering fleets based on meta-reinforcement learning

Maintenance policy optimization is crucial for ensuring the efficient functioning of structures and systems and mitigating the risk of deterioration. Reinforcement learning methods, especially when combined with deep neural networks, have seen significant progress in supporting maintenance decisions. However, deep reinforcement learning (DRL) typically necessitates an extensive number of interactions with the system to acquire the optimal policy, resulting in data inefficiency issues that limit the application of DRL in practical engineering fleet problems. Deriving optimal policies with DRL repeatedly for every individual in the engineering fleet can be computationally expensive or even prohibitive. To address the data inefficiency issues, this study proposes a novel maintenance optimization approach that can transfer knowledge from previously learned maintenance cases to the new cases to accelerate the DRL process. Meta-reinforcement learning (Meta-RL) method is proposed to realize the concept of knowledge transfer within a fleet by learning a meta-learned policy. In particular, the meta-learned policy can be quickly adapted to each individual case of the engineering fleet, thereby reducing the required computational burden for maintenance policy optimization. Two examples are used to demonstrate the effectiveness of knowledge transfer.

Application of Deep Reinforcement Learning (DRL) for Malware Detection

Malware poses a significant threat to computer systems and networks, necessitating advanced detection methods to safeguard against potential cyber-attacks. This paper investigates the application of Deep Reinforcement Learning (DRL) for malware detection, leveraging its ability to learn complex patterns and behaviours from raw data. The study employs a DRL framework to train an agent to identify malicious software based on dynamic features extracted from executable files. A comprehensive evaluation is conducted using a diverse dataset, encompassing various types of malware samples. The experimental results demonstrate the effectiveness of the proposed DRL based approach in accurately detecting malware, achieving competitive performance compared to traditional methods and state-of-the-art techniques. Additionally, the paper discusses the interpretability and scalability of the model, along with potential challenges and future research directions in applying DRL to cybersecurity.

Deep Reinforcement Learning: A Survey.

Deep reinforcement learning (DRL) integrates the feature representation ability of deep learning with the decision-making ability of reinforcement learning so that it can achieve powerful end-to-end learning control capabilities. In the past decade, DRL has made substantial advances in many tasks that require perceiving high-dimensional input and making optimal or near-optimal decisions. However, there are still many challenging problems in the theory and applications of DRL, especially in learning control tasks with limited samples, sparse rewards, and multiple agents. Researchers have proposed various solutions and new theories to solve these problems and promote the development of DRL. In addition, deep learning has stimulated the further development of many subfields of reinforcement learning, such as hierarchical reinforcement learning (HRL), multiagent reinforcement learning, and imitation learning. This article gives a comprehensive overview of the fundamental theories, key algorithms, and primary research domains of DRL. In addition to value-based and policy-based DRL algorithms, the advances in maximum entropy-based DRL are summarized. The future research topics of DRL are also analyzed and discussed.

Deep reinforcement learning based computing offloading in unmanned aerial vehicles for disaster management

Abstract The emergence of Internet of Things enabled with mobile computing has the applications in the field of unmanned aerial vehicle (UAV) development. The development of mobile edge computational offloading in UAV is dependent on low latency applications such as disaster management, Forest fire control and remote operations. The task completion efficiency is improved by means of using edge intelligence algorithm and the optimal offloading policy is constructed on the application of deep reinforcement learning (DRL) in order to fulfill the target demand and to ease the transmission delay. The joint optimization curtails the weighted sum of average energy consumption and execution delay. This edge intelligence algorithm combined with DRL network exploits computing operation to increase the probability that at least one of the tracking and data transmission is usable. The proposed joint optimization significantly performs well in terms of execution delay, offloading cost and effective convergence over the prevailing methodologies proposed for UAV development. The proposed DRL enables the UAV to real-time decisions based on the disaster scenario and computing resources availability.

Efficient deep reinforcement learning under task variations via knowledge transfer for drone control

Despite the growing interest in using deep reinforcement learning (DRL) for drone control, several challenges remain to be addressed, including issues with generalization across task variations and agent training (which requires significant computational power and time). When the agent’s input changes owing to the drone’s sensors or mission variations, significant retraining overhead is required to handle the changes in the input data pattern and the neural network architecture to accommodate the input data. These difficulties severely limit their applicability in dynamic real-world environments. In this paper, we propose an efficient DRL method that leverages the knowledge of the source agent to accelerate the training of the target agent under task variations. The proposed method consists of three phases: collecting training data for the target agent using the source agent, supervised pre-training of the target agent, and DRL-based fine-tuning. Experimental validation demonstrated a remarkable reduction in the training time (up to 94.29%), suggesting a potential avenue for the successful and efficient application of DRL in drone control.

Based on Deep Reinforcement Learning to path planning in uncertain ocean currents for Underwater Gliders

This paper studies the application of Deep Reinforcement Learning (DRL) in the field of underwater glider (UG) formation path planning and solves the path planning of glider formation in uncertain ocean currents. Facing the uncertain ocean current environment, based on the observation and collection of ocean current data by the overall ocean observation system, the glider formation needs to have the ability of path planning. The original DRL algorithm network architecture is simple, and the feature extraction ability is limited. According to the characteristics of ocean current and path planning, a Deep Convolutional Neural Network for uncertain ocean current environment (Duo-CNN) network architecture with a dual-branch structure is proposed. Based on the DRL, the glider model and the interaction between the glider and the environment are used to guide the path planning through the implicit prediction of the known ocean current data. A solution for formation construction, obstacle avoidance, and path planning of glider formation in a dynamic uncertain ocean current environment is provided. The simulation verifies the effectiveness of the path planning of the glider formation using the DRL of the dual-branch Duo-CNN architecture. Compared with other architectures with feature extraction ability, Duo-CNN has advantages.

Research on intelligent fault diagnosis for railway point machines using deep reinforcement learning

Abstract The advanced diagnosis of faults in railway point machines is momentous to ensure the efficient and stable operation of the turnout conversion system. Numerous mature deep learning methods have been extensively applied in this domain. While robust perception has yielded excellent diagnostic outcomes, the deficiency in decision-making capability has led to a lack of overall intelligence. To deal with this, this study proposes an end-to-end deep reinforcement learning (DRL) framework for diagnosing faults in railway point machines. Firstly, a one-dimensional convolutional neural network (1DCNN) is used for the automatic extraction of features from the current signal. Subsequently, the deep Q network (DQN) algorithm is introduced as the core of the diagnostic framework. This involves designing an interactive environment for fault classification and optimizing the agent training network. Finally, leveraging fault data, the agent and the environment engage in continuous interactive learning to produce the ideal classification policy. Multiple comparative experiments are conducted to validate the proposed method. The results demonstrate that the diagnostic accuracy reaches 98.41%, and the average accuracy after many iterations is as high as 99.12%. Notably, this research introduces a creative application of DRL to address the challenge of diagnosing faults in railway point machines. The incorporation of decision thought effectively enhances the intelligence of fault diagnosis.

Deep reinforcement learning enhanced skeleton based pipe routing for high-throughput transmission in flying ad-hoc networks

Artificial Intelligence (AI) is a huge step towards the emergence of powerful and smart decision-makers which can optimally tune the network parameters. In this work, a novel idea of an intelligent pipe (iPipe) routing scheme for a network of unmanned aerial vehicles (UAVs) is presented by considering the availability of a centralized entity with enough computational resources. The proposed scheme deploys artificial intelligence to optimally decide on the direction of the routing pipe (i.e. the medial axis), based on the current network information and the prediction of the immediate future. To address the potential influx of input data, the application of Deep Reinforcement Learning (DRL) is proposed to enhance the throughput performance and adapt more effectively to network dynamics. The scheme benefits from the concept of flying ad hoc network (FANET) to provide nodes with geometric coordinates in order to facilitate the data forwarding mechanism. Compared to the conventional AI-based routing strategies, the proposed scheme does not require making decisions for the routing table of all the nodes. Instead, it only finds the trend as a set of geometric indices which will be included in the packet headers. However, the pipe nodes still act in a distributed manner. They build their routing tables according to the trend suggested by the controller and adjust the transmission probability of each link based on the local feedback. This reduces the complexity of the centralized solution. The simulation results show that the proposed iPipe method can outperform the other state-of-the-art SSR routing scheme, which is a distributed but low-cost pipe routing.

Deep Reinforcement Learning Approach for Cyberattack Detection

Recently, there has been a growing concern regarding the detrimental effects of cyberattacks on both infrastructure and users. Conventional safety measures, such as encryption, firewalls, and intrusion detection, are inadequate to safeguard cyber systems against emerging and evolving threats. To address this issue, researchers have turned to reinforcement learning (RL) as a potential solution for complex decision-making problems in cybersecurity. However, the application of RL faces various obstacles, including a lack of suitable training data, dynamic attack scenarios, and challenges in modeling real-world complexities. This paper suggests applying deep reinforcement learning (DRL), a deep framework, to simulate malicious cyberattacks and enhance cybersecurity. Our framework utilizes an agent-based model that is capable of continuous learning and adaptation within a dynamic network security environment. The agent determines the most optimal course of action based on the network’s state and the corresponding rewards received for its decisions. We present the outcomes of our experimentation with the application of DRL on a specific model, double deep Q-network (DDQN), utilizing policy gradient (PG) on three distinct datasets: NSL-KDD, CIC-IDS-2018, and AWID. Our research demonstrates that DRL can effectively improve cyberattack detection outcomes through our model and specific parameter adjustments.

A Survey of Deep Reinforcement Learning Based on Multi-Particle Environments

This paper extensively explores the application of deep reinforcement learning within multi-agent environments, a domain marked by intricate challenges and burgeoning opportunities. Multi-agent environments inherently entail the presence of multiple interacting intelligent agents, thereby amplifying the complexity of decision-making and control. Traditional reinforcement learning methods, when confronted with these intricate settings, manifest limitations that have prompted a compelling shift towards the fusion of deep learning and reinforcement learning. The journey commences by elucidating the multifaceted challenges and intricacies pervasive in multi-agent environments. It systematically delineates the constraints of conventional reinforcement learning techniques, underscoring their inadequacies in effectively addressing the multifarious complexities presented by these environments. As the narrative unfolds, it places a spotlight on the transformative potential of deep learning in surmounting the formidable challenges associated with decision-making in multi-agent settings. A critical examination ensues, encompassing a thorough review of pioneering applications of methodologies such as value functions, policy gradients, and actor-critic approaches within the realm of multi-agent systems. Through an exhaustive exploration of existing literature, this article endeavors to unveil the vanguard developments and persisting challenges within the domain of deep reinforcement learning in multi-agent environments. Furthermore, this paper conducts a comparative analysis of the performance of two prominent algorithms, namely Q-learning and the Deep Q Network (DQN) algorithm. This analysis serves as a compass, elucidating the trajectory of development within the sphere of deep reinforcement learning. Ultimately, this paper acts as a compass, guiding future research and innovation in the dynamic and promising field of deep reinforcement learning within multi-agent environments.

Towards Risk-Aware Real-Time Security Constrained Economic Dispatch: A Tailored Deep Reinforcement Learning Approach

In the presence of increasing uncertainties brought by intermittent renewable energy sources, security and risk management continue to be the most critical concern in modern power system operation. Risk-aware, real-time security constrained economic dispatch (RT-SCED) provides an efficient solution towards promptly, economically and robustly responding to the changes in the power system operating state. Despite different model-based methods have been developed to handle uncertainties, significant computation burden arise to incorporate <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$N$</tex-math></inline-formula> -1 contingency constraints with a higher temporal resolution in RT-SCED. Driven by similar computational challenges, risk evaluation is often overlooked in the current application of deep reinforcement learning (DRL) based data-driven methods in RT-SCED. This paper proposes a DRL-based risk-aware RT-SCED methodological framework by incorporating a novel data-driven risk evaluation model to foster efficient agent-environment interactions. The real-time dispatch policies are constructed with an improved twin delayed deep deterministic policy gradient method. The policy network features a residual network architecture and incorporates an active power allocation mechanism to integrate empirical dispatch knowledge, preventing early termination and fostering more efficient learning behavior. Case studies validate the superior performance of the proposed method in risk-aware RT-SCED on cost efficiency, uncertainty adaptability and computational efficiency, through benchmarking against model-based and data-driven baseline methods.

Autonomous Vehicles: Applications of Deep Reinforcement Learning

This research paper offers a concise exploration into the pivotal role played by deep reinforcement learning in propelling advancements in both perception and decision-making aspects of autonomous vehicles. Through the fusion of deep learning and reinforcement learning techniques, we delve into how these methodologies synergistically contribute to augmenting the navigational proficiency of autonomous vehicles amidst intricate and dynamically changing environments. With a focus on a streamlined presentation, this paper provides a succinct yet insightful overview of various perception algorithms utilized by autonomous vehicles, emphasizing object detection, semantic segmentation, and LiDAR-based techniques.

Security and Privacy Issues in Deep Reinforcement Learning: Threats and Countermeasures

Deep Reinforcement Learning (DRL) is an essential subfield of Artificial Intelligence (AI), where agents interact with environments to learn policies for solving complex tasks. In recent years, DRL has achieved remarkable breakthroughs in various tasks, including video games, robotic control, quantitative trading, and autonomous driving. Despite its accomplishments, security and privacy-related issues still prevent us from deploying trustworthy DRL applications. For example, by manipulating the environment, an attacker can influence an agent’s actions, misleading it to behave abnormally. Additionally, an attacker can infer private training data and environmental information by maliciously interacting with DRL models, causing a privacy breach. In this survey, we systematically investigate the recent progress of security and privacy issues in the context of DRL. First, we present a holistic review of security-related attacks within DRL systems from the perspectives of single-agent and multi-agent systems and review privacy-related attacks. Second, we review and classify defense methods used to address security-related challenges, including robust learning, anomaly detection, and game theory approaches. Third, we review and classify privacy-preserving technologies, including encryption, differential privacy, and policy confusion. We conclude the survey by discussing open issues and possible directions for future research in this field.

Uncertainty-aware human-like driving policy learning with deep Bayesian inverse reinforcement learning

The application of deep reinforcement learning in driving policy learning for automated vehicles is limited by the difficulty of designing reward functions. Most existing inverse reinforcement learning (IRL) methods make a structure assumption on the reward function and omit the uncertainty of neural networks. In view of this, this paper proposes a novel deep Bayesian IRL method that addresses both the reward function designing and the uncertainty-measuring issues by learning an approximate posterior distribution over the reward function. Furthermore, we propose to train uncertainty-aware human-like driving policies by maximising the predicted reward and penalising its uncertainty. Finally, the proposed methods were validated in simulated highway driving scenarios. The results show that AVRL models uncertainty and learns reward functions significantly outperforming the existing IRL method applied in automated driving. It was also found that penalising the uncertainty of the reward function during policy training improves the success rate and human likeness of the learned policy.

Abstract PR001: Learning to adapt: Rational personalization of adaptive therapy using deep reinforcement learning

Abstract Standard-of-care treatment regimens have long been designed for maximal cell kill, yet these strategies often fail when applied to metastatic cancers due to the emergence of drug resistance. Adaptive treatment strategies have been developed as an alternative approach, dynamically adjusting treatment to suppress the growth of treatment-resistant populations, delaying, or even preventing, tumor progression. Promising clinical results in prostate cancer indicate the potential to optimize adaptive treatment protocols. We propose the application of deep reinforcement learning (DRL) to guide adaptive drug scheduling, and demonstrate that these treatment schedules can outperform the current adaptive protocols in a mathematical model calibrated to prostate cancer dynamics, more than doubling the time to progression. We show that the DRL strategies are robust to patient variability, including both tumor dynamics and clinical monitoring schedules. We demonstrate that the DRL framework can produce interpretable, adaptive strategies based on a single tumor burden threshold, replicating and informing optimal treatment strategies. Given the DRL framework has no knowledge of the underlying mathematical tumor model, this demonstrates the capability of DRL to help develop treatment strategies in novel or complex settings. Finally, we propose a five-step pathway combining mechanistic modeling with the DRL framework to translate this approach to the clinic with limited information for a given patient. Integrating conventional tools to ensure the interpretability of traditionally `black-box' DRL models, we generate personalized treatment schedules that consistently outperform clinical standard-of-care protocols. Citation Format: Kit Gallagher, Maximilian Strobl, Robert Gatenby, Philip Maini, Alexander Anderson. Learning to adapt: Rational personalization of adaptive therapy using deep reinforcement learning [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Translating Cancer Evolution and Data Science: The Next Frontier; 2023 Dec 3-6; Boston, Massachusetts. Philadelphia (PA): AACR; Cancer Res 2024;84(3 Suppl_2):Abstract nr PR001.

Artificial Intelligence Gamers Based on Deep Reinforcement Learning

This study investigates the design and implementation of Artificial Intelligence (AI) game players based on deep reinforcement learning, offering a novel approach to autonomous decision-making and strategy acquisition in intelligent games. Initially, the fundamental principles and algorithms of deep reinforcement learning are introduced, along with the fusion of deep learning and reinforcement learning. Subsequently, existing research is reviewed, and the pros and cons of current methodologies are examined, highlighting the underlying issues and challenges. The utilization of AI players in mainstream games is then introduced, and the influence of AI players on contemporary games is analyzed. Through this analysis of AI players in mainstream games, the strengths and weaknesses of current AI players are identified, and recommendations for optimizing them are provided. This study holds significant implications for guiding the design and development of intelligent game players, while also enriching the application of deep reinforcement learning within the gaming domain.

Deep reinforcement learning identifies personalized intermittent androgen deprivation therapy for prostate cancer.

The evolution of drug resistance leads to treatment failure and tumor progression. Intermittent androgen deprivation therapy (IADT) helps responsive cancer cells compete with resistant cancer cells in intratumoral competition. However, conventional IADT is population-based, ignoring the heterogeneity of patients and cancer. Additionally, existing IADT relies on pre-determined thresholds of prostate-specific antigen to pause and resume treatment, which is not optimized for individual patients. To address these challenges, we framed a data-driven method in two steps. First, we developed a time-varied, mixed-effect and generative Lotka-Volterra (tM-GLV) model to account for the heterogeneity of the evolution mechanism and the pharmacokinetics of two ADT drugs Cyproterone acetate and Leuprolide acetate for individual patients. Then, we proposed a reinforcement-learning-enabled individualized IADT framework, namely, I$^{2}$ADT, to learn the patient-specific tumor dynamics and derive the optimal drug administration policy. Experiments with clinical trial data demonstrated that the proposed I$^{2}$ADT can significantly prolong the time to progression of prostate cancer patients with reduced cumulative drug dosage. We further validated the efficacy of the proposed methods with a recent pilot clinical trial data. Moreover, the adaptability of I$^{2}$ADT makes it a promising tool for other cancers with the availability of clinical data, where treatment regimens might need to be individualized based on patient characteristics and disease dynamics. Our research elucidates the application of deep reinforcement learning to identify personalized adaptive cancer therapy.