Deep Reinforcement Learning Model Research Articles

A DRL-Based Satellite Service Allocation Method in LEO Satellite Networks

Satellite computing represents a recent computational paradigm in the development of low Earth orbit (LEO) satellites. It aims to augment the capabilities of LEO satellites beyond their current transparent relay functions by enabling real-time processing, thereby providing low-latency computational services to end users. In LEO constellations, a significant deployment of computationally capable satellites is orchestrated to offer enhanced computational resources. Challenges arise in the optimal allocation of terminal services to the most suitable satellite due to overlapping coverage among neighboring satellites, compounded by constraints on satellite energy and computational resources. The satellite service allocation (SSA) problem is recognized as NP-hard, yet assessing allocation methods through results allows for the application of deep reinforcement learning (DRL) to obtain improved solutions, partially addressing the SSA challenge. In this paper, we introduce a satellite computing capability model to quantify satellite computational resources. A DRL model is proposed to address service demands, computational resources, and resolve service allocation conflicts, strategically placing each service on appropriate servers. Through simulation experiments, numerical results demonstrate the superiority of our proposed method over baseline approaches in service allocation and satellite resource utilization, showcasing advancements in this field.

F rugal L ight : Symmetry-Aware Cyclic Heterogeneous Intersection Control using Deep Reinforcement Learning with Model Compression, Distillation and Domain Knowledge

Developing countries need to better manage fast increasing traffic flows, owing to rapid urbanization. Else, increasing traffic congestion would increase fatalities due to reckless driving, as well as keep vehicular emissions and air pollution critically high in cities like New Delhi. State-of-the-art traffic signal control methods in developed countries, however, use expensive sensing, computation, and communication resources. How far can control algorithms go, under resource constraints, is explored through the design and evaluation of FrugalLight (FL) in this article. We also captured and processed a real traffic dataset at a busy intersection in New Delhi, India, using efficient techniques on low cost embedded devices. This dataset ( https://delhi-trafficdensity-dataset.github.io ) contains traffic density information at fine time granularity of one measurement every second, from all approaches of the intersection for 40 days. FrugalLight ( https://github.com/sachin-iitd/FrugalLight ) is evaluated on the collected traffic dataset from New Delhi and another open source traffic dataset from New York. FrugalLight matches the performance of state-of-the-art Convolutional Neural Network (CNN) based sensing and Deep Reinforcement Learning (DRL) based control algorithms, while utilizing resources less by an order of magnitude. We further explore improvements using a careful combination of knowledge distillation and domain knowledge based DRL model compression, with employing Model-Agnostic Meta-Learning to quickly adapt to traffic at new intersections. The collected real dataset and FrugalLight therefore opens up opportunities for resource efficient RL based intersection control design for the ML research community, where the controller should have limited carbon footprint. Such intelligent, green, intersection controllers can help reduce traffic congestion and associated vehicular emissions, even if compute and communication infrastructure is limited in low resource regions. This is a critical step toward achieving two of the United Nations Sustainable Development Goals (SDG), namely sustainable cities and communities and climate action.

A deep reinforcement learning algorithm framework for solving multi-objective traveling salesman problem based on feature transformation

As a special type of multi-objective combinatorial optimization problems (MOCOPs), the multi-objective traveling salesman problem (MOTSP) plays an important role in practical fields such as transportation and robot control. However, due to the complexity of its solution space and the conflicts between different objectives, it is difficult to obtain satisfactory solutions in a short time. This paper proposes an end-to-end algorithm framework for solving MOTSP based on deep reinforcement learning (DRL). By decomposing strategies, solving MOTSP is transformed into solving multiple single-objective optimization subproblems. Through linear transformation, the features of the MOTSP are combined with the weights of the objective function. Subsequently, a modified graph pointer network (GPN) model is used to solve the decomposed subproblems. Compared with the previous DRL model, the proposed algorithm can solve all the subproblems using only one model without adding weight information as input features. Furthermore, our algorithm can output a corresponding solution for each weight, which increases the diversity of solutions. In order to verify the performance of our proposed algorithm, it is compared with four classical evolutionary algorithms and two DRL algorithms on several MOTSP instances. The comparison shows that our proposed algorithm outperforms the compared algorithms both in terms of training time and the quality of the resulting solutions.

Integrating few-shot personalized thermal comfort model and reinforcement learning for HVAC demand response optimization

Heating, ventilation, and air conditioning (HVAC) is the most common equipment to maintain personalized thermal comfort (PTC), and a valuable demand response (DR) resource. The challenges to be addressed in HVAC DR optimization include uncertainties of environment, PTC varies by users and rapid decision-making. This paper proposed a fast HVAC DR optimization method that integrates deep reinforcement learning (DRL) and few-shot PTC model. Firstly, we established an HVAC DR optimization framework integrated few-shot PTC model and DRL, in which the few-shot PTC model was utilized to generate user’s PTC temperature range served as the constraint and state variable of DRL. Secondly, we proposed a few-shot PTC model, which only required several field samples and had excellent engineering usability. Thirdly, we designed the HVAC DR optimization algorithm considering user’s PTC based on proximal policy optimization, which automatically dealt with the uncertainties and made rapid decisions for setting HVAC setpoints of temperature dynamically. Simulation results show that the few-shot PTC model only requires 9 field samples, which is less than 1% of existing advanced methods. Furthermore, the proposed method converges within 12.6 min under uncertainties on an i7-12700 processor, which is only about 1/2 of the standard deep deterministic policy gradient.

Low Earth Orbit Satellite Network Routing Algorithm Based on Graph Neural Networks and Deep Q-Network

Low Earth orbit (LEO) satellite networks are characterized by rapid topological changes, numerous network nodes and varying states of node resource constraints, which have resulted in traditional routing algorithms no longer being suitable for LEO satellite network routing. Therefore, this paper proposes an inductive learning architecture based on Graph Sample and Aggregate (GraphSAGE), which can significantly reduce the number of topology nodes to be trained, thereby reducing the computational complexity of the nodes. Then deep reinforcement learning (DRL) is employed for the continuous learning optimization of routing algorithms, and its generalization is improved by selecting GraphSAGE to construct the DRL agent. In the proposed graph neural-network-based routing optimization algorithm for LEO satellite networks, each Deep Q-Network (DQN) agent independently generates the hidden states of the nodes through the GraphSAGE model and uses them as inputs to the DRL model to make routing decisions. After a simulation and comparison, the proposed algorithm not only improves the overall network throughput, but also reduces the average end-to-end delay. The average throughput of the proposed algorithm increases by 29.47% and 18.42% compared to that of Dijkstra and the DQN, respectively. The average end-to-end delay is reduced by 39.76% and 15.29%, respectively, and can also adapt to changing topologies.

DRL-GAN: A Hybrid Approach for Binary and Multiclass Network Intrusion Detection.

Our increasingly connected world continues to face an ever-growing number of network-based attacks. An Intrusion Detection System (IDS) is an essential security technology used for detecting these attacks. Although numerous Machine Learning-based IDSs have been proposed for the detection of malicious network traffic, the majority have difficulty properly detecting and classifying the more uncommon attack types. In this paper, we implement a novel hybrid technique using synthetic data produced by a Generative Adversarial Network (GAN) to use as input for training a Deep Reinforcement Learning (DRL) model. Our GAN model is trained on the NSL-KDD dataset, a publicly available collection of labeled network traffic data specifically designed to support the evaluation and benchmarking of IDSs. Ultimately, our findings demonstrate that training the DRL model on synthetic datasets generated by specific GAN models can result in better performance in correctly classifying minority classes over training on the true imbalanced dataset.

Artificial Intelligence-Assisted Optimization of Antipigmentation Tyrosinase Inhibitors: De Novo Molecular Generation Based on a Low Activity Lead Compound.

Artificial intelligence (AI) de novo molecular generation is a highly promising strategy in the drug discovery, with deep reinforcement learning (RL) models emerging as powerful tools. This study introduces a fragment-by-fragment growth RL forward molecular generation and optimization strategy based on a low activity lead compound. This process integrates fragment growth-based reaction templates, while target docking and drug-likeness prediction were simultaneously performed. This comprehensive approach considers molecular similarity, internal diversity, synthesizability, and effectiveness, thereby enhancing the quality and efficiency of molecular generation. Finally, a series of tyrosinase inhibitors were generated and synthesized. Most compounds exhibited more improved activity than lead, with an optimal candidate compound surpassing the effects of kojic acid and demonstrating significant antipigmentation activity in a zebrafish model. Furthermore, metabolic stability studies indicated susceptibility to hepatic metabolism. The proposed AI structural optimization strategies will play a promising role in accelerating the drug discovery and improving traditional efficiency.

Mathematical Model-Driven Deep Learning Enables Personalized Adaptive Therapy.

Standard-of-care treatment regimens have long been designed for maximal cell killing, yet these strategies often fail when applied to metastatic cancers due to the emergence of drug resistance. Adaptive treatment strategies have been developed as an alternative approach, dynamically adjusting treatment to suppress the growth of treatment-resistant populations and thereby delay, or even prevent, tumor progression. Promising clinical results in prostate cancer indicate the potential to optimize adaptive treatment protocols. Here, we applied deep reinforcement learning (DRL) to guide adaptive drug scheduling and demonstrated that these treatment schedules can outperform the current adaptive protocols in a mathematical model calibrated to prostate cancer dynamics, more than doubling the time to progression. The DRL strategies were robust to patient variability, including both tumor dynamics and clinical monitoring schedules. The DRL framework could produce interpretable, adaptive strategies based on a single tumor burden threshold, replicating and informing optimal treatment strategies. The DRL framework had no knowledge of the underlying mathematical tumor model, demonstrating the capability of DRL to help develop treatment strategies in novel or complex settings. Finally, a proposed five-step pathway, which combined mechanistic modeling with the DRL framework and integrated conventional tools to improve interpretability compared to traditional "black-box" DRL models, could allow translation of this approach to the clinic. Overall, the proposed framework generated personalized treatment schedules that consistently outperformed clinical standard-of-care protocols.

Optimal parallelization strategies for active flow control in deep reinforcement learning-based computational fluid dynamics

Deep reinforcement learning (DRL) has emerged as a promising approach for handling highly dynamic and nonlinear active flow control (AFC) problems. However, the computational cost associated with training DRL models presents a significant performance bottleneck. To address this challenge and enable efficient scaling on high-performance computing architectures, this study focuses on optimizing DRL-based algorithms in parallel settings. We validate an existing state-of-the-art DRL framework used for AFC problems and discuss its efficiency bottlenecks. Subsequently, by deconstructing the overall framework and conducting extensive scalability benchmarks for individual components, we investigate various hybrid parallelization configurations and propose efficient parallelization strategies. Moreover, we refine input/output (I/O) operations in multi-environment DRL training to tackle critical overhead associated with data movement. Finally, we demonstrate the optimized framework for a typical AFC problem where near-linear scaling can be obtained for the overall framework. We achieve a significant boost in parallel efficiency from around 49% to approximately 78%, and the training process is accelerated by approximately 47 times using 60 central processing unit (CPU) cores. These findings are expected to provide valuable insight for further advancements in DRL-based AFC studies.

A distributed deep reinforcement learning-based longitudinal control strategy for connected automated vehicles combining attention mechanism

ABSTRACT With the rapid development of connected automated vehicles (CAVs), the trajectory control of CAVs has become a focus in traffic engineering. This paper proposes a distributed deep reinforcement learning-based longitudinal control strategy for CAVs combining attention mechanism, which enhances the stability of mixed traffic, car-following efficiency, energy efficiency, and safety. A longitudinal control strategy is built using a deep reinforcement learning model. The CAVs gradually learn optimal car-following strategy in training process to improve safety, stability, fuel economy, mobility, and driving comfort. To further capture the interactions among vehicles in each platoon, the graph attention network is introduced to facilitate the car-following control strategy. To verify the effectiveness of the proposed method, a comparative analysis is conducted, which indicates that the proposed method can dramatically dampen oscillations, enhance traffic efficiency, reduce fuel consumption, and improve driving safety under different scenarios.

Optimal Treatment Strategies for Critical Patients with Deep Reinforcement Learning

Personalized clinical decision support systems are increasingly being adopted due to the emergence of data-driven technologies, with this approach now gaining recognition in critical care. The task of incorporating diverse patient conditions and treatment procedures into critical care decision-making can be challenging due to the heterogeneous nature of medical data. Advances in Artificial Intelligence (AI), particularly Reinforcement Learning (RL) techniques, enables the development of personalized treatment strategies for severe illnesses by using a learning agent to recommend optimal policies. In this study, we propose a Deep Reinforcement Learning (DRL) model with a tailored reward function and an LSTM-GRU-derived state representation to formulate optimal treatment policies for vasopressor administration in stabilizing patient physiological states in critical care settings. Using an ICU dataset and the Medical Information Mart for Intensive Care (MIMIC-III) dataset, we focus on patients with Acute Respiratory Distress Syndrome (ARDS) that has led to Sepsis, to derive optimal policies that can prioritize patient recovery over patient survival. Both the DDQN ( RepDRL-DDQN ) and Dueling DDQN ( RepDRL-DDDQN ) versions of the DRL model surpass the baseline performance, with the proposed model’s learning agent achieving an optimal learning process across our performance measuring schemes. The robust state representation served as the foundation for enhancing the model’s performance, ultimately providing an optimal treatment policy focused on rapid patient recovery.

Integrating Neural Pathways for Learning in Deep Reinforcement Learning Models

Considering that the human brain is the most powerful, generalizable, and energy-efficient computer we know of, it makes the most sense to look to neuroscience for ideas regarding deep learning model improvements. I propose one such idea, augmenting a traditional Advantage-Actor-Critic (A2C) model with additional learning signals akin to those in the brain. Pursuing this direction of research should hopefully result in a new reinforcement learning (RL) control paradigm that can learn from fewer examples, train with greater stability, and possibly consume less energy.

Learning-driven hybrid scaling for multi-type services in cloud

In order to deal with the fast changing requirements of container based services in clouds, auto-scaling is used as an essential mechanism for adapting the number of provisioned resources with the variable service workloads. However, the latest auto-scaling approaches lack the comprehensive consideration of variable workloads and hybrid auto-scaling for multi-type services. Firstly, the historical data based proactive approaches are widely used to handle complex and variable workloads in advance. The decision-making accuracy of proactive approaches depends on the prediction algorithm, which is affected by the anomalies, missing values and errors in the historical workload data, and the unexpected workload cannot be handled. Secondly, the trigger based reactive approaches are seriously affected by workload fluctuation which causes the frequent invalid scaling of service resources. Besides, due to the existence of scaling time, there are different completion delays of different scaling actions. Thirdly, the latest approaches also ignore the different scaling time of hybrid scaling for multi-type services including stateful services and stateless services. Especially, when the stateful services are scaled horizontally, the neglected long scaling time causes the untimely supply and withdrawal of resources. Consequently, all three issues above can lead to the degradation of Quality of Services (QoS) and the inefficient utilization of resources. This paper proposes a new hybrid auto-scaling approach for multi-type services to resolve the impact of service scaling time on decision making. We combine the proactive scaling strategy with the reactive anomaly detection and correction mechanism. For making a proactive decision, the ensemble learning model with the structure improved deep network is designed to predict the future workload. On the basis of the predicted results and the scaling time of different types of services, the auto-scaling decisions are made by a Deep Reinforcement Learning (DRL) model with heterogeneous action space, which integrates horizontal and vertical scaling actions. Meanwhile, with the anomaly detection and correction mechanism, the workload fluctuation and unexpected workload can be detected and handled. We evaluate our approach against three different proactive and reactive auto-scaling approaches in the cloud environment, and the experimental results show the proposed approach can achieve the better scaling behavior compared to state-of-the-art approaches.

Multi-agent deep reinforcement learning for joint dynamic conservation voltage reduction and Q-sharing in inverter-based autonomous microgrids

Conservation voltage reduction (CVR) is implemented in power systems to mitigate power consumption in steady-state time scales. We propose dynamic-scale CVR (DCVR) as a potent solution to provide cost-effective frequency support in inverter-interfaced (micro) grids. The proposed DCVR reduces the voltage profile in dynamics and consequently power reduction helps to maintain instant production-consumption balance for dynamic frequency support. However, DCVR implementation in autonomous MGs (AMGs) faces critical control and stability problems. DCVR is implemented by controlling the voltage, which is a local variable, and makes it difficult to be realized through a decentralized structure. Besides, preserving accurate reactive power sharing (Q-sharing) through conventional droop controllers while employing the DCVR is another critical concern. In this light, this paper proposes a novel artificial intelligence (AI)-based decentralized control structure to implement the DCVR in AMGs for tackling the existing issues. The multi-agent deep reinforcement learning (DRL) model, with a deep Q-network (DQN) algorithm, is adopted to address the grid instabilities and inaccurate Q-sharing issues that arise due to incorporating the DCVR. The proposed method is able to handle the nonlinearity and complexity of the system while maintaining proper dynamic performance and AMG stability. Simulation results in MATLAB/Simulink prove the effectiveness of the proposed control method.

Deep reinforcement learning based on balanced stratified prioritized experience replay for customer credit scoring in peer-to-peer lending

In recent years, deep reinforcement learning (DRL) models have been successfully utilised to solve various classification problems. However, these models have never been applied to customer credit scoring in peer-to-peer (P2P) lending. Moreover, the imbalanced class distribution in experience replay, which may affect the performance of DRL models, has rarely been considered. Therefore, this article proposes a novel DRL model, namely a deep Q-network based on a balanced stratified prioritized experience replay (DQN-BSPER) model, for customer credit scoring in P2P lending. Firstly, customer credit scoring is formulated as a discrete-time finite-Markov decision process. Subsequently, a balanced stratified prioritized experience replay technology is presented to optimize the loss function of the deep Q-network model. This technology can not only balance the numbers of minority and majority experience samples in the mini-batch by using stratified sampling technology but also select more important experience samples for replay based on the priority principle. To verify the model performance, four evaluation measures are introduced for the empirical analysis of two real-world customer credit scoring datasets in P2P lending. The experimental results show that the DQN-BSPER model can outperform four benchmark DRL models and seven traditional benchmark classification models. In addition, the DQN-BSPER model with a discount factor γ of 0.1 has excellent credit scoring performance.

Visual analytics system for understanding DeepRL-based charged particle tracking

AbstractIn this work, we propose a visual analytics system to analyze deep reinforcement learning (deepRL) models working on the track reconstruction of charged particles in the field of particle physics. The data of these charged particles are in the form of point clouds with high-dimensional features. We use one of the existing post hoc saliency methods of explainable artificial intelligence (XAI) and extend its adaptation to compute saliency attributions for the input data corresponding to the output of the model. Our proposed system helps users to explore these saliency attributions corresponding to the high-dimensional input data of the machine learning model and interpret the decision-making process of the model. In particular, we provide the users with multiple task-oriented components, different types of linked views and interactive tools to analyze the model. We explain how to use the system by outlining a typical user workflow and demonstrate the system’s usefulness using several case studies which address specific analysis tasks.

Deep Reinforcement Learning Model for Stock Portfolio Management Based on Data Fusion

Deep reinforcement learning (DRL) can be used to extract deep features that can be incorporated into reinforcement learning systems to enable improved decision-making; DRL can therefore also be used for managing stock portfolios. Traditional methods cannot fully exploit the advantages of DRL because they are generally based on real-time stock quotes, which do not have sufficient features for making comprehensive decisions. In this study, in addition to stock quotes, we introduced stock financial indices as additional stock features. Moreover, we used Markowitz mean-variance theory for determining stock correlation. A three-agent deep reinforcement learning model called Collaborative Multi-agent reinforcement learning-based stock Portfolio management System (CMPS) was designed and trained based on fused data. In CMPS, each agent was implemented with a deep Q-network to obtain the features of time-series stock data, and a self-attention network was used to combine the output of each agent. We added a risk-free asset strategy to CMPS to prevent risks and referred to this model as CMPS-Risk Free (CMPS-RF). We conducted experiments under different market conditions using the stock data of China Shanghai Stock Exchange 50 and compared our model with the state-of-the-art models. The results showed that CMPS could obtain better profits than the compared benchmark models, and CMPS-RF was able to accurately recognize the market risk and achieved the best Sharpe and Calmar ratios. The study findings are expected to aid in the development of an efficient investment-trading strategy.

DESiRED — Dynamic, Enhanced, and Smart iRED: A P4-AQM with Deep Reinforcement Learning and In-band Network Telemetry

Active Queue Management (AQM) is a mechanism employed to alleviate transient congestion in network device buffers, such as routers and switches. Traditional AQM algorithms use fixed thresholds, like target delay or queue occupancy, to compute random packet drop probabilities. A very small target delay can increase packet losses and reduce link utilization, while a large target delay may increase queueing delays while lowering drop probability. Due to dynamic network traffic characteristics, where traffic fluctuations can lead to significant queue variations, maintaining a fixed threshold AQM may not suit all applications. Consequently, we explore the question: What is the ideal threshold (target delay) for AQMs? In this work, we introduce DESiRED (Dynamic, Enhanced, and Smart iRED), a P4-based AQM that leverages precise network feedback from In-band Network Telemetry (INT) to feed a Deep Reinforcement Learning (DRL) model. This model dynamically adjusts the target delay based on rewards that maximize application Quality of Service (QoS). We evaluate DESiRED in a realistic P4-based test environment running an MPEG-DASH service. Our findings demonstrate up to a 90x reduction in video stall and a 42x increase in high-resolution video playback quality when the target delay is adjusted dynamically by DESiRED.

Dynamic path planning via Dueling Double Deep Q-Network (D3QN) with prioritized experience replay

Path planning is a key requirement for mobile robots employed for different tasks such as rescue or transport missions. Conventional methods such as A* or Dijkstra to tackle path planning problem need a premise map of the robot's environment. Nowadays, dynamic path planning is a popular research topic, which drives mobile robots without prior static requirements. Deep reinforcement learning (DRL), which is another popular research area, is being harnessed to solve dynamic path planning problem by the researchers. In this study, Deep Q-Networks, which is a subdomain of DRL are opted to solve dynamic path planning problem. We first employ well known techniques Double Deep Q-Networks (D2QN) and Dueling Double Deep Q-Networks (D3QN) to train a model which can drive a mobile robot in environments with static and dynamic obstacles within 3 different configurations. Then we propose D3QN with Prioritized Experience Replay (PER) extension in order to further optimize the DRL model. We created a test bed to measure the performance of the DRL models against 99 randomly generated goal locations. According to our experiments, D3QN-PER method performs better than D2QN and D3QN in terms of path length and travel time to the goal without any collisions. Robot Operating System and Gazebo simulation environment is utilized to realize the training and testing environments, thus, the trained DRL models can be deployed to any ROS compatible robot seamlessly.

Improved DRL-based energy-efficient UAV control for maximum lifecycle

Unmanned aerial vehicles (UAVs) operating as airborne base stations (UAV-BSs) provide efficient on-demand services to ground users. UAV-BSs are inherently flexible and mobile, allowing them to be strategically deployed based on ground user distribution and quality of service requirements, including coverage rate, system lifecycle, and user fairness. Owing to the limited battery capacity and coverage range of the UAVs, managing them to extend their operational lifecycle, ensure service fairness, and maintain a specific real-time coverage rate is challenging. Therefore, a multi-objective optimization problem with constrained Pareto dominance is formulated. Subsequently, a novel assisted deep reinforcement learning model is developed to maximize the minimum remaining energy while simultaneously considering user fairness and coverage-rate requirements. The particle swarm optimization algorithm is adopted to assist multi-agent cooperative deep reinforcement learning. Finally, the simulation results show that the proposed model outperforms the other popular methods in terms of user fairness, system lifecycle, coverage rate, and energy efficiency in the context of multi-objective, multi-agent cooperative coverage control deployment.