Deep Reinforcement Learning Algorithm Research Articles

Deep Reinforcement Learning for personalized diagnostic decision pathways using Electronic Health Records: A comparative study on anemia and Systemic Lupus Erythematosus

Background:Clinical diagnoses are typically made by following a series of steps recommended by guidelines that are authored by colleges of experts. Accordingly, guidelines play a crucial role in rationalizing clinical decisions. However, they suffer from limitations, as they are designed to cover the majority of the population and often fail to account for patients with uncommon conditions. Moreover, their updates are long and expensive, making them unsuitable for emerging diseases and new medical practices. Methods:Inspired by guidelines, we formulate the task of diagnosis as a sequential decision-making problem and study the use of Deep Reinforcement Learning (DRL) algorithms to learn the optimal sequence of actions to perform in order to obtain a correct diagnosis from Electronic Health Records (EHRs), which we name a diagnostic decision pathway. We apply DRL to synthetic yet realistic EHRs and develop two clinical use cases: Anemia diagnosis, where the decision pathways follow a decision tree schema, and Systemic Lupus Erythematosus (SLE) diagnosis, which follows a weighted criteria score. We particularly evaluate the robustness of our approaches to noise and missing data, as these frequently occur in EHRs. Results:In both use cases, even with imperfect data, our best DRL algorithms exhibit competitive performance compared to traditional classifiers, with the added advantage of progressively generating a pathway to the suggested diagnosis, which can both guide and explain the decision-making process. Conclusion:DRL offers the opportunity to learn personalized decision pathways for diagnosis. Our two use cases illustrate the advantages of this approach: they generate step-by-step pathways that are explainable, and their performance is competitive when compared to state-of-the-art methods.

A path planning method based on deep reinforcement learning for AUV in complex marine environment

The potential of autonomous underwater vehicle (AUV) on future applications is significant due to advancements in autonomy and intelligence. Path planning is a critical technology for AUV to perform operational missions in complex marine environments. To this end, this paper proposes a path planning method for AUV based on deep reinforcement learning. Initially, considering actual requirements, a complex marine environment model containing underwater terrain, sonobuoy detection, and ocean currents is established. Subsequently, the corresponding state space, action space, and reward function are formulated. Furthermore, to address the inherent limitations of existing deep reinforcement learning algorithms in terms of training efficiency, a mixed experience replay (MER) strategy is proposed. This strategy aims to enhance the efficiency of sample learning by integrating prior knowledge and exploration experience. Lastly, a novel HMER-SAC algorithm for AUV path planning is proposed by integrating the Soft Actor–Critic (SAC) algorithm with the hierarchical reinforcement learning strategy and the MER strategy. The results of the simulation and experiment demonstrate that the method is capable of efficiently planning executable paths in complex marine environments and exhibits superior training efficiency, stability, and performance.

Deep Reinforcement Learning with Local Attention for Single Agile Optical Satellite Scheduling Problem

This paper investigates the single agile optical satellite scheduling problem, which has received increasing attention due to the rapid growth in earth observation requirements. Owing to the complicated constraints and considerable solution space of this problem, the conventional exact methods and heuristic methods, which are sensitive to the problem scale, demand high computational expenses. Thus, an efficient approach is demanded to solve this problem, and this paper proposes a deep reinforcement learning algorithm with a local attention mechanism. A mathematical model is first established to describe this problem, which considers a series of complex constraints and takes the profit ratio of completed tasks as the optimization objective. Then, a neural network framework with an encoder–decoder structure is adopted to generate high-quality solutions, and a local attention mechanism is designed to improve the generation of solutions. In addition, an adaptive learning rate strategy is proposed to guide the actor–critic training algorithm to dynamically adjust the learning rate in the training process to enhance the training effectiveness of the proposed network. Finally, extensive experiments verify that the proposed algorithm outperforms the comparison algorithms in terms of solution quality, generalization performance, and computation efficiency.

Dynamics-Adaptive Continual Reinforcement Learning via Progressive Contextualization.

A key challenge of continual reinforcement learning (CRL) in dynamic environments is to promptly adapt the reinforcement learning (RL) agent's behavior as the environment changes over its lifetime while minimizing the catastrophic forgetting of the learned information. To address this challenge, in this article, we propose DaCoRL, that is, dynamics-adaptive continual RL. DaCoRL learns a context-conditioned policy using progressive contextualization, which incrementally clusters a stream of stationary tasks in the dynamic environment into a series of contexts and opts for an expandable multihead neural network to approximate the policy. Specifically, we define a set of tasks with similar dynamics as an environmental context and formalize context inference as a procedure of online Bayesian infinite Gaussian mixture clustering on environment features, resorting to online Bayesian inference to infer the posterior distribution over contexts. Under the assumption of a Chinese restaurant process (CRP) prior, this technique can accurately classify the current task as a previously seen context or instantiate a new context as needed without relying on any external indicator to signal environmental changes in advance. Furthermore, we employ an expandable multihead neural network whose output layer is synchronously expanded with the newly instantiated context and a knowledge distillation regularization term for retaining the performance on learned tasks. As a general framework that can be coupled with various deep RL algorithms, DaCoRL features consistent superiority over existing methods in terms of stability, overall performance, and generalization ability, as verified by extensive experiments on several robot navigation and MuJoCo locomotion tasks.

Personalized Health Management Strategies Based on Deep Reinforcement Learning in the Network Environment

In order to study the optimal personalized motion push target, the author proposes a personalized health managementstrategy based on deep reinforcement learning in the network environment. Firstly, the research problem is defined, and a real-time interactive personalized motion target decision-making model is constructed; Subsequently, in response to the uncertain characteristics of user behavior in the problem, a deep reinforcement learning algorithm was adopted, combined with the departure strategy temporal difference learning method and neural network nonlinear fitting method, in order to learn strategies from user historical data; Finally, the effectiveness of the proposed method was validated using a real dataset from Fitbit. The research results indicate that personalized motion goal push services based on deep reinforcement learning can help users cultivate a healthy lifestyle and improve their personal health management level by analyzing user behavior data in real-time, providing scientific guidance and timely incentives.

Real-time scheduling for two-stage assembly flowshop with dynamic job arrivals by deep reinforcement learning

Assembly flowshops differ from flowshops because their jobs have several components that are manufactured in the processing stage and then joined together in the assembly stage. Real-time scheduling of two-stage assembly flowshops is of great significance for many discrete manufacturing companies in customized order fulfilment. Therefore, we study two-stage assembly flowshop scheduling problem with the objective of total tardiness minimization considering dynamic job arrivals using deep reinforcement learning (DRL). First, an architecture of DRL-based real-time scheduling is proposed. Second, we establish a DRL-based real-time scheduling model by designing proper state space, action space and reward function. A scheduling agent is constructed and trained through the proximal policy optimization (PPO) using interactive simulation data. The experimental results show that the trained scheduling agent can learn to choose the appropriate dispatching rules efficiently so as to make a real-time scheduling quickly and effectively. The PPO algorithm outperforms six dispatching rules, genetic algorithm and two DRL algorithms (i.e., DDQN and DDPG) in terms of solution quality and computing speed. The proposed approach is highly applicable to many practical scheduling situations that requires real-time and quick decisions.

Joint intelligent optimizing economic dispatch and electric vehicles charging in 5G vehicular networks

In recent years, with the rapid development of 5G networks, the road traffic network composed of vehicles with different energy sources has become more and more complex, and the problems of environmental pollution and road congestion have also become increasingly serious. Electric vehicles are favored by people due to their environmental protection and energy-saving characteristics. However, improper charging dispatching will cause excess energy in charging stations, affecting the power grid and road traffic, such as energy shortages and lower traffic throughput. Therefore, how to design a reasonable charging strategy that can maximize the user’s charging satisfaction and consume the energy of the charging station as much as possible becomes a challenge. Meanwhile, this strategy should consider power economic dispatch to reduce power generation costs and polluting gas emissions. With the support of 5G’s high-bandwidth and low-latency characteristics, this paper designs an intelligent charging model which indirectly reflects the charging satisfaction through the time cost, energy consumption cost, charging cost, and the user’s range anxiety, while consuming the remaining energy of the charging station as much as possible. Due to the uncertainty of wind and photovoltaic power generation, this paper proposes a two-stage economic dispatch model to improve the accuracy of power dispatch and reduce power generation costs and carbon emissions. Due to the highly variable traffic environment and energy demand, we employ proximal policy optimization-based deep reinforcement learning algorithms to realize electric vehicle charging dispatching and charging station power dispatching. Numerical results show the efficiency of our proposed strategy for electric vehicle charging in terms of the convergence speed.

Routing cost-integrated intelligent handover strategy for multi-layer LEO mega-constellation networks

Low Earth Orbit (LEO) mega-constellation networks, exemplified by Starlink, are poised to play a pivotal role in future mobile communication networks, due to their low latency and high capacity. With the massively deployed satellites, ground users now can be covered by multiple visible satellites, but also face complex handover issues with such massive high-mobility satellites in multi-layer. The end-to-end routing is also affected by the handover behavior. In this paper, we propose an intelligent handover strategy dedicated to multi-layer LEO mega-constellation networks. Firstly, an analytic model is utilized to rapidly estimate the end-to-end propagation latency as a key handover factor to construct a multi-objective optimization model. Subsequently, an intelligent handover strategy is proposed by employing the Dueling Double Deep Q Network (D3QN)-based deep reinforcement learning algorithm for single-layer constellations. Moreover, an optimal cross-layer handover scheme is proposed by predicting the latency-jitter and minimizing the cross-layer overhead. Simulation results demonstrate the superior performance of the proposed method in the multi-layer LEO mega-constellation, showcasing reductions of up to 8.2% and 59.5% in end-to-end latency and jitter respectively, when compared to the existing handover strategies.

A Two-Stage Target Search and Tracking Method for UAV Based on Deep Reinforcement Learning

To deal with the complexities of decision-making for unmanned aerial vehicles (UAVs) in denial environments, this paper applies deep reinforcement learning algorithms to search and rescue (SAR) tasks. It proposes a two-stage target search and tracking method for UAVs based on deep reinforcement learning, which divides SAR tasks into search and tracking stages, and the controllers for each stage are trained based on the proposed deep deterministic policy gradient with three critic networks (DDPG-3C) algorithm. Simulation experiments are carried out to evaluate the performance of each stage in a two-dimensional rectangular SAR scenario, including search, tracking, and the integrated whole stage. The experimental results show that the proposed DDPG-3C model can effectively alleviate the overestimation problem, and hence results in a faster convergence and improved performance during both the search and tracking stages. Additionally, the two-stage target search and tracking method outperforms the traditional single-stage approach, leading to a more efficient and effective decision-making ability in SAR tasks.

An Efficient Multi-Agent Policy Self-Play Learning Method Aiming at Seize-Control Scenarios

Aiming at the problem of multi-agent cooperative confrontation in seize-control scenarios, we design an efficient multi-agent policy self-play (EMAP-SP) learning method. First, a multi-agent centralized policy model is constructed to command the agents to perform tasks cooperatively. Considering that the policy being trained and its historical policies usually have poor exploration capability under incomplete information in self-play trainings, the intrinsic reward mechanism based on random network distillation (RND) is introduced in the self-play learning method. In addition, we propose a multi-step on-policy deep reinforcement learning (DRL) algorithm assisted by off-policy policy evaluation (MSOAO) to learn the best response policy in the self-play. Compared with DRL algorithms commonly used in complex decision problems, MSOAO has more efficient policy evaluation capability, and efficient policy evaluation further improves the policy learning capability. The effectiveness of EMAP-SP is fully verified in MiaoSuan wargame simulation system, and the evaluation results show that EMAP-SP can learn the cooperative policy of effectively defeating the Blue side’s knowledge-based policy under incomplete information. Moreover, the evaluations results in DRL benchmark environments also show that the best response policy learning algorithm MSOAO can promote the agent to learn approximately optimal policies.

Energy management of fuel cell hybrid electric bus in mountainous regions: A deep reinforcement learning approach considering terrain characteristics

Environmental characteristics, particularly in mountainous regions with significant slopes, substantially impact vehicle power demand and fuel consumption, thereby influencing both fuel economy and vehicle lifespan. However, existing research lacks energy management strategies specifically designed for these challenging terrains. This study presents an innovative energy management strategy (EMS) for fuel cell hybrid electric buses (FCHEB) utilizing a deep reinforcement learning algorithm considering terrain characteristics. A comprehensive model that incorporates road fluctuations and steep slopes was developed to accurately represent the terrain features of mountainous regions. Driving cycle and road condition data specific to hydrogen fuel cell buses in these areas were collected and processed as for the training sets of EMSs. Subsequently, an EMS based on the Proximal Policy Optimization (PPO) algorithm was devised to address the unique characteristics of these mountainous regions. Simulation results indicate a 26.16 % improvement in fuel economy and a 44.98 % enhancement in convergence efficiency compared to traditional methods that do not consider road slopes. Further validation through new bus route trials and standard driving cycle tests confirms the strategy's robustness and adaptability.

Eco-driving control for connected plug-in hybrid electric vehicles in urban scenarios with enhanced lane change engagement

Eco-driving control techniques have shown significant potential in reducing energy consumption in urban scenarios. The presence of slow-moving vehicles typically disrupts ecological velocity planning, leading to an increase in energy consumption. To solve it, this study proposes a hierarchical eco-driving control strategy, that integrates speed optimization and lane change decision-making in urban scenarios, to not only ensure traffic efficiency, but also save the energy consumption. Firstly, a data-driven energy model is leveraged in the upper level to estimate the energy consumption of candidate lanes and generate ecological lane change decisions. Then, in the lower level, the preceding vehicles and traffic lights are considered to plan an ecological velocity profile via deep reinforcement learning algorithm after transitions to the target driving lane, thereby enhancing the fuel economy and travel efficiency. A virtual driving environment model is established to verify the proposed method through numerous simulation cases. The results indicate that the proposed method effectively reduces energy consumption while maintaining favorable travel efficiency, compared with conventional benchmarks. Furthermore, the notable improvements are observed particularly in free traffic conditions.

Clustering and Cooperative Guidance of Multiple Decoys for Defending a Naval Platform against Salvo Threats

The threat to naval platforms from missile systems is increasing due to recent advancements in radar seeker technology, which have significantly enhanced the accuracy and effectiveness of missile targeting. In scenarios where a naval platform with limited maneuverability faces salvo attacks, the importance of an effective defense strategy becomes crucial to ensuring the protection of the platform. In this study, we present a multi-agent reinforcement learning-based decoy deployment approach that employs six decoys to increase the survival likelihood of a naval platform against salvo missile strikes. Our approach entails separating the decoys into two clusters, each consisting of three decoys. Subsequently, every cluster is allocated to a related missile threat. This is accomplished by training the decoys with the multi-agent deep reinforcement learning algorithm. To compare the proposed approach across different algorithms, we use two distinct algorithms to train the decoys; multi-agent deep deterministic policy gradient (MADDPG) and multi-agent twin-delayed deep deterministic policy gradient (MATD3). Following training, the decoys learn to form groups and establish effective formation configurations within each group to ensure optimal coordination. We assess the proposed decoy deployment strategy using parameters including decoy deployment angle and maximum decoy speed. Our findings indicate that decoys positioned on the same side outperform those positioned on different sides relative to the target platform. In general, MATD3 performs slightly better than MADDPG. Decoys trained with MATD3 succeed in more successful formation configurations than those trained with the MADDPG method, which accounts for this enhancement.

Assessing generalizability of Deep Reinforcement Learning algorithms for Automated Vulnerability Assessment and Penetration Testing

Modern cybersecurity best practices and standards require continuous Vulnerability Assessment (VA) and Penetration Test (PT). These activities are human- and time-expensive. The research community is trying to propose autonomous or semi-autonomous solutions based on Deep Reinforcement Learning (DRL) agents, but current proposals require further investigations. We observe that related literature reports performance tests of the proposed agents against a limited subset of the hosts used to train the models, thus raising questions on their applicability in realistic scenarios. The main contribution of this paper is to fill this gap by investigating the generalization capabilities of existing DRL agents to extend their VAPT operations to hosts that were not used in the training phase. To this purpose, we define a novel VAPT environment through which we devise multiple evaluation scenarios. While evidencing the limited capabilities of shallow RL approaches, we consider three state-of-the-art deep RL agents, namely Deep Q-Network (DQN), Proximal Policy Optimization (PPO), and Advantage Actor–Critic (A2C), and use them as bases for VAPT operations. The results show that the algorithm using A2C DRL agent outperforms the others because it is more adaptable to unknown hosts and converges faster. Our methodology can guide future researchers and practitioners in designing a new generation of semi-autonomous VAPT tools that are suitable for real-world contexts.

A multi-objective hierarchical deep reinforcement learning algorithm for connected and automated HEVs energy management

Connected and autonomous vehicles have offered unprecedented opportunities to improve fuel economy and reduce emissions of hybrid electric vehicle (HEV) in vehicular platoons. In this context, a hierarchical control strategy is put forward for connected HEVs. Firstly, we consider a deep deterministic policy gradient (DDPG) algorithm to compute the optimized vehicle speed using a trained optimal policy via vehicle-to-vehicle communication in the upper level. A multi-objective reward function is introduced, integrating vehicle fuel consumption, battery state-of-the-charge, emissions, and vehicle car-following objectives. Secondly, an adaptive equivalent consumption minimization strategy is devised to implement vehicle-level torque allocation in the platoon. Two drive cycles, HWFET and human-in-the-loop simulator driving cycles are utilized for realistic testing of the considered platoon energy management. It is shown that DDPG runs the engine more efficiently than the widely-implemented Q-learning and deep Q-network, thus showing enhanced fuel savings. Further, the contribution of this paper is to speed up the higher-level vehicular control application of deep learning algorithms in the connected and automated HEV platoon energy management applications.

Integrated Intelligent Control of Redundant Degrees-of-Freedom Manipulators via the Fusion of Deep Reinforcement Learning and Forward Kinematics Models

Redundant degree-of-freedom (DOF) manipulators offer increased flexibility and are better suited for obstacle avoidance, yet precise control of these systems remains a significant challenge. This paper addresses the issues of slow training convergence and suboptimal stability that plague current deep reinforcement learning (DRL)-based control strategies for redundant DOF manipulators. We propose a novel DRL-based intelligent control strategy, FK-DRL, which integrates the manipulator’s forward kinematics (FK) model into the control framework. Initially, we conceptualize the control task as a Markov decision process (MDP) and construct the FK model for the manipulator. Subsequently, we expound on the integration principles and training procedures for amalgamating the FK model with existing DRL algorithms. Our experimental analysis, applied to 7-DOF and 4-DOF manipulators in simulated and real-world environments, evaluates the FK-DRL strategy’s performance. The results indicate that compared to classical DRL algorithms, the FK-DDPG, FK-TD3, and FK-SAC algorithms improved the success rates of intelligent control tasks for the 7-DOF manipulator by 21%, 87%, and 64%, respectively, and the training convergence speeds increased by 21%, 18%, and 68%, respectively. These outcomes validate the proposed algorithm’s effectiveness and advantages in redundant manipulator control using DRL and FK models.

Deep reinforcement learning for solving car resequencing with selectivity banks in automotive assembly shops

In the automobile manufacturing process, the complexity of the production environment and the different objectives of the upstream and downstream shops necessitate the resequencing of the car sequences using buffers. This paper addresses a car resequencing problem in automotive assembly shops by utilising selectivity banks to reshuffle the given upstream sequences to minimise the number of violations of sequencing rules in the downstream sequence. A new resequencing approach combining heuristics and deep reinforcement learning is proposed to solve the problem. Heuristics solve the automobile storage problem, while a deep reinforcement learning method based on proximal policy optimisation addresses the automobile release problem. Three state matrices, agent actions, and the reward function are designed based on the characteristics of the objective and selectivity banks. Additionally, a novel network structure combining convolutional neural networks and long short-term memory networks is proposed to train the agent. The results demonstrate that the proposed method outperforms two efficient deep reinforcement learning algorithms and two heuristics and obtains better solutions in handling dynamic events.

Method for Bottle Opening with a Dual-Arm Robot.

This paper introduces a novel approach to robotic assistance in bottle opening using the dual-arm robot TIAGo++. The solution enhances accessibility by addressing the needs of individuals with injuries or disabilities who may require help with common manipulation tasks. The aim of this paper is to propose a method involving vision, manipulation, and learning techniques to effectively address the task of bottle opening. The process begins with the acquisition of bottle and cap positions using an RGB-D camera and computer vision. Subsequently, the robot picks the bottle with one gripper and grips the cap with the other, each by planning safe trajectories. Then, the opening procedure is executed via a position and force control scheme that ensures both grippers follow the unscrewing path defined by the cap thread. Within the control loop, force sensor information is employed to control the vertical axis movements, while gripper rotation control is achieved through a Deep Reinforcement Learning (DRL) algorithm trained to determine the optimal angle increments for rotation. The results demonstrate the successful training of the learning agent. The experiments confirm the effectiveness of the proposed method in bottle opening with the TIAGo++ robot, showcasing the practical viability of the approach.

Intelligent control of the Magnus anti-rolling device: A co-simulation approach

In this study, we present an artificial intelligence control method specifically designed for the Magnus anti-rolling device. The core of our approach is the development of a co-simulation framework that integrates an intelligent algorithm with a three-dimensional numerical computational programme within a complex hydrodynamic environment. This integration is achieved through the use of a deep reinforcement learning algorithm, which allows for intelligent adaptation of the anti-rolling device's rotating speed in real time. Our research focuses on evaluating the performance of the intelligent anti-rolling control algorithm under a variety of conditions, including different ship-model roll angles, column geometry models, and varying swinging or slewing speeds. Through numerical studies, we compare the effects of these variables on the effectiveness of the control method. The co-simulation technology provides a platform for testing the intelligent control method. It allows a detailed examination of how the intelligent algorithm interacts with the hydrodynamic responses of the Magnus anti-rolling device, ensuring that the control method can adapt to a wide range of operational scenarios. This adaptability is crucial for maintaining stability and improving the performance of the anti-rolling device in real maritime environments. This study highlights the potential for integrating artificial intelligence with traditional maritime engineering solutions.

Optimized patient-specific immune checkpoint inhibitor therapies for cancer treatment based on tumor immune microenvironment modeling.

Enhancing patient response to immune checkpoint inhibitors (ICIs) is crucial in cancer immunotherapy. We aim to create a data-driven mathematical model of the tumor immune microenvironment (TIME) and utilize deep reinforcement learning (DRL) to optimize patient-specific ICI therapy combined with chemotherapy (ICC). Using patients' genomic and transcriptomic data, we develop an ordinary differential equations (ODEs)-based TIME dynamic evolutionary model to characterize interactions among chemotherapy, ICIs, immune cells, and tumor cells. A DRL agent is trained to determine the personalized optimal ICC therapy. Numerical experiments with real-world data demonstrate that the proposed TIME model can predict ICI therapy response. The DRL-derived personalized ICC therapy outperforms predefined fixed schedules. For tumors with extremely low CD8 + T cell infiltration ('extremely cold tumors'), the DRL agent recommends high-dosage chemotherapy alone. For tumors with higher CD8 + T cell infiltration ('cold' and 'hot tumors'), an appropriate chemotherapy dosage induces CD8 + T cell proliferation, enhancing ICI therapy outcomes. Specifically, for 'hot tumors', chemotherapy and ICI are administered simultaneously, while for 'cold tumors', a mid-dosage of chemotherapy makes the TIME 'hotter' before ICI administration. However, in several 'cold tumors' with rapid resistant tumor cell growth, ICC eventually fails. This study highlights the potential of utilizing real-world clinical data and DRL algorithm to develop personalized optimal ICC by understanding the complex biological dynamics of a patient's TIME. Our ODE-based TIME dynamic evolutionary model offers a theoretical framework for determining the best use of ICI, and the proposed DRL agent may guide personalized ICC schedules.