Deep Reinforcement Learning Framework Research Articles

Age-of-Information based Multi-UAV Trajectories Using Deep Reinforcement Learning

The use of unmanned aerial vehicles (UAVs) as a unified platform for sensing and communication is especially relevant in environments with inadequate infrastructure. In this paper, a multi-UAV system is constructed for dynamic data collection in a resource-constrained environment. The devised approach involves the implementation of an access platform referred to as the Access UAV (A_UAV). This A_UAV orchestrates the data collection process from Inspection-UAVs (I_UAVs), each equipped with a visual sensor, facilitating the relay of collected data to the cloud. Our approach jointly considers the trajectory scheduling of A_UAV and I_UAV to collect data from specific points in a decentralized manner. Specifically, a Deep Reinforcement Learning framework utilizing an actor-critic network is formulated for A_UAV, aiming to generate an equitable access schedule for I_UAVs. Moreover, trajectory scheduling of A_UAV ensures dynamic data collection while minimizing total system energy and the Age of Information (AoI) of data arriving from I_UAVs. The simulation results validate the performance of our proposed approach against several baselines under different parameter settings.

Federated deep reinforcement learning for mobile robot navigation

Navigation of a mobile robot in an unknown environment ensuring the safety of the robot and its surroundings is of utmost importance. Traditional methods, such as pathplanning algorithms, simultaneous localization and mapping, computer vision, and fuzzy techniques, have been employed to address this challenge. However, to achieve better generalization and self-improvement capabilities, reinforcement learning has gained significant attention. The concern of privacy issues in sharing data is also rising in various domains. In this study, a deep reinforcement learning strategy is applied to the mobile robot to move from its initial position to a destination. Specifically, the Deep Q-Learning algorithm has been used for this purpose. This strategy is trained using a federated learning approach to overcome privacy issues and to set a foundation for further analysis of distributed learning. The application scenario considered in this work involves the navigation of a mobile robot to a charging point within a greenhouse environment. The results obtained indicate that both the traditional deep reinforcement learning and federated deep reinforcement learning frameworks are providing 100% success rate. However federated deep reinforcement learning could be a better alternate since it overcomes the privacy issue along with other advantages discussed in this paper.

Multiagent-based deep reinforcement learning framework for multi-asset adaptive trading and portfolio management

The highly dynamic nature of stock markets has motivated researchers to propose various supervised learning models to assist investors to optimize financial performance. Machine learning models have been used to predict price trends, and approaches have been proposed for portfolio management. However, these studies focus on only one kind of financial issue, and the methods proposed exhibit poor generalizability. We address these problems with a multi-agent portfolio adaptive trading framework based on reinforcement learning to create an automated trading system with the best trading strategy that can be achieved by long-short situation judgment and adaptive capital allocation. We use the TD3 algorithm in the multi-agent algorithm to mitigate the overestimation and overfitting exhibited by traditional value functions and improve training stability. Experimental results show that the proposed framework outperforms single-agent reinforcement learning algorithms while achieving more stable returns.

Machine learning based on reinforcement learning for smart grids: Predictive analytics in renewable energy management

This paper introduces a groundbreaking approach to energy management in smart cities, leveraging deep reinforcement learning (DRL) and secured blockchain technology. The escalating demand for energy in urban areas necessitates intelligent solutions that optimize energy usage while ensuring data security and integrity. Our proposed technique integrates DRL algorithms with blockchain technology to achieve optimal energy management while addressing security concerns. The core innovation lies in the development of a DRL framework that dynamically adapts to changing energy demands and environmental factors. By continuously learning and optimizing energy allocation strategies, our approach enhances resource utilization efficiency and reduces wastage, contributing to sustainable smart city initiatives. Additionally, the incorporation of blockchain technology ensures data immutability, transparency, and secure transactions within the energy management system. We validate the effectiveness of our method through extensive simulations and real-world experiments, demonstrating significant improvements in energy efficiency, cost savings, and security compared to traditional approaches. Our research paves the way for future advancements in smart city infrastructure, offering a scalable and robust solution for optimal energy management with heightened security measures.

Intelligent Control of Building Vibrations: A Transformer-Based Deep Reinforcement Learning Framework

Intelligent Control of Building Vibrations: A Transformer-Based Deep Reinforcement Learning Framework

VizNav: A Modular Off-Policy Deep Reinforcement Learning Framework for Vision-Based Autonomous UAV Navigation in 3D Dynamic Environments

Unmanned aerial vehicles (UAVs) provide benefits through eco-friendliness, cost-effectiveness, and reduction of human risk. Deep reinforcement learning (DRL) is widely used for autonomous UAV navigation; however, current techniques often oversimplify the environment or impose movement restrictions. Additionally, most vision-based systems lack precise depth perception, while range finders provide a limited environmental overview, and LiDAR is energy-intensive. To address these challenges, this paper proposes VizNav, a modular DRL-based framework for autonomous UAV navigation in dynamic 3D environments without imposing conventional mobility constraints. VizNav incorporates the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm with Prioritized Experience Replay and Importance Sampling (PER) to improve performance in continuous action spaces and mitigate overestimations. Additionally, VizNav employs depth map images (DMIs) to enhance visual navigation by accurately estimating objects’ depth information, thereby improving obstacle avoidance. Empirical results show that VizNav, by leveraging TD3, improves navigation, and the inclusion of PER and DMI further boosts performance. Furthermore, the deployment of VizNav across various experimental settings confirms its flexibility and adaptability. The framework’s architecture separates the agent’s learning from the training process, facilitating integration with various DRL algorithms, simulation environments, and reward functions. This modularity creates a potential to influence RL simulation in various autonomous navigation systems, including robotics control and autonomous vehicles.

Integrated operations strategies for shared and privately-owned autonomous vehicles: A deep reinforcement learning framework

The shared autonomous vehicle (SAV), combining shared mobility and vehicle automation, will play a pivotal role in urban transportation. Effectively matching SAVs to stochastic customer demand poses a significant challenge, particularly when addressing microscopic multi-AV coordination to prevent conflicts and congestion within a network. To tackle this issue, we propose a novel traffic control scheme called the Integrated Operation Framework for Order Matching and AV Coordination (I-OFOMAC). This scheme introduces a third-party coordinator responsible for integrating online order matching for the SAV platform and real-time AV coordination for the road manager. Our approach utilizes an offline deep reinforcement learning method to train a state value neural network, which facilitates strategic order matching. Additionally, a neural network approximating bisimulation metrics is trained to enhance exploration and accelerate learning. To achieve scalability in network-level AV coordination, we adopt a scheduled transportation scheme. The coordinator solves the I-OFOMAC during each decision epoch, optimizing order matching and AV coordination. The I-OFOMAC aims to balance the goals of minimizing AV travel delay and maximizing SAV service revenue. We conduct extensive simulation studies in the realistic Wangjing network to evaluate the proposed I-OFOMAC scheme. Compared to two-step benchmark frameworks that implement order matching and AV coordination sequentially, the I-OFOMAC achieves an average increase of 13.28% in platform revenue (ranging from 2% to 22.53%) and a 4.48% reduction in average PAV travel delay (ranging from −6.55% to 20.17%).

Intelligent agent for hyperspectral image classification with noisy labels: a deep reinforcement learning framework

ABSTRACT Label accuracy plays an important role in supervised hyperspectral image classification problems. However, labelling can be prone to various types of noise, mainly including human errors and incorrect pseudo-label predictions. Learning from datasets with noisy labels is a critical challenge in practice. To address this issue, we first propose a deep reinforcement learning method named HyperRefine to tackle the label noise problem for hyperspectral image classification. The method utilizes an agent to learn the accuracy of each sample label and determine its suitability to be used in training the classifier model. The agent is trained using a reinforcement signal based on the reward obtained from a small validation set that reflects performance on the target task, where the validation set is generated from the training set by a momentum-based recruiter. Compared to complex hand-crafted methods of label noise cleaning, HyperRefine is easy to implement, yet highly effective. The robustness of the model to different types of label noises is significantly improved by further combining a novel label refinement approach. Experiments on artificially added label noise and pseudo-label noise hyperspectral datasets show that the proposed method can achieve promising performance.

Deep reinforcement learning framework for end-to-end semiconductor process control

Deep reinforcement learning framework for end-to-end semiconductor process control

A novel method-based reinforcement learning with deep temporal difference network for flexible double shop scheduling problem

This paper studies the flexible double shop scheduling problem (FDSSP) that considers simultaneously job shop and assembly shop. It brings about the problem of scheduling association of the related tasks. To this end, a reinforcement learning algorithm with a deep temporal difference network is proposed to minimize the makespan. Firstly, the FDSSP is defined as the mathematical model of the flexible job-shop scheduling problem joined to the assembly constraint level. It is translated into a Markov decision process that directly selects behavioral strategies according to historical machining state data. Secondly, the proposed ten generic state features are input into the deep neural network model to fit the state value function. Similarly, eight simple constructive heuristics are used as candidate actions for scheduling decisions. From the greedy mechanism, optimally combined actions of all machines are obtained for each decision step. Finally, a deep temporal difference reinforcement learning framework is established, and a large number of comparative experiments are designed to analyze the basic performance of this algorithm. The results showed that the proposed algorithm was better than most other methods, which contributed to solving the practical production problem of the manufacturing industry.

Optimizing Building Energy Management with Deep Reinforcement Learning for Smart and Sustainable Infrastructure

This study develops a new technique for optimising Energy Consumption (EC) and occupant satisfaction in business centres using Building Energy Management Systems (BEMS) that implement Deep Reinforcement Learning (DRL). Energy Management Models (EMM) are growing increasingly advanced and vital for intelligent power systems due to the growing demand for energy efficiency and the adoption of Renewable Energy Sources (RES), which are subject to variability. Flawed energy Consumption (EC) and problems are typical effects of traditional BEMS due to their unpredictability and failure to adapt to new environments. In this intended investigation, a DRL framework is demonstrated that may evolve its decision-making in real-time to control energy savings, electricity, and HVAC through input from the environment in which it operates. A pair of significant metrics, namely the cost of energy and room temperature stability, are employed to assess the effectiveness of the model compared to that provided by conventional rule-driven and predictive control systems. As investigated with different baseline models, the experimental findings proved that the DRL approach significantly reduced the cost of electricity while maintaining stable levels of comfort.

Mathematical Model-Driven Deep Learning Enables Personalized Adaptive Therapy.

Standard-of-care treatment regimens have long been designed for maximal cell killing, yet these strategies often fail when applied to metastatic cancers due to the emergence of drug resistance. Adaptive treatment strategies have been developed as an alternative approach, dynamically adjusting treatment to suppress the growth of treatment-resistant populations and thereby delay, or even prevent, tumor progression. Promising clinical results in prostate cancer indicate the potential to optimize adaptive treatment protocols. Here, we applied deep reinforcement learning (DRL) to guide adaptive drug scheduling and demonstrated that these treatment schedules can outperform the current adaptive protocols in a mathematical model calibrated to prostate cancer dynamics, more than doubling the time to progression. The DRL strategies were robust to patient variability, including both tumor dynamics and clinical monitoring schedules. The DRL framework could produce interpretable, adaptive strategies based on a single tumor burden threshold, replicating and informing optimal treatment strategies. The DRL framework had no knowledge of the underlying mathematical tumor model, demonstrating the capability of DRL to help develop treatment strategies in novel or complex settings. Finally, a proposed five-step pathway, which combined mechanistic modeling with the DRL framework and integrated conventional tools to improve interpretability compared to traditional "black-box" DRL models, could allow translation of this approach to the clinic. Overall, the proposed framework generated personalized treatment schedules that consistently outperformed clinical standard-of-care protocols.

Application of Deep Reinforcement Learning (DRL) for Malware Detection

Malware poses a significant threat to computer systems and networks, necessitating advanced detection methods to safeguard against potential cyber-attacks. This paper investigates the application of Deep Reinforcement Learning (DRL) for malware detection, leveraging its ability to learn complex patterns and behaviours from raw data. The study employs a DRL framework to train an agent to identify malicious software based on dynamic features extracted from executable files. A comprehensive evaluation is conducted using a diverse dataset, encompassing various types of malware samples. The experimental results demonstrate the effectiveness of the proposed DRL based approach in accurately detecting malware, achieving competitive performance compared to traditional methods and state-of-the-art techniques. Additionally, the paper discusses the interpretability and scalability of the model, along with potential challenges and future research directions in applying DRL to cybersecurity.

Balancing Awareness Fast Charging Control for Lithium-Ion Battery Pack Using Deep Reinforcement Learning

Minimizing charging time without damaging the batteries is significantly crucial for the large-scale penetration of electric vehicles. However, charging inconsistency caused by inevitable manufacture and usage inconsistencies can lead to lower efficiency, capacity, and shorter durability due to the “cask effect”. This goal can be achieved by solving a series of constrained optimization problems with the model-based framework. Nevertheless, the high computational complexity, identifiability, and observability issues still limit their fidelity and robustness. To overcome these limitations and provide end-to-end learning strategies, this paper proposes a balancing-aware fast-charging control framework based on deep reinforcement learning. In particular, a cell-to-pack equalization topology is first introduced to dispatch energy among in-pack cells. Then, the balancing awareness fast charging problem is formulated as a multiobjective optimization problem by considering charging time, consistency, and over-voltage safety constraints. Thirdly, a deep reinforcement learning framework using a deep Q-network is established to find the optimal policy. By using the generalization of a neural network, the learned policies can be transferred to real-time charging and balancing control, thus improving the applicability of the proposed strategy. Finally, numerous comparative simulations and experimental results illustrate its effectiveness and superiority in terms of charging rapidity and balancing.

Optimal parallelization strategies for active flow control in deep reinforcement learning-based computational fluid dynamics

Deep reinforcement learning (DRL) has emerged as a promising approach for handling highly dynamic and nonlinear active flow control (AFC) problems. However, the computational cost associated with training DRL models presents a significant performance bottleneck. To address this challenge and enable efficient scaling on high-performance computing architectures, this study focuses on optimizing DRL-based algorithms in parallel settings. We validate an existing state-of-the-art DRL framework used for AFC problems and discuss its efficiency bottlenecks. Subsequently, by deconstructing the overall framework and conducting extensive scalability benchmarks for individual components, we investigate various hybrid parallelization configurations and propose efficient parallelization strategies. Moreover, we refine input/output (I/O) operations in multi-environment DRL training to tackle critical overhead associated with data movement. Finally, we demonstrate the optimized framework for a typical AFC problem where near-linear scaling can be obtained for the overall framework. We achieve a significant boost in parallel efficiency from around 49% to approximately 78%, and the training process is accelerated by approximately 47 times using 60 central processing unit (CPU) cores. These findings are expected to provide valuable insight for further advancements in DRL-based AFC studies.

ToupleGDD: A Fine-Designed Solution of Influence Maximization by Deep Reinforcement Learning

Aiming at selecting a small subset of nodes with maximum influence on networks, the Influence Maximization (IM) problem has been extensively studied. Since it is #P-hard to compute the influence spread given a seed set, the state-of-the-art methods, including heuristic and approximation algorithms, faced with great difficulties such as theoretical guarantee, time efficiency, generalization, etc. This makes it unable to adapt to large-scale networks and more complex applications. On the other side, with the latest achievements of Deep Reinforcement Learning (DRL) in artificial intelligence and other fields, lots of works have been focused on exploiting DRL to solve combinatorial optimization problems. Inspired by this, we propose a novel end-to-end DRL framework, ToupleGDD, to address the IM problem in this paper, which incorporates three coupled graph neural networks for network embedding and double deep Q-networks for parameters learning. Previous efforts to solve IM problem with DRL trained their models on subgraphs of the whole network, and then tested on the whole graph, which makes the performance of their models unstable among different networks. However, our model is trained on several small randomly generated graphs with a small budget, and tested on completely different networks under various large budgets, which can obtain results very close to IMM and better results than OPIM-C on several datasets, and shows strong generalization ability. Finally, we conduct a large number of experiments on synthetic and realistic datasets, and experimental results prove the effectiveness and superiority of our model.

Multi-policy deep reinforcement learning for multi-objective multiplicity flexible job shop scheduling

This study considers the simultaneous minimization of makespan and total tardiness for the multi-objective multiplicity flexible job shop scheduling problem (MOMFJSP). A deep reinforcement learning framework employing a multi-policy proximal policy optimization algorithm (MPPPO) is developed to solve MOMFJSP. The MOMFJSP is treated as a Markov decision process, allowing an intelligent agent to make sequential decisions based on the current production status. This framework involves multiple policy networks with different objective weight vectors. Using MPPPO, these networks are optimized simultaneously to obtain a set of high-quality Pareto-optimal policies. Moreover, a fluid model is introduced to extract state features and devise composite dispatching rules as discrete actions. A multi-policy co-evolution mechanism (MPCEM) is proposed to facilitate collaborative evolution among policy networks, supported by a reward mechanism that considers the objective weights. A training algorithm based on MPPPO is designed for learning across multiple policy networks. The effectiveness and superiority of the proposed method are confirmed through comparisons with composite dispatching rules and other scheduling methods.

Deep reinforcement learning aided secure UAV communications in the presence of moving eavesdroppers

Integrating unmanned aerial vehicles (UAVs) with wireless systems is widely anticipated to bring enormous benefits to network users, and is considered as a key enabling technology for next generation wireless networks. However deployment of UAVs raises security concerns, as they are prone to eavesdropping attacks. In this paper we investigate secure data transmissions in a UAV-aided communication system from the physical layer security (PLS) perspective. The UAV is employed as a flying base station and transmit confidential information to a ground user. Specifically, an eavesdropper is moving in the vicinity of the ground user, attempting to intercept legitimate data transmission. We aim to optimize UAV trajectory to maximize secrecy throughput and due to the fast changing environment with unpredictable movements of the eave, this nonconvex problem is difficult to solve. Instead, we propose a deep reinforcement learning framework by reformulating this problem as a Markov Decision Process (MDP). By carefully designing state spaces, state-dependent action sets and time-dependant reward signals, we aim to help the agent learn an optimized trajectory and complete the flight task within time limit. We adapt the deep Q network (DQN) algorithm and extend it with prioritized experience replay (PER) and importance sampling method to ensure a stable and effective learning outcome. Simulation results validate the effectiveness of the learning algorithm and confirm that the agent is able to learn an effective policy with desirable and robust learning outcomes. Compared with benchmark schemes, our proposed scheme is shown to achieve considerable performance gain in secrecy throughput.

Multi-Agent Deep Reinforcement Learning Framework for Renewable Energy-Aware Workflow Scheduling on Distributed Cloud Data Centers

Multi-Agent Deep Reinforcement Learning Framework for Renewable Energy-Aware Workflow Scheduling on Distributed Cloud Data Centers

Active flow control of a turbulent separation bubble through deep reinforcement learning

The control efficacy of classical periodic forcing and deep reinforcement learning (DRL) is assessed for a turbulent separation bubble (TSB) at Reτ = 180 on the upstream region before separation occurs. The TSB can resemble a separation phenomenon naturally arising in wings, and a successful reduction of the TSB can have practical implications in the reduction of the aviation carbon footprint. We find that the classical zero-net-mas-flux (ZNMF) periodic control is able to reduce the TSB by 15.7%. On the other hand, the DRL-based control achieves 25.3% reduction and provides a smoother control strategy while also being ZNMF. To the best of our knowledge, the current test case is the highest Reynolds-number flow that has been successfully controlled using DRL to this date. In future work, these results will be scaled to well-resolved large-eddy simulation grids. Furthermore, we provide details of our open-source CFD–DRL framework suited for the next generation of exascale computing machines.