Deep Reinforcement Learning Approach Research Articles

A deep recurrent reinforcement learning approach for enhanced MPPT in PV systems

Under partial shading condition (PSC), for efficient usage, solar photovoltaic (PV) systems are required to operate at global maximum power point (GMPP). A maximum power point tracking (MPPT) controller is usually employed for this task. In this work, for MPPT under PSC, a recurrent deep reinforcement learning (DRL) approach has been developed and investigated. The optimized neural network, augmented by long short-term memory (LSTM), is trained using the proximal policy optimization (PPO) DRL algorithm. The MPPT performance of the proposed scheme was assessed using the GMPP attainment accuracy (%), on several realistic static and dynamic test cases. The presented investigations revealed the proposed controller's average MPPT accuracy of 97.79 % for the considered random static PSCs. Similarly, average MPPT accuracies of 95.69 % under changing irradiance, 95.70 % under change in temperature and 95.77 % under changes in both irradiance and temperature were recorded. These accuracies are found to be significantly superior to recent reported works. The resulting performance enhancement is attributed to the inclusion of LSTM, which allowed the agent to retain information about past states and actions, enabling it to make informed decisions. Therefore, based on the presented investigations, it is concluded that the proposed PPO-LSTM approach is an excellent MPPT alternative for PV systems.

Deep Reinforcement Learning for Optimizing Restricted Access Window in IEEE 802.11ah MAC Layer.

The IEEE 802.11ah standard is introduced to address the growing scale of internet of things (IoT) applications. To reduce contention and enhance energy efficiency in the system, the restricted access window (RAW) mechanism is introduced in the medium access control (MAC) layer to manage the significant number of stations accessing the network. However, to achieve optimized network performance, it is necessary to appropriately determine the RAW parameters, including the number of RAW groups, the number of slots in each RAW, and the duration of each slot. In this paper, we optimize the configuration of RAW parameters in the uplink IEEE 802.11ah-based IoT network. To improve network throughput, we analyze and establish a RAW parameters optimization problem. To effectively cope with the complex and dynamic network conditions, we propose a deep reinforcement learning (DRL) approach to determine the preferable RAW parameters to optimize network throughput. To enhance learning efficiency and stability, we employ the proximal policy optimization (PPO) algorithm. We construct network environments with periodic and random traffic in an NS-3 simulator to validate the performance of the proposed PPO-based RAW parameters optimization algorithm. The simulation results reveal that using the PPO-based DRL algorithm, optimized RAW parameters can be obtained under different network conditions, and network throughput can be improved significantly.

Collaborative Computation Offloading and Resource Management in Space–Air–Ground Integrated Networking: A Deep Reinforcement Learning Approach

With the increasing dissemination of the Internet of Things and 5G, mobile edge computing has become a novel scheme to assist terminal devices in executing computation tasks. To elevate the coverage and computation capability of edge computing, a collaborative computation offloading and resource management architecture was proposed in space–air–ground integrated networking (SAGIN). In this manuscript, we established a novel model considering the computation offloading cost constraints of the communication, computing and cache model in the SAGIN. To be specific, the joint optimization problem of collaborative computation offloading and resource management was modeled as a mixed integer nonlinear programming problem. To address this issue, this paper proposed a computation offloading and resource allocation strategy based on deep reinforcement learning (DRL). Differing from traditional methods, DRL does not need a well-established formulation or previous information, and it is capable of revising the strategy adaptively according to the environment. The simulation results demonstrate the proposed approach can achieve the optimal reward values in the case of different terminal device numbers. Furthermore, this manuscript provided the analysis with variant parameters of the proposed approach.

An adaptable and personalized framework for top-N course recommendations in online learning

In recent years, the proliferation of Massive Open Online Courses (MOOC) platforms on a global scale has been remarkable. Learners can now meet their learning demands with the help of MOOC. However, learners might not understand the course material well if they have access to a lot of information due to their inadequate expertise and cognitive ability. Personalized Recommender Systems (RSs), a cutting-edge technology, can assist in addressing this issue. It greatly increases resource acquisition through personalized availability for various people of all ages. Intelligent learning methods, such as machine learning and Reinforcement Learning (RL) can be used in RS challenges. However, machine learning needs supervised data and classical RL is not suitable for multi-task recommendations in online learning platforms. To address these challenges, the proposed framework integrates a Deep Reinforcement Learning (DRL) and multi-agent approach. This adaptive system personalizes the learning experience by considering key factors such as learner sentiments, learning style, preferences, competency, and adaptive difficulty levels. We formulate the interactive RS problem using a DRL-based Actor-Critic model named DRR, treating recommendations as a sequential decision-making process. The DRR enables the system to provide top-N course recommendations and personalized learning paths, enriching the student's experience. Extensive experiments on a MOOC dataset such as the 100 K Coursera course review validate the proposed DRR model, demonstrating its superiority over baseline models in major evaluation metrics for long-term recommendations. The outcomes of this research contribute to the field of e-learning technology, guiding the design and implementation of course RSs, to facilitate personalized and relevant recommendations for online learning students.

Collaborative Computing in Non-Terrestrial Networks: A Multi-Time-Scale Deep Reinforcement Learning Approach

Collaborative Computing in Non-Terrestrial Networks: A Multi-Time-Scale Deep Reinforcement Learning Approach

An Adaptive Route Guidance Model Considering the Effect of Traffic Signals Based on Deep Reinforcement Learning

Navigation or route guidance systems are designed to provide drivers with real-time travel information and the associated recommended routes for their trips. Classical route choice models typically rely on utility theory to represent drivers’ route choice behavior. Such choices, however, may not be optimal from both the individual and the system perspectives. This is simply due to the fact that drivers usually have imperfect knowledge about the time-varying traffic conditions. In this article, we explore and propose a new model-free deep reinforcement learning (DRL) approach to solving the adaptive route guidance problem based on microsimulation. The proposed approach consists of three interconnected algorithms, including a network edge labeling algorithm, a routing plan identification algorithm, and an adaptive route guidance algorithm. Simulation experiments on both a toy network and a real-world network of Suzhou, China, are performed to demonstrate the effectiveness of the proposed approach in terms of guiding a single vehicle as well as multiple vehicles through complex traffic environments. Comparative results confirm that the DRL approach outperforms the traditional shortest path method by further reducing the average travel time in the network.

Resource allocation in multi-user cellular networks: A transformer-based deep reinforcement learning approach

Resource allocation in multi-user cellular networks: A transformer-based deep reinforcement learning approach

Data distribution inference attack in federated learning via reinforcement learning support

Federated Learning(FL) is currently a widely used collaborative learning framework, and the distinguished feature of FL is that the clients involved in training do not need to share raw data, but only transfer the model parameters to share knowledge, and finally get a global model with improved performance. However, recent studies have found that sharing model parameters may still lead to privacy leakage. From the shared model parameters, local training data can be reconstructed and thus lead to a threat to individual privacy and security. We observed that most of the current attacks are aimed at client-specific data reconstruction, while limited attention is paid to the information leakage of the global model. In our work, we propose a novel FL attack based on shared model parameters that can deduce the data distribution of the global model. Different from other FL attacks that aim to infer individual clients’ raw data, the data distribution inference attack proposed in this work shows that the attackers can have the capability to deduce the data distribution information behind the global model. We argue that such information is valuable since the training data behind a well-trained global model indicates the common knowledge of a specific task, such as social networks and e-commerce applications. To implement such an attack, our key idea is to adopt a deep reinforcement learning approach to guide the attack process, where the RL agent adjusts the pseudo-data distribution automatically until it is similar to the ground truth data distribution. By a carefully designed MDP process, our implementation ensures our attack can have stable performance and experimental results verify the effectiveness of our proposed inference attack.

Optimization for Dynamic Laser Inter-Satellite Link Scheduling With Routing: A Multi-Agent Deep Reinforcement Learning Approach

Optimization for Dynamic Laser Inter-Satellite Link Scheduling With Routing: A Multi-Agent Deep Reinforcement Learning Approach

Federated Multiagent Deep Reinforcement Learning Approach via Physics-Informed Reward for Multimicrogrid Energy Management.

The utilization of large-scale distributed renewable energy (RE) promotes the development of the multimicrogrid (MMG), which raises the need of developing an effective energy management method to minimize economic costs and keep self energy sufficiency. The multiagent deep reinforcement learning (MADRL) has been widely used for the energy management problem because of its real-time scheduling ability. However, its training requires massive energy operation data of microgrids (MGs), while gathering these data from different MGs would threaten their privacy and data security. Therefore, this article tackles this practical yet challenging issue by proposing a federated MADRL (F-MADRL) algorithm via the physics-informed reward. In this algorithm, the federated learning (FL) mechanism is introduced to train the F-MADRL algorithm, thus ensures the privacy and the security of data. In addition, a decentralized MMG model is built, and the energy of each participated MG is managed by an agent, which aims to minimize economic costs and keep self energy sufficiency according to the physics-informed reward. At first, MGs individually execute the self-training based on local energy operation data to train their local agent models. Then, these local models are periodically uploaded to a server and their parameters are aggregated to build a global agent, which will be broadcasted to MGs and replace their local agents. In this way, the experience of each MG agent can be shared and the energy operation data are not explicitly transmitted, thus protecting the privacy and ensuring data security. Finally, experiments are conducted on Oak Ridge National Laboratory distributed energy control communication laboratory MG (ORNL-MG) test system, and the comparisons are carried out to verify the effectiveness of introducing the FL mechanism and the outperformance of our proposed F-MADRL.

Joint Offloading and Resource Allocation for Hybrid Cloud and Edge Computing in SAGINs: A Decision Assisted Hybrid Action Space Deep Reinforcement Learning Approach

Joint Offloading and Resource Allocation for Hybrid Cloud and Edge Computing in SAGINs: A Decision Assisted Hybrid Action Space Deep Reinforcement Learning Approach

A Deep Reinforcement Learning Approach for Microgrid Energy Transmission Dispatching

Optimal energy transmission dispatching of microgrid systems involves complicated transmission energy allocation and battery charging/discharging management and remains a difficult and challenging research problem subject to complex operation conditions and action constraints due to the randomness and volatility of new energy. Traditional microgrid transmission dispatching mainly considers the matching of the demand side and the supply side from a macro perspective, without considering the impact of line loss. Therefore, a Hierarchical Deep Q-network (HDQN) approach for microgrid energy dispatching is proposed to address this issue. The approach takes the power flow of each line and the battery charging/discharging behavior as decision variables to minimize the system operation cost. The proposed approach employs a two-layer agent optimization architecture for simultaneously processing the discrete and continuous variables, with one agent making upper layer decisions on the charging and discharging behavior of the batteries, and the other agent making lower layer decisions on the transmission energy allocation for the line. The experimental results indicate that the proposed approach achieves better performance than existing approaches.

Interactive learning for multi-finger dexterous hand: A model-free hierarchical deep reinforcement learning approach

When a multi-fingered dexterous hand interacts with the external environment, it encounters various challenges, including the utilization of complex control techniques and the intricate coordination of finger motion sequences. Previous studies have primarily concentrated on investigating the interaction between multi-fingered dexterous hands and external objects, usually using model-based control or model-free reinforcement learning techniques. However, during practical implementation, various constraining factors are encountered, such as intricate modeling and limited interaction capabilities. In practical scenarios, the utilization of multi-fingered dexterous hands is imperative for the swift and efficient execution of a wide range of interactive tasks, including but not limited to throwing a ball and playing rock-paper-scissors. These tasks require skilled manual dexterity to demonstrate both precise control and quick responsiveness. To tackle this issue, we propose a hierarchical control approach for multi-fingered dexterous hands with interactive functionalities, utilizing model-free deep reinforcement learning. The complex interaction task is decomposed into simple sub-tasks using hierarchical strategy and action primitive decomposition, which effectively reduces the complexity of the action space, and achieves the motion planning and end finger trajectory control of dexterous hand. In a simulated environment, the aforementioned method has successfully executed interactive tasks, including ball throwing and playing rock-paper-scissors. It achieved a maximum normalized reward of 0.83 and an 84 % success rate. These results are noteworthy in terms of both control accuracy and response speed. This study offers novel insights into the effective resolution of the intricate challenges associated with interactions involving multi-fingered dexterous hands and human-computer interaction.

A discrete event simulator to implement deep reinforcement learning for the dynamic flexible job shop scheduling problem

The job shop scheduling problem, which involves the routing and sequencing of jobs in a job shop context, is a relevant subject in industrial engineering. Approaches based on Deep Reinforcement Learning (DRL) are very promising for dealing with the variability of real working conditions due to dynamic events such as the arrival of new jobs and machine failures. Discrete Event Simulation (DES) is essential for training and testing DRL approaches, which are based on the interaction of an intelligent agent and the production system. Nonetheless, there are numerous papers in the literature in which DRL techniques, developed to solve the Dynamic Flexible Job Shop Problem (DFJSP), have been implemented and evaluated in the absence of a simulation environment. In the paper, the limitations of these techniques are highlighted, and a numerical experiment that demonstrates their ineffectiveness is presented. Furthermore, in order to provide the scientific community with a simulation tool designed to be used in conjunction with DRL techniques, an agent-based discrete event simulator is also presented.

Energy management for scalable battery swapping stations: A deep reinforcement learning and mathematical optimization cascade approach

The rapid growth of electric vehicles is driving the expansion of scalable Battery Swapping Stations (BSSs) to meet the demand for fast charging. However, existing BSS energy management struggles to adapt to the changes in battery counts online, the uncertainty of electricity prices and battery demand, as well as the complexity of demand response (DR). To address these issues, we propose a cascading approach that combines Deep Reinforcement Learning (DRL) with Mathematical Optimization (MO). Firstly, our method employs a two-layer optimization framework and establishes a BSS model. The upper controller uses DRL to make sequential decisions on total power for each time slot, addressing sequential decision-making under uncertainty. The lower controller uses MO to allocate total power to each charging bay for a single time slot, addressing space complexity. Secondly, we tailor proximal policy optimization algorithms and mixed-integer quadratic programming algorithms for BSS DR. The DRL is universally adaptable to scalable BSS, while the MO automatically generates and solves optimization models based on the working charging bay count. The proposed method is at least 33 times faster than existing methods and performs well in scalable BSSs under uncertain conditions. It can converge within a typical BSS DR scheduling slot of 15 min based on an i7-8950H CPU, even for a BSS with 3200 charging bays.

Cooperative lane-changing in mixed traffic: a deep reinforcement learning approach

Deep Reinforcement Learning (DRL) has made remarkable progress in autonomous vehicle decision-making and execution control to improve traffic performance. This paper introduces a DRL-based mechanism for cooperative lane changing in mixed traffic (CLCMT) for connected and automated vehicles (CAVs). The uncertainty of human-driven vehicles (HVs) and the microscopic interactions between HVs and CAVs are explicitly modelled, and different leader-follower compositions are considered in CLCMT, which provides a high-fidelity DRL learning environment. A feedback module is established to enable interactions between the decision-making layer and the manoeuvre control layer. Simulation results show that the increase in CAV penetration leads to safer, more comfort, and eco-friendly lane-changing behaviours. A CAV-CAV lane-changing scenario can enhance safety by 24.5%–35.8%, improve comfort by 8%–9%, and reduce fuel consumption and emissions by 5.2%–12.9%. The proposed CLCMT promises advantages in the lateral decision-making and motion control of CAVs.

A zero-shot reinforcement learning strategy for autonomous guidewire navigation.

The treatment of cardiovascular diseases requires complex and challenging navigation of a guidewire and catheter. This often leads to lengthy interventions during which the patient and clinician are exposed to X-ray radiation. Deep reinforcement learning approaches have shown promise in learning this task and may be the key to automating catheter navigation during robotized interventions. Yet, existing training methods show limited capabilities at generalizing to unseen vascular anatomies, requiring to be retrained each time the geometry changes. In this paper, we propose a zero-shot learning strategy for three-dimensional autonomous endovascular navigation. Using a very small training set of branching patterns, our reinforcement learning algorithm is able to learn a control that can then be applied to unseen vascular anatomies without retraining. We demonstrate our method on 4 different vascular systems, with an average success rate of 95% at reaching random targets on these anatomies. Our strategy is also computationally efficient, allowing the training of our controller to be performed in only 2h. Our training method proved its ability to navigate unseen geometries with different characteristics, thanks to a nearly shape-invariant observation space.

Energy-saving operation in urban rail transit: A deep reinforcement learning approach with speed optimization

The energy consumption of urban rail transit plays a significant role in the operating costs of trains. It is particularly crucial to decrease the energy consumption of the traction power supply in subway systems, as it accounts for approximately half of the total energy consumption of the subway operating organization. To overcome the limitations of traditional real-time speed profile generation methods and the limited exploration capabilities of popular reinforcement learning algorithms in the speed domain, this paper presents the Energy-Saving Maximum Entropy Deep Reinforcement Learning (ES-MEDRL) algorithm. The ES-MEDRL algorithm incorporates Lagrange multipliers and maximum policy entropy as penalties to formulate a novel objective function. This function aims to intensify exploration in the speed domain, minimize train traction energy consumption, and ensure a balance between ride comfort, punctuality, and safety within the subway system. This leads to the optimization of speed profile strategies. To further reduce energy consumption, this paper proposes a secondary optimization strategy for the energy-saving speed profile. This approach involves trading acceptable travel time for improved energy efficiency. To validate the performance of the proposed model and algorithm, numerical experiments are conducted using the Yizhuang Line of the Beijing Metro. The findings demonstrate a minimum 20 % increase in energy efficiency with the ES-MEDRL algorithm compared to manual driving. This algorithm can guide energy-efficient train operations at the planning level.

A Deep Reinforcement Learning Approach for Optimizing Inventory Management in the Agri-Food Supply Chain

This research aims to improve inventory management throughout the agri-food supply chain, through the use of Deep Reinforcement Learning algorithms. Three Deep Reinforcement Learning algorithms, including Deep Q-Networks, Deep Deterministic Policy Gradient, and Proximal Policy Optimization algorithms were implemented and tested in order to evaluate their ability to actively manage inventories and improve the performance of supply chains. The results of the experimental phase offered important information regarding the performance of each Deep Reinforcement Learning algorithm. The Deep Deterministic Policy Gradient algorithm was identified as a viable choice, offering the best results in terms of accuracy to set the optimal inventories for the supply chain and improving the efficiency of the supply chain. The percentage of cost efficiency improved from 92.5% to 95.8% in the case of the DDPG model. The inventory turnover was also improved, surpassing the level of 8.1 units from the original level of 7.3 units which means that the system converts the inventor into sales in less time. The metric for on-time delivery also beneficiated from several improvements, reaching 96.5% form the level of 93.2% return. The quality metrics also registered a significant improvement and reported level of 96.2% after the implementation of the DDPG algorithm and compared to the level of 94.6% prior to the implementation of the algorithm. These results suggest that using such a system will bring beneficial changes to the supply chain and will offer the possibility of implementing a data-driven inventory system based on Deep Reinforcement Learning.

Optimizing Building Energy Management with Deep Reinforcement Learning for Smart and Sustainable Infrastructure

This study develops a new technique for optimising Energy Consumption (EC) and occupant satisfaction in business centres using Building Energy Management Systems (BEMS) that implement Deep Reinforcement Learning (DRL). Energy Management Models (EMM) are growing increasingly advanced and vital for intelligent power systems due to the growing demand for energy efficiency and the adoption of Renewable Energy Sources (RES), which are subject to variability. Flawed energy Consumption (EC) and problems are typical effects of traditional BEMS due to their unpredictability and failure to adapt to new environments. In this intended investigation, a DRL framework is demonstrated that may evolve its decision-making in real-time to control energy savings, electricity, and HVAC through input from the environment in which it operates. A pair of significant metrics, namely the cost of energy and room temperature stability, are employed to assess the effectiveness of the model compared to that provided by conventional rule-driven and predictive control systems. As investigated with different baseline models, the experimental findings proved that the DRL approach significantly reduced the cost of electricity while maintaining stable levels of comfort.