Deep Reinforcement Learning Approach Research Articles

Privacy preservation in deep reinforcement learning: A training perspective

Reinforcement learning (RL) is a principled AI framework for autonomous, experience-driven learning. Deep reinforcement learning (DRL) enhances this by incorporating deep learning models, promoting a higher-level understanding of the visual world. However, privacy concerns are emerging in RL applications that involve vast amounts of private information. Recent studies have demonstrated that DRL can leak private information and be vulnerable to attacks aiming to infer the training environment from an agent’s behaviors without direct access to the environment. To address these privacy concerns, we propose a differentially private DRL approach that obfuscates the agent’s observations from each visited state. This defends against privacy leakage attacks and prevents the inference of the agent’s training environment from its optimized policy. We provide a theoretical analysis and design comprehensive experiments to thoroughly reproduce the privacy leakage attack. Both the theoretical analysis and experimental results demonstrate that our method effectively defends against privacy leakage attacks while maintaining the model utility of the RL agent.

Team formation in large organizations: A deep reinforcement learning approach

Efficient team formation is critical to human resource management, particularly as large enterprise organizations continue to flatten and are increasingly driven by projects. Efficiently scheduling internal departments and reducing employee scheduling costs are essential objectives. This paper addresses the challenge of extracting employees from the existing network who possess the necessary skills to meet project requirements while minimizing the disruption to the original department network. To tackle this problem, we model the organization as a graph, where each employee is a node, and edges represent communication between them. We formulate team formation as a combinatorial optimization problem on the graph. We first innovatively design the employee replacement and organizational measures for changing structures on the graph. To overcome the complexity of team formation under vast organizational structures and resource constraints, we propose the Graph Combinatorial Optimization DQN framework. This novel approach combines reinforcement learning and graph neural networks. By leveraging graph neural networks, we learn employee representations based on their basic information, skills, and communication patterns with other employees. Furthermore, during testing, we enable the agent to continuously improve its solutions through learning and avoid the pitfall of optimizing early decisions that may hinder the modification of later decisions. This is achieved by incrementally building subsets of solutions. We demonstrate the superiority of the GCO-DQN framework using both the real-world enterprise dataset and a synthetic dataset by comparing GCO-DQN with five state-of-the-art methods.

Reinforcement learning for heliostat aiming: Improving the performance of Solar Tower plants

Solar Tower (ST) systems use heliostats to concentrate solar radiation onto a tower-mounted receiver. Optimizing the aiming strategy for these heliostats over the receiver remains a critical challenge due to the dynamic nature of solar radiation and the need to maximize energy capture while ensuring operational safety. This paper introduces a novel, model-free deep Reinforcement Learning (RL) approach to optimize heliostat aiming strategies, utilizing the Soft Actor–Critic (SAC) algorithm. This advanced RL method enhances the traditional Actor–Critic framework with two neural networks. The proposal dynamically adjusts the aiming points across the receiver surface in real time, trying to improve the overall performance of the ST plant. The strategy was simulated and evaluated over a full operational year and compared with traditional methods. The results show an increase of more than 8.8% in yearly absorbed power, a significant improvement that directly enhances performance and contributes to better economic outcomes for the technology. This technique also eliminates the need for constant human intervention and is applicable to both existing and future plants.

A Deep Reinforcement Learning Approach to Injection Speed Control in Injection Molding Machines with Servomotor-Driven Constant Pump Hydraulic System

The control of the injection speed in hydraulic injection molding machines is critical to product quality and production efficiency. This paper analyzes servomotor-driven constant pump hydraulic systems in injection molding machines to achieve optimal tracking control of the injection speed. We propose an efficient reinforcement learning (RL)-based approach to achieve fast tracking control of the injection speed within predefined time constraints. First, we construct a precise Markov decision process model that defines the state space, action space, and reward function. Then, we establish a tracking strategy using the Deep Deterministic Policy Gradient RL method, which allows the controller to learn optimal policies by interacting with the environment. Careful attention is also paid to the network architecture and the definition of states/actions to ensure the effectiveness of the proposed method. Extensive numerical results validate the proposed approach and demonstrate accurate and efficient tracking of the injection velocity. The controller’s ability to learn and adapt in real time provides a significant advantage over the traditional Proportion Integration Differentiation controller. The proposed method provides a practical solution to the challenge of maintaining accurate control of the injection speed in the manufacturing process.

Enhancing the Minimum Awareness Failure Distance in V2X Communications: A Deep Reinforcement Learning Approach.

Vehicle-to-everything (V2X) communication is pivotal in enhancing cooperative awareness in vehicular networks. Typically, awareness is viewed as a vehicle's ability to perceive and share real-time kinematic information. We present a novel definition of awareness in V2X communications, conceptualizing it as a multi-faceted concept involving vehicle detection, tracking, and maintaining their safety distances. To enhance this awareness, we propose a deep reinforcement learning framework for the joint control of beacon rate and transmit power (DRL-JCBRTP). Our DRL-JCBRTP framework integrates LSTM-based actor networks and MLP-based critic networks within the Soft Actor-Critic (SAC) algorithm to effectively learn optimal policies. Leveraging local state information, the DRL-JCBRTP scheme uses an innovative reward function to increase the minimum awareness failure distance. Our SLMLab-Gym-VEINS simulations show that the DRL-JCBRTP scheme outperforms existing beaconing schemes in minimizing awareness failure probability and maximizing awareness distance, ultimately improving driving safety.

DRL-assisted task offloading in enhanced time-expanded graph (eTEG)-modeled aerial computing

Space–air–ground integrated networks (SAGINs), categorized under aerial computing (AC), are emerging as a promising hierarchical platform designed to meet the seamless connectivity demands of the forthcoming 6G era. However, efficiently offloading ground tasks to space entities via SAGINs presents unprecedented challenges, primarily due to the mobility of these networks. In response, an enhanced time-expanded graph (eTEG) is proposed to model the dynamic distribution of heterogeneous SAGIN resources, including transmission bandwidth, computation, and storage, thereby optimizing task offloading and resource allocation by employing eTEG. Specifically, this optimization challenge is addressed using a deep reinforcement learning (DRL) approach, aimed at streamlining decision-making for task offloading and resource management to significantly reduce end-to-end delay and enhance network performance. Simulation experiments conducted to evaluate the proposed DRL-based method demonstrate its effectiveness in reducing energy consumption and improving stability, thereby outperforming other methods by achieving reduced delays and satisfying user requirements.

Deep Reinforcement Learning Approach to Portfolio Optimization in the Australian Stock Market

The future of portfolio management is evolving from relying on human expertise to incorporating artificial intelligence techniques. Traditional techniques such as fundamental and technical analysis will eventually be replaced by more sophisticated deep reinforcement learning (DRL) algorithms. However, it is still a long way from designing a profitable strategy in the complex and dynamic stock market. While previous studies have focused on the American stock market, this paper applies two DRL algorithms, the proximal policy optimization (PPO) and the advantage actor–critic (A2C), to trade the constituent stocks of the Australian Securities Exchange 50 (ASX50) Index. This paper also incorporates a weighted moving average into the action space and introduces a transaction threshold to help agents minimize trivial trades that lead to high transaction costs. The results are presented and benchmarked against the ASX50 Index. The A2C agent was better at following trends and had the higher upside potential but can suffer from more severe damage during bearish markets. On the other hand, the PPO agent had the lowest annual volatility and the highest maximum drawdown, which is more helpful in a bearish or volatile market.

Advancements in UAV Path Planning: A Deep Reinforcement Learning Approach with Soft Actor-Critic for Enhanced Navigation

This paper tackles the intricate challenge of autonomous navigation for Unmanned Aerial Vehicles (UAVs) through dynamically changing environments. We focus on a sophisticated Deep Reinforcement Learning (DRL) approach using the Soft Actor-Critic (SAC) algorithm, optimized for UAV path planning within a continuous action space. This methodology leverages environmental image data to enhance the precision of flight maneuvers and effective obstacle avoidance. Our approach, validated through extensive simulations in Gazebo and field tests, demonstrates the algorithm’s efficacy in enabling UAVs to adeptly navigate obstacles using depth maps. The study further explores the robustness of the SAC algorithm by comparing it with traditional DRL methods, emphasizing its superior performance in real-world applications. This research contributes significantly to advancing UAV technology, particularly in autonomous motion planning, by integrating cutting-edge machine learning techniques. The video link is: https://www.youtube.com/watch?v=Nd_aMzejNXY .

Multi-Stream Scheduling of Inference Pipelines on Edge Devices - a DRL Approach

Low-power edge devices equipped with Graphics Processing Units (GPUs) are a popular target platform for real-time scheduling of inference pipelines. Such application-architecture combinations are popular in Advanced Driver-assistance Systems for aiding in the real-time decision-making of automotive controllers. However, the real-time throughput sustainable by such inference pipelines is limited by resource constraints of the target edge devices. Modern GPUs, both in edge devices and workstation variants, support the facility of concurrent execution of computation kernels and data transfers using the primitive of streams , also allowing for the assignment of priority to these streams. This opens up the possibility of executing computation layers of inference pipelines within a multi-priority, multi-stream environment on the GPU. However, manually co-scheduling such applications while satisfying their throughput requirement and platform memory budget may require an unmanageable number of profiling runs. In this work, we propose a Deep Reinforcement Learning (DRL)-based method for deciding the start time of various operations in each pipeline layer while optimizing the latency of execution of inference pipelines as well as memory consumption. Experimental results demonstrate the promising efficacy of the proposed DRL approach in comparison with the baseline methods, particularly in terms of real-time performance enhancements, schedulability ratio, and memory savings. We have additionally assessed the effectiveness of the proposed DRL approach using a real-time traffic simulation tool IPG CarMaker.

Optimizing Energy and Delay in Task Offloading for Connected Vehicles with Proximal Policy Optimization

Electric-powered intelligent connected vehicles are becoming the pivotal point of the automotive industry. As vehicles integrate more applications, the computation tasks they generate increase substantially. Cloud servers cannot handle these tasks promptly, and current electric vehicles (EVs) have limited energy and computing resources. Multi-Access edge computing (MEC) performs various activities in proximity to the vehicles, resulting in decreased latency and the preservation of EV battery power. However, MEC servers have finite processing resources and may be unable to satisfy the required latency restrictions. We propose a task offloading scheme to optimize the allocation of computational resources from roadside servers across several EVs. We develop a mathematical model to optimize both computation latency and EV energy, represented as a Markov decision process (MDP). To address this, we employ the deep reinforcement learning-proximal policy optimization (DRL-PPO) algorithm. The implementation of our mathematical model, which is based on an MDP, together with the use of the DRL-PPO algorithm, showcases notable decreases in both energy consumption and latency when compared to alternative benchmark deep reinforcement learning (DRL) approaches.

An adaptive testing item selection strategy via a deep reinforcement learning approach.

Computerized adaptive testing (CAT) aims to present items that statistically optimize the assessment process by considering the examinee's responses and estimated trait levels. Recent developments in reinforcement learning and deep neural networks provide CAT with the potential to select items that utilize more information across all the items on the remaining tests, rather than just focusing on the next several items to be selected. In this study, we reformulate CAT under the reinforcement learning framework and propose a new item selection strategy based on the deep Q-network (DQN) method. Through simulated and empirical studies, we demonstrate how to monitor the training process to obtain the optimal Q-networks, and we compare the accuracy of the DQN-based item selection strategy with that of five traditional strategies-maximum Fisher information, Fisher information weighted by likelihood, Kullback‒Leibler information weighted by likelihood, maximum posterior weighted information, and maximum expected information-on both simulated and real item banks and responses. We further investigate how sample size and the distribution of the trait levels of the examinees used in training affect DQN performance. The results show that DQN achieves lower RMSE and MAE values than traditional strategies under simulated and real banks and responses in most conditions. Suggestions for the use of DQN-based strategies are provided, as well as their code.

Optimizing Energy Harvesting in Wireless Body Area Networks: A Deep Reinforcement Learning Approach to Dynamic Sampling

In energy-harvesting wireless body area networks (EH-WBAN), the self-sustainability of body sensor nodes without compromising the service quality criteria is very important. In remote vital signs monitoring applications, the sampling rate of body nodes in each period should be determined to achieve this goal. There are two fundamental challenges in determining the sampling rate: 1) In EH-WBAN, the rate of the harvestable energy is time-dependent and unpredictable. 2) The vital signs of the patient have different change rates relevant to the time. Therefore, a technique is required that can learn the changes of the harvestable energy and the change rates of vital signs simultaneously and specify the proper sampling rate for BNs. The reinforcement learning algorithms are among the efficient solutions for this problem because they are capable of learning environmental uncertainties. Previous RL-based methods for determining the BN's sampling rate have three fundamental problems: 1) focus on the optimization of the transmission energy and ignore the sensing energy, 2) only affect one of the two aspects of energy or sensed data in determining the sampling rate and 3) discretization of the problem space does not ensure the determination of the optimal sampling rate. Therefore, this paper proposes a method named “deep reinforcement learning-based dynamic sampling” (DRDS) which first formulates the sampling rate determination problem as a Markov decision process (MDP) and proposes a deep deterministic policy gradient algorithm (DDPG) to solve it. The proposed method considers both energy and data variability aspects in determining the sampling rate. Two parameters, the super-capacitor’s voltage level, and ambient light intensity, are considered for the energy aspect; for the data variability aspect, the long short-term memory (LSTM) algorithm is developed to predict the change rate of data in the next round. Simulations indicate that by preserving the data integrity, the proposed method can decrease the sampling rate and unnecessary data transmission by about 49.95% and 89.7%, respectively, compared with state-of-the-art methods, and allow the sensor to achieve self-sustainability.

Towards various occupants with different thermal comfort requirements: A deep reinforcement learning approach combined with a dynamic PMV model for HVAC control in buildings

Reinforcement learning (RL) has great potential in achieving energy-efficient, comfortable and intelligent control of heating, ventilation and air conditioning (HVAC) systems. Although research on RL-based HVAC control has attracted increasing interest, current studies generally use simple building simulation as the environment to train agents, and the definition of thermal comfort is limited to a wide temperature range, which cannot meet the different thermal comfort requirements of various occupants. This study proposes a deep reinforcement learning (DRL) control framework based on the Dueling Deep Q-network (DQN) algorithm, combined with a self-designed environmental model and reward function, for HVAC control meeting different thermal comfort requirements. Specifically, based on the theory of building thermal dynamics, a nonlinear equation modified by experimental data is used for the environmental model that reflects the actual thermal change of building. Different thermal comfort requirements are considered and analysed through a dynamic predicted mean vote (PMV) model that focuses on the metabolic rate and clothing level of occupants. By systematically exploring different heating modes for occupants and control time intervals, the proposed framework demonstrates that heating energy consumption can be reduced by 4.8%-39.58% under various conditions compared to rule-based control. In addition, the study found that the HVAC control based on DRL has greater potential in saving energy when the heating demand of building is higher. Our study is helpful for researchers to make HVAC control more energy-efficient and user-friendly with the help of artificial intelligence.

Dynamic Fault Reconfiguration of Distribution Networks in Ship Power Systems Based on Deep Reinforcement Learning Approach

Dynamic Fault Reconfiguration of Distribution Networks in Ship Power Systems Based on Deep Reinforcement Learning Approach

Real-Time Scheduling of Electric Bus Flash Charging at Intermediate Stops: A Deep Reinforcement Learning Approach

Real-Time Scheduling of Electric Bus Flash Charging at Intermediate Stops: A Deep Reinforcement Learning Approach

Intelligent joint communication and computation scheme of UAV-assisted Offloading in high speed rail scenarios

With the continuous development of communication and computation technologies, large bandwidth services place higher demands on computing capacity and throughput in High-Speed Rail (HSR) scenarios. On the one hand, Millimetre-Wave enables high data transmission rates, but leads to high Doppler frequency deviation as well as large path loss. On the other hand, the development of Mobile Edge Computing greatly alleviates user computing congestion. In this paper, we propose the Adaptive Joint Communication and Computation Resource Allocation scheme to solve the energy optimization problem of a dual-band UAV and Mobile Relay relay-assisted HSR offloading system. This scheme works well to optimize the performance of the system through resource allocation. In addition, since the optimization problem is modeled as a Mixed-Integer Nonlinear Program, we propose the Pre- and Post-state connected Parameterized Deep Q-Network algorithm, which is based on Deep Reinforcement Learning approach, for offloading decision and bandwidth resource allocation. Simulation results show that the proposed algorithms result in lower system energy consumption, while ensuring a high task completion rate as well.

Latency-Aware Container Scheduling in Edge Cluster Upgrades: A Deep Reinforcement Learning Approach

Latency-Aware Container Scheduling in Edge Cluster Upgrades: A Deep Reinforcement Learning Approach

Autonomous Navigation for Cellular-Connected UAV in Highly Dynamic Environments: A Deep Reinforcement Learning Approach

Autonomous Navigation for Cellular-Connected UAV in Highly Dynamic Environments: A Deep Reinforcement Learning Approach

Trajectory tracking control based on deep reinforcement learning and ensemble random network distillation for robotic manipulator

Abstract In general, the trajectory tracking of robotic manipulator is exceptionally challenging due to the complex and strongly coupled mechanical architecture. In this paper, precise track control of the robotic manipulator is formulated as a dense reward problem for reinforcement learning(RL). A deep RL(DRL) approach combining the soft actor-critic (SAC) algorithm and ensemble random network distillation (ERND) is proposed to address the tracking control problem for robotic manipulator. Firstly, an ERND model is designed, consisting of a module list of multiple RND models. Each RND model obtains the error by learning the target features and the predicted features of the environment. The resulting error serves as internal rewards that drive the robotic agent to explore unknown and unpredictable environmental states. The ensemble model obtains the total internal reward by summing the internal rewards of each RND model, thereby obtaining more accurately reflecting the characteristics of the manipulator in tracking control tasks and improving control performance. Secondly, combining the SAC algorithm with ERND facilitates more robust exploration capabilities in environments with input saturation and joint angle constraints, thereby enabling faster learning of effective policies and enhancing the performance and efficiency of robotic manipulator tracking control tasks. Finally, the simulation results demonstrate that the robotic manipulator tracking control task is effectively completed in dense reward problems through the combination of the SAC algorithm and ERND.

Energy-Efficient and Accelerated Resource Allocation in O-RAN Slicing Using Deep Reinforcement Learning and Transfer Learning

Abstract Next Generation Wireless Networks (NGWNs) have two main components: Network Slicing and Open Radio Access Networks (O-RAN). NS is needed to handle various Quality of Services (QoS). O-RAN adopts an open environment for network vendors and Mobile Network Operators (MNOs). In recent years, Deep Reinforcement Learning (DRL) approaches have been proposed to solve some key issues in NGWNs. The primary obstacles preventing the DRL deployment are being slowly converged and unstable. Additionally, these algorithms have enormous carbon emissions that negatively impact climate change. This paper tackles the dynamic allocation problem of O-RAN radio resources for better QoS, faster convergence, stability, lower energy and power consumption, and reduced carbon emissions. Firstly, we develop an agent with a newly designed latency-based reward function and a top-k filtration mechanism for actions. Then, we propose a policy Transfer Learning approach to accelerate agent convergence. We compared our model to another two models.