Actor-critic Deep Reinforcement Learning Research Articles

Optimal planning for electric vehicle fast charging stations placements in a city scale using an advantage actor-critic deep reinforcement learning and geospatial analysis

The transition to Electric Vehicles (EVs) for reducing urban greenhouse gas emissions is hindered by the lack of public charging infrastructure, particularly fast-charging stations. Given that electric vehicle fast charging stations (EVFCS) can burden the electricity grid, it is crucial for EVFCS to adopt sustainable energy supply methods while accommodating the growing demands of EVs. Despite recent research efforts to optimize the placement of renewable-powered EV charging stations, current planning methods face challenges when applied to a complex city scale and integrating with renewable energy resources. This study thus introduces a robust decision-making model for optimal EVFCS placement planning integrated with solar power supply in a large and complex urban environment (e.g., Chicago), utilizing an advantage actor-critic (A2C) deep reinforcement learning (DRL) approach. The model balances traffic demand with energy supply, strategically placing charging stations in areas with high traffic density and solar potential. As a result, the model is used to optimally place 1,000 charging stations with a random starting search approach, achieving total reward values of 74.30 %, and estimated the capacities of potential EVFCS. This study can inform the identification of suitable locations to advance the microgrid-based charging infrastructure systems in large urban environments.

Multi-IRSs enhanced UAV secure semantic communication system

This paper delves into a unmanned aerial vehicle (UAV) semantic communication system featuring multiple intelligent reflecting surfaces (IRSs). The UAV transmits semantic symbols to users in the presence of eavesdropping, and thus has inevitable security problems. In order to measure secure communication performance in semantic communication, the security semantic transmission rate (SSR) based on the semantic similarity is employed. An optimization problem is formulated to enhance the average SSR by optimizing UAV trajectory, IRS reflecting coefficients, number of semantic symbols, and transmission power, while satisfying position, speed, anti-collision, and power constraints. Due to the highly dynamic nature of the environment and the difficulties in solving complex modeling problems, a novel approach based on hybrid soft actor–critic (H-SAC) deep reinforcement learning (DRL) is proposed to address this non-convex optimization problem. The algorithm addresses the challenge of dynamic environments in UAV scenarios by seamlessly accommodating both discrete and continuous action spaces such as the trajectory and the IRS reflecting coefficient within the network. This enables effective processing of actions taken by intelligent agents, ensuring adaptability to the diverse requirements of the environment without the need for separate handling mechanisms. Simulation results show that the proposed intelligent algorithm exhibits excellent convergence and achieves a well-balanced pairing of appropriate IRSs and selection of flight paths, significantly improving the average SSR of the system. Specifically, our proposed scheme enhances the secure semantic transfer rate by more than fifty percent compared to when the number of semantic symbols is fixed at two.

Off-policy actor-critic deep reinforcement learning methods for alert prioritization in intrusion detection systems

Alert prioritization plays a very important role in network security as it helps security teams manage and respond to the overwhelming volume of alerts generated by intrusion detection systems (IDSs). Although current deep reinforcement learning (DRL) based deep deterministic policy gradient (DDPG) has achieved good performance for alert prioritization, there are still limitations in dealing with DDPG's instability and limited exploration capabilities. In this paper, we propose TD3-AP and SAC-AP, two DRL empowered off-policy actor-critic alert prioritization methods based on twin delayed deep deterministic policy gradient (TD3) and soft actor-critic (SAC), respectively. We model the interaction between an adversary and a defender as a zero-sum game and use a double oracle framework to obtain mixed strategy Nash equilibrium (MSNE). Our objective is to minimize the defender's loss which represents the inability of the defender to investigate alerts generated due to the attacks. To demonstrate the benefits and scalability of the proposed approaches, we conduct extensive experiments on three datasets: MQTT-IoT-IDS2020, DARPA 2000 LLDOS 1.0 and CSE-CIC-IDS2018. The results depict that TD3-AP and SAC-AP exhibit a decrease in the defender's loss by 50% and 14.28%, respectively, in comparison to the alert prioritization method based on DDPG. The improvements are significantly higher when the proposed approaches are compared with traditional alert prioritization methods including Uniform, Snort, Fuzzy and RAP. Additionally, we also provide an analysis of the interpretability of our results through the utilization of SHapley Additive exPlanations (SHAP). We assess the sensitivity of our proposed approaches to different hyperparameters and evaluate the training time and computational resources necessary for their implementation.

Explorer-Actor-Critic: Better actors for deep reinforcement learning

Actor-critic deep reinforcement learning methods have demonstrated significant performance in many challenging decision-making and control tasks, but also suffer from high sample complexity and overestimation bias. Current researches focus on using underestimation to balance overestimation and reducing bias through ensemble learning, but introducing underestimation bias and excessive network costs. In this paper, we first analyze the effect of action selection policy on estimation bias. Then, we propose the Explorer-Actor-Critic (EAC) method that gives a more conservative objective for the actor to reduce overestimation, introduces a learnable explorer to improve exploration ability, and uses an action mixing mechanism to mitigate experience distribution bias. Furthermore, we apply the EAC method to TD3 and SAC and verify its effectiveness through extensive comparison and ablation experiments. Our algorithm not only outperforms state-of-the-art algorithms, but also is compatible with other actor-critic methods.

ASSESSING TROPHIC STATE AND WATER QUALITY OF SMALL LAKES AND PONDS IN PERAK

Nowadays, the advancement in autonomous robots is the latest influenced by the development of a world surrounded by new technologies. Deep Reinforcement Learning (DRL) allows systems to operate automatically, so the robot will learn the next movement based on the interaction with the environment. Moreover, since robots require continuous action, Soft Actor Critic Deep Reinforcement Learning (SAC DRL) is considered the latest DRL approach solution. SAC is used because its ability to control continuous action to produce more accurate movements. SAC fundamental is robust against unpredictability, but some weaknesses have been identified, particularly in the exploration process for accuracy learning with faster maturity. To address this issue, the study identified a solution using a reward function appropriate for the system to guide in the learning process. This research proposes several types of reward functions based on sparse and shaping reward in SAC method to investigate the effectiveness of mobile robot learning. Finally, the experiment shows that using fusion sparse and shaping rewards in the SAC DRL successfully navigates to the target position and can also increase accuracy based on the average error result of 4.99%.

FUSION SPARSE AND SHAPING REWARD FUNCTION IN SOFT ACTOR-CRITIC DEEP REINFORCEMENT LEARNING FOR MOBILE ROBOT NAVIGATION

Nowadays, the advancement in autonomous robots is the latest influenced by the development of a world surrounded by new technologies. Deep Reinforcement Learning (DRL) allows systems to operate automatically, so the robot will learn the next movement based on the interaction with the environment. Moreover, since robots require continuous action, Soft Actor Critic Deep Reinforcement Learning (SAC DRL) is considered the latest DRL approach solution. SAC is used because its ability to control continuous action to produce more accurate movements. SAC fundamental is robust against unpredictability, but some weaknesses have been identified, particularly in the exploration process for accuracy learning with faster maturity. To address this issue, the study identified a solution using a reward function appropriate for the system to guide in the learning process. This research proposes several types of reward functions based on sparse and shaping reward in SAC method to investigate the effectiveness of mobile robot learning. Finally, the experiment shows that using fusion sparse and shaping rewards in the SAC DRL successfully navigates to the target position and can also increase accuracy based on the average error result of 4.99%.

A numerical simulation research on fish adaption behavior based on deep reinforcement learning and fluid–structure coupling: Implementation of the “perceive-feedback-memory” control system

The autonomous swimming of fish in a complex flow environment is a nonlinear and intricate system, which is the focus and challenge in various fields. This study proposed a novel simulation framework for artificial intelligence fish. It employed a high-precision immersed boundary-lattice Boltzmann coupling scheme to simulate the interactions between fish and flow in real time, and utilized the soft actor-critic (SAC) deep reinforcement learning algorithm for fish brain decision-making module, which was further divided into a vision-based directional navigation and a lateral line-based flow perception modules, each matched with its corresponding macro-action space. The flow features were extracted using a deep neural network based on a multi-classification algorithm from the data perceived by the lateral line and were linked to the fish actions. The predation swimming and the various Kármán gait swimming were explored in terms of training, simulation, and generalization. Numerical results demonstrated significant advantages in the convergence speed and training efficiency of the SAC algorithm. Owing to the closed-loop “perceive-feedback-memory” mode, intelligent fish can respond in real-time to changes in flow fields based on reward-driven requirements and experience, and the accumulated experience can be directly utilized in other flow fields, and its adaptability, model training efficiency, and generalization were substantially improved.

Comparing actor-critic deep reinforcement learning controllers for enhanced performance on a ball-and-plate system

The optimization of controller parameters remains an ongoing challenge in the field of control system applications. This study introduces a novel approach involving the creation of custom actor-critic deep reinforcement learning (DRL) based PID controllers. These controllers are designed with the goal of achieving adaptive tuning, precise trajectory tracking, and stability in a ball-and-plate system. To achieve this objective, multiple actor-critic reinforcement learning agents were developed using different learning algorithms: Soft actor critic (SAC), deep deterministic policy gradient (DDPG), and twin delayed deep deterministic policy gradient (TD3). These agents incorporate multilayer-perceptron (MLP) policy learning algorithms in both the actor and critic network architectures, employing non-linear activation functions. This enables them to fine-tune PID control parameters within an infinite search space. Additionally, a custom reward function derived from the system’s environment was integrated into the learning process. The performance of the proposed methods was compared against a benchmark method, specifically, an existing deep reinforcement learning controllers reported in the literature. The evaluation of these controllers and other approaches was based on error metrics and time response analysis. Results demonstrate that the proposed controller denoted as SAC-PID(5) excelled in trajectory tracking and outperformed other methods. It exhibited minimal predictive errors and the shortest time responses in the majority of experiments. This highlights the significance of designing a customized SAC agents with appropriate network architecture, which positively impacts the learning process for intelligent tuning of controllers for classical control systems.

An advanced actor critic deep reinforcement learning technique for gamification of WiFi environment

An advanced actor critic deep reinforcement learning technique for gamification of WiFi environment

Dynamic User-Scheduling and Power Allocation for SWIPT Aided Federated Learning: A Deep Learning Approach

Federated learning (FL) has been considered as a promising paradigm for enabling distributed machine learning (ML) in wireless networks. To address the limited energy capacity of wireless devices, we propose a simultaneous wireless information and power transfer (SWIPT) aided FL, in which one FL server (FLS) co-located at a cellular base station (BS) uses SWIPT to simultaneously broadcast the global model to wireless user-devices (UDs) and provide wireless power transfer to them. The UDs then use the harvested energy to train their local models and further transmit the local models to the FLS for aggregation. To improve the spectrum efficiency, we consider that the UDs form a non-orthogonal multiple access (NOMA) group for simultaneously sending their local models over the same spectrum channel. Taking the UDs' time-varying available energy and channel conditions into account, we propose a dynamic optimization of the UDs-scheduling, the BS's transmit-power allocation, and the UDs' power-splitting factors for SWIPT, with the objective of minimizing the long-term energy consumption while ensuring the FL convergence. The optimization problem, however, is challenging to solve since it is a finite-horizon dynamic programming problem but with an unknown stopping time, and moreover, the action space covers both discrete and continuous variables. To address these difficulties, we first execute a series of equivalent transformations to reduce the number of decision variables and then formulate the problem as a stochastic shortest path problem, based on which we propose an actor-critic deep reinforcement learning algorithm with the proximal policy optimization to efficiently learn the policy that dynamically adjusts the UDs-scheduling for FL as well as the BS's transmit-power for SWIPT. Numerical results validate the effectiveness and performance of our proposed algorithm. The results demonstrate that our proposed algorithm can effectively reduce the long-term energy consumption in comparison with two baseline algorithms.

Feasibility Constrained Online Calculation for Real-Time Optimal Power Flow: A Convex Constrained Deep Reinforcement Learning Approach

Due to the increasing uncertainties of renewable energy and stochastic demands, quick-optimal control actions are necessary to retain the system stability and economic operation. Existing optimal power flow (OPF) solution methods need to be enhanced to guarantee the solution optimality and feasibility in real-time operation under such uncertainties. This paper proposes a convex constrained soft actor-critic (CC-SAC) deep reinforcement learning (DRL) algorithm for the AC-OPF problem. First, this problem is standardized as a Markov decision process model to be solved by DRL algorithms. Second, the operational constraints are satisfied by a novel convex safety layer based on the penalty convex-concave procedure (P-CCP). Then, the control policy is updated by the state-of-the-art off-policy entropy maximization-based SAC algorithm. Therefore, the CC-SAC is a combination of data-driven and physics-driven approaches. The former speedups the solution time by predicting near-optimum control actions through a deep neural network. The latter effectively guarantees the solution feasibility. Simulation results demonstrate the computational performance of the proposed CC-SAC to effectively find AC-OPF decisions with no constraint violation, zero optimality gap and high speed up to 34 times compared to a state-of-the-art solver. The proposed approach indicates its practicability for power system real-time operation and marketing.

GraphSAGE with deep reinforcement learning for financial portfolio optimization

Portfolio optimization is an active management strategy that aims to maximize returns and control risk within reasonable limits. The Proximal Policy Optimization (PPO), a robust on-policy actor-critic deep reinforcement learning (DRL) model, is gaining popularity in portfolio optimization because it can help reduce emotional biases and take systematic investment actions. However, some research has found that the PPO model cannot achieve such remarkable performance in portfolio optimization as in games or robot control. In this paper, a novel GraphSAGE and DRL coupled model (GRL) is proposed to improve the architecture of the PPO agent by introducing a GraphSAGE-based feature extractor to capture the complex non-Euclidean relationships among market indexes, industry indexes and stocks. In addition, the explainable model SHAP is used to select a few but important features for GRL learning, and a method for generating a static financial graph is defined. This improves the robustness and training efficiency of the GRL model. We provide a holistic performance evaluation for GRL on three datasets using five metrics, i.e., Return On Investment (ROI), Sharpe Ratio, Sortino Ratio, Maximum Drawdown, and Calmar Ratio. The results show that the GRL model outperforms the Equal Weight strategy and the S&P 500 index. In addition, the results of the comparative analysis show that the Share-Extractor GRL and the Separate-Extractor GRL significantly outperform the PPO baseline without a feature extractor. This implies that integrating a GraphSAGE-based feature extractor into the PPO agent can improve its performance and robustness in portfolio optimization tasks.

Soft actor-critic DRL algorithm for interval optimal dispatch of integrated energy systems with uncertainty in demand response and renewable energy

The collaborative optimization dispatching of multiple energy flows plays a crucial role in achieving the economic and efficient low-carbon operation of integrated energy systems (IESs). However, the dispatching problem for IESs is characterized by high dimensionality, non-linearity, and complex coupling. Furthermore, the integration of renewable energy sources and flexible loads has transformed the IES into a complex dynamic system with significant uncertainty. Traditional intelligent optimization algorithms exhibit poor adaptability and lengthy solution computation time when tackling such problems. In contrast, deep reinforcement learning (DRL), as an interactive trial-and-error learning method, has shown improved decision-making capabilities. In view of this, a data-driven soft actor-critic (SAC) deep reinforcement learning-based approach is proposed in this paper for interval optimal dispatch of IESs considering multiple uncertainties. First, the basic principle of SAC reinforcement learning is introduced in detail, and the basic framework of reinforcement learning for interval optimal scheduling of IESs is constructed. Then, the environment model of agent interaction is constructed, and the action and state space of DRL, as well as the reward mechanism and neural network structure, are designed. Finally, a typical IES case is experimentally analyzed and compared with three popular DRL algorithms and five state-of-the-art intelligent optimization algorithms. The experimental results demonstrate the advantages and effectiveness of the proposed method in solving the optimal dispatching of IESs.

DAG-based workflows scheduling using Actor–Critic Deep Reinforcement Learning

High-Performance Computing (HPC) is essential to support the advance in multiple research and industrial fields. Despite the recent growth in processing and networking power, the HPC Data Centers (DCs) are finite, and should be carefully managed to host multiple jobs. The scheduling of tasks (composing a job) is a crucial and complex task, once the reflexes of the scheduler’s decisions are perceptible both for users (e.g., slowdown) and for infrastructure administrators (e.g., use of resources and queue length). In fact, the process of scheduling workflows atop a DC can be modeled as a graph mapping problem. While an undirected graph is used to represent the DC, a Directed Acyclic Graph (DAG) is used to express the tasks dependencies. Each vertex and edge from both graphs can have weights associated with them, denoting the residual capacities for DC resources, as well as computing and networking demands for workflows. Motivated by the combinatorial explosion of the aforementioned scheduling problem, the integration of Machine Learning (ML) for generating or improving scheduling policies is a reality, however the proposals in the specialized literature opt, mostly, for using simplified models to reduce the search space or are trained to specific scenarios, which leads to policies that eventually fall short of real DCs expectations. Given this challenge, this work applies Actor–Critic (AC) Reinforcement Learning (RL) to schedule DAG-based workflows. Instead of proposing a new policy, the AC RL is used to select the appropriated scheduling policy from a pool of consolidated algorithms, guided by the DAGs workload and DC usage. The AC RL-based scheduler analyzes the DAGs queue and the DC status to define which algorithms are better suited to improve the overall performance indicators in each scenario instance. The simulation protocol comprises multiple analysis with distinct workload configurations, number of jobs, queue ordering polices and strategies to select the target DC servers. The results demonstrated that the AC RL selects the scheduling policy which fits the current workload and DC status.

Benchmarking Actor-Critic Deep Reinforcement Learning Algorithms for Robotics Control With Action Constraints

This study presents a benchmark for evaluating action-constrained reinforcement learning (RL) algorithms. In action-constrained RL, each action taken by the learning system must comply with certain constraints. These constraints are crucial for ensuring the feasibility and safety of actions in real-world systems. We evaluate existing algorithms and their novel variants across multiple robotics control environments, encompassing multiple action constraint types. Our evaluation provides the first in-depth perspective of the field, revealing surprising insights, including the effectiveness of a straightforward baseline approach. The benchmark problems and associated code utilized in our experiments are made available online at github.com/omron-sinicx/action-constrained-RL-benchmark for further research and development.

Lexicographic Actor-Critic Deep Reinforcement Learning for Urban Autonomous Driving

Urban autonomous driving is a difficult task because of its complex road scenarios and the interaction between multiple vehicles. Autonomous vehicles need to balance multiple objectives in these complex scenarios, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.</i> , safety and speed. Traditional reinforcement learning methods deal with the multi-objective problem by optimizing agents with a single objective reward. However, these methods are sensitive to the reward scale and require huge experiments to design reward weights. In this paper, we propose the Lexicographical Proximal Policy Optimization algorithm (LPPO), which can express people's preference relationship through the lexicographical ordering between objectives. The proposed method has two main advantages. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">On the one hand,</i> LPPO has a smaller parameter adjustment space, which makes it easy to find the optimal solution that satisfies the actual problem preference. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">On the other hand,</i> the proposed method is less affected by the reward scale and easy to deploy in various driving scenarios. We evaluate our algorithm in two driving simulation environments, and the results show that the proposed method has better performance in urban driving tasks than previous reinforcement learning algorithms. In addition, we illustrate that the proposed method has better stability even if the reward scale changes.

Reward Shaping-Based Actor–Critic Deep Reinforcement Learning for Residential Energy Management

Residential energy consumption continues to climb steadily, requiring intelligent energy management strategies to reduce power system pressures and residential electricity bills. However, it is challenging to design such strategies due to the random nature of electricity pricing, appliance demand, and user behavior. This article presents a novel reward shaping (RS)-based actor–critic deep reinforcement learning (ACDRL) algorithm to manage the residential energy consumption profile with limited information about the uncertain factors. Specifically, the interaction between the energy management center and various residential loads is modeled as a Markov decision process that provides a fundamental mathematical framework to represent the decision-making in situations where outcomes are partially random and partially influenced by the decision-maker control signals, in which the key elements containing the agent, environment, state, action, and reward are carefully designed, and the electricity price is considered as a stochastic variable. An RS-ACDRL algorithm is then developed, incorporating both the actor and critic network and an RS mechanism, to learn the optimal energy consumption schedules. Several case studies involving real-world data are conducted to evaluate the performance of the proposed algorithm. Numerical results demonstrate that the proposed algorithm outperforms state-of-the-art RL methods in terms of learning speed, solution optimality, and cost reduction.

Vision-Based Robotic Object Grasping—A Deep Reinforcement Learning Approach

This paper focuses on developing a robotic object grasping approach that possesses the ability of self-learning, is suitable for small-volume large variety production, and has a high success rate in object grasping/pick-and-place tasks. The proposed approach consists of a computer vision-based object detection algorithm and a deep reinforcement learning algorithm with self-learning capability. In particular, the You Only Look Once (YOLO) algorithm is employed to detect and classify all objects of interest within the field of view of a camera. Based on the detection/localization and classification results provided by YOLO, the Soft Actor-Critic deep reinforcement learning algorithm is employed to provide a desired grasp pose for the robot manipulator (i.e., learning agent) to perform object grasping. In order to speed up the training process and reduce the cost of training data collection, this paper employs the Sim-to-Real technique so as to reduce the likelihood of damaging the robot manipulator due to improper actions during the training process. The V-REP platform is used to construct a simulation environment for training the deep reinforcement learning neural network. Several experiments have been conducted and experimental results indicate that the 6-DOF industrial manipulator successfully performs object grasping with the proposed approach, even for the case of previously unseen objects.

Machine learning-based fast charging of lithium-ion battery by perceiving and regulating internal microscopic states

Fast charging of the lithium-ion battery (LIB) is an enabling technology for the popularity of electric vehicles. However, high-rate charging regardless of the physical limits can induce irreversible degradation or even hazardous safety issues to the LIB system. Motivated by this, this paper proposes a machine learning-based fast charging strategy with multi-physical awareness within a battery-to-cloud framework. In particular, a reduced-order electrochemical-thermal model is built in the cloud to perceive the microscopic states of LIB, leveraging which the soft actor-critic (SAC) deep reinforcement learning (DRL) algorithm is exploited for the first time to train a fast charging strategy. Hardware-in-Loop tests and experiments with practical LIBs are carried out for validation. Results suggest that the battery-to-cloud architecture can mitigate the risk of a heavy computing burden in the real-time controller. The proposed strategy can effectively mitigate the unfavorable over-temperature and lithium deposition, which benefits the safety and longevity during fast charging. Given a similar charging speed, the proposed machine learning approach extends the LIB cycle life by about 75% compared to the commonly-used empirical protocol. Meanwhile, the proposed strategy is proven superior to the state-of-the-art rule-based and the model-based strategies in terms of charging rapidity, charging safety and computational complexity. Moreover, the trained low-complexity strategy is highly adaptive to the ambient temperature and initial charging state, which promises robust performance in practical applications.

Decentralized Joint Pilot and Data Power Control Based on Deep Reinforcement Learning for the Uplink of Cell-Free Systems

While the problem of jointly controlling the pilot-and-data power in cell-based systems has been extensively studied, this problem is difficult to solve in cell-free systems due to two reasons. First, both the large- and small-scale fading are markedly different between a served user and the multiple serving access points. Second, due to the user-centric architecture, there is a need for decentralized algorithms that scale well in the cell-free environment. In this work, we study the impact of joint pilot-and-data power control and receive filter design in the uplink of cell-free systems. The problem is formulated as optimization tasks considering two different objectives: 1) maximization of the minimum spectral efficiency (SE) and 2) maximization of the total SE. Since these problems are non-convex, we resort to successive convex approximation and geometric programming to obtain a local optimal centralized solution for benchmarking purposes. We also propose a decentralized solution based on actor-critic deep reinforcement learning, in which each user acts as an agent to locally obtain the best policy relying on minimum information exchange. Practical signaling aspects are provided for such a decentralized solution. Finally, numerical results indicate that the decentralized solution performs very close to the centralized one and outperforms state-of-the-art algorithms in terms of minimum SE and total system SE.