Soft Actor-critic Algorithm Research Articles

Multi-Agent attention-based deep reinforcement learning for demand response in grid-responsive buildings

Integrating renewable energy resources and deploying energy management devices offer great opportunities to develop autonomous energy management systems in grid-responsive buildings. Demand response can promote enhancing demand flexibility and energy efficiency while reducing consumer costs. In this work, we propose a novel multi-agent deep reinforcement learning (MADRL) based approach with an agent assigned to individual buildings to facilitate demand response programs with diverse loads, including space heating/cooling and electrical equipment. Achieving real-time autonomous demand response in networks of buildings is challenging due to uncertain system parameters, the dynamic market price, and complex coupled operational constraints. To develop a scalable approach for automated demand response in networks of interconnected buildings, coordination between buildings is necessary to ensure demand flexibility and the grid's stability. We propose a MADRL technique that utilizes an actor-critic algorithm incorporating shared attention mechanism to enable effective and scalable real-time coordinated demand response in grid-responsive buildings. The presented case studies demonstrate the ability of the proposed approach to obtain decentralized cooperative policies for electricity costs minimization and efficient load shaping without knowledge of building energy systems. The viability of the proposed control approach is also demonstrated by a reduction of over 6% net load demand compared to standard reinforcement learning approaches, deep deterministic policy gradient, and soft actor-critic algorithm, as well as a tailored MADRL approach for demand response.

Automatic Casting Control Method of Continuous Casting Based on Improved Soft Actor–Critic Algorithm

Continuous casting production is an important stage in smelting high-quality steel, and automatic casting control based on artificial intelligence is a key technology to improve the continuous casting process and the product quality. By controlling the opening degree of the stopper rod reasonably, the mold can be filled with liquid steel stably in the specified time window, and automatic casting can be realized. In this paper, an automatic casting control method of continuous casting based on an improved Soft Actor–Critic (SAC) algorithm is proposed. Firstly, a relational model of the stopper rod opening degree and the liquid steel outflow velocity is established according to historical casting data. Then the Markov Decision Process (MDP) model of the automatic casting problem and the reinforcement learning framework based on the SAC algorithm are established. Finally, a Heterogeneous Experience Pool (HEP) is introduced to improve the SAC algorithm. According to the simulation results, the proposed algorithm can predict the stopper rod opening degree sequence under the constraint of the target liquid level curve. Under different billet specifications and interference conditions, an accuracy of 80% of liquid level in the mold and a stopper rod opening degree stability rate of 75% can be achieved, which is 4.29% and 3.17% higher than those for the baseline algorithms, respectively.

Multi-Microgrid Collaborative Optimization Scheduling Using an Improved Multi-Agent Soft Actor-Critic Algorithm

The implementation of a multi-microgrid (MMG) system with multiple renewable energy sources enables the facilitation of electricity trading. To tackle the energy management problem of an MMG system, which consists of multiple renewable energy microgrids belonging to different operating entities, this paper proposes an MMG collaborative optimization scheduling model based on a multi-agent centralized training distributed execution framework. To enhance the generalization ability of dealing with various uncertainties, we also propose an improved multi-agent soft actor-critic (MASAC) algorithm, which facilitates energy transactions between multi-agents in MMG, and employs automated machine learning (AutoML) to optimize the MASAC hyperparameters to further improve the generalization of deep reinforcement learning (DRL). The test results demonstrate that the proposed method successfully achieves power complementarity between different entities and reduces the MMG system’s operating cost. Additionally, the proposal significantly outperforms other state-of-the-art reinforcement learning algorithms with better economy and higher calculation efficiency.

Framework of Active Obstacle Avoidance for Autonomous Vehicle Based on Hybrid Soft Actor-Critic Algorithm

Framework of Active Obstacle Avoidance for Autonomous Vehicle Based on Hybrid Soft Actor-Critic Algorithm

Battery thermal- and cabin comfort-aware collaborative energy management for plug-in fuel cell electric vehicles based on the soft actor-critic algorithm

While the air conditioning system provides a comfortable thermal environment for vehicle occupants, it is also one of the most energy-intensive auxiliary devices in modern vehicles. Meanwhile, the operating characteristic of the battery system significantly impacts the economic and safety performance of the vehicle. To improve the energy efficiency of plug-in fuel cell electric vehicles, this paper proposes a collaborative energy management strategy based on the soft actor-critic algorithm, considering battery thermal management and cabin thermal comfort management. First, the soft actor-critic algorithm is implemented in energy management research for fuel cell electric vehicles. Second, by establishing a control-oriented vehicle-integrated thermal management model and adopting a novel air conditioning system design, the coordinated optimization of the battery thermal management and cabin thermal comfort management is achieved. Finally, the collaborative energy management strategy based on the soft actor-critic algorithm is presented to minimize the total operating cost, while maintaining the optimal battery temperature range and cabin thermal comfort. The online testing results demonstrate that the implemented energy management strategy based on the soft actor-critic algorithm can achieve 96.06% fuel economy of the dynamic programming benchmark methodology, while significantly improving computational efficiency. Moreover, the proposed collaborative energy management strategy can effectively improve the overall vehicle fuel economy, manage the battery temperature within the optimal operating range, and maintain cabin thermal comfort. However, with the prolonged operation of the air conditioning system, the total operating cost of the vehicle increases considerably, up to 28.12% or even more, indicating that it is a highly energy-consuming component.

Hierarchical reinforcement learning from competitive self-play for dual-aircraft formation air combat

AbstractThe recent development of technology helps in the revolutionary war and it controls the war which is influenced by brilliant planning. The maneuver aircraft of intelligent algorithm aids the pilot to decide the particular position on the battlefield. Nowadays the hardware components of radar and missiles are widely used and the beyond visual range is the most popular method applied in air combat. The introduction of close-range air combat maneuver decisions generates the attention of researchers in artificial intelligence. Most of the existing methods are based on autonomous aircraft focused in air combat scenario but manual air combats are widely applied in dual aircraft. Based on the factors mentioned above, a novel hierarchical maneuver decision architecture is applied to a dual-aircraft close-range air combat scenario. Subsequently, the soft actor-critic algorithm is merged with competitive self-play which integrates the knowledge of sub-strategies. Further, the reinforcement learning technique is employed to achieve an approximate Nash equilibrium master strategy. The experimental results show that the hierarchical architecture exhibits good performance, symmetry, and robustness. The research generates a solution for intelligent formation of air combat in the future and guidance for manned or unmanned aircraft cooperative combat.

Manoeuvre decision‐making of unmanned aerial vehicles in air combat based on an expert actor‐based soft actor critic algorithm

AbstractThe demand for autonomous motion control of unmanned aerial vehicles in air combat is boosted as taking the initiative in combat appears more and more crucial. Unmanned aerial vehicles inability to manoeuvre autonomously during air combat that features highly dynamic and uncertain manoeuvres of the enemy; however, limits their combat capabilities, which proves to be very challenging. To meet the challenge, this article proposes an autonomous manoeuvre decision model using an expert actor‐based soft actor critic algorithm that reconstructs empirical replay buffer with expert experience. Specifically, the algorithm uses a small amount of expert experience to increase the diversity of the samples, which can largely improve the exploration and utilisation efficiency of deep reinforcement learning. And to simulate the complex battlefield environment, a one‐to‐one air combat model is established and the concept of missile's attack region is introduced. The model enables the one‐to‐one air combat to be simulated under different initial battlefield situations. Simulation results show that the expert actor‐based soft actor critic algorithm can find the most favourable policy for unmanned aerial vehicles to defeat the opponent faster, and converge more quickly, compared with the soft actor critic algorithm.

Cooperative Resource Allocation Based on Soft Actor–Critic With Data Augmentation in Cellular Network

This letter investigates the cooperative resource allocation of cellular networks with simultaneous wireless information and power transfer in the time-varying channel environment. The soft actor-critic (SAC) algorithm is exploited to tackle the optimization problem which aims to find a feasible resource allocation policy to maximize the data rate and system fairness while minimizing the channel switching penalty. Considering the costly agent-to-environment interactions and the restricted empirical dataset of the SAC algorithm, this letter explores the permutation equivalence of the optimization objective, and designs two data augmentation schemes for the experience replay buffer of SAC. The cumulative discount reward shows that data augmentation assisted algorithms outperform the baseline in the learning speed. The simulation results referring to the average data rate and system fairness show that the proposed schemes benefit to the training model and effectively improve the performance of algorithms.

Coordinated carbon capture systems and power-to-gas dynamic economic energy dispatch strategy for electricity–gas coupled systems considering system uncertainty: An improved soft actor–critic approach

Due to uncertainties in renewable energy generation and load demands, traditional energy dispatch schemes for an integrated electricity–gas system (IEGS) considerably depend on explicit forecast mathematical models. In this study, a novel data-driven deep reinforcement learning method is applied to solve the IEGS dynamic dispatch problem with the targets of minimizing carbon emission and operating cost. Moreover, a flexible operation of carbon capture system and power-to-gas facility is proposed to attain low operating costs. The IEGS dynamic dispatch problem is formulated as a Markov game, and a soft actor–critic (SAC) algorithm is applied to learn the optimal dispatch solution. To improve training efficiency and convergence, prioritized experience replay (PER) is employed. In the simulation, the proposed PER–SAC algorithm compared with deep Q-network and SAC has fast and stable learning performance. In contrast to a modified sequential quadratic programming based on uncertainty prediction, the proposed method can reduce the target cost by 11.62% when the prediction error exceeds 10%. The computational time of scenario analysis solution on the same hardware platform is 4.58 times than that of training the PER–SAC method. Finally, the simulation results under different scenarios demonstrate that the PER–SAC-based dispatch strategy has satisfactory generalization and adaptability.

Deep reinforcement learning-based two-timescale Volt-VAR control with degradation-aware smart inverters in power distribution systems

Higher penetration of intermittent solar photovoltaic (PV) systems in the distribution grid results in frequent voltage fluctuations. The conventional voltage regulating devices operating on a slow-timescale need to be supplemented with the fast-operating smart inverters with adjustable reactive power setpoints. Complete and accurate information about distribution network topology and line parameters is necessary for conventional model-based Volt-VAR control (VVC) methods. However, such information is often unavailable. To tackle these challenges, a reinforcement learning-based two-timescale VVC algorithm is proposed in this paper that jointly controls the conventional voltage regulating devices at the slow-timescale and the smart inverters at the fast-timescale. Our proposed VVC algorithm simultaneously minimizes voltage violation costs and system operation costs in a model-free manner utilizing historical operational data. Two hierarchically organized agents are set up for the slow-timescale and fast-timescale problems, which are coupled through a communication scheme. The two sets of control policies are learned concurrently by a deep deterministic policy gradient and multi-agent soft actor-critic algorithm respectively. Comprehensive numerical studies performed with the IEEE 123-bus distribution test feeder show that the proposed framework can identify near optimal control actions of voltage regulating devices and smart inverters in real-time operations.

6D (2,0) bootstrap with the soft-actor-critic algorithm

We study numerically the 6D (2,0) superconformal bootstrap using the soft-Actor-Critic (SAC) algorithm as a stochastic optimizer. We focus on the four-point functions of scalar superconformal primaries in the energy-momentum multiplet. Starting from the supergravity limit, we perform searches for adiabatically varied central charges and derive two curves for a collection of 80 CFT data (70 of these data correspond to unprotected long multiplets and 10 to protected short multiplets). We conjecture that the two curves capture the A- and D-series (2,0) theories. Our results are competitive when compared to the existing bounds coming from standard numerical bootstrap methods, and data obtained using the OPE inversion formula. With this paper we are also releasing our Python implementation of the SAC algorithm, BootSTOP. The paper discusses the main functionality features of this package.

Naturalistic data-driven and emission reduction-conscious energy management for hybrid electric vehicle based on improved soft actor-critic algorithm

Energy management strategies (EMSs) are critical to saving fuel and reducing emissions for hybrid electric vehicles (HEVs). Given that, this article proposes a naturalistic data-driven and emission reduction-conscious EMS based on deep reinforcement learning (DRL) for a power-split HEV. In this article, for the purpose of evaluating the practical fuel economy of an HEV driving in a certain city region, a specific driving cycle is constructed by using a naturalistic data-driven method. Furthermore, to realize the multi-objective optimization in terms of fuel conservation and emission reduction as well as the state of charge (SOC) sustaining, an intelligent EMS based on the improved soft actor-critic (SAC) algorithm with a novel experience replay method is innovatively proposed. Finally, the effectiveness and optimality of the proposed EMS are verified. Simulation results indicate that the constructed driving cycle can effectively reflect the real traffic scenarios of the test region. Moreover, the proposed EMS achieves 95.25% fuel economy performance of the global optimum, improving the fuel economy by 5.29% and reducing the emissions by 10.42% compared with the emission reduction-neglecting EMS based on standard SAC. This article contributes to energy conservation and emission reduction for the transportation industry through advanced DRL methods.

Suspension Control Strategies Using Switched Soft Actor-Critic Models for Real Roads

In this article, we propose learning and control strategies for a semiactive suspension system in a full car using soft actor-critic (SAC) models on real roads, where many road profiles with various power of disturbance exist (e.g., speed bumps and general roads). Therefore, a technique that enables deep reinforcement learning to cover different domains with largely different reward functions is proposed. This concept was first realized in a simulation environment. Our proposed switching learning system continuously identifies two different road disturbance profiles in real time such that the appropriately designed SAC model can be learned and applied accordingly. The results of the proposed switching SAC algorithm were compared against those of advanced and conventional benchmark suspension systems. Based on the results, the proposed algorithm showed smaller root-mean-square values of the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$z$</tex-math></inline-formula> -directional acceleration and pitch at the center of the body mass. Finally, we also presented our successfully implemented SAC training system in a real car on real roads. The trained SAC model outperforms conventional controllers reducing the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$z$</tex-math></inline-formula> -directional acceleration and pitch, similar to the simulation results, which is highly related to the riding comfort and vehicle maneuverability.

Offline reinforcement learning control for electricity and heat coordination in a supercritical CHP unit

With a high proportion of renewable energy generation connected to the power grid, the combined heat and power (CHP) units need to have flexible operation and control capabilities over a wide range of variable load conditions. In China, the proportion of supercritical combined heat and power (S-CHP) units gradually increases due to their high thermal efficiency. In this paper, we propose a data-driven environment modeling method and an offline reinforcement learning-based electricity–heat coordinated control approach for the wide and flexible load adjustment capabilities of the S-CHP unit. First of all, a modeling method based on the multiple multilayer perceptron (MLP) ensemble is proposed to address the possible over-fitting produced by a single MLP. Then, the state and reward are set considering the dynamic characteristics of the S-CHP unit. Moreover, policy training is implemented through a soft actor-critic algorithm with a maximum entropy model to ensure more robust search capabilities under variable load conditions and targets. The simulation results show that the generalization ability of the environment in the multiple MLP ensemble mode is more substantial than that in the single MLP mode. In addition, when the electric load command is between 267 MW and 325 MW, the offline reinforcement learning can significantly reduce the integral of absolute error matrices of the output parameters, demonstrating that the proposed strategy can achieve electricity–heat coordinated control under variable load conditions.

Real-time route planning of unmanned aerial vehicles based on improved soft actor-critic algorithm.

With the application and development of UAV technology and navigation and positioning technology, higher requirements are put forward for UAV maneuvering obstacle avoidance ability and real-time route planning. In this paper, for the problem of real-time UAV route planning in the unknown environment, we combine the ideas of artificial potential field method to modify the state observation and reward function, which solves the problem of sparse rewards of reinforcement learning algorithm, improves the convergence speed of the algorithm, and improves the generalization of the algorithm by step-by-step training based on the ideas of curriculum learning and transfer learning according to the difficulty of the task. The simulation results show that the improved SAC algorithm has fast convergence speed, good timeliness and strong generalization, and can better complete the UAV route planning task.

Data-efficient deep reinforcement learning with expert demonstration for active flow control

Deep reinforcement learning (RL) is capable of identifying and modifying strategies for active flow control. However, the classic active formulation of deep RL requires lengthy active exploration. This paper describes the introduction of expert demonstration into a classic off-policy RL algorithm, the soft actor-critic algorithm, for application to vortex-induced vibration problems. This combined online-learning framework is applied to an oscillator wake environment and a Navier–Stokes environment with expert demonstration obtained from the pole-placement method and surrogate model optimization. The results show that the soft actor-critic framework combined with expert demonstration enables rapid learning of active flow control strategies through a combination of prior demonstration data and online experience. This study develops a new data-efficient RL approach for discovering active flow control strategies for vortex-induced vibration, providing a more practical methodology for industrial applications.

System Frequency Control Method Driven by Deep Reinforcement Learning and Customer Satisfaction for Thermostatically Controlled Load

The intermittence and fluctuation of renewable energy aggravate the power fluctuation of the power grid and pose a severe challenge to the frequency stability of the power system. Thermostatically controlled loads can participate in the frequency regulation of the power grid due to their flexibility. Aiming to solve the problem of the traditional control methods, which have limited adjustment ability, and to have a positive influence on customers, a deep reinforcement learning control strategy based on the framework of soft actor–critic is proposed, considering customer satisfaction. Firstly, the energy storage index and the discomfort index of different users are defined. Secondly, the fuzzy comprehensive evaluation method is applied to evaluate customer satisfaction. Then, the multi-agent models of thermostatically controlled loads are established based on the soft actor–critic algorithm. The models are trained by using the local information of thermostatically controlled loads, and the comprehensive evaluation index fed back by users and the frequency deviation. After training, each agent can realize the cooperative response of thermostatically controlled loads to the system frequency only by relying on the local information. The simulation results show that the proposed strategy can not only reduce the frequency fluctuation, but also improve customer satisfaction.

Deep reinforcement learning based active disturbance rejection load frequency control of multi-area interconnected power systems with renewable energy

The use of renewable energy to replace traditional coal, oil and natural gas for power generation is of great significance to the alleviation of environmental pollution and resource depletion. As the core of automatic generation control (AGC), load frequency control (LFC) is of great significance to the stable operation of the power system. However, adding renewable energy sources such as wind turbines and solar energy will bring unstable factors to the frequency response, while frequency fluctuations will reduce the power quality. To minimize the adverse effects caused by renewable energy, the linear active disturbance rejection control (LADRC) based on the soft actor-critic (SAC) algorithm is designed. Through the ability of LADRC to eliminate unknown disturbances and the decision-making ability of the SAC algorithm, an intelligent LFC controller is applied to a two-area interconnected power system with a solar thermal power plant (STPP) and a four-area interconnected power system with wind turbine generator (WTG), photovoltaic (PV) cells and hydro turbine. The simulation results are compared with that of the PID, MPC, and conventional LADRC controllers, highlighting the effectiveness of the proposed method.

Deep Reinforcement Learning Based Decision Making for Complex Jamming Waveforms

With the development of artificial intelligence, intelligent communication jamming decision making is an important research direction of cognitive electronic warfare. In this paper, we consider a complex intelligent jamming decision scenario in which both communication parties choose to adjust physical layer parameters to avoid jamming in a non-cooperative scenario and the jammer achieves accurate jamming by interacting with the environment. However, when the situation becomes complex and large in number, traditional reinforcement learning suffers from the problems of failure to converge and a high number of interactions, which are fatal and unrealistic in a real warfare environment. To solve this problem, we propose a deep reinforcement learning based and maximum-entropy-based soft actor-critic (SAC) algorithm. In the proposed algorithm, we add an improved Wolpertinger architecture to the original SAC algorithm in order to reduce the number of interactions and improve the accuracy of the algorithm. The results show that the proposed algorithm shows excellent performance in various scenarios of jamming and achieves accurate, fast, and continuous jamming for both sides of the communication.

Path Planning for Multi-Arm Manipulators Using Soft Actor-Critic Algorithm with Position Prediction of Moving Obstacles via LSTM

This paper presents a deep reinforcement learning-based path planning algorithm for the multi-arm robot manipulator when there are both fixed and moving obstacles in the workspace. Considering the problem properties such as high dimensionality and continuous action, the proposed algorithm employs the SAC (soft actor-critic). Moreover, in order to predict explicitly the future position of the moving obstacle, LSTM (long short-term memory) is used. The SAC-based path planning algorithm is developed using the LSTM. In order to show the performance of the proposed algorithm, simulation results using GAZEBO and experimental results using real manipulators are presented. The simulation and experiment results show that the success ratio of path generation for arbitrary starting and goal points converges to 100%. It is also confirmed that the LSTM successfully predicts the future position of the obstacle.