Soft Actor-critic Research Articles

Stabilization of Phasor Measurement Sensor-Based Markovian Jump CPSs Through Soft Actor–Critic

Stabilization of Phasor Measurement Sensor-Based Markovian Jump CPSs Through Soft Actor–Critic

Low-Carbon Dispatch Method for Active Distribution Network Based on Carbon Emission Flow Theory

In the context of integrating renewable energy sources such as wind and solar energy sources into distribution networks, this paper proposes a proactive low-carbon dispatch model for active distribution networks based on carbon flow calculation theory. This model aims to achieve accurate carbon measurement across all operational aspects of distribution networks, reduce their carbon emissions through controlling unit operations, and ensure stable and safe operation. First, we propose a method for measuring carbon emission intensity on the source and network sides of active distribution networks with network losses, allowing for the calculation of total carbon emissions throughout the operation of networks and their equipment. Next, based on the carbon flow distribution of distribution networks, we construct a low-carbon dispatch model and formulate its optimization problem within a Markov Decision Process framework. We improve the Soft Actor–Critic (SAC) algorithm by adopting a Gaussian-distribution-based reward function to train and deploy agents for optimal low-carbon dispatch. Finally, the effectiveness of the proposed model and the superiority of the improved algorithm are demonstrated using a modified IEEE 33-bus distribution network test case.

Evade Unknown Pursuer via Pursuit Strategy Identification and Model Reference Policy Adaptation (MRPA) Algorithm

The game of pursuit–evasion has always been a popular research subject in the field of Unmanned Aerial Vehicles (UAVs). Current evasion decision making based on reinforcement learning is generally trained only for specific pursuers, and it has limited performance for evading unknown pursuers and exhibits poor generalizability. To enhance the ability of an evasion policy learned by reinforcement learning (RL) to evade unknown pursuers, this paper proposes a pursuit UAV attitude estimation and pursuit strategy identification method and a Model Reference Policy Adaptation (MRPA) algorithm. Firstly, this paper constructs a Markov decision model for the pursuit–evasion game of UAVs that includes the pursuer’s attitude and trains an evasion policy for a specific pursuit strategy using the Soft Actor–Critic (SAC) algorithm. Secondly, this paper establishes a novel relative motion model of UAVs in pursuit–evasion games under the assumption that proportional guidance is used as the pursuit strategy, based on which the pursuit UAV attitude estimation and pursuit strategy identification algorithm is proposed to provide adequate information for decision making and policy adaptation. Furthermore, a Model Reference Policy Adaptation (MRPA) algorithm is presented to improve the generalizability of the evasion policy trained by RL in certain environments. Finally, various numerical simulations imply the precision of pursuit UAV attitude estimation and the accuracy of pursuit strategy identification. Also, the ablation experiment verifies that the MRPA algorithm can effectively enhance the performance of the evasion policy to deal with unknown pursuers.

Soft Actor-Critic Approach to Self-Adaptive Particle Swarm Optimisation

Particle swarm optimisation (PSO) is a swarm intelligence algorithm that finds candidate solutions by iteratively updating the positions of particles in a swarm. The decentralised optimisation methodology of PSO is ideally suited to problems with multiple local minima and deceptive fitness landscapes, where traditional gradient-based algorithms fail. PSO performance depends on the use of a suitable control parameter (CP) configuration, which governs the trade-off between exploration and exploitation in the swarm. CPs that ensure good performance are problem-dependent. Unfortunately, CPs tuning is computationally expensive and inefficient. Self-adaptive particle swarm optimisation (SAPSO) algorithms aim to adaptively adjust CPs during the optimisation process to improve performance, ideally while reducing the number of performance-sensitive parameters. This paper proposes a reinforcement learning (RL) approach to SAPSO by utilising a velocity-clamped soft actor-critic (SAC) that autonomously adapts the PSO CPs. The proposed SAC-SAPSO obtains a 50% to 80% improvement in solution quality compared to various baselines, has either one or zero runtime parameters, is time-invariant, and does not result in divergent particles.

A deep reinforcement learning based charging and discharging scheduling strategy for electric vehicles

Grid security is threatened by the uncontrolled access of large-scale electric vehicles (EVs) to the grid. The scheduling problem of EVs charging and discharging is described as a Markov decision process (MDP) to develop an efficient charging and discharging scheduling strategy. Furthermore, a deep reinforcement learning (DRL)-based model-free method is suggested to address such issue. The proposed method aims to enhance EVs charging and discharging profits while reducing drivers' electricity anxiety and ensuring grid security. Drivers' electricity anxiety is described by fuzzy mathematical theory, and analyze the effect of EV current power and remaining charging time on it. Variable electricity pricing is calculated by real-time residential load. A dynamic charging environment is constructed considering the stochasticity of electricity prices, driver’s behavior, and residential load. A soft actor-critic (SAC) framework is used to train the agent, which learns the optimal charging and discharging scheduling strategies by interacting with the dynamic charging environment. Finally, simulation with actual data is used to verify that the suggested approach can reduce drivers' charging costs and electricity anxiety while avoiding transformer overload.

Optimization of emergency frequency control strategy for power systems considering both source and load uncertainties

With the increasing integration of renewable energy sources and the presence of numerous controllable loads such as electric vehicles and energy storage in the modern power system, higher nonlinearities and uncertainty both sources and loads are introduced. These factors pose challenges in achieving fast and accurate emergency frequency control. Therefore, this paper addresses the issue of dual source-load uncertainties in power system and presents an optimization strategy based on the Soft Actor Critic (SAC) algorithm that involves the participation of controllable loads in emergency frequency control. Firstly, the spatio-temporal uncertainties of wind farm power output on power supply side and power demand on the load side are described using Weibull and normal probability distributions, respectively. Secondly, an improved Markov Decision Process (MDP) model for emergency frequency control is established, which considers the characteristics of the dual source-load uncertainties. Finally, an optimization of the SAC algorithm is conducted based on Deep Reinforcement Learning (DRL), aiming to achieve rapid system frequency recovery and minimize the cost of removing controllable loads. The presented approach in the paper enhances the emergency frequency control strategy for uncertain power systems and effectively addresses the issue of source-load uncertainty compounded by fault power shortages.

A smart home energy management system based on human activity recognition and deep reinforcement learning

Smart home energy management systems (SHEMS) can help users save on energy costs by controlling various household loads in response to environmental changes. However, human activity is closely related to household energy consumption, which can lead to potential energy waste. Additionally, smart homes often feature multiple types of distributed energy resources (DER), and how to leverage their potential for participating in grid scheduling without significantly impacting users remains an unresolved issue. This paper proposes an intelligent home energy management method based on human activity recognition (HAR) and deep reinforcement learning. By using activity recognition results derived from intrusive load monitoring (ILM), we categorize contexts into two groups. The agent employs the soft actor-critic (SAC) algorithm for real-time control. In the first context, the goal is to ensure user satisfaction while reducing energy costs. In the second context, the focus is on controlling DER to participate in grid ancillary services to generate revenue. The proposed agent is trained using a real dataset, and simulations validate the effectiveness of the algorithm in home energy management.

Research on obstacle avoidance of underactuated autonomous underwater vehicle based on offline reinforcement learning

Abstract The autonomous navigation and obstacle avoidance capabilities of autonomous underwater vehicles (AUVs) are essential for ensuring their safe navigation and long-term, efficient operation. However, the complexity of the marine environment poses significant challenges to safe and effective obstacle avoidance. To address this issue, this study proposes an AUV obstacle avoidance control algorithm based on offline reinforcement learning. This method adopts the Conservative Q-learning (CQL) algorithm, which is based on the Soft Actor-Critic (SAC) framework. It learns from obtained historical obstacle avoidance data and ultimately achieves a favorable obstacle avoidance control strategy. In this method, PID and SAC control algorithms are utilized to generate expert obstacle avoidance data to construct a diversified offline database. Additionally, based on the line-of-sight (LOS) guidance method and artificial potential field (APF) method, information regarding the distance and orientation of targets and obstacles is incorporated into the state space, and heading and obstacle avoidance reward terms are integrated into the reward function design. The algorithm successfully guides the AUV in autonomous navigation and dynamic obstacle avoidance in three-dimensional space. Furthermore, the algorithm exhibits a certain degree of anti-interference capability against uncertain disturbances and ocean currents, enhancing the safety and robustness of the AUV system. Simulation results fully demonstrate the feasibility and effectiveness of the intelligent obstacle avoidance method based on offline reinforcement learning. This study highlights the profound significance of offline reinforcement learning in enabling robust and reliable control systems for AUVs, paving the way for enhanced operational capabilities in challenging marine environments.

Empowering legal justice with AI: A reinforcement learning SAC-VAE framework for advanced legal text summarization.

Automated summarization of legal texts poses a significant challenge due to the complex and specialized nature of legal documentation. Despite the recent progress in reinforcement learning for natural language text summarization, its application in the legal domain has been less effective. This paper introduces SAC-VAE, a novel reinforcement learning framework specifically designed for legal text summarization. We leverage a Variational Autoencoder (VAE) to condense the high-dimensional state space into a more manageable lower-dimensional feature space. These compressed features are subsequently utilized by the Soft Actor-Critic (SAC) algorithm for policy learning, facilitating the automated generation of summaries from legal texts. Through comprehensive experimentation, we have empirically demonstrated the effectiveness and superior performance of the SAC-VAE framework in legal text summarization.

Research on robust decision making for intelligent connected vehicle at highway on-ramp

Reinforcement learning has demonstrated its potential in the decision-making field of autonomous driving. By engaging in trial-and-error learning and interacting with the environment, we can adapt our behavioral strategies based on reward and enhance driving decisions. Nevertheless, real-world vehicle decision-making involves intricate and diverse information, and the limited state information hinders agents from making optimal decisions, which may result in catastrophic consequences like collisions. Therefore, this paper proposes a robust deep reinforcement learning method based on the Soft Actor-Critic (SAC) algorithm for highway intelligent connected vehicle ramp merging decision-making. The model represents the vehicle features of the target lane and its adjacent lanes as an environmental state space. Additionally, a hybrid action space is designed, which combines discrete lateral actions and continuous longitudinal actions. Finally, an on-ramp simulation platform is built using actual roads and SUMO to verify the feasibility of the model. Multiple sets of comparative analysis experimental results demonstrate that the proposed method outperforms others in terms of the average reward return value, robustness value, collision rate, and success rate. Moreover, the model exhibits a stable lane change success rate under various road traffic density conditions, which indicates its robustness. The code can be obtained at https://github.com/ColinFanghz/sac-on-ramp.git .

Deep Reinforcement Learning for Autonomous Driving Systems

Autonomous driving systems (ADS) are poised to revolutionize the future of transportation, promising increased safety, efficiency, and convenience. Deep Reinforcement Learning (DRL) has emerged as a powerful approach to solving complex decision-making tasks in dynamic environments, making it a promising candidate for the development of intelligent autonomous vehicles. This paper explores the application of DRL techniques in autonomous driving, focusing on the integration of perception, planning, and control. We review state-of-the-art DRL algorithms, including Deep Q-Networks (DQN), Proximal Policy Optimization (PPO), and Soft Actor-Critic (SAC), and examine their roles in enabling end-to-end learning for driving policies. Furthermore, we discuss the challenges inherent in deploying DRL in real-world autonomous driving scenarios, including sample inefficiency, safety constraints, and the sim-to-real gap. Finally, the paper presents case studies and experimental results that highlight the potential of DRL to improve autonomous vehicle performance in complex environments, while identifying future research directions to address open problems in the field.

A path planning method based on deep reinforcement learning for AUV in complex marine environment

The potential of autonomous underwater vehicle (AUV) on future applications is significant due to advancements in autonomy and intelligence. Path planning is a critical technology for AUV to perform operational missions in complex marine environments. To this end, this paper proposes a path planning method for AUV based on deep reinforcement learning. Initially, considering actual requirements, a complex marine environment model containing underwater terrain, sonobuoy detection, and ocean currents is established. Subsequently, the corresponding state space, action space, and reward function are formulated. Furthermore, to address the inherent limitations of existing deep reinforcement learning algorithms in terms of training efficiency, a mixed experience replay (MER) strategy is proposed. This strategy aims to enhance the efficiency of sample learning by integrating prior knowledge and exploration experience. Lastly, a novel HMER-SAC algorithm for AUV path planning is proposed by integrating the Soft Actor–Critic (SAC) algorithm with the hierarchical reinforcement learning strategy and the MER strategy. The results of the simulation and experiment demonstrate that the method is capable of efficiently planning executable paths in complex marine environments and exhibits superior training efficiency, stability, and performance.

Optimization of geological carbon storage operations with multimodal latent dynamic model and deep reinforcement learning

Identifying the time-varying control schemes that maximize storage performance is critical to the commercial deployment of geological carbon storage (GCS) projects. However, the optimization process typically demands extensive resource-intensive simulation evaluations, which poses significant computational challenges and practical limitations. In this study, we presented the multimodal latent dynamic (MLD) model, a novel deep learning framework for fast flow prediction and well control optimization in GCS operations. The MLD model implicitly characterizes the forward compositional simulation process through three components: a representation module that learns compressed latent representations of the system, a transition module that approximates the evolution of the system states in the low-dimensional latent space, and a prediction module that forecasts the flow responses for given well controls. A novel model training strategy combining a regression loss and a joint-embedding consistency loss was introduced to jointly optimize the three modules, which enhances the temporal consistency of the learned representations and ensures multi-step prediction accuracy. Unlike most existing deep learning models designed for systems with specific parameters, the MLD model supports arbitrary input modalities, thereby enabling comprehensive consideration of interactions between diverse types of data, including dynamic state variables, static spatial system parameters, rock and fluid properties, as well as external well settings. Since the MLD model mirrors the structure of a Markov decision process (MDP) that computes state transitions and rewards (i.e., economic calculation for flow responses) for given states and actions, it can serve as an interactive environment to train deep reinforcement learning agents. Specifically, the soft actor-critic (SAC) algorithm was employed to learn an optimal control policy that maximizes the net present value (NPV) from the experiences gained by continuous interactions with the MLD model. The efficacy of the proposed approach was first compared against commonly used simulation-based evolutionary algorithm and surrogate-assisted evolutionary algorithm on a deterministic GCS optimization case, showing that the proposed approach achieves the highest NPV, while reducing the required computational resources by more than 60%. The framework was further applied to the generalizable GCS optimization case. The results indicate that the trained agent is capable of harnessing the knowledge learned from previous scenarios to provide improved decisions for newly encountered scenarios, demonstrating promising generalization performance.

Simultaneous Optimization of Discrete and Continuous Parameters Defining a Robot Morphology and Controller.

The morphology and controller design of robots is often a labor-intensive task performed by experienced and intuitive engineers. Automatic robot design using machine learning is attracting increasing attention in the hope that it will reduce the design workload and result in better-performing robots. Most robots are created by joining several rigid parts and then mounting actuators and their controllers. Many studies limit the possible types of rigid parts to a finite set to reduce the computational burden. However, this not only limits the search space, but also prohibits the use of powerful optimization techniques. To find a robot closer to the global optimal design, a method that explores a richer set of robots is desirable. In this article, we propose a novel method to efficiently search for various robot designs. The method combines three different optimization methods with different characteristics. We apply proximal policy optimization (PPO) or soft actor-critic (SAC) as the controller, the REINFORCE algorithm to determine the lengths and other numerical parameters of the rigid parts, and a newly proposed method to determine the number and layout of the rigid parts and joints. Experiments with physical simulations confirm that when this method is used to handle two types of tasks-walking and manipulation-it performs better than simple combinations of existing methods. The source code and videos of our experiments are available online (https://github.com/r-koike/eagent).

Reinforcement learning for heliostat aiming: Improving the performance of Solar Tower plants

Solar Tower (ST) systems use heliostats to concentrate solar radiation onto a tower-mounted receiver. Optimizing the aiming strategy for these heliostats over the receiver remains a critical challenge due to the dynamic nature of solar radiation and the need to maximize energy capture while ensuring operational safety. This paper introduces a novel, model-free deep Reinforcement Learning (RL) approach to optimize heliostat aiming strategies, utilizing the Soft Actor–Critic (SAC) algorithm. This advanced RL method enhances the traditional Actor–Critic framework with two neural networks. The proposal dynamically adjusts the aiming points across the receiver surface in real time, trying to improve the overall performance of the ST plant. The strategy was simulated and evaluated over a full operational year and compared with traditional methods. The results show an increase of more than 8.8% in yearly absorbed power, a significant improvement that directly enhances performance and contributes to better economic outcomes for the technology. This technique also eliminates the need for constant human intervention and is applicable to both existing and future plants.

Enhancing the Minimum Awareness Failure Distance in V2X Communications: A Deep Reinforcement Learning Approach.

Vehicle-to-everything (V2X) communication is pivotal in enhancing cooperative awareness in vehicular networks. Typically, awareness is viewed as a vehicle's ability to perceive and share real-time kinematic information. We present a novel definition of awareness in V2X communications, conceptualizing it as a multi-faceted concept involving vehicle detection, tracking, and maintaining their safety distances. To enhance this awareness, we propose a deep reinforcement learning framework for the joint control of beacon rate and transmit power (DRL-JCBRTP). Our DRL-JCBRTP framework integrates LSTM-based actor networks and MLP-based critic networks within the Soft Actor-Critic (SAC) algorithm to effectively learn optimal policies. Leveraging local state information, the DRL-JCBRTP scheme uses an innovative reward function to increase the minimum awareness failure distance. Our SLMLab-Gym-VEINS simulations show that the DRL-JCBRTP scheme outperforms existing beaconing schemes in minimizing awareness failure probability and maximizing awareness distance, ultimately improving driving safety.

Advancements in UAV Path Planning: A Deep Reinforcement Learning Approach with Soft Actor-Critic for Enhanced Navigation

This paper tackles the intricate challenge of autonomous navigation for Unmanned Aerial Vehicles (UAVs) through dynamically changing environments. We focus on a sophisticated Deep Reinforcement Learning (DRL) approach using the Soft Actor-Critic (SAC) algorithm, optimized for UAV path planning within a continuous action space. This methodology leverages environmental image data to enhance the precision of flight maneuvers and effective obstacle avoidance. Our approach, validated through extensive simulations in Gazebo and field tests, demonstrates the algorithm’s efficacy in enabling UAVs to adeptly navigate obstacles using depth maps. The study further explores the robustness of the SAC algorithm by comparing it with traditional DRL methods, emphasizing its superior performance in real-world applications. This research contributes significantly to advancing UAV technology, particularly in autonomous motion planning, by integrating cutting-edge machine learning techniques. The video link is: https://www.youtube.com/watch?v=Nd_aMzejNXY .

Sea-Based UAV Network Resource Allocation Method Based on an Attention Mechanism

As humans continue to exploit the ocean, the number of UAV nodes at sea and the demand for their services are increasing. Given the dynamic nature of marine environments, traditional resource allocation methods lead to inefficient service transmission and ping-pong effects. This study enhances the alignment between network resources and node services by introducing an attention mechanism and double deep Q-learning (DDQN) algorithm that optimizes the service-access strategy, curbs action outputs, and improves service-node compatibility, thereby constituting a novel method for UAV network resource allocation in marine environments. A selective suppression module minimizes the variability in action outputs, effectively mitigating the ping-pong effect, and an attention-aware module is designed to strengthen node-service compatibility, thereby significantly enhancing service transmission efficiency. Simulation results indicate that the proposed method boosts the number of completed services compared with the DDQN, soft actor–critic (SAC), and deep deterministic policy gradient (DDPG) algorithms and increases the total value of completed services.

Achieving Robust Learning Outcomes in Autonomous Driving with DynamicNoise Integration in Deep Reinforcement Learning

The advancement of autonomous driving technology is becoming increasingly vital in the modern technological landscape, where it promises notable enhancements in safety, efficiency, traffic management, and energy use. Despite these benefits, conventional deep reinforcement learning algorithms often struggle to effectively navigate complex driving environments. To tackle this challenge, we propose a novel network called DynamicNoise, which was designed to significantly boost the algorithmic performance by introducing noise into the deep Q-network (DQN) and double deep Q-network (DDQN). Drawing inspiration from the NoiseNet architecture, DynamicNoise uses stochastic perturbations to improve the exploration capabilities of these models, thus leading to more robust learning outcomes. Our experiments demonstrated a 57.25% improvement in the navigation effectiveness within a 2D experimental setting. Moreover, by integrating noise into the action selection and fully connected layers of the soft actor–critic (SAC) model in the more complex 3D CARLA simulation environment, our approach achieved an 18.9% performance gain, which substantially surpassed the traditional methods. These results confirmed that the DynamicNoise network significantly enhanced the performance of autonomous driving systems across various simulated environments, regardless of their dimensionality and complexity, by improving their exploration capabilities rather than just their efficiency.

Smart energy management for hybrid electric bus via improved soft actor-critic algorithm in a heuristic learning framework

Deep reinforcement learning (DRL) is currently the cutting-edge artificial intelligence approach in the field of energy management for hybrid electric vehicles. However, inefficient offline training limits the energy-saving efficacy of DRL-based energy management strategies (EMSs). Motivated by this, this article proposes a smart DRL-based EMS in a heuristic learning framework for an urban hybrid electric bus. In order to enhance the sampling efficiency, the prioritized experience replay technique is introduced into soft actor-critic (SAC) for the innovative formulation of an improved SAC algorithm. Additionally, to strengthen the generalizability of the improved SAC agent to real driving scenarios, a stochastic training environment is constructed. Afterward, curriculum learning is employed to develop a heuristic learning framework that expedites convergence. Experimental simulations reveal that the designed EMS expedites convergence by 85.58 % and saves fuel by 6.43 % compared with the cutting-edge baseline EMS. Moreover, the computation complexity test demonstrates that the designed EMS holds significant promise for real-time implementation. These findings highlight the contribution of this article in facilitating fuel conservation for urban hybrid electric buses through the application of emerging artificial intelligence technologies.