Soft Actor-critic Research Articles

Real-time automatic control of multi-energy system for smart district community: A coupling ensemble prediction model and safe deep reinforcement learning

Energy system autonomous control is influenced by day-ahead forecasting, despite being carried out independently in the literature. This paper develops an energy management modular platform that integrates multi-variable timeseries prediction and autonomous energy infrastructure scheduling in real-time. Firstly, an ensemble prediction model is developed for the day-ahead multi-energy and renewable power prediction, which is implemented by coupling variants of CNN, GRU, and BiLSTM models into a global model using an ensemble approach. Secondly, a deep reinforcement learning (DRL) with a soft actor critic (SAC) algorithm that include safety-guided network to make the policy network constraint-aware is proposed. A multi-energy system with renewable energy and carbon capture technology is then anticipated as the energy infrastructure for achieving a carbon neutral community and evaluating our proposed model. The proposed models are trained and tested on a real-world dataset from Arizona, USA. The ensemble prediction model achieved the least root mean squared error (RMSE). On the other hand, the improved DRL method exhibits superior performance in reducing energy cost, minimum constraint violation, and fast deployment compared to state-of-the-art DRL methods. Finally, the two models are coupled and carry out generalization performance on the prediction and energy management scheme, including the sensitivity analysis on carbon capture price, in the case studies.

Penetration Strategy for High-Speed Unmanned Aerial Vehicles: A Memory-Based Deep Reinforcement Learning Approach

With the development and strengthening of interception measures, the traditional penetration methods of high-speed unmanned aerial vehicles (UAVs) are no longer able to meet the penetration requirements in diversified and complex combat scenarios. Due to the advancement of Artificial Intelligence technology in recent years, intelligent penetration methods have gradually become promising solutions. In this paper, a penetration strategy for high-speed UAVs based on improved Deep Reinforcement Learning (DRL) is proposed, in which Long Short-Term Memory (LSTM) networks are incorporated into a classical Soft Actor–Critic (SAC) algorithm. A three-dimensional (3D) planar engagement scenario of a high-speed UAV facing two interceptors with strong maneuverability is constructed. According to the proposed LSTM-SAC approach, the reward function is designed based on the criteria for successful penetration, taking into account energy and flight range constraints. Then, an intelligent penetration strategy is obtained by extensive training, which utilizes the motion states of both sides to make decisions and generate the penetration overload commands for the high-speed UAV. The simulation results show that compared with the classical SAC algorithm, the proposed algorithm has a training efficiency improvement of 75.56% training episode reduction. Meanwhile, the LSTM-SAC approach achieves a successful penetration rate of more than 90% in hypothetical complex scenarios, with a 40% average increase compared with the conventional programmed penetration methods.

Deep reinforcement learning algorithms for dynamic pricing and inventory management of perishable products

A perishable product has a limited shelf life, and inefficient management often leads to waste. This paper focuses on dynamic pricing and inventory management strategies for perishable products. By implementing effective inventory control and the right pricing policy, it is possible to maximize expected revenue. However, the exponential growth of the problem size due to the shelf life of the products makes it impractical to use methods that guarantee optimal solutions, such as Dynamic Programming (DP). Therefore, approximate solution algorithms become necessary. We use Deep Reinforcement Learning (DRL) algorithms to address the dynamic pricing and ordering problem for perishable products, considering price and age-dependent stochastic demand. We investigate Deep Q Learning (DQL) solutions for discrete action spaces and Soft Actor-Critic (SAC) solutions for continuous action spaces. To mitigate the negative impact of the stochastic environment inherent in the problem, we propose two different DQL approaches. Our results show that the proposed DQL and SAC algorithms effectively address inventory control and dynamic pricing for perishable products, even when products of different ages are offered simultaneously. Compared to dynamic programming, our proposed DQL approaches achieve an average approximation of 95.5 % and 96.6 %, and reduce solution times by 71.5 % and 79.9 %, respectively, for the largest problem. In addition, the SAC algorithm achieves on average 4.6 % and 1.7 % better results and completes the task 56.1 % and 48.2 % faster than the proposed DQL algorithms.

Constant force grinding controller for robots based on SAC optimal parameter finding algorithm

Since conventional PID (Proportional–Integral–Derivative) controllers hardly control the robot to stabilize for constant force grinding under changing environmental conditions, it is necessary to add a compensation term to conventional PID controllers. An optimal parameter finding algorithm based on SAC (Soft-Actor-Critic) is proposed to solve the problem that the compensation term parameters are difficult to obtain, including training state action and normalization preprocessing, reward function design, and targeted deep neural network design. The algorithm is used to find the optimal controller compensation term parameters and applied to the PID controller to complete the compensation through the inverse kinematics of the robot to achieve constant force grinding control. To verify the algorithm's feasibility, a simulation model of a grinding robot with sensible force information is established, and the simulation results show that the controller trained with the algorithm can achieve constant force grinding of the robot. Finally, the robot constant force grinding experimental system platform is built for testing, which verifies the control effect of the optimal parameter finding algorithm on the robot constant force grinding and has specific environmental adaptability.

A Soft Actor-Critic Deep Reinforcement-Learning-Based Robot Navigation Method Using LiDAR

When there are dynamic obstacles in the environment, it is difficult for traditional path-generation algorithms to achieve desired obstacle-avoidance results. To solve this problem, we propose a robot navigation control method based on SAC (Soft Actor-Critic) Deep Reinforcement Learning. Firstly, we use a fast path-generation algorithm to control the robot to generate expert trajectories when the robot encounters danger as well as when it approaches a target, and we combine SAC reinforcement learning with imitation learning based on expert trajectories to improve the safety of training. Then, for the hybrid data consisting of agent data and expert data, we use an improved prioritized experience replay method to improve the learning efficiency of the policies. Finally, we introduce RNN (Recurrent Neural Network) units into the network structure of the SAC Deep Reinforcement-Learning navigation policy to improve the agent’s transfer inference ability in a new environment and obstacle-avoidance ability in dynamic environments. Through simulation and practical experiments, it is fully verified that our method has a higher training efficiency and navigation success rate compared to state-of-the-art reinforcement-learning algorithms, which further enhances the obstacle-avoidance capability of the robot system.

AUV Obstacle Avoidance Framework Based on Event-Triggered Reinforcement Learning

Autonomous Underwater Vehicles (AUVs), as a member of the unmanned intelligent ocean vehicle group, can replace human beings to complete dangerous tasks in the ocean. It is of great significance to apply reinforcement learning (RL) to AUVs to realize intelligent control. This paper proposes an AUV obstacle avoidance framework based on event-triggered reinforcement learning. Firstly, an environment perception model is designed to judge the relative position relationship between the AUV and all unknown obstacles and known targets. Secondly, considering that the detection range of AUVs is limited, and the proposed method needs to deal with unknown static obstacles and unknown dynamic obstacles at the same time, two different event-triggered mechanisms are designed. Soft actor–critic (SAC) with a non-policy sampling method is used. Then, improved reinforcement learning and the event-triggered mechanism are combined in this paper. Finally, a simulation experiment of the obstacle avoidance task is carried out on the Gazebo simulation platform. Results show that the proposed method can obtain higher rewards and complete tasks successfully. At the same time, the trajectory and the distance between each obstacle confirm that the AUV can reach the target well while maintaining a safe distance from static and dynamic obstacles.

Online EVs Vehicle-to-Grid Scheduling Coordinated with Multi-Energy Microgrids: A Deep Reinforcement Learning-Based Approach

The integration of electric vehicles (EVs) into vehicle-to-grid (V2G) scheduling offers a promising opportunity to enhance the profitability of multi-energy microgrid operators (MMOs). MMOs aim to maximize their total profits by coordinating V2G scheduling and multi-energy flexible loads of end-users while adhering to operational constraints. However, scheduling V2G strategies online poses challenges due to uncertainties such as electricity prices and EV arrival/departure patterns. To address this, we propose an online V2G scheduling framework based on deep reinforcement learning (DRL) to optimize EV battery utilization in microgrids with different energy sources. Firstly, our approach proposes an online scheduling model that integrates the management of V2G and multi-energy flexible demands, modeled as a Markov Decision Process (MDP) with an unknown transition. Secondly, a DRL-based Soft Actor-Critic (SAC) algorithm is utilized to efficiently train neural networks and dynamically schedule EV charging and discharging activities in response to real-time grid conditions and energy demand patterns. Extensive simulations are conducted in case studies to testify to the effectiveness of our proposed approach. The overall results validate the efficacy of the DRL-based online V2G scheduling framework, highlighting its potential to drive profitability and sustainability in multi-energy microgrid operations.

An adaptive PID controller for path following of autonomous underwater vehicle based on Soft Actor–Critic

In recent years, autonomous underwater vehicles (AUVs) have witnessed rapid development, and its motion control has garnered increasing attention. Meanwhile, in industries, PID controllers are still wildly employed by most AUVs due to their simplicity, ease of deployment, and a certain level of robustness. However, they are facing significant challenges in parameter tuning, especially when dealing with various control missions and changing external environments. Deep reinforcement learning, as a data-driven approach, has gradually made its impact in AUV control. However, its lack of interpretability has hindered its deployment in relevant experiments. To address these issues, this paper proposed an adaptive PID controller for path following of AUVs based on the Soft Actor–Critic (SAC). This controller combines the interpretability of PID with the intelligence of reinforcement learning. A simulation platform was established and compared with other typical control methods, demonstrating the superiority of the proposed controller. Finally, the feasibility of the proposed SAC-PID controller was validated by lake trials. The results showed that the SAC-PID controller significantly outperformed the PID and Proximal Policy Optimization (PPO) PID controllers in multiple dimensions, such as control precision and convergence speed.

Deep reinforcement learning-based optimal scheduling of integrated energy systems for electricity, heat, and hydrogen storage

The increasing load demands and the extensive usage of renewable energy in integrated energy systems pose a challenge to the most efficient scheduling of integrated energy systems (IES) because of the unpredictability and volatility of both the load side and renewable energy.Integrating heat storage and hydrogen storage technologies into integrated energy systems can effectively alleviate the phenomena of wind and solar power desertion caused by the instability of renewable energy sources. This study introduces a real-time optimal scheduling method based on the Soft Actor-Critic (SAC) algorithm, aimed at reducing system operating costs and enhancing the consumption rate of renewable energy. First, the established mathematical model is converted into a Markov Decision Process (MDP), the objective function is converted by designing the action state space and the reward function, and the deep reinforcement learning framework is established. Finally, the decision-making outcomes of intelligence in various energy storage scenarios of renewable energy consumption and extreme cases are analyzed and compared, and the results show that the heat storage and hydrogen storage system significantly improve the rate of renewable energy consumption and the economy of the system. This method is also shown to significantly improve the rate of renewable energy consumption.

Model-Free Optimal Vibration Control of a Nonlinear System Based on Deep Reinforcement Learning

Optimal control of nonlinear vibration requires precise knowledge of the system and the solution to Hamilton–Jacobi–Bellman (HJB) equation. However, in practical engineering applications, acquiring precise system parameters poses challenges, and the analytical solutions for the HJB equation are difficult to obtain. In this paper, two reinforcement learning algorithms, Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm and Soft Actor-Critic (SAC) algorithm, are employed to train neural network-based optimal controllers for the van der Pol vibration system in the presence of unknown system parameters. To validate their performance, the controllers undergo testing in a series of experiments, including assessments of free vibration, frequency sweep excitation, and Gaussian noise excitation. The results indicate that both the TD3-trained and SAC-trained neural network-based controller are capable of proficiently suppress the vibration of the van der Pol oscillator. Additionally, these two model-free controllers can approximate the optimal control law which solved based on the dynamic model of the nonlinear system.

Social Learning with Actor–Critic for dynamic grasping of underwater robots via digital twins

Dynamic grasping is a crucial technology in the field of underwater robotics, playing an essential role in executing complex tasks. However, traditional control methods encounter challenges in achieving an efficient and stable dynamic grasping process due to the complexities and uncertainties of underwater environments. This paper introduces a novel black-box intelligent optimization algorithm named Social Learning with Actor–Critic (SLAC) for the dynamic grasping of underwater robots. The core architecture of SLAC is based on the integration of two key algorithms: Intelligent Social Learning (ISL) for intelligent optimization and Soft Actor–Critic (SAC) for reinforcement learning. ISL enhances SAC by supplying a larger number of transitions, whereas SAC improves ISL with more effective strategies. These algorithms interact synergistically, augmenting their respective strengths throughout the learning process. To evaluate SLAC’s performance, a comparison is made with six state-of-the-art methods across eight continuous control benchmark cases. The results highlight SLAC’s exceptional learning capability and performance benefits. Furthermore, virtual reality software for the underwater robot and a corresponding digital twin system have been developed. The SLAC algorithm is trained in the digital twin environment before its application in the actual underwater setting. Through interactive training and iterative learning, both simulated and experimental results demonstrate the robot’s proficiency in achieving efficient and stable dynamic grasping, effectively adapting to various underwater environments’ variations and complexities.

Blockchain-Based Zero-Trust Supply Chain Security Integrated with Deep Reinforcement Learning for Inventory Optimization

Modern supply chain systems face significant challenges, including lack of transparency, inefficient inventory management, and vulnerability to disruptions and security threats. Traditional optimization methods often struggle to adapt to the complex and dynamic nature of these systems. This paper presents a novel blockchain-based zero-trust supply chain security framework integrated with deep reinforcement learning (SAC-rainbow) to address these challenges. The SAC-rainbow framework leverages the Soft Actor–Critic (SAC) algorithm with prioritized experience replay for inventory optimization and a blockchain-based zero-trust mechanism for secure supply chain management. The SAC-rainbow algorithm learns adaptive policies under demand uncertainty, while the blockchain architecture ensures secure, transparent, and traceable record-keeping and automated execution of supply chain transactions. An experiment using real-world supply chain data demonstrated the superior performance of the proposed framework in terms of reward maximization, inventory stability, and security metrics. The SAC-rainbow framework offers a promising solution for addressing the challenges of modern supply chains by leveraging blockchain, deep reinforcement learning, and zero-trust security principles. This research paves the way for developing secure, transparent, and efficient supply chain management systems in the face of growing complexity and security risks.

Stochastic Integrated Actor-Critic for Deep Reinforcement Learning.

We propose a deep stochastic actor-critic algorithm with an integrated network architecture and fewer parameters. We address stabilization of the learning procedure via an adaptive objective to the critic's loss and a smaller learning rate for the shared parameters between the actor and the critic. Moreover, we propose a mixed on-off policy exploration strategy to speed up learning. Experiments illustrate that our algorithm reduces the sample complexity by 50%-93% compared with the state-of-the-art deep reinforcement learning (RL) algorithms twin delayed deep deterministic policy gradient (TD3), soft actor-critic (SAC), proximal policy optimization (PPO), advantage actor-critic (A2C), and interpolated policy gradient (IPG) over continuous control tasks LunarLander, BipedalWalker, BipedalWalkerHardCore, Ant, and Minitaur in the OpenAI Gym.

Robust and adaptive deep reinforcement learning for enhancing flow control around a square cylinder with varying Reynolds numbers

The present study applies a Deep Reinforcement Learning (DRL) algorithm to Active Flow Control (AFC) of a two-dimensional flow around a confined square cylinder. Specifically, the Soft Actor-Critic (SAC) algorithm is employed to modulate the flow of a pair of synthetic jets placed on the upper and lower surfaces of the confined squared cylinder in flow configurations characterized by Re of 100, 200, 300, and 400. The investigation starts with an analysis of the baseline flow in the absence of active control. It is observed that at Re = 100 and Re = 200, the vortex shedding exhibits mono-frequency characteristics. Conversely, at Re = 300 and Re = 400, the vortex shedding is dominated by multiple frequencies, which is indicative of more complex flow features. With the application of the SAC algorithm, we demonstrate the capability of DRL-based control in effectively suppressing vortex shedding, while significantly diminishing drag and fluctuations in lift. Quantitatively, the data-driven active control strategy results in a drag reduction of approximately 14.4%, 26.4%, 38.9%, and 47.0% for Re = 100, 200, 300, and 400, respectively. To understand the underlying control mechanism, we also present detailed flow field comparisons, which showcase the adaptability of DRL in devising distinct control strategies tailored to the dynamic conditions at varying Re. These findings substantiate the ability of DRL to control chaotic, multi-frequency dominated vortex shedding phenomena, underscoring the robustness of DRL in complex AFC problems.

Sim-to-Real Application of Reinforcement Learning Agents for Autonomous, Real Vehicle Drifting

Enhancing the safety of passengers by venturing beyond the limits of a human driver is one of the main ideas behind autonomous vehicles. While drifting is mostly witnessed in motorsports as an advanced driving technique, it could provide many possibilities for improving traffic safety by avoiding accidents in extreme traffic situations. The purpose of the research presented in this article is to provide a machine learning-based solution to autonomous drifting as a proof of concept for vehicle control at the limits of handling. To achieve this, reinforcement learning (RL) agents were trained for the task in a MATLAB/Simulink-based simulation environment, using the state-of-the-art Soft Actor–Critic (SAC) algorithm. The trained agents were tested in reality at the ZalaZONE proving ground on a series production sports car with zero-shot transfer. Based on the test results, the simulation environment was improved through domain randomization, until the agent could perform the task both in simulation and in reality on a real test car.

Continuous trading strategy based on deep reinforcement learning

Automatic trading policy has been researched with reinforcement learning (RL). Designing a profitable and applicable policy is of great significance for research in quantitative finance. Incoprating with deep learning, typical deep reinforcement learning (DRL) algorithms such as Proximal Policy Optimization (PPO) have shown their effectiveness. To improve the practical applicability, the manner in which we train our model should better simulate the dynamic of the stock market. A sliding window training strategy is a better solution, which employes training, validation and trading procedures on dataset with a sliding window. However, this empirical strategy can still be further investigated in algorithm evaluation and the experiment designation such as choice of sliding window. In this paper, we further investigated the continuous trading strategy (CTS). We evaluated the performance of a wider range of algorithms, including PPO, Deep Deterministic Policy Gradient (DDPG), Advantage Actor-Critic (A2C), Soft Actor-Critic (SAC). Moreover, we trained our models on a longer term period. We also provide detailed observations and suggestions on experiment settings. These discussions will facilitate researchers in their future work.

A fast balance optimization approach for charging enhancement of lithium-ion battery packs through deep reinforcement learning

This paper presents an innovative strategy that utilizes reinforcement learning to enhance the fast balance charging of lithium-ion battery packs. We develop an interactive framework for lithium-ion batteries by utilizing an electro-thermal coupled model that incorporates hysteresis and temperature impacts. This framework provides precise simulation within a user-friendly gym environment. We propose a novel priority-objective reward function to address the joint challenge of battery pack balancing and fast charging. This reward function is then integrated into the Soft Actor–Critic (SAC) algorithm, delivering an optimal solution within a unified problem framework for the first time. Through a series of comprehensive simulations, we validate the effectiveness of our approach in terms of both performance and computational costs. Our method consistently achieves accelerated charging times, maintains a uniform cell state of charge distribution, and minimizes safety hazards.

Off-policy actor-critic deep reinforcement learning methods for alert prioritization in intrusion detection systems

Alert prioritization plays a very important role in network security as it helps security teams manage and respond to the overwhelming volume of alerts generated by intrusion detection systems (IDSs). Although current deep reinforcement learning (DRL) based deep deterministic policy gradient (DDPG) has achieved good performance for alert prioritization, there are still limitations in dealing with DDPG's instability and limited exploration capabilities. In this paper, we propose TD3-AP and SAC-AP, two DRL empowered off-policy actor-critic alert prioritization methods based on twin delayed deep deterministic policy gradient (TD3) and soft actor-critic (SAC), respectively. We model the interaction between an adversary and a defender as a zero-sum game and use a double oracle framework to obtain mixed strategy Nash equilibrium (MSNE). Our objective is to minimize the defender's loss which represents the inability of the defender to investigate alerts generated due to the attacks. To demonstrate the benefits and scalability of the proposed approaches, we conduct extensive experiments on three datasets: MQTT-IoT-IDS2020, DARPA 2000 LLDOS 1.0 and CSE-CIC-IDS2018. The results depict that TD3-AP and SAC-AP exhibit a decrease in the defender's loss by 50% and 14.28%, respectively, in comparison to the alert prioritization method based on DDPG. The improvements are significantly higher when the proposed approaches are compared with traditional alert prioritization methods including Uniform, Snort, Fuzzy and RAP. Additionally, we also provide an analysis of the interpretability of our results through the utilization of SHapley Additive exPlanations (SHAP). We assess the sensitivity of our proposed approaches to different hyperparameters and evaluate the training time and computational resources necessary for their implementation.

Soft Actor-Critic and Risk Assessment-Based Reinforcement Learning Method for Ship Path Planning

Ship path planning is one of the most important themes in waterway transportation, which is deemed as the cleanest mode of transportation due to its environmentally friendly and energy-efficient nature. A path-planning method that combines the soft actor-critic (SAC) and navigation risk assessment is proposed to address ship path planning in complex water environments. Specifically, a continuous environment model is established based on the Markov decision process (MDP), which considers the characteristics of the ship path-planning problem. To enhance the algorithm’s performance, an information detection strategy for restricted navigation areas is employed to improve state space, converting absolute bearing into relative bearing. Additionally, a risk penalty based on the navigation risk assessment model is introduced to ensure path safety while imposing potential energy rewards regarding navigation distance and turning angle. Finally, experimental results obtained from a navigation simulation environment verify the robustness of the proposed method. The results also demonstrate that the proposed algorithm achieves a smaller path length and sum of turning angles with safety and fuel economy improvement compared with traditional methods such as RRT (rapidly exploring random tree) and DQN (deep Q-network).

Safe Hybrid-Action Reinforcement Learning-Based Decision and Control for Discretionary Lane Change

Autonomous lane-change, a key feature of advanced driver-assistance systems, can enhance traffic efficiency and reduce the incidence of accidents. However, safe driving of autonomous vehicles remains challenging in complex environments. How to perform safe and appropriate lane change is a popular topic of research in the field of autonomous driving. Currently, few papers consider the safety of reinforcement learning in discretionary lane-change scenarios. We introduce safe hybrid-action reinforcement learning into discretionary lane change for the first time and propose the Parameterized Soft Actor–Critic with PID Lagrangian (PASAC-PIDLag) algorithm. Furthermore, we conduct a comparative analysis with Parameterized Soft Actor–Critic (PASAC), which is an unsafe version of PASAC-PIDLag. Both algorithms are employed to train the lane-change strategy to output both discrete lane-change decisions and continuous longitudinal vehicle acceleration. Our simulation results indicate that at a traffic density of 15 vehicles per kilometer (15 veh/km), the PASAC-PIDLag algorithm exhibits superior safety with a collision rate of 0%, outperforming the PASAC algorithm, which has a collision rate of 1%. The generalization assessments reveal that at low traffic density levels, both the PASAC-PIDLag and PASAC algorithms are proficient in attaining zero collision rates. However, at high traffic density levels, although both algorithms result in collisions, PASAC-PIDLag has a much lower collision rate than PASAC.