Soft actor-critic algorithm with adaptive normalization,

Accurately measuring cable length in cable-driven continuum robots is challenging due to cable elongation from elasticity and friction in the transmission mechanisms. Due to above reasons, traditional kinematics-based control methods hard to achieve precise control for cable driven robots. This paper proposes a model-free kinematic control method using the Soft Actor-Critic (SAC) algorithm. PyBullet based software has been used to build simulation environment, and defines the mapping between the robot’s state space and cable space as the agent’s learning objective. The SAC algorithm autonomously learns the optimal policy through interaction with the environment. By introducing a dual-critic network, weight clipping, and a specifically designed reward, we achieved the cable driven robot non-model control. Experimental results show that compared with traditional model-based control designed by our previously designed robot, the SAC algorithm is able to improves the end-effector position tracking accuracy by 12%, and shown excellent anti-interference capability and adaptive learning characteristics.

An adaptive proportional integral derivative (PID) controller based on the soft actor-critic (SAC) algorithm for trajectory control of unmanned surface vehicles (USV) is proposed in this paper. The gains of the PID controller need to be manually adjusted based on experience in the original formulation. Furthermore, once tuned, these gains remain fixed and making further modifications becomes time-consuming and labor-intensive. To address these limitations, the SAC algorithm is introduced, enabling online tuning of PID gains through agent-environment interaction. Additionally, the strategy of combining SAC algorithm with PID controller mitigates concerns regarding interpretability and security often associated with DRL. In this study, stability analysis of the adaptive trajectory controller based on the SAC-PID algorithm is conducted. This paper horizontally compares the proposed method with traditional PID tuning methods, genetic algorithms (GA), and deep deterministic policy gradient (DDPG) algorithm to highlight the superiority of the SAC-PID approach. Finally, experiments in different scenarios are performed to compare generalization capabilities between DDPG and SAC algorithms. Results demonstrate that the proposed SAC-PID algorithm exhibits excellent stability properties, fast convergence speed, and strong generalization ability.

Aiming at the frequency instability and the increase of control cost caused by the uncertainty of the island microgrid, this paper proposes a multi-objective control strategy for the island microgrid based on the Soft Actor Critic (SAC) algorithm. Firstly, considering the influence of the output distribution of micro gas turbines on the control cost, a multi-objective control model that coordinates control cost and frequency stability is constructed. Secondly, in order to cope with the sparse reward task of multi-objective control, this paper takes the frequency deviation and control cost as the state set, takes the unit output as the action set, and designs the corresponding reward function, thus completing the process design of the multi-objective control strategy. Finally, the calculation example shows that the SAC controller can effectively reduce the control cost on the premise of ensuring the frequency regulation requirements of the microgrid, and the comprehensive control effect is far superior to traditional control methods such as PI control and fuzzy control.

Great attention has been attracted to the growing energy consumption of the subway system. It is significant to investigate the energy-efficient train control methodology which is regarded as an effective way to cut down the energy cost of the subway system. In this paper, a soft actor-critic-based method is proposed to determine the optimal driving strategy. Firstly, the inverse problem of the energy-efficient train control problem, i.e., minimizing the trip time with constant traction energy, is presented, and it is converted to a finite Markov decision process which can be solved with deep reinforcement learning (DRL) algorithms. Secondly, the soft actor-critic (SAC) algorithm is proposed to determine the optimal driving strategy. Finally, the effectiveness of the proposed method is verified via the case studies. It is also illustrated that the SAC-based method has a good performance in robustness and stability. Moreover, compared with other DRL algorithms such as the deep Q network (DQN), the SAC-based method saves about 66% training steps on average to achieve the optimum, which reduces the computational resources.

Shop scheduling is deeply involved in manufacturing. In order to improve the efficiency of scheduling and fit dynamic scenarios, many Deep Reinforcement Learning (DRL) methods are studied to solve scheduling problems like job shop and flow shop. But most studies focus on using the latest algorithms while ignoring that the environment plays an important role in agent learning. In this paper, we design an effective, robust and size-agnostic environment for job shop scheduling. The proposed design of environment uses centralised training and decentralised execution (CTDE) to implement a multi-agent architecture. Together with the observation space we design, environmental information that is irrelevant to the current decision is eliminated as much as possible. The proposed action space enlarges the decision space of agents, which performs better than the traditional way. Finally, Soft Actor-Critic (SAC) algorithm is adapted to learning within this environment. By comparing with traditional scheduling rules, other reinforcement learning algorithms, and relevant literature, the superiority of the results obtained in this study is demonstrated.

The eco-driving strategy based on deep reinforcement learning holds significant potential for achieving energy efficiency, longevity, and safety of fuel cell hybrid electric vehicle (FCHEV). This paper proposes a health-aware eco-driving strategy for FCHEV based on the soft actor-critic (SAC) algorithm. Building upon health awareness of multiple electric components including power battery, fuel cell, and driving motor, this eco-driving strategy integrates energy management system (EMS) and adaptive cruise control (ACC) to comprehensively optimize vehicle performance. By incorporating health-awareness into the eco-driving approach, this study aims to maximize the lifespan of electric components, enhance energy utilization efficiency, and ensure driving comfort and safety. SAC algorithm not only enhances optimization performance in complex nonlinear multi-objective optimization problems but also accommodates real-time control requirements under diverse driving conditions. The simulation results demonstrate that the proposed strategy achieves 0.41% reduction in H2 consumption and same level health maintenance of electric components compared with the dynamic programming benchmark of EMS, while maintaining the comfort within 6% of the gap but safer following performance compared with the intelligent driver model benchmark of ACC. Moreover, the comparative experiments demonstrate that the effectiveness and adaptability of proposed eco driving strategy.

Although Tor is designed to provide anonymity, website fingerprinting (WF) attacks have posed significant threats to user privacy. In response, various defense approaches have been developed. Randomization and regularization-based defenses are criticized to be inefficient due to their bandwidth-consuming nature. Some adversarial learning-based defenses are impractical because the generation of perturbation depends on the complete traffic traces. Other adversarial learning-based defenses have weaknesses of lacking adaptability because their perturbations are input-agnostic. To overcome these shortcomings, we propose RUDOLF, an efficient and adaptive WF defense based on the soft actor-critic (SAC) algorithm of reinforcement learning (RL). We train the agent that can incrementally output perturbations synchronously following each burst of real-time traffic. Different from previous defenses, RUDOLF’s perturbation does not depend on the integrity of the traffic and concerns the actual real-time traffic, which ensures the practicality of implementation and adaptability. Besides, we take advantage of the exploratory characteristics of the SAC algorithm to obtain the optimal policy of adding perturbations that can efficiently balance defense effects and bandwidth consumption. Experiments on synthetic datasets show that with less than 30% bandwidth overhead (BWO), RUDOLF can reduce the average attack accuracy to around 15%–20%, which is superior to previous works. We also have implemented RUDOLF as a Tor pluggable transport. The performance in the real Tor network shows that RUDOLF can reduce the average accuracy of WF classifier to around 24% with about 25% BWO and almost no time delay.

To address bus voltage fluctuations and state-of-charge (SOC) constraint violations in hybrid electric-hydrogen energy storage systems (HEH-ESS) for islanded DC microgrids, this paper proposes an intelligent control strategy based on the Soft Actor-Critic (SAC) algorithm. The objective is to achieve long-term system stability and optimized energy scheduling. First, a detailed DC microgrid model is developed, incorporating photovoltaic (PV) panels, fuel cells, electrolyzers, hydrogen storage tanks, lithiumion batteries, and DC loads. Second, a hierarchical control architecture is designed. The primary control layer utilizes droop control and Virtual Synchronous Generator (VSG) techniques to enable grid-forming capabilities. The secondary control layer implements an SAC-based reinforcement learning strategy, modeled as a Markov Decision Process (MDP), to optimize global power distribution. To enable coordinated voltage stabilization and SOC management, a dynamic, weighted single-objective reward function is formulated. This reward function integrates system constraints such as power limits, storage capacity, bus voltage range, and ramp rate, enhancing both the convergence speed and robustness of the control strategy. Simulation results validate the effectiveness of the proposed method. The SAC-based control significantly improves voltage stability, reduces the risks of battery overcharging and overdischarging, and enhances the system's adaptability to complex load disturbances across diverse operational scenarios.

To address the challenges of coordinated defense amid dynamic topology evolution and multidimensional security threats in decomposed optical networks, this study introduces the Graph-Entangled Security Actor-Critic (GESAC) model. GESAC is built on spatiotemporal modeling of evolving topologies and leverages a cross-layer spatiotemporal Graph Neural Network (GNN) to capture causal dependencies between optical path switching and access requests. Additionally, it enables adaptive delineation of security boundaries across multiple domains through federated representation learning. Within this framework, the Soft Actor-Critic (SAC) algorithm is employed to construct a policy optimization mechanism. By integrating entropy-guided multi-objective reinforcement learning, GESAC maps encoded network states to access control strategies, jointly optimizing for security, service quality, and system resilience. Experimental validation is conducted on a heterogeneous dataset comprising Cooperative Association for Internet Data Analysis (CAIDA) topology data, Canadian Institute for Cybersecurity Intrusion Detection Systems (CIC-IDS) access logs, and International Telecommunication Union Telecommunication Standardization Sector threat characteristics. The dataset encompasses 12 attack scenarios, 57,000 dynamic topology sequences, and 2.8 million cross-domain authentication events. Key findings include: (1) Threat Detection: GESAC achieves an F1-score of 0.915–0.931 in identifying physical-layer attacks such as wavelength eavesdropping and cross-domain privilege escalation, with a false positive rate as low as 0.7%. (2) Resource Optimization: Compared to greedy strategies, GESAC improves wavelength utilization variance by up to 58.9% and reduces end-to-end latency standard deviation by up to 57.7% under high-load conditions. (3) Policy Robustness: In scenarios involving topological mutations, the model increases Pareto frontier coverage by over 100% and reduces policy entropy decay rate by more than 65%, indicating strong robustness. (4) Scalability: At a scale of 100,000 network nodes, GESAC achieves a single-step decision latency of just 25.6µs and significantly reduces communication overhead, demonstrating excellent scalability. GESAC is designed to overcome the limitations of static security policies in the face of dynamic decomposition and large-scale attacks in optical networks. Integrating causal inference with game-theoretic equilibrium redefines the security control paradigm—shifting from passive defense to proactive resilience—and provides an interpretable, highly adaptive foundation for next-generation architectures such as multi-domain collaboration and computing-network convergence.Supplementary InformationThe online version contains supplementary material available at 10.1038/s41598-025-15225-z.

Hybrid vehicles offer significant advantages in range and power due to their multiple operating modes. This paper proposes a configuration for an electro-hydraulic hybrid vehicle. However, due to the frequent mode switching of its power sources, a complex longitudinal impact control issue arises. To address this problem, an optimization algorithm is introduced to reduce the impact during vehicle mode switching. To enhance the generalization ability of the algorithm, the principal component-K-means method is employed to extract key features. A control management system framework for hybrid vehicles is constructed by integrating rule-based control with a multi-condition model trained using the soft actor-critic (SAC) algorithm. To validate the superiority of the SAC algorithm, five deep reinforcement learning algorithms, Trust Region Policy Optimization, SAC, Proximal Policy Optimization, Deep Deterministic Policy Gradient, and Twin Delayed Deep Deterministic Policy Gradient, were compared and trained. The results show that the optimization algorithm reduces longitudinal impact while ensuring optimal energy distribution. This leads to an average reduction of 70.56% in the energy decay rate across various typical operating conditions. Meanwhile, the average energy consumption efficiency increased by 8.9%. Furthermore, real driving data confirms the robustness and superior performance of this strategy in practical applications.

This study aims to explore the enterprise fission path optimization strategy based on the Soft Actor-Critic (SAC) algorithm and analyze its impact on the parent company and the overall operational efficiency. Firstly, the enterprise finance and marketing data provided by the National Bureau of Statistics public dataset are used for data pre-processing. Secondly, a multi-level reward function is designed that covers short-term financial and market indices. Meanwhile, it incorporates long-term indices that measure dynamic capabilities, such as innovation, market agility, and resource integration. Finally, by introducing the reinforcement learning algorithm of SAC, the enterprise fission scenario is constructed into a complicated decision environment, in which the state space includes the current financial situation, market performance, and dynamic capability level of the enterprise. The action space encompasses various strategic choices of enterprise fission to simulate the enterprise fission decision process. The SAC algorithm’s entropy regularization feature prompts the model to strike a balance between exploration and utilization to optimize the dynamic capability construction. The experimental results show that the fission path optimized by deep reinforcement learning (DRL) markedly improves the resource allocation efficiency and market response speed by an average of 20.4% and 25.2%, respectively. More importantly, dynamic capability construction has been significantly enhanced, with the innovation capability index increasing by 15.4%, market agility improving by 12.3%, and resource integration capability also enhancing by 10.5%. This indicates that the strategy can help accelerate the formation of industrial clusters. Therefore, the SAC algorithm-based enterprise fission path optimization strategy constructed in this study can bring lasting competitive advantages to enterprises.

In automated factories, dynamic obstacle avoidance and trajectory planning of robotic manipulators are critical to achieving safe and efficient operations. However, traditional obstacle avoidance methods, such as the artificial potential field, element decomposition, viewable, Voronoi diagram, and probabilistic road map, face many challenges in dealing with complex dynamic environments, such as target unreachable and local minimum problems. A new dynamic obstacle avoidance and trajectory planning framework based on the Soft Actor-Critic (SAC) algorithm is proposed in this paper to address these problems. The framework combines the fast-scaling random tree (RRT) algorithm for global path planning and the SAC algorithm to optimize the local path to adapt to the changes in the dynamic environment. Specifically, the simulation uses Python to construct a URDF (Unified Robot Description Format) model of an open-source robot arm. It applies the SAC algorithm to the model's dynamic obstacle avoidance trajectory planning. The simulation results show that the proposed framework combining RRT and SAC algorithms achieved a high success rate in reaching the target point. This method can effectively find the right trajectory in a complex dynamic environment.

Efficient traffic signal control plays a critical role in reducing urban congestion and improving transportation efficiency. This paper presents an adaptive traffic signal control approach based on reinforcement learning that employs the Soft Actor-Critic (SAC) algorithm to optimize traffic signal phase durations. By focusing on phase duration optimization rather than discrete phase selection, the proposed method ensures a more predictable and adaptive traffic management system. Unlike traditional methods that select discrete signal phases, our approach continuously adjusts phase duration based on real-time traffic conditions, providing a more flexible and responsive control strategy. The proposed framework uses connected vehicle data, including position, speed, and acceleration, to predict vehicle arrival times at intersections. These predictions, combined with aggregated traffic characteristics such as queue length and waiting time, form the state representation for the reinforcement learning model. The SAC algorithm is then used to determine optimal phase durations. We evaluated the proposed approach using the SUMO traffic simulator in three different urban scenarios: a single intersection, a three-intersection arterial road, and a small road network. Experimental results demonstrate that the proposed method outperforms baseline approaches, including Deep Q-Network and a heuristic-based method, in terms of average travel time, time loss, and waiting time. Specifically, the SAC-based algorithm achieves reductions of up to 1.5\% in average travel time and up to 13\% in average waiting time across various simulation scenarios compared to the baseline methods. Furthermore, training convergence analysis and visualizations confirm the stability and effectiveness of the learned policy.

Due to the excessive degree of freedom of the space flexible manipulator, we can hardly obtain its accurate dynamic model for its motion planning. In this work, we formulate the precise motion control of the free-floating space piecewise constant curvature (FSPCC) continuum manipulator (i.e. space flexible manipulator) as a sparse reward problem in reinforcement learning, and use the Soft Actor-Critic (SAC) algorithm along with the Random Network Distillation (RND) method to train the optimal policies. Firstly, we use the RND method to jointly train a predictive network and a fixed network. The discrepancy between the output values of the two networks is served as an internal reward for the environment. Secondly, the SAC algorithm aims to maximize the expected return and the entropy of the policy. Policies with high entropy will successfully complete the task while acting as randomly as possible. Finally, the internal rewards tend to incentivize the agent to explore more widely for faster convergence of the algorithm. We applied this method to the FSPCC continuum manipulator simulation model and the results demonstrate that the SAC algorithm together with RND method can control the FSPCC continuum manipulator to catch the target quickly, even in the presence of sparse reward.

The complexity of dynamics raises challenges in deformable object manipulation, particularly the soft tissue shape control in in vivo environment. Previous studies assume the soft tissue is in an obstacle-free space, and the constraints are constant. To go one step further, a Soft Actor-Critic (SAC) algorithm is presented to manipulate the soft tissue under external disturbances while avoiding obstacles. K-Means is applied to reduce the dimension of the state space. Furthermore, self-attention is introduced to focus on the external forces and the constraints. Experiment results show that the agent can shape the soft tissue into the target deformation with the proposed approach under unknown external force, where the agent can automatically increase attention to the external force. The agent can find an obstacle-free path while manipulating the soft tissue. Compared with the SAC algorithm, the proposed approach with self-attention requires less training time, and the critic network is more stable.