Actor-critic Algorithm Research Articles

Research on adaptive circuit structure optimization in electronic design based on Asynchronous Advantage Actor Critic (A3C) algorithm

The circuit structure optimizationed with the traditional method is often difficult to meet the complex and changeable design requirements. In this paper the A3C algorithm has been applied to integrate strategy learning and value learning for the circuit structure optimization. This integration can facilitate continuous interaction with the environment, enabling automatic adjustment of circuit structures to meet the complex design requirements. Gain, bandwidth, latency, and power consumption have been set as the optimization objectives, and the actions of the intelligent agent, which include adding, deleting, modifying connection lines, and adjusting component parameters have been introduced in detail. Once the Actor and Critic networks have been established, multiple agents can operate concurrently, translating optimization objectives into reward signals and providing direction and motivation for agent learning. Then with the proposed method, the circuit structure of one switch audio power amplifier has been designed in a simulation environment. The structure optimization results demonstrated that the gain can reach 78.4 dB at convergence of the A3C algorithm, while the bandwidth can reach 156.2 MHz at convergence, and both the circuit delay and power consumption have been reduced significantly. Obviously, the application of the A3C algorithm can effectively optimize the circuit structures through offering more flexible and efficient solutions.

Optimization of emergency frequency control strategy for power systems considering both source and load uncertainties

With the increasing integration of renewable energy sources and the presence of numerous controllable loads such as electric vehicles and energy storage in the modern power system, higher nonlinearities and uncertainty both sources and loads are introduced. These factors pose challenges in achieving fast and accurate emergency frequency control. Therefore, this paper addresses the issue of dual source-load uncertainties in power system and presents an optimization strategy based on the Soft Actor Critic (SAC) algorithm that involves the participation of controllable loads in emergency frequency control. Firstly, the spatio-temporal uncertainties of wind farm power output on power supply side and power demand on the load side are described using Weibull and normal probability distributions, respectively. Secondly, an improved Markov Decision Process (MDP) model for emergency frequency control is established, which considers the characteristics of the dual source-load uncertainties. Finally, an optimization of the SAC algorithm is conducted based on Deep Reinforcement Learning (DRL), aiming to achieve rapid system frequency recovery and minimize the cost of removing controllable loads. The presented approach in the paper enhances the emergency frequency control strategy for uncertain power systems and effectively addresses the issue of source-load uncertainty compounded by fault power shortages.

A smart home energy management system based on human activity recognition and deep reinforcement learning

Smart home energy management systems (SHEMS) can help users save on energy costs by controlling various household loads in response to environmental changes. However, human activity is closely related to household energy consumption, which can lead to potential energy waste. Additionally, smart homes often feature multiple types of distributed energy resources (DER), and how to leverage their potential for participating in grid scheduling without significantly impacting users remains an unresolved issue. This paper proposes an intelligent home energy management method based on human activity recognition (HAR) and deep reinforcement learning. By using activity recognition results derived from intrusive load monitoring (ILM), we categorize contexts into two groups. The agent employs the soft actor-critic (SAC) algorithm for real-time control. In the first context, the goal is to ensure user satisfaction while reducing energy costs. In the second context, the focus is on controlling DER to participate in grid ancillary services to generate revenue. The proposed agent is trained using a real dataset, and simulations validate the effectiveness of the algorithm in home energy management.

Empowering legal justice with AI: A reinforcement learning SAC-VAE framework for advanced legal text summarization.

Automated summarization of legal texts poses a significant challenge due to the complex and specialized nature of legal documentation. Despite the recent progress in reinforcement learning for natural language text summarization, its application in the legal domain has been less effective. This paper introduces SAC-VAE, a novel reinforcement learning framework specifically designed for legal text summarization. We leverage a Variational Autoencoder (VAE) to condense the high-dimensional state space into a more manageable lower-dimensional feature space. These compressed features are subsequently utilized by the Soft Actor-Critic (SAC) algorithm for policy learning, facilitating the automated generation of summaries from legal texts. Through comprehensive experimentation, we have empirically demonstrated the effectiveness and superior performance of the SAC-VAE framework in legal text summarization.

A study of electricity sales offer strategies applicable to the participation of multi-energy generators in short- and medium-term markets

Due to the increasing proportion of renewable energy, a multi-layered and multi-timescale energy market has emerged in many countries such as China. In the meanwhile, power generation companies must develop more intelligent and dynamic offer strategies to adapt to today's intricate energy trading. Because of the difficulty in describing the dynamic trading environment caused by the uncertainty of renewable energy, previous studies have not fully explored the offer strategy especially in both short-term and medium-term electricity markets. In response to this challenge, this research introduces a novel biding strategy framework leveraging a Asynchronous Advantage Actor-Critic (A3C) algorithm, which can effectively address the decision making in dynamic and uncertain energy markets. The framework focuses on intra-monthly transaction clearing mechanisms with the aim of optimally enhancing earnings. The research formulates an offer model both for thermal and renewable power generation enterprises, which is applicable to medium-term monthly and intra-monthly trading. The study then validates this framework through three distinct analyses: the returns of various bid methods under standard scenarios, the offer strategies return of power generation companies with diverse cost profiles, and the impact of varying renewable energy proportions. The multi-angle simulations confirm that the model presented in this paper offers a scientific basis for the development of offer strategies for power generation companies and enable power generating firms to effectively adopt to the current power market.

A hierarchical deep reinforcement learning method for solving urban route planning problems under large-scale customers and real-time traffic conditions

As urbanization and economic growth advance, large-scale customers and real-time traffic conditions have become crucial factors in urban route planning. Deep reinforcement learning is considered the most effective method for solving urban route planning problems involving large-scale customers and real-time traffic conditions. Due to memory usage limitations, existing deep reinforcement learning methods cannot identify candidate customers or determine optimal travel routes in large-scale and real-time environments. To tackle these problems, this study introduces a hierarchical deep reinforcement learning method utilizing an improved transformer model (HDRLITF) based on the divide-and-conquer concept. Graph attention networks and gate mechanisms are integrated into the transformer model to capture dynamic features and improve the model’s performance. A two-stage training method, based on the actor-critic algorithm, is proposed to determine the optimal policy function. To evaluate the HDRLITF method, experiments were conducted using datasets from the cities of Shenzhen and Chengdu in China. The experimental results suggest that the HDRLITF method can effectively interact with real-time traffic environments and obtain high-quality solutions compared to other deep reinforcement learning methods. The robustness and reliability of HDRLITF were further validated across multiple traffic scenarios and indicators.

Research on robust decision making for intelligent connected vehicle at highway on-ramp

Reinforcement learning has demonstrated its potential in the decision-making field of autonomous driving. By engaging in trial-and-error learning and interacting with the environment, we can adapt our behavioral strategies based on reward and enhance driving decisions. Nevertheless, real-world vehicle decision-making involves intricate and diverse information, and the limited state information hinders agents from making optimal decisions, which may result in catastrophic consequences like collisions. Therefore, this paper proposes a robust deep reinforcement learning method based on the Soft Actor-Critic (SAC) algorithm for highway intelligent connected vehicle ramp merging decision-making. The model represents the vehicle features of the target lane and its adjacent lanes as an environmental state space. Additionally, a hybrid action space is designed, which combines discrete lateral actions and continuous longitudinal actions. Finally, an on-ramp simulation platform is built using actual roads and SUMO to verify the feasibility of the model. Multiple sets of comparative analysis experimental results demonstrate that the proposed method outperforms others in terms of the average reward return value, robustness value, collision rate, and success rate. Moreover, the model exhibits a stable lane change success rate under various road traffic density conditions, which indicates its robustness. The code can be obtained at https://github.com/ColinFanghz/sac-on-ramp.git .

Optimization of geological carbon storage operations with multimodal latent dynamic model and deep reinforcement learning

Identifying the time-varying control schemes that maximize storage performance is critical to the commercial deployment of geological carbon storage (GCS) projects. However, the optimization process typically demands extensive resource-intensive simulation evaluations, which poses significant computational challenges and practical limitations. In this study, we presented the multimodal latent dynamic (MLD) model, a novel deep learning framework for fast flow prediction and well control optimization in GCS operations. The MLD model implicitly characterizes the forward compositional simulation process through three components: a representation module that learns compressed latent representations of the system, a transition module that approximates the evolution of the system states in the low-dimensional latent space, and a prediction module that forecasts the flow responses for given well controls. A novel model training strategy combining a regression loss and a joint-embedding consistency loss was introduced to jointly optimize the three modules, which enhances the temporal consistency of the learned representations and ensures multi-step prediction accuracy. Unlike most existing deep learning models designed for systems with specific parameters, the MLD model supports arbitrary input modalities, thereby enabling comprehensive consideration of interactions between diverse types of data, including dynamic state variables, static spatial system parameters, rock and fluid properties, as well as external well settings. Since the MLD model mirrors the structure of a Markov decision process (MDP) that computes state transitions and rewards (i.e., economic calculation for flow responses) for given states and actions, it can serve as an interactive environment to train deep reinforcement learning agents. Specifically, the soft actor-critic (SAC) algorithm was employed to learn an optimal control policy that maximizes the net present value (NPV) from the experiences gained by continuous interactions with the MLD model. The efficacy of the proposed approach was first compared against commonly used simulation-based evolutionary algorithm and surrogate-assisted evolutionary algorithm on a deterministic GCS optimization case, showing that the proposed approach achieves the highest NPV, while reducing the required computational resources by more than 60%. The framework was further applied to the generalizable GCS optimization case. The results indicate that the trained agent is capable of harnessing the knowledge learned from previous scenarios to provide improved decisions for newly encountered scenarios, demonstrating promising generalization performance.

Knowledge-Assisted Actor Critic Proximal Policy Optimization-Based Service Function Chain Reconfiguration Algorithm for 6G IoT Scenario.

Future 6G networks will inherit and develop Network Function Virtualization (NFV) architecture. With the NFV-enabled network architecture, it becomes possible to establish different virtual networks within the same infrastructure, create different Virtual Network Functions (VNFs) in different virtual networks, and form Service Function Chains (SFCs) that meet different service requirements through the orderly combination of VNFs. These SFCs can be deployed to physical entities as needed to provide network functions that support different services. To meet the highly dynamic service requirements in the future 6G Internet of Things (IoT) scenario, the highly flexible and efficient SFC reconfiguration algorithm is the key research direction. Deep-learning-based algorithms have shown their advantages in solving this type of dynamic optimization problem. Considering that the efficiency of the traditional Actor Critic (AC) algorithm is limited, the policy does not directly participate in the value function update. In this paper, we use the Proximal Policy Optimization (PPO) clip function to restrict the difference between the new policy and the old policy, to ensure the stability of the updating process. We combine PPO with AC, and further bring the historical decision information as the network knowledge to offer better initial policies, to accelerate the training speed. We also propose the Knowledge = Assisted Actor Critic Proximal Policy Optimization (KA-ACPPO)-based SFC reconfiguration algorithm to ensure the Quality of Service (QoS) of end-to-end services. Simulation results show that the proposed KA-ACPPO algorithm can effectively reduce computing cost and power consumption.

Enhancing the Minimum Awareness Failure Distance in V2X Communications: A Deep Reinforcement Learning Approach.

Vehicle-to-everything (V2X) communication is pivotal in enhancing cooperative awareness in vehicular networks. Typically, awareness is viewed as a vehicle's ability to perceive and share real-time kinematic information. We present a novel definition of awareness in V2X communications, conceptualizing it as a multi-faceted concept involving vehicle detection, tracking, and maintaining their safety distances. To enhance this awareness, we propose a deep reinforcement learning framework for the joint control of beacon rate and transmit power (DRL-JCBRTP). Our DRL-JCBRTP framework integrates LSTM-based actor networks and MLP-based critic networks within the Soft Actor-Critic (SAC) algorithm to effectively learn optimal policies. Leveraging local state information, the DRL-JCBRTP scheme uses an innovative reward function to increase the minimum awareness failure distance. Our SLMLab-Gym-VEINS simulations show that the DRL-JCBRTP scheme outperforms existing beaconing schemes in minimizing awareness failure probability and maximizing awareness distance, ultimately improving driving safety.

Efficient deep reinforcement learning based task scheduler in multi cloud environment

Task scheduling problem (TSP) is huge challenge in cloud computing paradigm as number of tasks comes to cloud application platform vary from time to time and all the tasks consists of variable length, runtime capacities. All these tasks may generated from various heterogeneous resources which comes onto cloud console directly effects the performance of cloud paradigm with increase in makespan, energy consumption, resource costs. Traditional task scheduling algorithms cannot handle these type of complex workloads in cloud paradigm. Many authors developed Task Scheduling algorithms by using metaheuristic techniques, hybrid approaches but all these algorithms give near optimal solutions but still TSP is a highly challenging and dynamic scenario as it resembles NP hard problem. Therefore, to tackle the TSP in cloud computing paradigm and schedule the tasks in an effective way in cloud paradigm, we formulated Adaptive Task scheduler which segments all the tasks comes to cloud console as sub tasks and fed these to the scheduler which is modeled by Improved Asynchronous Advantage Actor Critic Algorithm(IA3C) to generate schedules. This scheduling process is carried out in two stages. In first stage, all incoming tasks are segmented as sub tasks. After segmentation, all these sub tasks according to their size, execution time, communication time are grouped together and fed to the (ATSIA3C) scheduler. In the second stage, it checks for the above said constraints and disperse them onto the corresponding suitable processing capacity VMs resided in datacenters. Proposed ATSIA3C is simulated on Cloudsim. Extensive simulations are conducted using both fabricated worklogs and as well as realtime supercomputing worklogs. Our proposed mechanism evaluated over baseline algorithms i.e. RATS-HM, AINN-BPSO, MOABCQ. From results it is evident that our proposed ATSIA3C outperforms existing task schedulers by improving makespan by 70.49%. Resource cost is improved by 77.42%. Energy Consumption is improved over compared algorithms 74.24% in multi cloud environment by proposed ATSIA3C.

Advancements in UAV Path Planning: A Deep Reinforcement Learning Approach with Soft Actor-Critic for Enhanced Navigation

This paper tackles the intricate challenge of autonomous navigation for Unmanned Aerial Vehicles (UAVs) through dynamically changing environments. We focus on a sophisticated Deep Reinforcement Learning (DRL) approach using the Soft Actor-Critic (SAC) algorithm, optimized for UAV path planning within a continuous action space. This methodology leverages environmental image data to enhance the precision of flight maneuvers and effective obstacle avoidance. Our approach, validated through extensive simulations in Gazebo and field tests, demonstrates the algorithm’s efficacy in enabling UAVs to adeptly navigate obstacles using depth maps. The study further explores the robustness of the SAC algorithm by comparing it with traditional DRL methods, emphasizing its superior performance in real-world applications. This research contributes significantly to advancing UAV technology, particularly in autonomous motion planning, by integrating cutting-edge machine learning techniques. The video link is: https://www.youtube.com/watch?v=Nd_aMzejNXY .

Smart energy management for hybrid electric bus via improved soft actor-critic algorithm in a heuristic learning framework

Deep reinforcement learning (DRL) is currently the cutting-edge artificial intelligence approach in the field of energy management for hybrid electric vehicles. However, inefficient offline training limits the energy-saving efficacy of DRL-based energy management strategies (EMSs). Motivated by this, this article proposes a smart DRL-based EMS in a heuristic learning framework for an urban hybrid electric bus. In order to enhance the sampling efficiency, the prioritized experience replay technique is introduced into soft actor-critic (SAC) for the innovative formulation of an improved SAC algorithm. Additionally, to strengthen the generalizability of the improved SAC agent to real driving scenarios, a stochastic training environment is constructed. Afterward, curriculum learning is employed to develop a heuristic learning framework that expedites convergence. Experimental simulations reveal that the designed EMS expedites convergence by 85.58 % and saves fuel by 6.43 % compared with the cutting-edge baseline EMS. Moreover, the computation complexity test demonstrates that the designed EMS holds significant promise for real-time implementation. These findings highlight the contribution of this article in facilitating fuel conservation for urban hybrid electric buses through the application of emerging artificial intelligence technologies.

URLLC-aware and energy-efficient data offloading strategy in high-mobility vehicular mobile edge computing environments

The integration of Internet of Things (IoT) technologies into the vehicular industry has initiated a new era of connected and autonomous vehicles, revolutionizing transportation systems. However, this transformation introduces significant challenges, especially in 5 G networks, such as achieving Ultra-Reliable Low-Latency Communications (URLLC) and maintaining energy efficiency within the high mobility of vehicular environments. These are essential for supporting sustainable and environmentally friendly computing practices. To address these challenges, this paper introduces a URLLC-aware and energy-efficient data offloading strategy, utilizing the Asynchronous Advantage Actor-Critic (A3C) algorithm to navigate the complex dynamics of vehicular Mobile Edge Computing (MEC) environments. Our proposed method balances latency and energy consumption trade-offs while ensuring robust communication reliability. Technical evaluations reveal that our approach significantly outperforms other algorithms, achieving up to 8.2 % energy savings and a reduction of over 29 % in latency.

Trajectory tracking control based on deep reinforcement learning and ensemble random network distillation for robotic manipulator

Abstract In general, the trajectory tracking of robotic manipulator is exceptionally challenging due to the complex and strongly coupled mechanical architecture. In this paper, precise track control of the robotic manipulator is formulated as a dense reward problem for reinforcement learning(RL). A deep RL(DRL) approach combining the soft actor-critic (SAC) algorithm and ensemble random network distillation (ERND) is proposed to address the tracking control problem for robotic manipulator. Firstly, an ERND model is designed, consisting of a module list of multiple RND models. Each RND model obtains the error by learning the target features and the predicted features of the environment. The resulting error serves as internal rewards that drive the robotic agent to explore unknown and unpredictable environmental states. The ensemble model obtains the total internal reward by summing the internal rewards of each RND model, thereby obtaining more accurately reflecting the characteristics of the manipulator in tracking control tasks and improving control performance. Secondly, combining the SAC algorithm with ERND facilitates more robust exploration capabilities in environments with input saturation and joint angle constraints, thereby enabling faster learning of effective policies and enhancing the performance and efficiency of robotic manipulator tracking control tasks. Finally, the simulation results demonstrate that the robotic manipulator tracking control task is effectively completed in dense reward problems through the combination of the SAC algorithm and ERND.

Actor-Critic With Synthesis Loss for Solving Approximation Biases.

Approximation biases of value functions are considered a key problem in reinforcement learning (RL). In particular, existing RL algorithms are hindered by overestimation and underestimation biases, i.e., value mismatching between RL's actual returns and action-value approximations limits the performance of RL algorithms. In this article, we first develop a new synthesis loss function for RL's action-value estimation integrating a regularization term and a modified "clipped double Q-learning" structure for solving overestimation and underestimation biases. To minimize the differences between action-value estimations and actual returns in RL, we develop a new discrepancy function to determine the type and magnitude of approximation biases. Then, two coefficients embedded in the synthesis loss are automatically tuned by minimizing the discrepancy function during training to minimize approximation biases. We further design a new actor-critic (AC) algorithm, named AC with synthesis loss (ACSL), by integrating the synthesis loss function and an error-controlled mechanism. Experimental results on continuous control tasks illustrate that the proposed ACSL algorithm outperforms other cutting-edge RL methods in many tasks and that the proposed synthesis loss function is easily implemented into other algorithms and significantly reduces approximation biases while improving performance. The proposed method can successfully handle many complex continuous control tasks and can greatly outperform other state-of-the-art algorithms on most tasks.

Stratospheric airship trajectory planning in wind field using deep reinforcement learning

Stratospheric airships, with their long endurance, high flight altitude, and large payload capacity, show promise in earth observation and mobile internet applications. However, challenges arise due to their low flight speed, limited maneuverability and energy constraints when planning trajectories in dynamic wind fields. This paper proposes a deep reinforcement learning-based method for trajectory planning of stratospheric airships. The model considers the motion characteristics of stratospheric airships and environmental factors like wind fields and solar radiation. The soft actor-critic (SAC) algorithm is utilized to assess the effectiveness of the method in various scenarios. A comparison between time-optimized and energy-optimized trajectories reveals that time-optimized trajectories are smoother with a higher speed, while energy-optimized trajectories can save up to 10% energy by utilizing wind fields and solar energy absorption. Overall, the deep reinforcement learning approach proves effective in trajectory planning for stratospheric airships in deterministic and dynamic wind fields, offering valuable insights for flight design and optimization.

Learning-based joint recommendation, caching, and transmission optimization for cooperative edge video caching in Internet of Vehicles

In an era dominated by multimedia information, achieving efficient video transmission in the Internet of Vehicles (IoV) is crucial because of the inherent bandwidth constraints and network volatility within vehicular environments. In this paper, we propose a cooperative edge video caching framework designed to enhance video delivery efficiency in IoV by integrating joint recommendation, caching, and transmission optimization. Leveraging deep reinforcement learning with the discrete soft actor–critic algorithm, our methodology dynamically adapts to fluctuating network conditions and diverse user preferences, aiming to optimize content delivery efficiency and quality of experience. The proposed approach combines recommendation and caching strategies with transmission optimization to provide a comprehensive solution for high-performance video services. Extensive simulation results demonstrate that our framework significantly outperforms traditional baseline methods, achieving superior outcomes in terms of service utility, delivery rate, and delay reduction. These results highlight the robust potential of our solution to facilitate seamless and high-quality video experiences in the complex and dynamic landscape of vehicular networks, advancing the capabilities of IoV content delivery.

TAG: Teacher-Advice Mechanism With Gaussian Process for Reinforcement Learning.

Reinforcement learning (RL) still suffers from the problem of sample inefficiency and struggles with the exploration issue, particularly in situations with long-delayed rewards, sparse rewards, and deep local optimum. Recently, learning from demonstration (LfD) paradigm was proposed to tackle this problem. However, these methods usually require a large number of demonstrations. In this study, we present a sample efficient teacher-advice mechanism with Gaussian process (TAG) by leveraging a few expert demonstrations. In TAG, a teacher model is built to provide both an advice action and its associated confidence value. Then, a guided policy is formulated to guide the agent in the exploration phase via the defined criteria. Through the TAG mechanism, the agent is capable of exploring the environment more intentionally. Moreover, with the confidence value, the guided policy can guide the agent precisely. Also, due to the strong generalization ability of Gaussian process, the teacher model can utilize the demonstrations more effectively. Therefore, substantial improvement in performance and sample efficiency can be attained. Considerable experiments on sparse reward environments demonstrate that the TAG mechanism can help typical RL algorithms achieve significant performance gains. In addition, the TAG mechanism with soft actor-critic algorithm (TAG-SAC) attains the state-of-the-art performance over other LfD counterparts on several delayed reward and complicated continuous control environments.

Learning to drive as humans do: Reinforcement learning for autonomous navigation

This article introduces an approach aimed at enabling self-driving cars to emulate human-learned driving behavior. We propose a method where the navigation challenge of autonomous vehicles, from starting to ending positions, is framed as a series of decision-making problems encountered in various states negating the requirement for high-precision maps and routing systems. Utilizing high-quality images and sensor-derived state information, we design rewards to guide an agent’s movement from the initial to the final destination. The soft actor-critic algorithm is employed to learn the optimal policy from the interaction between the agent and the environment, informed by these states and rewards. In an innovative approach, we apply the variational autoencoder technique to extract latent vectors from high-quality images, reconstructing a new state space with vehicle state vectors. This method reduces hardware requirements and enhances training efficiency and task success rates. Simulation tests conducted in the CARLA simulator demonstrate the superiority of our method over others. It enhances the intelligence of autonomous vehicles without the need for intermediate processes such as target detection, while concurrently reducing the hardware footprint, even though it may not perform as well as the currently available mature techniques.